Showing posts with label docs. Show all posts
Showing posts with label docs. Show all posts
Database schema diagrams in 2022
Suggestions
Whenever you publish a diagram, also include a comment explaining which software/website was used.
If you need to log into a website in order to update the diagram, post the login details… somewhere.
Ideally you would first create an account that is intended to be shared within you team.
If the diagram was generated from source code, publish the source code (or a link to it) along with the diagram.
For bonus points, include this information in the image itself.
These suggestions apply equally to email, chat, wiki, printouts, etc.
Don’t send the source to clients, only to colleagues.
Read The C4 model for visualising software architecture for a good system and suggestions on diagrams (then look at using plantUML with the C4 extentions).
This is a good talk on the topic:
Types of diagram
Database schema diagram
Flowchart
Sequence diagram
Component/Class diagram
Code workflow
…
Desktop software
Flowcharts
VIsual-only
Dia - beware, it's Dia by name and dire by nature
Omnigraffle - Mac only
Source + Visual
VS Code - has nice plantuml integration - live preview
PlantUML (example)
Summary: Full featured language designed for visualising software architecture, build multiple diagrams from a single model, etc.
See also
Structurizr
Summary: Simpler, general purpose language for diagrams (dot language)
Database
…
Online software
Flowcharts
Gliffy.com (now paid)
moqups (has a free plan)
PlantUML
Graphviz
Database
- diagrams.net (formerly draw.io)
- Quite heavy
- Like an online omnigraffle
- Doesn't specialise in databases
- No import function for SQL
- Free for up to 10 diagrams :-|
- Fairly good
- Has multiple import/export
- Importing is quite easy (although doesn't support all syntax)
- Diagrams don't look quite as good as dbdesigner.net
- dbdesigner.net
- Free for 2 diagrams? :-/
- Slightly better looking diagrams
- But more buggy (Login & save before starting!)
- Importing is more painful than dbdiagram.io
Hardware
Pencil and paper - consider this for a first draft (scan/photo if necessary)
Comments
Note: PlantUML uses Graphviz to draw some diagrams.
(sorry, the formatting of this post is a complete mess)
POD servers for Perl
You want to browse Perl embedded documentation (POD)
You have some software that was designed for browsing information: a web browser
You need a server to turn the POD into web pages and serve it to the browser.
You try one of these modules:
- https://metacpan.org/pod/Pod::Browser
- Uses Catalyst
- https://metacpan.org/pod/Pod::Server
- ?
- https://metacpan.org/pod/Pod::Webserver
- Very basic
- Parsing much better than Pod::POM::Web, but still fails in places.
- Contains all the Perl built-in modules, which I don't want to see.
- How to run: podwebserver
- https://metacpan.org/pod/Pod::POM::Web
- Advanced design
- Fails to parse nested tags
- Contains all the Perl built-in modules, which I don't want to see.
- How to run: perl -I $DOC_PATH -I $POD_POM_WEB_PATH -MPod::POM::Web -e server
Questions:
- What is the reference implementation of a POD interpreter? `perldoc` just removes links and it is plain text only. So in what viewer do the links actually link to something?
Documentation formatting, presentation and distribution
Points relating to formatting and presentation (not content):
- Documents need version numbers. This comes for free with a wiki-based system.
- Need a "changes" section at the top, including dates & authors.
- A table of contents with clickable (and bookmarkable) links
- Code samples to be in a consistent style, monospace font, and limited to a single page
- Web page or PDF, not .docx because .docx will be formatted differently by every application, and might appear corrupted or unreadable on some computers.
- Host the document on company servers and send around a link to download it. Don't email as an attachment because it will become out of date quickly.
Personal Software Development Checklists, 2008
To do: Fix formatting
They should be matched to each milestone in the development process. These aren't strictly defined.
Will Sheppard, BBC, 2008
They should be matched to each milestone in the development process. These aren't strictly defined.
- Requirements discussion : Are the requirements ok? Requirements checklist
- Sprint planning / estimation : Are you planning and estimating well? Sprint planning and Estimation checklist
- Adding an item to the backlog : No checklist, to make it as easy as possible to add ideas?
- Construction : Are you designing and constructing well? Design checklist / Construction checklist / Database checklist / Monitoring checklist
- Bug fixing/refactoring : Are you maintaining code well? Maintenance/Editing checklist
- Deployment : Are you finishing up neatly? Agile checklist / I'm finished checklist / Deployment checklist / Documentation checklist
Requirements checklist
- Have you removed any requirements that are not strictly necessary?
- Have you removed any requirements which can be postponed until a second phase?
- Have you made a decision on absolutely all of the open questions?
Remember that if one person doesn't make the decision, someone else will be forced to.
i.e. if Client doesn't decide, Engineer must. Or if Engineer doesn't decide, Client must.
SCRUM
- Is the white card specified in the form "As an XXX I want to YYY". (This forces you to ask the question "When will this project really be considered 'done'?")
- Have you identified the most risky requirements? (dependencies, unknown, etc).
- Have you stated any assumptions explicitly?
Volere
- Have you noted a rationale for the requirement?
- Have you given the requirement a unique ID? (e.g. tracked in a ticketing system).
- Have you noted who raised the requirement (person's role, if not their name).
- Have you written fit criterion (How will I know when I'm done?)
- Have you noted any dependencies?
- Have you recorded the history of the requirement (changes made)?
Sprint planning and Estimation checklist
This checklist should be used in Sprint Planning.
NOTE: As Sprint Planning is often the first time we hear of a requirement, also use the Requirements checklist here.
- Have you created a task to clarify requirements further, if necessary?
- Is the white card in the form of a story?
- Are the requirements framed as a problem, or are they really a suggested solution?
Preparation
- Have you made some rough technical designs and considered several solutions?
Estimating
- Has enough research been done to give a good enough estimate?
- Have you tried to counteract your usual bias?
- Have you compared it to previously completed tasks? (Add or subtract time depending on the differences between this task and the known one).
- Have you thought of the best case, and the worst case, and done the calculation? (best + 2*worst) / 3
Have you created tasks to:
- write test cases before writing code (Test Driven Development)
- model the environment
- only write code to pass tests
- test the solution end-to-end
- get a Dev/Test environment set up that is identical to live (e.g. Forge / ExtDev)
- QA (testing by someone else)
- check that foreign characters are encoded/decoded correctly (tests)
- build in monitoring
- deploy to a production environment
- write documentation
- get the solution signed off
- clarify the requirements?
Design checklist
This should be read before you have completed the design of the system.
- Have you looked over the other Technical Specification checklist?
- Have you asked around to see if anyone's solved the problem already?
Have you considered:
- using any off-the-shelf solutions (e.g. FeedEngine, Cocoon, IDOL)
- whether or not using a database would be beneficial?
- the use of any design patterns?
- using the most appropriate language for the job?
- Have you identified any gaps in knowledge, e.g. HTTP connection in C#
Maintainability
- Have you made sure that especially any data-based parts of the system can be easily modified?
- e.g. Consider how much work is involved to add a new dataset to clickthrough tracking
Modularisation
- Is any functional routine able to be called on its own from a stand-alone script?
- Have you set out proper objects and their relationships?
On paper
- Have you drawn a diagram of the information flow, at the very least?
- Any amount of UML diagrams at the most
- Have you drawn a class diagram to begin with?
- Have you defined the major data structures?
- Have you written a pseudo-configuration file?
- Has the main routine been written in pseudo-code?
- Has each sub-routine been written in pseudo-code?
- Has each routine been broken down in enough detail so that it almost seems simpler to just write the code instead?
- Has care been taken to contain all the business logic separately from the internal logic?
- Does the design extend far enough in each direction?
- i.e. are there defined mechanisms for acquiring input, and publishing output?
- Does the design take account of any possible non-ASCII characters?
Investigation
- Have you fully investigated all of the options? (i.e. followed through on each design idea until you're sure it either will or won't work, possibly using prototypes)
- Have you asked around for previous solutions to the various possible solutions? (engineers/leads from different teams, and people with experience).
- Have you highlighted any risk? (risk = unknowns)
think about testing, during design
- Have you thought about how you would test the system?
- NB: a test consists of an input and an expected output.
Agreement
- Has the client explicitly agreed to the inputs they'll provide, and outputs they'll receive?
Construction checklist
- Have you entered the RCS keywords into the file for Perforce? (replace with your own VCS)
# $File$
# $Revision$
# $Change$
# $DateTime$
# $Author$
- Are you using Log4Perl?
- Have you got all the modules/libraries you need?
- Are you using strict/warnings?
Database checklist
- Have you drawn it out on paper first?
- Have you used foreign key constraints for referential integrity?
- Have you avoided optimising early?
From the perspective of data warehousing:
- Have you run every query with EXPLAIN to check that the correct index is being used and it's as efficient as possible?
- Have you run every query with small and large ranges of data? (different indexes get used under different circumstances).
- Have you considered the effect of indexes (or lack of them) on INSERT, DELETE and UPDATE as well as SELECT?
- Have you built a set of test queries representing the full range of data, in order to test performance?
- Have you considered using memcache to improve query response time?
- Have you compared more than one type of each query? (sub-queries, temporary tables, etc).
- Have you considered keeping lookup data tables in the application memory rather than using expensive joins?
- Have you considered redesigning the data gathering, loading and schema to facilitate more efficient queries?
- Data warehousing specifically: Have you avoided ENUMs and allowed any data to be inserted? (add to lookup tables and be given a description later).
Monitoring checklist
- Have you considered using a framework that periodically publishes its status via HTTP? (e.g. BBC perforce //depot/osd/Importer)
- Have you considered the need to write a CGI script to make the necessary checks and report a simple error code back for use in Nagios?
- Are there fine-grained, customised, complex or non-standard requirements for monitoring? If so, consider using your own team's Nagios installation instead of Livesite's.
- Livesite like to send alerts to group email addresses, every hour, and use only simple regex matches. They are also geared towards critical systems that need to be supported 24/7.
Example of monitoring criteria:
For Search+ PAL/JBoss:
For Search+ PAL/JBoss:
- Load - warning at 15 load, critical at 30
- Memory/Swap - warning at 70% used, critical at 85%
- Diskspace - warning if any disk 80% full, critical at 90%
IDOLs (e.g. nolidol301):
IDOL [Collections2] - ACI STATUS OK 07-02-2009 16:57:10 30d 1h 55m 42s 1/3 AUTONOMY OK
IDOL [Collections2] - INDEX QUEUE OK 07-02-2009 16:59:58 1d 16h 41m 29s 1/3 DRE_INDEX_QUEUE OK
IDOL [Collections2] - PERFORMANCE OK 07-02-2009 16:59:19 30d 1h 56m 3s 1/3 DRE_STATISTICS OK
IDOL [iPlayer] - QUERY OK 07-02-2009 16:56:58 3d 17h 56m 11s 1/3 HTTP OK
Maintenance checklist
To cover any changes made to code / configuration: Adding new features or debugging issues
- have you sync'd from the repository? (checking first to see if there are any unsubmitted changes in your workspace)
- have you re-read the design checklist with the proposed changes in mind?
- is the system adequately documented?
- have the requirements been specified in enough detail?
- are you modifying the design, then implementing the design? (as opposed to modifying the code)
Agile checklist
To check that you're benefitting from agile methodologies
- Have you split up tasks into MMFs? (Minimum Marketable Features)
- Have you deployed as early as possible? (As soon as there is a stable feature to show the client).
I'm finished checklist
- Could you rename some variables or routines to make them clearer?
- Could you refactor the code to make it clearer?
- put code into subroutines
- clean up comments
- Perl: Have all print, die and warn statements been replaced with $LOG->info, $LOG->logdie and $LOG->warn statements? (log4perl/log4j)
- is all the code checked into the versioning system?
- are only the changes since the last release checked in and tagged with a new release number?
- does the system satisfy the original requirements?
- How can you demonstrate it? (Check against original requirements)
- has the system been tested?
- Has it passed an end-to-end test in the production environment, with realistic levels of traffic?
- How confident are you it will hold up?
- Have you tried to break it with bad input?
- are all paths set to the live environment?
- grep -r your_username packagedir/
- is the system been deployed into a suitably monitored environment?
- who will be notified of errors that occur during execution?
- have you documented (from least technical to most technical):
- product details
- user operation
- support operation
- flow of logic / requests / component interaction
- database structure
- data structures in the code
- have you told everyone about it that needs to know?
- have you considered the need to restrict access?
- will you be able to tell who is using it / how they are using it?
Deployment checklist
General:
- Has it been installed onto an empty VM and run successfully?
- Have all the steps to get it running on the VM been documented for Ops?
- Have you run unit tests?
- Have you run functional test manifests?
- Have you performed the set of sanity checks or manual checks?
- Have you updated the package with any hot fixes you've quickly made to get it working for the first time?
BBC Livesite conventions:
- Have you considered to what data the app will need access, and which modules it will need?
- Have you consulted Livesite on any alternative architecture that might simplify their deployment and maintenance work?
- Have you set the configuration to write data and logs to the /data partition?
- Have you considered packaging as an RPM for ease of install and upgrade?
- Have you considered writing an init script instead of a crontab entry?
- Have you arranged for access to the server and application logs in order to resolve issues faster? (e.g. they could be made available on the web servers at /apache_logs/).
Afterwards:
- Have you updated the deployment log on the wiki?
- Have you updated the operational info on the wiki?
Documentation checklist
Template for documentation
Written with a wiki in mind.
For each product/project (landing page)
- At which stage of development is it?
- Summary: What is it? What does it do? Why do we need it? Who uses it?
- Contacts: Who owns it? Who develops it? Who supports it?
- Access: How do you access it? Do you need permission?
- Context/Scope: What similar systems are there? Why weren't they used? What does it integrate with? Where is the system used?
- Components: What are the different parts of the system? What do they do?
- Architecture: Is there an overview diagram?
- Troubleshooting: What might go wrong? What has gone wrong in the past? How does one fix it?
For each component (separate page)
- Summary: What is it? What does it do?
- Contacts: Who develops and supports it?
- Operational: Where is it hosted? Who administers the servers?
- Architecture: Is there a diagram of this component?
Component development information (separate page)
- Where is the code stored?
- How does one install it/set it up?
- What is the workflow/dataflow? (pseudocode/flowcharts)
- What are the features?
- What are the known bugs?
- What work is planned?
Working notes (separate page)
- Anything your colleagues would need to know, should you win the lottery and never be seen again!
- An area to save ongoing development notes to share with your colleagues
Will Sheppard, BBC, 2008
Subscribe to:
Posts (Atom)
Posts
There are a number of ways to evaluate these software packages:
Flowchart vs Database
Online vs Desktop
Visual-only vs Generated from source
Source language: General (e.g. Dot) vs Specific (e.g. PlantUML + C4)
Source-generated diagrams have many benefits:
Dot is the easiest to learn, and PlantUML has the most features.