You have a massive legacy spaghetti mess of code, and you need to add a new feature.
Write tests first.
Put all your new code in a new module, and only insert single lines into the legacy code to call the new code. This allows you to properly encapsulate the new code, wrap it in try/catch blocks, etc. and not break the legacy system.
Pass whole objects into the new code, this is a necessary consequence of encapsulation.
# First method: 131TiB
$ wget https://phillm.net/libgen-stats.php -O libgen-stats.json
$ cat libgen-stats.json | jq '.[]|.size_bytes' | perl -lne'$x+=$_;END{print$x}'
144116297318150
$ bc -l
144116297318150 / 1024 / 1024 / 1024 / 1024
131.07300884998949186410 Tebibytes
$ bc -l
144116297318150 / 1000 / 1000 / 1000 / 1000
144.11629731815000000000 Terabytes
## Languages
- Perl: best for fast prototyping
- Ruby: best for having fun
- Node: best for public projects
- Python: best for experiments
## Frameworks
### Perl
- DBIx::Class 💯
- Mojolicious 💯
### Node
- Express (popular)
- React?
- Koa?
- NestJS?
-....? for testing: Qunit?
-....? for web framework
### Python
- Django? (this one)
- Flask?
### Ruby
- Rails?
- Active Record?
## Libraries
### Perl
- just everything
### Node
- minimist reads command line arguments
- cheerio parses HTML
- daysjs manages dates/times
- customParseFormat plugin parses dates
- knex does CRUD on databases
- fs perfoms operations on the file system
- import process from 'node:process'; // process.exit(1);
Ludo: Here's the online slide deck from Mark Jason Dominus’ 2002 YAPC lightning talk “Design Patterns Aren’t”, on Professor Christoper Alexander, his book A Pattern Language, the ‘Gang of Four’ and design patterns.
APL was book two of a trilogy also incorporating The Timeless Way of Building (volume 1) and The Oregon Experiment (volume 3).
Alexander’s earlier work Notes on the Synthesis of Form was also a big hit with computer scientists in the 1960s, even though it, too, was primarily aimed at designers and architects. (And if you still have any empty shelves on your bookcase, Alexander’s 2002-2004 four-volume work The Nature of Order: An Essay on the Art of Building and the Nature of the Universe is apparently considered his magum opus; and he also published The Battle for the Life and Beauty of the Earth: A Struggle between Two World-Systems, about how his team designed & constructed a Japanese school, in 2012.)
Will: There's another book with a possibly contrasting view: Semantic Software Design. It says the main work in software is coming up with the concepts/paradigm/metaphor which explains what the software is doing. And also we shouldn't think of developers as manufacturers, builders or architects. But rather creative artists. Mainly because we never produce the same piece of work twice.
A list of cloud servers for basic hobby-level software development.
All prices per month:
Whenever you publish a diagram, also include a comment explaining which software/website was used.
If you need to log into a website in order to update the diagram, post the login details… somewhere.
Ideally you would first create an account that is intended to be shared within you team.
If the diagram was generated from source code, publish the source code (or a link to it) along with the diagram.
For bonus points, include this information in the image itself.
These suggestions apply equally to email, chat, wiki, printouts, etc.
Don’t send the source to clients, only to colleagues.
Read The C4 model for visualising software architecture for a good system and suggestions on diagrams (then look at using plantUML with the C4 extentions).
This is a good talk on the topic:
Database schema diagram
Flowchart
Sequence diagram
Component/Class diagram
Code workflow
…
VIsual-only
Dia - beware, it's Dia by name and dire by nature
Omnigraffle - Mac only
Source + Visual
VS Code - has nice plantuml integration - live preview
PlantUML (example)
Summary: Full featured language designed for visualising software architecture, build multiple diagrams from a single model, etc.
See also 
Structurizr
Summary: Simpler, general purpose language for diagrams (dot language)
…
Gliffy.com (now paid)
moqups (has a free plan)
PlantUML
Graphviz
Pencil and paper - consider this for a first draft (scan/photo if necessary)
Note: PlantUML uses Graphviz to draw some diagrams.
(sorry, the formatting of this post is a complete mess)
A list:
Javascript philosophy and execution flow is quite different to Perl:
Recommendations for your javascript application:
Inspired by Semantic Software Design.
Program Name Design Definition
Executive Summary
State the purpose of the program and what this document covers.
About this Document
This section contains metadata about the design document itself.
Authors
Names of people who did most of the writing.
Contributors
Names of people who contributed some writing or key ideas.
Reviewers
Names of people who did some quality control on this document.
Document State
Is this a draft, in review, or published?
Use of IETF Keywords
This document employs a subset of the Internet Engineering Task Force key‐ words found in RFC 2119. These words are MUST, SHOULD, MAY, and their counterparts MUST NOT, SHOULD NOT, MAY NOT. They are capitalized throughout the document to draw attention to their special status as keywords used to indicate requirement levels.
Business Design
Describe the business objectives, drivers, and expected benefits for undertaking this program.
Capabilities
Describe what capabilities the solution will have. What new benefits are provided in functional and nonfunctional terms? What new scalability, reliability, global distribution, performance, resilience, extensibility, manageability, portability, security, or other benefits are expected? Don’t advertise or market to your audi‐ ence; be concrete and concise.
Strategic Fit
What element of the business and technology strategies does this help realize? How?
Business Drivers
Why are we doing this? What priority is it among known others?
Assumptions
What is expected to be in place for this solution approach to be successful? Refer to funding, availability of key resources, operations support, contracts, key pro‐ cesses, procurement, Global Network Operations Center (GNOC), regulations, stated standards and guidance or technology patterns from the leadership team, and so on.
Constraints Applicable laws
Does the Americans with Disabilities Act (ADA) apply?
Applicable regulations
Reference need to maintain compliance with Payment Card Industry (PCI), Personally Identifiable Information (PII), General Data Protection Regula‐ tion (GDPR), and System and Organization Controls (SOC). Do not merely
state that these laws exist, state how your design specifically accounts for supporting them so that the developers can be sure to implement it accordingly.
Risks
List business risks in doing the project as envisioned, risks to the customer or existing business prospects or processes, how it can be maintained and operated properly, availability of staffing resources, securing funding, countries/markets, the inherent risks in trade-offs made, and so forth.
Impacts
What will this project or this architecture create in terms of organizational, train‐ ing, and process needs that affect the business? How will the business need to change to best support this? Are there new processes that are expected to be adopted internally within product development to support a new technology or skillset? Should any roles and responsibilities change?
Stakeholders
List the organization, roles, and named individuals who stand to win or lose by a good or bad outcome on this program. Who must change to accommodate it? This will eventually feed your Responsible, Accountable, Consulted, Informed (RACI) document.
Governance
What is the framework you’ll use to govern this project? Must you invent a new committee? Are there design review processes or cloud governance or financial governance processes to follow?
Application Design
Application overview and general strategy description.
Applicable Standards and Policies
List of links to published guidelines and conventions for development teams to follow. For example, any internal policies, industry-specific standards, PCI guide‐ lines, ADA guidelines, and so forth that you expect teams to follow.
Guidelines and Conventions
Links to published guidelines and conventions for Dev teams to follow, such as published internal standards, Google Java coding guidelines, JavaScript conven‐ tions, code quality guidelines, and so forth.
Patterns
Diagrams, links, and descriptions of patterns for Dev teams to follow in the implementation. Reference any design patterns for designing and developing services as applicable.
Services
List services to be created or existing services to be reused, along with the owners of those services.
Security
Security requirements and design: how data will be secured, encrypted, author‐ ized, authenticated at rest, in transmission, or in processing. Use of Open Web Application Security Project (OWASP) Top Ten and how those are addressed. What security groups are required? Highlight security requirements for develop‐ ment such as bastion hosts. List transport or Transport Layer Security (TLS)/ Secure Sockets Layer (SSL) requirements.
Availability
Target SLA in terms of 9’s uptime and how specifically the architecture will sup‐ port such numbers. How recoverability, disaster recovery, and the like is being supported. Consider these questions:
What are the compensating actions taken in a failure?
Will a circuit breaker be used?
What redundancy is there?
How is caching used to support certain failure events?
Health-check page?
Multideployments?
How are specific key components particularly designed for resilience and high availability? Are you using multiple cloud zones?
What is your service replacement and versioning strategy to support zero- downtime releases?
Scalability and Performance
What number of transactions per second at this latency and CPU utilization must the solution support? What is the unit of scale (container, virtual machine, cluster)? Use actual numbers and calculate the impact on server footprint that will have. What are the ways in which the application and services can scale through statelessness, Auto Scaling Groups (ASGs)? State the ASG threshold so developers aren’t left to guess.
Extensibility
Will you use certain patterns in your services such as Strategy or Specification? How will APIs specifically support the ways that the application affords future change, how will the application support customizing per customer, and how are configurations afforded?
Testability
How will this be tested, what tools will be used, and what specific automation and targets will be in place? Include functional testing, regression testing, integration testing, and chaos/resilience testing. Load testing plan? How will this be automa‐ ted? What specific toolset used in what process?
Maintainability
What software guidance for developers will help make the code base easier, cleaner, simple to maintain in the long term? Code repository needs or project needs? What is the maintenance schedule anticipated or downtime for upgrades strategy?
Monitorability and Metrics
What tools and dashboards are required, logging requirements, how the software itself must support event publishing to increase visibility. What are the specific metrics that will indicate system uptime, health, and proper performance? How will alerts be triggered at what threshold? Consider CPU, memory, drive/filesys‐ tem volumes, process monitoring, logs, event logs, and required procedures.
Data Design
Database Strategy
Overview the general data management strategy for this application. What spe‐ cial customer requirements should be called out for careful handling? What are the risks involved with this strategy? Expected benefits?
Standards and Guidelines
What is the applicable set of data technology standards, guidelines, and tools?
Technologies
What database technologies will be used for what purposes, what specific serv‐ ices? Are there any new database technologies being used, or new versions? Where can developers learn more? What guidance exists for them?
What instance types are being used? Are the available networking modes the proper type for that database? Have you used the cost calculator to size and price instances?
Import/Export
How will customers get data into your system?
If you’re porting data, what are the source and target maps?
How will customers get their data back out of your system for their own use?
Replication, Backup, and Recovery
How is data replication supported? How will you distribute data globally?
What is the backup/recovery strategy? Will you store full or partial backups? For how long? On what systems, for what services?
What is the bulk data replication strategy? What is the multiregion strategy?
Data Versioning
How will your database be versioned? What about the data itself?
Database Automation
How will the database farm be automated and updated? What tools, pipelines, and processes are in place or should be developed?
Database Performance Considerations
What performance thresholds for transactions within the database are expected? Do you have a known level you must support?
Data Warehousing, Storage, and Management Requirements
What data volumes must be supported? Data movement policies and require‐ ments. Reference how logs will be stored, managed, or forwarded.
Data Maintenance
How data will be maintained, data retention policies, scripting to offload, data restoration. How will data be populated for different environments for this appli‐ cation? Will data be truncated? At what interval? How will data be encrypted? Are there GDPR or PII/PCI requirements to be stated for development teams or infrastructure admins?
Data Migration
How will data get into the system? Is connecting to a legacy system required? Is Golden Gate or Kafka or Extract, Transform, and Load (ETL) or another tool in use? What time period is anticipated for this? Will data need to be synchronized over a certain period of time?
Data Volume
What is the expected starting volume of data? What size database is anticipated? Will there be multiple data stores? How many rows are anticipated to be added or removed daily for the key services?
Logging
What are your log rotation policies? Are there requirements for Splunk or another monitoring tool?
Reporting
How will we do reporting for customer use? For internal use?
Auditing
How must the system audit user changes to track and report in case of a security breach or for compliance?
Security
How is the database farm itself specifically secured? Will data be encrypted at rest, and how will that be managed? How will access to different environments be managed, including at the development level?
Analytics
What data must be exposed by the application to support business analytics. How must that data be exposed to support analytics tools? Will you have a data lake? How will that be maintained, accessed, and used properly?
Caching Strategy
Requirements for caching and the locations and technology to support caching. How distributed is it? Any guidance for application developers as applicable. Guidelines on caching for implementers and for developers subsequently using it.
Machine Learning
What specific processes must be called out for machine learning? How will data pipelines be supported? What is that entire subsystem design? Feature engineer‐ ing considerations.
Infrastructure Design
Infrastructure overview statement.
Infrastructure Strategy
What are the patterns you employ? Are there deployment diagrams, component diagrams, other UML? Cloud infrastructure design? Do you need to include load balancers, DNS, application servers, networks, CIDR allowances, security, data servers, firewalls, storage, queues, and so on?
Latency and Performance
How does your infrastructure support the specific application and customer-level SLAs described earlier?
Infrastructure Security
How will the environments be secured?
Maintenance
How will teams perform patching (hopefully no-patching)? Related policies such as operating system (OS) upgrades, replacements.
Standards, Guidelines, Conventions
References or links to existing guidelines that you want the teams to follow.
Infrastructure as Code (IaC)
References to templates, best practices, samples.
Environment Guidelines
When/how to use reserve instances, autoscaling groups, VPN access to different environments by different teams, and so on.
Global Distribution
How do we enable and maintain the ability to distribute our applications glob‐ ally? Are there any particular concerns, such as regarding the Chinese firewall? Concerns associated with performance and security in globally distributing? Server replication processes and automation?
Immutable Infrastructure
Describe how your infrastructure is immutable, including process automation.
Hybrid
Are there any necessary strategies for hybrid architectures (solutions that span on-premise and cloud)?
SLAs
State SLAs that you must meet. Put them in mathematically testable terms, from a customer perspective. Include the panoply of customers, not only the end user.
END OF TEMPLATE
Writing a hundred pages up front is not the answer. What is the answer is this:
Proposition 0
By definition, any purposive compound of objects and their relations is called a system. (Examples can include a software application, a datacenter, a business organization, a business process, a chemical compound, a written document, a play, a music composition, and so forth.)
Proposition 0a
These compounded elements and their relations are not innate, but are proposed,
socially constructed, captured, augmented, determined, and filtered by the design‐
ers of that system.
Proposition 0b
Any system is either designed explicitly (purposively), or implicitly. If the design is implicit, its design is regarded and comprehended only after the fact, after the system is in place, as a result of a series of accidents, which is likely non-optimal.
Proposition 1
Certain principles apply to well-designed systems, and these same principles can
be employed across the design of any system, though seemingly disparate.
Proposition 1a
The attributes of any well-designed system include, at a minimum:
Fitness to purpose
Felicity
It must afford that purpose in a way that minimizes friction and noise, mak‐ ing it easy and delightful to use, consume, and participate in.
Flexibility
Given that the system operates in a world of frequent change, it should be designed in a way to allow modifications, updates, and extension according to future needs.
Proposition 1b
An additional set of attributes contemplated for a well-designed system (software or otherwise) include the following:
Maintainability
It should be easy to correct faults, improve performance, and adapt to a changed environment.
Manageability
You should be able to keep the system safe, secure, and operating smoothly.
Monitorability
You should be able to see into the system, to measure and understand how it is working.
Performance
It must excel at its purposes.
Portability
It should be able to operate in a variety of contexts.
Scalability
It should be able to operate at the same level, even under increased load.
Proposition 2
The software system you design will operate within a business context, and there‐ fore, to be optimally designed, the software system must be designed to support and operate within this business context or a new business context the software, in its innovation, potentially requires the creation of.
Proposition 3
The business is a system of systems (these are business elements that are also sys‐ tems: your service-oriented development organization, the sales delivery process, the architecture review board, the strategic funding process, local executive steer‐ ing committee meetings, the joint venture strategy, the project execution plan, and so forth).
Proposition 4
The business therefore can be designed as systems; it operates according to these same principles.
Proposition 5
Because the semantic designer (creative architect) is foremost a designer of sys‐
tems, the purview of the role includes the proper design of the software as well as
the design of the business systems themselves.
Conclusion
The business is a system just like the application is, so you as the creative director
must help design the business itself as a cohesive and coherent system according
to these principles, to achieve a better overall business outcome. The resulting
business, as a context in which software is developed, will help improve the soft‐
ware itself, and help you make it on time and on budget and according to user
needs. They inform and help (or hurt) each other.
So there are two points here:
You might not have historically considered it part of your job, but to be especially effective, consider your purview to include the design of the organization itself and its processes according to received architecture and design principles.
When you design especially effective software, you not only consider the applica‐ tion frameworks and software attributes, but consider the impact the business will have on your system, and the impact your system will have on the business.
Therefore, now we turn our attention to the business itself, to ask specifically:
How can you see your organization and processes as systems in themselves to be understood and purposively designed?
After you begin to see your organization through the lens of systems, how can you optimize the organization and processes toward maximum effectiveness?
How can you determine the impact your burgeoning system might have on the business?
• How aligned is the business with the system you are creating? As you bring it to life, can it be properly supported?
By the end of this chapter, you will be able to answer these questions with the practi‐ cal tools we’ll introduce.
Go and buy the book to read more.
Posts
All Comments
There are a number of ways to evaluate these software packages:
Flowchart vs Database
Online vs Desktop
Visual-only vs Generated from source
Source language: General (e.g. Dot) vs Specific (e.g. PlantUML + C4)
Source-generated diagrams have many benefits:
Dot is the easiest to learn, and PlantUML has the most features.