How to maintain software

You have a massive legacy spaghetti mess of code, and you need to add a new feature.

Write tests first.

Put all your new code in a new module, and only insert single lines into the legacy code to call the new code. This allows you to properly encapsulate the new code, wrap it in try/catch blocks, etc. and not break the legacy system.

Pass whole objects into the new code, this is a necessary consequence of encapsulation.

Books about software design theory

Ludo: Here's the online slide deck from Mark Jason Dominus’ 2002 YAPC lightning talk “Design Patterns Aren’t”, on Professor Christoper Alexander, his book A Pattern Language, the ‘Gang of Four’ and design patterns.

APL was book two of a trilogy also incorporating The Timeless Way of Building (volume 1) and The Oregon Experiment (volume 3).

Alexander’s earlier work Notes on the Synthesis of Form was also a big hit with computer scientists in the 1960s, even though it, too, was primarily aimed at designers and architects. (And if you still have any empty shelves on your bookcase, Alexander’s 2002-2004 four-volume work The Nature of Order: An Essay on the Art of Building and the Nature of the Universe is apparently considered his magum opus; and he also published The Battle for the Life and Beauty of the Earth: A Struggle between Two World-Systems, about how his team designed & constructed a Japanese school, in 2012.)

Will: There's another book with a possibly contrasting view: Semantic Software Design. It says the main work in software is coming up with the concepts/paradigm/metaphor which explains what the software is doing. And also we shouldn't think of developers as manufacturers, builders or architects. But rather creative artists. Mainly because we never produce the same piece of work twice.

Software logging philosophy

Ideas:

• Always set up logging from the start of the application, even if there is nothing to log.
• Imagine in future there is an error reported in the front-end.
• To investigate, there could be more error information in the logs.
• And even if not, to investigate further, more debug info could be added to the log temporarily, even for a specific client.
• If that requires you to set up logging infrastructure, it's much less likely to be done at all
• So it's important to have a logging system in place for that reason.
• Distinguish types of logging (see other post):
• Event - this can include debug, info, warn, error. It is logged and that's it.
• Exception - unexpected error that requires action to resolve. A human should be notified.
• Request - every request made (higher/framework level)
• ...

How to write tickets

Anatomy of a good Jira ticket

User story

AS A [role]
I WANT [something]
SO THAT [end goal]

Background/details

• This was attempted earlier in ATS-123 but it didn't work

• This is to solve the larger problem that __________ (e.g. we are spending too much time supporting this product)

• Who originally raised the requirement and why (person's role, if not their name)

• Any potential blockers/risks highlighted here

• Names of functions or files to look at

• Links to the relevant code repos

Acceptance Criteria

• (it's done when) all passwords are validated

• (it's done when) the publish button is no longer visible

• (it's done when) the page loads in under 2 seconds

Attachments

• Screenshot of the relevant page / error message / whatever

• Email chain where this has been previously discussed

• Any open questions should be noted here so that the implementer is aware of them

• One possible solution was to use the ABC widget to achieve this.

• Here are some extra details from product

• You will need to ask Alex for the access details

• Summaries of subsequent online discussions about the story should be copied here instead of remaining hidden in email or Slack

• Summaries of decisions made at meetings about the story (after starting it) should be put here too

---

Example of a good bug report

The key points of a bug report are:

1. Steps to reproduce

2. What currently happens

3. What is expected to happen instead

Description

Users should be able to select (and apply) the ‘disabled’ filter on the ‘Agents’ page without being redirected to another page.

Scenario (what the user did / steps to reproduce)

• Log in and navigate to 'Agents' section.

• Select the ‘disabled’ filter button.

Expected result

The agents table should only show ‘disabled’ agents.

Actual result

The user is redirected to another page -- this is an undefined ‘Agent’ page.

---

Other general notes

• Have all stakeholders communicate via the ticket and @ people to get their attention. Any communication via email or Slack, etc. risks that info being lost, unless copied/attached to the ticket.

• Ensure that all requirements/decisions/etc which are discussed one-to-one or during a meeting are added back into the ticket so nothing is lost or forgotten.

• Conduct all conversations in open channels that allow visibility into the decisions for others working on related issues.

• Continuously keep tickets updated with the current status so the whole team is aware of where everyone else is with their tasks.

• Highlight any risks or blockers ahead of time.

Code context

• "Providing other developers with the names of functions or files to look for from someone with deeper knowledge of the codebase can save an immense amount of time up front and avoids potential refactoring down the road. It also reduces guesswork and more importantly, might reinforce a best practice when there are multiple approaches that would technically work. Examples of previous implementations or before-and-after code samples are also great things to consider providing"

Why spend time writing good tickets

• Regardless of the expected assignee’s knowledge, the extra time spent to write a good ticket is rarely wasted. Tickets often get passed around as people take vacations, flat tires happen, and children get sick. When these things inevitably occur, we don’t want to rely upon assumptions, which, no matter how good, only need to be wrong once to cause potentially large amounts of wasted time, embarrassment, liability, etc.

• Treat it as an opportunity to show respect to each of the many professionals that will touch the ticket through its lifecycle by providing them the information and tools they need to perform their job to the best of their abilities.

source: https://chromatichq.com/insights/anatomy-good-ticket

---

• Note the background: What has been done, what exists, and the general situation. Explain it here.

• Note the questions: As I'm writing the ticket, I'll have questions so I just list them here until I have a chance to speak to someone who can answer them.

• Note the business value: Often companies would like to keep track of the ROI or business value of a feature so that it can be used during backlog prioritisation and sprint planning meetings. You can talk about revenue gains, cost savings, engagement, customer satisfaction, and other benefits here.

• Note the metrics: It's good to indicate which metrics we are measuring in relation to this story. For example, how many users are affected by this and what's the current engagement? This gives an overview of the scale of the feature and a snapshot of the situation before the feature is released.

source: https://www.taigeair.com/JIRA-Ticket.html

---

See also "10 powerful strategies for breaking down Product Backlog Items in Scrum" (with cheatsheet)

And "Writing Better User Stories and Bug Tickets"

Coding wisdom

While working with Broadbean, I was reminded of some of these pearls for Perl, and learnt some others for the first time:

Database:

• Use timestamp for when something happened: updated_time, created_time.
• Use datetime for when something is scheduled to happen: reminder_time, renew_time - this is why datetime depends on the timezone; the point-in-time of the reminder depends on the daylight savings in that timezone at the point the reminder should fire.

Version control:

• The commit message should explain WHY the change was needed, not WHAT the change does (because what the change does should already be apparent from the code and in-line comments).

Code:

• If you have an if/unless conditional you can fit on one line, use a post-fix:
• if ( $i == 3 ) { return $i }
• is better written as
• return $i if $i == 3;
• The second form is better because it discourages nested logic (which is harder to read and maintain)

• Generally prefer subroutines and "return" statements to control flow, instead of "if/then/else" statements. This way the code is factored into smaller chunks that are easier to understand, modify and test.

• Always include a dry-run option in stand-alone scripts meant for production.

• Every pull request must contain one and only one feature/behaviour that needed changing. This reduces cognitive drain on human reviewers who are required to scan everything and may therefore miss some details.

Perl module preferences

Preferences:

• Never use Switch, it has some truly awful bugs.
• Use Try::Tiny instead of TryCatch, it's less magical/scary.
• Instead of JSON, use Cpanel::JSON::XS or JSON::XS
• It's safer to explicitly name the package being used, because JSON picks one depending on what's already installed.

General API design principles

API principles

Status: Draft

Working notes:

• Read the Heroku HTTP API design guide
• And the The twelve-factor app methodology for building SaaS
• Use jsonapi.org
• Consider JSON PATCH
• Endpoints are all nouns, use the HTTP actions as verbs.
• All endpoint nouns must always be singular, to match with database tables. Or plural (explanation). The point is they should remain consistent with other APIs from the same team, organisation or whatever.
• Article: Your API versioning is wrong. Conclusion: Use the headers for versioning.
• Cool ideas:
• (for ease of development) Provide a special undocumented "override" URL path for humans, that sets the header appropriately:
• i.e. /api/v2/nodes --> redirects to --> /api/nodes and automatically sets header: api-version: 2
• or /api/nodes?version=2 --> redirects the same as above
• Caution: Don't overload the content-type header.
• (for ease of development) Provide a similar parameter for the Accept headers. The reason for this is to make it easier during development to send a URL to someone non-technical, or who does have the right dev environment set up, but the URL will still works without any special software like curl or browser plugins. Example: /api/nodes?accept=application/json
• Return 2xx status code to indicate the success of the HTTP request
• Return a status field in the content body to indicate the progress of the business domain request
  • Use a "status" field, not a "state" field. Status refers to a progression.

APIs should conform to previously created APIs where that doesn't contradict the principles above.

Where legacy APIs don't follow the principles above, they should be updated to conform as a pre-requisite to any changes.

Minimal JSONAPI examples

Success response:

(the data array is optional, it could be ommitted if the response is always a single object).

Failure response:

Questions

• The response should be valid JSON API. But must the request be JSON API too?

References

• If the examples above seem unnecessarily verbose (even though they have been cut down the most they can be), try JSend instead, it's simpler.
• How to validate JSON API: Take the schema and your JSON, and input them at JSON schema lint

More stuff

https://www.youtube.com/watch?v=aAb7hSCtvGw

Some of the key takeaways:

• An api should be easy to learn, easy to use, hard to misuse
• Continue to write to the API early and often
• Example programs should be exemplary - this code will end up being copied everywhere
• When in doubt, leave it out
• Be consistent - same word means the same thing across the api 
• Documentation matters - reuse is something is easier to say than to do - doing it requires both good design and good documentation
• Do what is customary - obey standard naming conventions - it should feel like one of the core APIs, know the common pitfalls for the language and avoid them

Reasons to use "git pull --rebase" by default

Rebase is a really sharp knife. Sometimes a really sharp knife is what you need, but it's also possible to accidentally injure yourself.

(thanks to Bill Blunn for the imagery).

Reasons to use "git pull --rebase" and not "git pull" (which merges by default):

• Prevents massive weird conflicting-with-yourself problems if anyone rebases that branch later, e.g. just before merging to master
• Keeps all the commits together in a bunch at the top of the tree so you can easily identify them visually
• Allows "git diff master..HEAD" to work without including unrelated changes

Reasons not to use "git pull --rebase" and stick with "git pull" (using merges):

• It's the default, so most people will use it, and their merges will clash with your rebases

Comparison table:

issue	git pull (default: merge)	git pull --rebase
git log, in a feature branch	pro: you always see when master was merged con: your commits will be interleaved with others commits (less of a problem for short-lived branches)	pro: root of feature branch is transparently moved to head of master pro: all your commits are kept bunched together con: you don't know when master was merged con: you have to notify everyone downstream
diff feature branch against master	con: it's more difficult to diff against master	pro: "git diff master" just works
mixing rebase and merge	con: you can't rebase without weird conflicts	con: you can't merge downstream without weird conflicts (not sure about this one)

(thank you tablesgenerator.com)

What good Perl looks like

This is a selection (not an exhaustive list) of points which make Perl code a joy to maintain:

QUALITY
    * Named parameters to subroutines, passed as a hashref. Some cases of subroutines taking only one or two positional parameters may be acceptable, if it's obvious what they should be.
    * Parameters to subroutines are validated (e.g. Params::Validate, MooseX::Params::Validate, Type::Tiny with Moo
    * use strict; use warnings; # or equivalent (e.g. Moose or NAP::policy) for every file. A test to ensure this, e.g. Perl::Critic's RequireUseStrict and RequireUseWarnings.
    * Does not use the Switch module, the bugs in that module are extremely dangerous and even affect code which doesn't appear to use switch at all.
    * Well factored - write several subroutines or subclasses with meaningful names and avoid large "if/then" blocks.
    * Semantic classes, not utility classes
    * Modules have @EXPORT_OK (instead of @EXPORT), so imported methods have to be declared whenever used, and it's easy to trace where they came from
    * Variable names are full words with underscores separating words, no abbreviations just "to save typing"
    * No magic numbers or magic strings, use constants with comments explaining them. Even better, encapsulate in a method so constants don't have to be defined everywhere.
    * Do not re-use variables, i.e. don't use them for different purposes in different parts of the code. Use two variables instead.
    * Code written in the language of the problem domain (e.g. $album->{artist}) and not the solution domain (e.g. $hash->{lookup_field}).
    * Never perform an eval without checking the error and re-throwing the exception if it's not recognised
    * Code written to be easily understandable by other developers. No cleverness or "magic" without detailed explanatory comments.
    * Logging via at least one abstraction layer so log output can be easily controlled. Ideally use Log::Any which implements the Observer pattern, so you don't have to pass a log object into your classes.

DOCUMENTATION
    * Comments to explain the intention of every distinct section of the code
    * At least a few words of POD for _all_ classes and methods explaining the reason for existence, and what it represents in the real world
    * POD for functional tests (not needed for unit tests)
    * Everything built with a developer in mind who has never seen the system before (there will be many of these in the software's future).

TESTS
    * Test suite can run anywhere (checkout dir, dev env, test env, etc.)
    * Test suite not brittle, tests pass when run in any order
    * Tests clearly separated into:
        * environment - if these fail, bail out
        * unit tests - only test one module, with dependencies mocked out
        * integration tests - tests involving multiple in-team systems
        * monitoring - "tests" of external systems - not really tests but a form of monitoring
    * Appropriate mix of unit and integration tests
    * Tests for POD, syntax, etc. even if with a blacklist

ARCHITECTURE
    * No logic in the templates, proper MVC separation
    * No logic in scripts, only in re-usable modules

FORMATTING
    * No mixed tabs and spaces
    * Indent with 4 spaces
    * Consistent indenting

    * No indenting for longer than about 50 lines. If that happens, break some of the logic out into a different subroutine.

The opinions above are my own.

A nod of the head to Damian Conway's "Perl Best Practices".

Temporarily skip a test until a specific date

The job of a test suite is to alert the developers of a potential problem. Once that's happened, the only value in continuing to fail is to nag and remind them to make a change. In real life sometimes fixing issues takes a long time. If they're already tracking changes in a ticketing system, or actively working on a fix, the failure becomes noise, and can hide other unexpected failures. There should be no such thing as an "okay failure" which can be ignored - you need to stop it failing if the fix is already planned. But you don't want to risk forgetting about it. So you can disable it for a short time, to allow the developer to fix the underlying issue. In the future, perhaps CI frameworks might feature a Nagios-style "acknowledge" function.

This is one solution for now:

use Test::More;
use DateTime;

my $expected_to_fail = (
    ($test_data = 'foo')
    &&
    ! deadline_has_passed( DateTime->new( year => 2014, month => 11, day => 23 ) # the future
) ? 1 : 0;

SKIP: {
    skip "while we do something (ticket ID)", 1 if $expected_to_fail;
    ok( "some test" );
}

sub deadline_has_passed {
    my ($deadline) = @_;
    my $today = DateTime->now; #->add( days => 30 );
    my $deadline_has_passed = (DateTime->compare( $today, $deadline ) < 1) ? 0 : 1;

    # debug
    #note("today    = ".$today->ymd);
    #note("deadline = ".$deadline->ymd);
    #note("compare today,deadline => ". DateTime->compare( $today, $deadline ));
    #note("has deadline passed? $deadline_has_passed");

    return $deadline_has_passed;
}

How to communicate within an IT department or tech team

Communication is difficult, it's easy to get wrong. Here are some ideas about how to get it right.

• Mailing lists: Make them public (within the company). Archive them. Make them discoverable. Let users administer them. See GNU mailman.
• Wiki: Only have one. Keep the software up-to-date. Have an 'information architect' role, who manages the structure, so that the information is in predictable places. Make sure the search works (test it regularly). Unlock all the permissions. Let people delete pages. React to user feedback.
• When you upgrade, ideally migrate/import the data to the new version. If that doesn't happen, then export or otherwise safely archive the old content. Publish it internally, make sure it's available, even if static.
• Use the Gliffy plugin or equivalent (e.g. on Atlassian Confluence an open source wiki) or equivalent to allow users to publish diagrams of unlimited complexity, that can be edited and kept up-to-date by anyone.
• Issue tracking system: Keep the software up-to-date. Don't install a million plugins (looking at you Jira) that make it difficult to upgrade. Make sure the workflows match reality, give people standard ones and document how they work. Be very careful when making changes to workflows, try to keep them as general and re-usable as possible.
• Write high quality tickets.
• Service Desk: Use the same system as the bug tracker, which will allow all staff to track their tickets' progress in a way they understand. If this is not possible, make the service desk ticket system no less usable/visible than the staff issue tracker system.
• Ensure the service desk issue system is no less usable than email, i.e. original message chain quoted in all replies, and CCs honoured with reply-all.
• Have a special email address for people to CC which will copy the email to a ticket. Publicise it regularly.
• Choose software that makes a healthy development environment.
• Things to publish internally, and regularly re-post:
• instructions for transferring phonecalls internally
• instructions for setting up a conference calls (for all phone models in use)
• instructions for setting up video calls from conference rooms
• instructions for setting up a laptop/desktop with the big screen in meeting rooms/presentations
• List all regular non-stream tasks, define roles to triage, co-ordinate and communicate them to the rest of the team, with a view to improving and streamlining the work of the team. Examples:
• Failing tests
• Environmental issues
• Live bugs
• Supporting the release
• Inter-team liaison
• Monitoring reports
• Business investigations
• Ideally don't have any remote members of the team. But at the very least, don't inconvenience the co-located members for the sake of remote members. This means:
• Don't hold scrum via phone or skype, hold it in person. Figure out some way for the remote people to attend that doesn't take anything away from the people in the room. If there's no way, they can attend their own scrum, or not at all.
• Update: Video chat (e.g. Zoom) works well when there is a large TV screen and dedicated room microphone device. Whoever is in the office goes over to that TV & mic area and starts the video call so anyone working remotely can dial in and participate.
• Don't force people to communicate via headsets when they're standing next to each other. That would discard the many benefits of face-to-face communication.
• Make sure everyone actually stands up around a physical board (Update: Jira scrum/kanban board works fine over video chat, if updated outside the meeting). If headset range is too far to work, don't use them.
• Don't have the scrum at a weird time of day to accommodate other time zones.
• If it's difficult to hear the remote members then have them email points to the team.
• Try to arrange for the remote members to work one-on-one with each co-located member, to build rapport
• Put ALL work in the issue-tracking system to make it visible, even non-development work. If it's not tracked, you didn't really do it. Each task must link back up to team "epics" or long-term goals, and ultimately to a company goal. Must be extremely easy to create tasks with one line of text and no other info, i.e. prevent unnecessary overhead for simple tasks.
• Specific to large companies:
• Instant chat client for ALL members of staff, with zero configuration, it's just there waiting to be used. Hooked into Outlook and the O/S so that it knows when you're in a meeting or away from your desk and updates your status accordingly (e.g. MS Office Communicator)
• Have a meeting room booking system that is painless (is this even possible?) and transparent with regard to: higher-ranked employees permission to "override" bookings, policy for block-booking rooms every week, etc. Above all it should be obvious what is happening and why.
• IRC/Slack or equivalent for all. Make it clear that all communications will be logged.
• (crazy idea) Employ a person/team whose sole remit is to make it easier to communicate within the company. This is definitely not the usual "Internal Communications" team that sits with HR and serves the Executive. It's a team that is there to support IT workers.
• If the office is very large, publish a map of where each team sits. Make sure it's always up-to-date, either by giving a team the responsibility to do it, or making it editable by anyone on the wiki.
• Link from the team map to official wiki pages, etc. org chart, make everyone's names and roles discoverable and visually represented.
• Put signs up around each area so you can see which team sits where.
• Use scrumblr.ca or funretro.io or similar (virtual post-it note boards) for retrospectives, to allow ideas to be captured throughout a sprint, not just on the day. Also a wiki page does just fine. But it's also important to keep real, physical post-it notes and pens in the meeting for people to jot down ideas they just think of at that time.
• Create the following types of wiki pages, for each team or development group:
• Tech Debt / Yak Shaving / Infrastructure to-do list / Ideas / Suggestions (use your judgement if this should be one page or several). This is a list of things that people want to do, changes they want to make that are not on the roadmap, or don't obviously provide direct value to the business. The team may need help explaining where the value lies.
• Proposed technical discussion topics. Topics worthy of group discussion. Schedule a regular meeting (every 2-4 weeks) to discuss whatever is on this list and make a decision as a team about how to proceed.
• Developer FAQ. Either per-team or per-department.

Source: My own experience working in IT.

What a healthy development environment looks like

In my opinion, the best set up would include:

Software

• Jira for issue tracking 
• A wiki for information sharing 
• Git for version control
• Continuous integration and continuous deployment (e.g. with Jenkins)
• Use an existing package provider like Redhat, don't compile publicly available software yourself

Environment

• Full commit history available with details of developer name and change ticket ID
• Full ticket history available to read, with no historical tickets in the "old" system and unreadable
• /etc/init.d scripts ('service') for starting, stopping and restarting software
• VMs for development 
• Identical production, development and test environments
• Continuous Integration: Every check-in to any development branch is run through the test suite, and any failures are emailed to the author
• Immutable infrastructure: Servers are not changed. All changes begin with pushing to a repo, and a new server is then built and deployed to a subset of users, tested, then to all users.
• One-touch deployment, identical in both production and development environments 
• Continuous deployment: Every commit to trunk that passes the test suite is deployed
• Refactoring code as required, restricted only by the need to pass the test suite
• All major software components described on wiki with details of source code, restarting, troubleshooting, owner, etc.

Tests

• Easy to run the entire test suite for any branch
• Test suite should complete quickly
• A test suite that is comprehensive and well understood by all
• Test coverage stats are visible to everyone
• Tests for hallmarks of quality like 'use strict' and embedded documentation (even if a blacklist of legacy code is required)

Code

• Clear model-view-controller separation
• All code contained within re-usable modules (scripts are thin wrappers around classes)
• No "utility classes" (inherit/use roles if common methods are needed)
• No "god classes" (split up methods into meaningul sublclasses or roles)
• All logic contained within meaningful "noun" classes designed for problem domain
• Meaningful variable names in domain language and no unnecessary abbreviations
• Plentiful comments, written in the language of the problem domain
• All classes/modules to have embedded documentation

People

• For every piece of work, three people should discuss it together for a few minutes after a technical design has been proposed, and before any development work is started - a representative of the business, a QA tester and a developer.
• They should ask questions like "Are the requirements accurate and high quality?", "Does the proposed solution fulfill the requirements?", "Is the proposed solution high quality?", "Can the proposed solution be easily tested?", etc.
• The product manager as well as the people coming up with requirements must be readily available to answer questions.
• To facilitate communication. teams should *not* be split across timezones
• (separate) Scrums should take place at the beginning of the local day

Don't indent so much

TL;DR: Don't indent over more than one page.

When programming, please factor your subroutines enough. One test for this is the level of indentation. If you find the code is indented for longer than one page (i.e. around 40-50 lines), then split that indented code out into its own subroutine.

This applies to loops, if-then statements, try-catch blocks, and given-when blocks. Especially when any combination of those are nested over several pages, it makes the code very difficult to follow.

Anything that indents the code indicates a logical grouping which you may find fits happily into a new subroutine. You can even give the subroutine a meaningful name which will serve as a signpost to future maintainers (not to mention a useful checkpoint to assert that variable contents are correct by validating the subroutine arguments).