Development notes: 2019

How to copy off images from Chrome's cache (on Mac)

How to find Chrome cache on Mac

export YOU=your_username
export PROFILE=Default

mkdir ~/chrome_images
cp /Users/$YOU/Library/Caches/Google/Chrome/$PROFILE/Cache/* ~/chrome_images/
for k in `/bin/ls`; do mv $k $k.jpg; done # rename all files to jpg

open ~/chrome_images

# Now browse the image folder and find the images you want

# More sophisticated renaming example:

if [[ `file $filename | grep JPEG` ]]; then mv $filename{,.jpg} ; fi

YAML tricks to DRY

Use YAML anchor to copy & paste, or deduplicate data using DRY principle for YAML:

credentials::app-name: &app_credentials_anchor # <-- amp="" anchor="" b="" copies="">
db:
username: foo
password: bar

credentials::app-name-worker: *app_credentials_anchor # <-- alias="" b="" pastes="">

Docker basics

# Download an image and spin up a container, run it, connect to it

docker pull [name pf image] # download an image

docker run -d -p 80:80 --name pintail-whoami pintailai/pintail-whoami:0.0.1 # download + run

docker run --rm -v /path/on/machine:/app/out image-name:stable params to app

# List running containers

docker ps # show running containers

docker ps | cut -c-$(tput cols) # show running containers without wrapping to the next line

docker ps -q # show just the IDs of the running containers

docker ps -q | head -1 # show ID of most recently started container (how to sort)

# Run a shell on a container

docker run -i -t [Container ID] /bin/bash

# Connect to a shell on an already-runnning docker container.
# (Shell: Use /bin/bash for ubuntu or /bin/ash for alpine)

docker exec -it [Container ID] [shell]

docker exec -it `docker ps -q | head -1` /bin/bash # run shell on the most recently started container

# Copy a file off

docker cp <container>:<src-path> <local-dest-path>

# Delete all stopped containers and images

docker system prune -a

# List all images

docker image ls

Install perl really fast

Fastest way to install perl:

perlbrew install -j 5 --notest perl-5.16.3

Only takes about 2 mins.

Thanks perlbrew.

How to build a CPAN module

Not really, this is just a few rough notes.

https://metacpan.org/pod/Dist::Zilla::Tutorial - shows the format of dist.ini

http://dzil.org/tutorial/start.html

Still use a cpanfile, and also Dist::Zilla::Plugin::Prereqs::FromCPANfile

With dzil you can choose whether you want to generate a Makefile.PL or Build.PL

Changes:

Thanks to Nelo & the team.

Test that all your module dependencies are listed in the cpanfile

Put this at xt/author/cpanfile.t: Thanks Anton & the team.

When Postgres allows any password

Problem: Postgres does not check my password, i.e. it accepts all passwords.

Solution: pg_hba.conf and change trust to md5. Restart postgres. That is all.

(source)

SSH tunnel using a jump host

Access a service on a remote machine via an intermediary

ssh -v -L 4444:app.example.com:5000 $USER@jump.example.com -nNT

Now you can access the service running on app.example.com:5000 by going to localhost:4444 in your browser.

Explanation of the command

from the host machine (where you are running the command)
connect to jump.example.com as user $USER
once there, access service app.example.com on port 5000
then make that service available on the host machine on port 4444

Advanced usage - Two jumps

ssh -J user@jump.example.com user@app.example.com -L 1111:database.example.com:3306 -nNT -vvv

Notes:

-J jumps to another host

-L makes a tunnel to a service that's already running

Now you can do:

mysql --protocol=tcp --host=127.0.0.1 --port=1111

Notes:

- you must specify protocol because of the tunnel

- specifying 127.0.0.1 (instead of "localhost") prevents MySQL trying to use a local socket and failing

Use a proxy on a remote machine via an intermediary

If there's a proxy you need to use: proxy.example.com:8888 -- but you can only access it from jump.example.com -- then set up a tunnel like this:

ssh -A -L 4444:proxy.example.com:8888 $USER@jump.example.com -nNTv

Now you can use http://localhost:4444 as your proxy server, instead of http://proxy.example.com:8888

Vim mappings for commands like JSON pretty-print and perltidy

imap :!python -m json.tool
map :%!perltidy

?
nmap :%!python -m json.tool

# AB
nmap :%!python -m json.tool
map :!perl -cW %
map :%!perltidy
map :resize +10

How to work with dist.ini as a user

You are a developer who needs to work with a Perl package. Instead of a cpanfile it has a dist.ini. Pop quiz, hot shot: What do you do?

Answer:

cpm install Dist::Zilla
dzil build
Follow instructions to install dependencies using cpanm

Or if you prefer cpm (and you should), then try this: dzil listdeps | xargs cpm install

Or: dzil authordeps --missing | xargs cpm install

Mac clients for AWS RedShift

JetBrains just released DataGrip ( JetBrains DataGrip: Your Swiss Army Knife for Databases and SQL ). I've been using it full time during the EAP (aka "beta"). It's expensive ($90/yr or $9/mth) but very, very good. Never loses your work. Starts up right where it left off. Dark and light themes. DB specific syntax highlighting. Good stuff.

Before that I was using Navicat Premium Essentials. I got it for ~$20. It's now ~$150 which is too much given it's limited functionality and spotty reliability IMO.

re:dash is overly simple for serious use IMO. Edit: We're actually using this now. It's more of service than an SQL client. It allows casual users (e.g. not full time analysts) to run queries without any setup. Great for centralised dashboards and sharing queries. Try this before you commit to something like Periscope, Looker or Mode.

Portico (was PG Commander) looks interesting but I (personally) need to work with multiple database flavors so it's a non-starter.

SQuirreL and SQL Workbench/J are both pretty crunchy and old feeling Java apps. If you've been on a Mac for a while they will make your eyes bleed.

There is also Toad on the Mac App Store. The screenshots look Mac native but it's pretty crunchy and unpleasant. It feels like going back in time.

Update 2016–09–02: I would suggest that you also look into the “code notebook” applications that are available now, e.g. Jupyter, Zeppelin, and Beaker. You may well find these tools more useful unless you’re a full time DBA. We have replaced Re:Dash with Zeppelin internally.

Why Android is better then iPhone

A list:

Homescreen widgets
Custom homescreen launchers, e.g. Nova Launcher
Custom homescreen icon arrangement
Notifications persist visibly in status bar so you don't forget them
Plentiful game console emulators
Easily download files from web pages
Easily load mp3 music without special software
Save any file to the phone storage, like you can with Windows/Mac/Linux

Migrate everything from iPhone to Android

How to transfer all your stuff from iPhone to Android:

Photos - install Google Photos app on iPhone and it will back up everything to the Google cloud.
Contacts - Add a gmail account on your iPhone and configure it to sync contacts (alternatively, you could export a vCard/VCF).
Calendar - Add a gmail account to the iPhone and configure it to sync calendar.
Notes - Add a gmail account to the iPhone and configure it to sync notes (there are also other ways).
SMS text messages - Use iSMS2droid
Email - on the Android phone, go to Settings | Accounts | Add account, and choose Personal IMAP/POP3 or Exchange, according to the type of email account you have
WhatsApp chat history - Use WhazzapMigrator (source, source)
Apps - you will have to install new Android apps manually. Your app data will be lost unless they have their own cloud backup.

Remember to TURN OFF iMessage before you switch over, otherwise your SMS may get lost (source)

Afterward, you can easily check some of your data has been migrated at:

photos.google.com
contacts.google.com
calendar.google.com
mail.google.com - for notes, if you used that method

Finally, bask in the satisfaction that Android is better than iPhone.

Get command history in the perl debugger

Problem:

When you press "up", you don't get the previous command.

Solution:

Install module Term::ReadLine::Gnu

"up" will now work.

(source)

Get www and https back in the Chrome URL bar

Restore to a configuration useful to web developers:

Adjust the Chrome settings by disabling the following flags:

Omnibox UI Hide Steady-State URL Scheme

In the omnibox, hide the scheme from steady state displayed URLs. It is restored during editing. – Mac, Windows, Linux, Chrome OS, Android

#omnibox-ui-hide-steady-state-url-scheme

Omnibox UI Hide Steady-State URL Trivial Subdomains

In the omnibox, hide trivial subdomains from steady state displayed URLs. Hidden portions are restored during editing. – Mac, Windows, Linux, Chrome OS, Android

#omnibox-ui-hide-steady-state-url-trivial-subdomains

Omnibox UI Hide Steady-State URL Path, Query, and Ref

In the omnibox, hide the path, query and ref from steady state displayed URLs. Hidden portions are restored during editing. – Mac, Windows, Linux, Chrome OS, Android

#omnibox-ui-hide-steady-state-url-path-query-and-ref

A good try/catch pattern for exceptions and failures

This Perl code neatly captures (1) unexpected exceptions, and (2) expected failures:

Add headers or cookies to request in Mojolicious tests

When using Test::Mojo, sometimes you want to modify the request to add a cookie or custom headers.

.

How to write tickets

Anatomy of a good Jira ticket

User story

AS A [role]
I WANT [something]
SO THAT [end goal]

Background/details

This was attempted earlier in ATS-123 but it didn't work
This is to solve the larger problem that __________ (e.g. we are spending too much time supporting this product)
Who originally raised the requirement and why (person's role, if not their name)
Any potential blockers/risks highlighted here
Names of functions or files to look at
Links to the relevant code repos

Acceptance Criteria

(it's done when) all passwords are validated
(it's done when) the publish button is no longer visible
(it's done when) the page loads in under 2 seconds

Attachments

Screenshot of the relevant page / error message / whatever
Email chain where this has been previously discussed

Comments

Any open questions should be noted here so that the implementer is aware of them
One possible solution was to use the ABC widget to achieve this.
Here are some extra details from product
You will need to ask Alex for the access details
Summaries of subsequent online discussions about the story should be copied here instead of remaining hidden in email or Slack
Summaries of decisions made at meetings about the story (after starting it) should be put here too

Example of a good bug report

The key points of a bug report are:

Steps to reproduce
What currently happens
What is expected to happen instead

Description

Users should be able to select (and apply) the ‘disabled’ filter on the ‘Agents’ page without being redirected to another page.

Scenario (what the user did / steps to reproduce)

Log in and navigate to 'Agents' section.
Select the ‘disabled’ filter button.

Expected result

The agents table should only show ‘disabled’ agents.

Actual result

The user is redirected to another page -- this is an undefined ‘Agent’ page.

Other general notes

Have all stakeholders communicate via the ticket and @ people to get their attention. Any communication via email or Slack, etc. risks that info being lost, unless copied/attached to the ticket.
Ensure that all requirements/decisions/etc which are discussed one-to-one or during a meeting are added back into the ticket so nothing is lost or forgotten.
Conduct all conversations in open channels that allow visibility into the decisions for others working on related issues.
Continuously keep tickets updated with the current status so the whole team is aware of where everyone else is with their tasks.
Highlight any risks or blockers ahead of time.

Code context

"Providing other developers with the names of functions or files to look for from someone with deeper knowledge of the codebase can save an immense amount of time up front and avoids potential refactoring down the road. It also reduces guesswork and more importantly, might reinforce a best practice when there are multiple approaches that would technically work. Examples of previous implementations or before-and-after code samples are also great things to consider providing"

Why spend time writing good tickets

Regardless of the expected assignee’s knowledge, the extra time spent to write a good ticket is rarely wasted. Tickets often get passed around as people take vacations, flat tires happen, and children get sick. When these things inevitably occur, we don’t want to rely upon assumptions, which, no matter how good, only need to be wrong once to cause potentially large amounts of wasted time, embarrassment, liability, etc.
Treat it as an opportunity to show respect to each of the many professionals that will touch the ticket through its lifecycle by providing them the information and tools they need to perform their job to the best of their abilities.

source: https://chromatichq.com/insights/anatomy-good-ticket

Note the background: What has been done, what exists, and the general situation. Explain it here.
Note the questions: As I'm writing the ticket, I'll have questions so I just list them here until I have a chance to speak to someone who can answer them.
Note the business value: Often companies would like to keep track of the ROI or business value of a feature so that it can be used during backlog prioritisation and sprint planning meetings. You can talk about revenue gains, cost savings, engagement, customer satisfaction, and other benefits here.
Note the metrics: It's good to indicate which metrics we are measuring in relation to this story. For example, how many users are affected by this and what's the current engagement? This gives an overview of the scale of the feature and a snapshot of the situation before the feature is released.

source: https://www.taigeair.com/JIRA-Ticket.html

And "Writing Better User Stories and Bug Tickets"

Coding wisdom

While working with Broadbean, I was reminded of some of these pearls for Perl, and learnt some others for the first time:

Database:

Use timestamp for when something happened: updated_time, created_time.
Use datetime for when something is scheduled to happen: reminder_time, renew_time - this is why datetime depends on the timezone; the point-in-time of the reminder depends on the daylight savings in that timezone at the point the reminder should fire.

Version control:

The commit message should explain WHY the change was needed, not WHAT the change does (because what the change does should already be apparent from the code and in-line comments).

Code:

If you have an if/unless conditional you can fit on one line, use a post-fix:

if ( $i == 3 ) { return $i }

is better written as

return $i if $i == 3;

The second form is better because it discourages nested logic (which is harder to read and maintain)

Generally prefer subroutines and "return" statements to control flow, instead of "if/then/else" statements. This way the code is factored into smaller chunks that are easier to understand, modify and test.

Always include a dry-run option in stand-alone scripts meant for production.

Every pull request must contain one and only one feature/behaviour that needed changing. This reduces cognitive drain on human reviewers who are required to scan everything and may therefore miss some details.

Perl module preferences

Preferences:

Never use Switch, it has some truly awful bugs.
Use Try::Tiny instead of TryCatch, it's less magical/scary.
Instead of JSON, use Cpanel::JSON::XS or JSON::XS

It's safer to explicitly name the package being used, because JSON picks one depending on what's already installed.

Mock time with Test::Time for Perl

Voila:

use Test::Time time => 1;

my $now = time; # $now is equal to 1
sleep 300; # returns immediately, displaying a note
my $then = time; # $then equals to 301

Sending email with Perl

use Email::Stuffer; # everything else is a pain

How OG fixed his Mac AGAIN

`for i in ../opt/openssl/lib/lib*; do ln -vs $i .; done`

and also installing *pkg-config*

and also deleting a local directory from `~/`

nginx basics

How to serve up a directory on the web

sudo apt-get install nginx
vi /etc/nginx/nginx.conf # observe how the html { } section has: include /etc/nginx/sites-enabled/*;
vi /etc/nginx/sites-available/foo # build the server { } section

server {
listen 80 default_server;
root /var/www/html/foo;
index index.html
location / {
autoindex on;
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd;
}
}

# build .htpasswd file
ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/foo
service nginx restart

How to use Type::Params to validate your subroutine parameters in Perl

How to:
(Type::Params module)

Easy/Hard vs Complex/Simple

This may help estimating your stories in agile:

	Easy	Hard
Simple	Washing car	Pushing car
Complex	Driving car	Building car

How to do authentication with Mojolicious

It's like this:

401 vs 403 HTTP codes

Explanation:

401 'Unauthorized' should actually be 401 'Unauthenticated'
It should be used for missing or bad authentication.

403 'Forbidden' is the server telling you, "I know who you are, and you don't have permission to access this resource".

(source)

Investigating Jira

They use Jira for issue tracking where I'm working.

Jira displays little coloured dots that seem to indicate the length of time an issue has spent in the column. If you hover over the dots it displays a tooltip telling you how long the issue has been in that column.

That would all be great except I don't know what the number and colour of the dots actually means, without hovering over them to see the tooltip. What's the point in even having coloured dots if the exact meaning of them is kept a mystery?

I don't like mysteries, so I set about trying to reverse engineer the meaning.

Yes I did try RTFM first, but the documentation is horrendously confusing and frankly quite disappointing for a popular application in 2019. I can't even see what version of Jira (Cloud?) I'm using - the about screen says "Jira 2c6be9a9" which looks like a version control commit ID. It also mentions "Jira Service Desk Application 3.3.0-OD" but that doesn't seem right because Wikipedia says the latest version of 7.13.0.

Investigation:

View source of Jira board, drill down to find the widget:

Search for the "more-then-5" (sic) class across the CSS assets of the page, brings us to base.css and:

I schlepped through all the page assets but I couldn't find the actual images used. Still at least we now know the "time taken" thresholds: 1, 2, 3, 5, 8, 12 and 20 days.

Based on empirical observation, I would tentatively map as follows:

1 day: O
2 days: OO
3 days: OOX (yellow)
5 days: OOOX (red)
8 days: OOXX (red)
12 days: OXXX (red)
20+ days: XXXX (red)

Key:

O = grey

X = colour

Batch deleting files in Windows

List all the jpg files:

dir /s *.jpg /b > jpgs.txt

Delete all the jpg files:

del /s /q /f /a *.jpg

How to disable Mojo debug logging

When setting MOJO_LOG_LEVEL does not work... Make the following hacks:

removing debug output

~ ============================================================================ Tuesday 12th February 2019

~~~ FIXES FOR OLD VERSION OF MOJO (Supervillain 8.03)

../local/lib/perl5/Mojolicious.pm
132: $self->log->debug(qq{$method "$path" ($id)}) if $ENV{BB_DEBUG} || $ENV{MOJO_DEBUG};

../local/lib/perl5/Mojolicious/Controller.pm
209: $app->log->debug("$code $msg (${elapsed}s, $rps/s)") if $ENV{BB_DEBUG} || $ENV{MOJO_DEBUG};

../local/lib/perl5/Mojolicious/Plugin/EPLRenderer.pm
18: $c->app->log->debug("Rendering cached @{[$mt->name]}") if $ENV{BB_DEBUG} || $ENV{MOJO_DEBUG};
30: $c->app->log->debug(qq{Rendering inline template "$name"}) if $ENV{BB_DEBUG} || $ENV{MOJO_DEBUG};
40: $c->app->log->debug(qq{Rendering template "$name"}) if $ENV{BB_DEBUG} || $ENV{MOJO_DEBUG};
46: $c->app->log->debug(qq{Rendering template "$name" from DATA section}) if $ENV{BB_DEBUG} || $ENV{MOJO_DEBUG};

../local/lib/perl5/Mojolicious/Routes.pm
95: $app->log->debug('Routing to a callback') if $ENV{BB_DEBUG} || $ENV{MOJO_DEBUG};
150: $log->debug(qq{Routing to application "$class"}) if $ENV{BB_DEBUG} || $ENV{MOJO_DEBUG};
164: $log->debug(qq{Routing to controller "$class" and action "$method"}) if $ENV{MOJO_DEBUG} || $ENV{BB_DEBUG};

~ ============================================================================ Thursday 7th March 2019

local/lib/perl5/Mojolicious.pm
123: #return unless $ENV{MOJO_DEBUG}; # WS hack

local/lib/perl5/Mojolicious/Controller.pm
187: return unless $ENV{MOJO_DEBUG};

local/lib/perl5/Mojolicious/Routes.pm
103: $app->log->debug('Routing to a callback') if $ENV{MOJO_DEBUG};
170: $log->debug(qq{Routing to controller "$class" and action "$method"}) if $ENV{MOJO_DEBUG};

x
491:>> /Users/will/dev/ats-broadcast-hub/local/lib/perl5/Mojolicious/Routes.pm:103: $app->log->debug('Routing to a callback') if $ENV{MOJO_DEBUG};

~~~~ AND ~~~~

To fix CODE(0x7fde03335ca0) now:

vi local/lib/perl5/Mojolicious.pm +122

~~~ OLD VERSION OF MOJO

local/lib/perl5/Mojolicious.pm

60 our $CODENAME = 'Supervillain';
61 our $VERSION = '8.03';

119 # Start timer (ignore static files)
120 my $stash = $c->stash;
121 unless ($stash->{'mojo.static'} || $stash->{'mojo.started'}) {
122 my $req = $c->req;
123 my $method = $req->method;
124 my $path = $req->url->path->to_abs_string;
125 my $id = $req->request_id;
126 $self->log->debug(qq{$method "$path" ($id)});
127 $c->helpers->timing->begin('mojo.timer');
128 }

~~~ NEW VERSION OF MOJO

119 # Start timer (ignore static files)
120 my $stash = $c->stash;
121 $self->log->debug(sub {
122 my $req = $c->req;
123 my $method = $req->method;
124 my $path = $req->url->path->to_abs_string;
125 my $id = $req->request_id;
126 $c->helpers->timing->begin('mojo.timer');
127 return qq{$method "$path" ($id)};
128 }) unless $stash->{'mojo.static'};
129

~~~ FIX FOR NEW VERSION (8.12)

vi local/lib/perl5/Mojolicious.pm

61 our $CODENAME = 'Supervillain';
62 our $VERSION = '8.12';

119 # Start timer (ignore static files)
120 my $stash = $c->stash;
121 my $req = $c->req;
122 my $method = $req->method;
123 my $path = $req->url->path->to_abs_string;
124 my $id = $req->request_id;
125 $c->helpers->timing->begin('mojo.timer');
126 if ($ENV{MOJO_DEBUG}) {
127 $self->log->debug(qq{$method "$path" ($id)}) unless $stash->{'mojo.static'};
128 }

vi local/lib/perl5/Mojolicious/Controller.pm

184 # Disable auto rendering and stop timer
185 my $app = $self->render_later->app;
186 # $app->log->debug(sub {
187 my $timing = $self->helpers->timing;
188 my $elapsed = $timing->elapsed('mojo.timer') // 0;
189 my $rps = $timing->rps($elapsed) // '??';
190 my $code = $res->code;
191 my $msg = $res->message || $res->default_message($code);
192 # return "$code $msg (${elapsed}s, $rps/s)";
193 # }) unless $stash->{'mojo.static'};
194 if ($ENV{MOJO_DEBUG}) {
195 $app->log->debug("$code $msg (${elapsed}s, $rps/s)") unless $stash->{'mojo.static'}; # MOJO_DEBUG
196 }