POD servers for Perl

You want to browse Perl embedded documentation (POD)

You have some software that was designed for browsing information: a web browser

You need a server to turn the POD into web pages and serve it to the browser.

You try one of these modules:

https://metacpan.org/pod/Pod::Browser

Uses Catalyst

https://metacpan.org/pod/Pod::Server

https://metacpan.org/pod/Pod::Webserver

Very basic
Parsing much better than Pod::POM::Web, but still fails in places.
Contains all the Perl built-in modules, which I don't want to see.
How to run: podwebserver

https://metacpan.org/pod/Pod::POM::Web

Advanced design
Fails to parse nested tags
Contains all the Perl built-in modules, which I don't want to see.
How to run: perl -I $DOC_PATH -I $POD_POM_WEB_PATH -MPod::POM::Web -e server

Questions:

What is the reference implementation of a POD interpreter? `perldoc` just removes links and it is plain text only. So in what viewer do the links actually link to something?

Docker guidance, 2020

DOCKER

Link: https://www.docker.com

Description: Docker download

Link: https://github.com/jesseduffield/lazydocker

Description: Lazy docker - a super cool management of containers (found this weekend)

Link: https://github.com/docker/machine

Description: Docker Machine - easy remote running of docker containers

Link: https://github.com/bcicen/ctop

Description: Like htop but for containers (found this weekend)

Link: https://www.youtube.com/watch?v=3c-iBn73dDE

Description:This women is nothing short of amazing, she does loads of free videos on docker and k8

Level: Advanced

Link: https://www.youtube.com/watch?v=eGz9DS-aIeY

Description: A quick and dirty getting started

level: beginner

Link: https://www.youtube.com/watch?v=IbUXb4pQbPY

Description: a quick overview of docker - Tech lead can be hard to watch at times tbh

level: mid

Link: https://github.com/docker/awesome-compose

Description: A curated list of Docker Compose samples

Link: https://github.com/veggiemonk/awesome-docker

Description: if you are just not a video person

my fav containers (you should check them out):

- mysql

- postgres

- grafana

- Redis

- portainer

- plex

- Rancher

- traefik

- nextcloud

- influxDB

- loki

- cadvisor

- pi-hole

- ubuntu/debian

(source: Ash-J B)

Mojo::UserAgent request format

The get(), post(), etc. methods are both too magic, and not magic enough.

They do what I mean, but only if I remember the order or parameters. Luckily Perl has an easy solution to this problem: Named parameters! AKA Passing a hash, like a normal person.

Good hex values

For UUIDs:

c0ffeeee-cafe-cafe-cafe-c0ffeec0ffee

fadedfad-fade-fade-fade-decaffdecaff

g00dg00d-g00d-g00d-g00d-g00dg00dg00d

Perl Postgres test fixtures

Test::PostgreSQL approach:

Test::DBIx::Class approach:

MySQL basics - creating a user, etc.

Creating a user:

CREATE USER 'user'@'hostname';

ALTER USER 'user'@'hostname' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON dbName.* To 'user'@'localhost' IDENTIFIED BY 'password';

(source)

for file in `grep -r mysqld t|cut -d: -f1|uniq`; do sed -i -e '/use Test::mysqld/d; /Test::mysqld->new/N; /Test::mysqld->new/N; /Test::mysqld->new/N; /Test::mysqld->new/N; /Test::mysqld->new/N; /Test::mysqld->new/d; /traits/d' $file; done; find t -name '*-e' -delete

Result:

Install postgres on Ubuntu 16

How to do it:

sudo apt-get install postgresql postgresql-contrib
sudo -i -u postgres
psql

(source)

What is a professional attitude to software development?

Professional attitude:

Confidence without arrogance
Humility without self-pity
Politeness, approachability
Communicative, hard-working
Competence and reliability above all

Doesn't seem too hard does it!?

Documentation formatting, presentation and distribution

Points relating to formatting and presentation (not content):

Documents need version numbers. This comes for free with a wiki-based system.
Need a "changes" section at the top, including dates & authors.
A table of contents with clickable (and bookmarkable) links
Code samples to be in a consistent style, monospace font, and limited to a single page
Web page or PDF, not .docx because .docx will be formatted differently by every application, and might appear corrupted or unreadable on some computers.
Host the document on company servers and send around a link to download it. Don't email as an attachment because it will become out of date quickly.

Add Javascript to a security protected page

If you want to run some JQuery (or any other script) on a page that doesn't have it already, you can load it like this in the web browser console:

var script = document.createElement('script');
script.src = "https://ajax.googleapis.com/ajax/libs/jquery/1.6.3/jquery.min.js";
document.getElementsByTagName('head')[0].appendChild(script);

But on pages with a strict security policy, you may get an error:

Refused to load the script '......' because it violates the following Content Security Policy directive: .....

Solution: Edit the DOM and replace the security policy with this one:

<meta http-equiv="Content-Security-Policy" content="default-src * self blob: data: gap:; style-src * self 'unsafe-inline' blob: data: gap:; script-src * 'self' 'unsafe-eval' 'unsafe-inline' blob: data: gap:; object-src * 'self' blob: data: gap:; img-src * self 'unsafe-inline' blob: data: gap:; connect-src self * 'unsafe-inline' blob: data: gap:; frame-src * self blob: data: gap:;">

(source)

Breakdown of policy from that guy:

default-src * self blob: data: gap:;
style-src * self 'unsafe-inline' blob: data: gap:;
script-src * 'self' 'unsafe-eval' 'unsafe-inline' blob: data: gap:;
object-src * 'self' blob: data: gap:;
img-src * self 'unsafe-inline' blob: data: gap:;
connect-src self * 'unsafe-inline' blob: data: gap:;
frame-src * self blob: data: gap:;

Hmm, that doesn't look quite right. Seems inconsistent.

Devel::Cover patterns for Perl

One:

cover -delete
PERL5OPT=-MDevel::Cover=-ignore,^local/ prove -r t
cover -report html_basic

Another:

perl -MDevel::Cover=+ignore,.*,-select,^lib/,-select,^t/,-silent,on t/path/to/test.t
cover

Or:

for file in `find t -name '*.t`; do perl -MDevel::Cover=+ignore,.*,-select,^lib/,-select,^t/,-silent,on $file; done

In full:

unset PERL5LIB; unset PERL_LOCAL_LIB_ROOT; eval $(perl -Mlocal::lib=./local)

export PERL5LIB="/home/will/alt/cpan/devel-cover/local/lib/perl5:$PERL5LIB"

PERL5OPT=-MDevel::Cover=-ignore,^local/ prove -I t/lib -lr t

perl -I /home/will/alt/cpan/devel-cover/local/lib/perl5 /home/will/alt/cpan/devel-cover/local/bin/cover

Using your bastion server as a socks proxy

How to access web services via a jump host:

ssh -v -N -C -D 9090 username@bastion.example.com

* `-v` verbose, see any issues as they arise
* `-N` don't execute a remote command
* `-C` turn on compression
* `-D #` open a SOCKS proxy on this local port

Browser setup: FoxyProxy (Configure to "Use Enabled Proxies By Patterns and Order")

NOTE: It's a bad idea to route all web traffic through your company's bastion server / jump host. Only route the hosts you absolutely need to.

Or for command line stuff:

ssh -v -N -C -D 9090 mad@bastard.example.com
hpts -s 127.0.0.1:9090 -p 8989 --level info
export http_proxy=http://127.0.0.1:8989/
export https_proxy=http://127.0.0.1:8989/

Recording HTTP responses with Perl

There are two main ways. If your HTTP requests are being done in a different way, you will need to build your own solution.

The LWP way:

use LWP::UserAgent::Mockable;

BEGIN {
# Options: record|playback|passthrough
$ENV{LWP_UA_MOCK} ||= 'playback';
$ENV{LWP_UA_MOCK_FILE} ||= "$0-lwp-mock.out";
}

END {
LWP::UserAgent::Mockable->finished();
}

# You might be able to normalise the requests using the callbacks

The Mojo way:

use Mojo::UserAgent::Mockable;

my $mock_ua = Mojo::UserAgent::Mockable->new(
mode => $ENV{MOJO_UA_MOCK},
file => "$0-mojo-ua-mock.out",
# ignore_body => 1,
# or:
request_normalizer => sub {
my ( $req, $recorded_req ) = @_;

my ($time) = $recorded_req->body =~ m{(.*)};
my ($hash) = $recorded_req->body =~ m{(.*)};

my $body = $req->body;
$body =~ s{.*}{$time};
$body =~ s{.*}{$hash};

$req->body($body);
},

);

END {
$mock_ua->save if $ENV{MOJO_UA_MOCK} eq 'record';
}

ok my $t = Test::Mojo->new("My::App");

$t->app->mock( useragent => sub { $mock_ua } );

# Configure the proxy in record/passthrough mode only

if ( $MODE =~ /record|passthrough/ ) {

local $ENV{HTTPS_PROXY} = $t->app->config->{http_proxy};

local $ENV{HTTP_PROXY} = $t->app->config->{http_proxy};

my $proxy = Mojo::UserAgent::Proxy->new;

$proxy->detect;

$mock_ua->proxy( $proxy );

}

How to specify a library path when installing perl modules

Examples:

brew install openssl
export LDFLAGS=-L/usr/local/opt/openssl/lib
export CPPFLAGS=-I/usr/local/opt/openssl/include
export PKG_CONFIG_PATH=/usr/local/opt/openssl/lib/pkgconfig
cpanm IO::Socket::SSL

Or:

cpanm --look Crypt::OpenSSL::X509
vim Makefile.PL
 + inc '-I/usr/local/opt/openssl/include -I/usr/include/openssl -I/usr/local/include/ssl -I/usr/local/ssl/include';
 + libs '-L/usr/local/opt/openssl/lib -L/usr/lib -L/usr/local/lib -L/usr/local/ssl/lib -lcrypto';
perl Makefile.PL
make
make test
make install

(source)

Requirements discussion : Are the requirements ok? Requirements checklist
Sprint planning / estimation : Are you planning and estimating well? Sprint planning and Estimation checklist
Adding an item to the backlog : No checklist, to make it as easy as possible to add ideas?
Construction : Are you designing and constructing well? Design checklist / Construction checklist / Database checklist / Monitoring checklist
Bug fixing/refactoring : Are you maintaining code well? Maintenance/Editing checklist
Deployment : Are you finishing up neatly? Agile checklist / I'm finished checklist / Deployment checklist / Documentation checklist

Requirements checklist

Have you removed any requirements that are not strictly necessary?

Have you removed any requirements which can be postponed until a second phase?

Have you made a decision on absolutely all of the open questions?
Remember that if one person doesn't make the decision, someone else will be forced to.
i.e. if Client doesn't decide, Engineer must. Or if Engineer doesn't decide, Client must.

SCRUM

Is the white card specified in the form "As an XXX I want to YYY". (This forces you to ask the question "When will this project really be considered 'done'?")

Have you identified the most risky requirements? (dependencies, unknown, etc).

Have you stated any assumptions explicitly?

Volere

Have you noted a rationale for the requirement?

Have you given the requirement a unique ID? (e.g. tracked in a ticketing system).

Have you noted who raised the requirement (person's role, if not their name).

Have you written fit criterion (How will I know when I'm done?)

Have you noted any dependencies?

Have you recorded the history of the requirement (changes made)?

Sprint planning and Estimation checklist

This checklist should be used in Sprint Planning.

NOTE: As Sprint Planning is often the first time we hear of a requirement, also use the Requirements checklist here.

Have you created a task to clarify requirements further, if necessary?

Is the white card in the form of a story?

Are the requirements framed as a problem, or are they really a suggested solution?

Preparation

Have you made some rough technical designs and considered several solutions?

Estimating

Has enough research been done to give a good enough estimate?

Have you tried to counteract your usual bias?

Have you compared it to previously completed tasks? (Add or subtract time depending on the differences between this task and the known one).

Have you thought of the best case, and the worst case, and done the calculation? (best + 2*worst) / 3

Have you created tasks to:

write test cases before writing code (Test Driven Development)

model the environment

only write code to pass tests

test the solution end-to-end

get a Dev/Test environment set up that is identical to live (e.g. Forge / ExtDev)

QA (testing by someone else)

check that foreign characters are encoded/decoded correctly (tests)

build in monitoring

deploy to a production environment

write documentation

get the solution signed off

clarify the requirements?

Design checklist

This should be read before you have completed the design of the system.

Have you looked over the other Technical Specification checklist?

Have you asked around to see if anyone's solved the problem already?

Have you considered:

using any off-the-shelf solutions (e.g. FeedEngine, Cocoon, IDOL)
whether or not using a database would be beneficial?
the use of any design patterns?
using the most appropriate language for the job?

Have you identified any gaps in knowledge, e.g. HTTP connection in C#

Maintainability

Have you made sure that especially any data-based parts of the system can be easily modified?

e.g. Consider how much work is involved to add a new dataset to clickthrough tracking

Modularisation

Design the Module's Interface First (Damien Conway)

Is any functional routine able to be called on its own from a stand-alone script?

Have you set out proper objects and their relationships?

On paper

Have you drawn a diagram of the information flow, at the very least?

Any amount of UML diagrams at the most

Have you drawn a class diagram to begin with?

Have you defined the major data structures?

Have you written a pseudo-configuration file?

Has the main routine been written in pseudo-code?

Has each sub-routine been written in pseudo-code?

Has each routine been broken down in enough detail so that it almost seems simpler to just write the code instead?

Has care been taken to contain all the business logic separately from the internal logic?

Does the design extend far enough in each direction?

i.e. are there defined mechanisms for acquiring input, and publishing output?

Does the design take account of any possible non-ASCII characters?

Investigation

Have you fully investigated all of the options? (i.e. followed through on each design idea until you're sure it either will or won't work, possibly using prototypes)

Have you asked around for previous solutions to the various possible solutions? (engineers/leads from different teams, and people with experience).

Have you highlighted any risk? (risk = unknowns)

think about testing, during design

Have you thought about how you would test the system?

NB: a test consists of an input and an expected output.

Agreement

Has the client explicitly agreed to the inputs they'll provide, and outputs they'll receive?

Construction checklist

Have you entered the RCS keywords into the file for Perforce? (replace with your own VCS)

# $File$

# $Revision$

# $Change$

# $DateTime$

# $Author$

Write the Test Cases Before the Code (Damien Conway)

Are you using Log4Perl?

Have you got all the modules/libraries you need?

Are you using strict/warnings?

Are you avoiding creating any unnecessary dependencies?
(loose coupling)

Agree Upon a Coherent Layout Style and Automate It with perltidy (Damien Conway)

Code in Commented Paragraphs (Damien Conway)

Throw Exceptions Instead of Returning Special Values or Setting Flags (Damien Conway)

Don't Optimize Code--Benchmark It (Damien Conway)

Database checklist

Have you drawn it out on paper first?
Have you used foreign key constraints for referential integrity?
Have you avoided optimising early?

From the perspective of data warehousing:

Have you run every query with EXPLAIN to check that the correct index is being used and it's as efficient as possible?
Have you run every query with small and large ranges of data? (different indexes get used under different circumstances).
Have you considered the effect of indexes (or lack of them) on INSERT, DELETE and UPDATE as well as SELECT?
Have you built a set of test queries representing the full range of data, in order to test performance?
Have you considered using memcache to improve query response time?
Have you compared more than one type of each query? (sub-queries, temporary tables, etc).
Have you considered keeping lookup data tables in the application memory rather than using expensive joins?
Have you considered redesigning the data gathering, loading and schema to facilitate more efficient queries?
Data warehousing specifically: Have you avoided ENUMs and allowed any data to be inserted? (add to lookup tables and be given a description later).

Monitoring checklist

Have you considered using a framework that periodically publishes its status via HTTP? (e.g. BBC perforce //depot/osd/Importer)

Have you considered the need to write a CGI script to make the necessary checks and report a simple error code back for use in Nagios?

Are there fine-grained, customised, complex or non-standard requirements for monitoring? If so, consider using your own team's Nagios installation instead of Livesite's.

Livesite like to send alerts to group email addresses, every hour, and use only simple regex matches. They are also geared towards critical systems that need to be supported 24/7.

Example of monitoring criteria:
For Search+ PAL/JBoss:

Load - warning at 15 load, critical at 30
Memory/Swap - warning at 70% used, critical at 85%
Diskspace - warning if any disk 80% full, critical at 90%

IDOLs (e.g. nolidol301):

IDOL [Collections2] - ACI STATUS OK 07-02-2009 16:57:10 30d 1h 55m 42s 1/3 AUTONOMY OK

IDOL [Collections2] - INDEX QUEUE OK 07-02-2009 16:59:58 1d 16h 41m 29s 1/3 DRE_INDEX_QUEUE OK

IDOL [Collections2] - PERFORMANCE OK 07-02-2009 16:59:19 30d 1h 56m 3s 1/3 DRE_STATISTICS OK

IDOL [iPlayer] - QUERY OK 07-02-2009 16:56:58 3d 17h 56m 11s 1/3 HTTP OK

Maintenance checklist

To cover any changes made to code / configuration: Adding new features or debugging issues

have you sync'd from the repository? (checking first to see if there are any unsubmitted changes in your workspace)

Add New Test Cases Before you Start Debugging (Damien Conway)

have you re-read the design checklist with the proposed changes in mind?

is the system adequately documented?

have the requirements been specified in enough detail?

are you modifying the design, then implementing the design? (as opposed to modifying the code)

Agile checklist

To check that you're benefitting from agile methodologies

Have you split up tasks into MMFs? (Minimum Marketable Features)

Have you deployed as early as possible? (As soon as there is a stable feature to show the client).

I'm finished checklist

Could you rename some variables or routines to make them clearer?

Could you refactor the code to make it clearer?

put code into subroutines
clean up comments

Perl: Have all print, die and warn statements been replaced with $LOG->info, $LOG->logdie and $LOG->warn statements? (log4perl/log4j)

is all the code checked into the versioning system?

are only the changes since the last release checked in and tagged with a new release number?

does the system satisfy the original requirements?

How can you demonstrate it? (Check against original requirements)

has the system been tested?

Has it passed an end-to-end test in the production environment, with realistic levels of traffic?
How confident are you it will hold up?
Have you tried to break it with bad input?

are all paths set to the live environment?
grep -r your_username packagedir/

is the system been deployed into a suitably monitored environment?

who will be notified of errors that occur during execution?

have you documented (from least technical to most technical):

product details
user operation
support operation
flow of logic / requests / component interaction
database structure
data structures in the code

Create Standard POD Templates for Modules and Applications (Damien Conway)

have you told everyone about it that needs to know?

have you considered the need to restrict access?

will you be able to tell who is using it / how they are using it?

Deployment checklist

General:

Has it been installed onto an empty VM and run successfully?

Have all the steps to get it running on the VM been documented for Ops?

Have you run unit tests?

Have you run functional test manifests?

Have you performed the set of sanity checks or manual checks?

Have you updated the package with any hot fixes you've quickly made to get it working for the first time?

BBC Livesite conventions:

Have you considered to what data the app will need access, and which modules it will need?

Have you consulted Livesite on any alternative architecture that might simplify their deployment and maintenance work?

Have you set the configuration to write data and logs to the /data partition?

Have you considered packaging as an RPM for ease of install and upgrade?

Have you considered writing an init script instead of a crontab entry?

Have you arranged for access to the server and application logs in order to resolve issues faster? (e.g. they could be made available on the web servers at /apache_logs/).

Afterwards:

Have you updated the deployment log on the wiki?

Have you updated the operational info on the wiki?

Documentation checklist

Template for documentation

Written with a wiki in mind.

For each product/project (landing page)

At which stage of development is it?
Summary: What is it? What does it do? Why do we need it? Who uses it?
Contacts: Who owns it? Who develops it? Who supports it?
Access: How do you access it? Do you need permission?
Context/Scope: What similar systems are there? Why weren't they used? What does it integrate with? Where is the system used?
Components: What are the different parts of the system? What do they do?
Architecture: Is there an overview diagram?
Troubleshooting: What might go wrong? What has gone wrong in the past? How does one fix it?

For each component (separate page)

Summary: What is it? What does it do?
Contacts: Who develops and supports it?
Operational: Where is it hosted? Who administers the servers?
Architecture: Is there a diagram of this component?

Component development information (separate page)

Where is the code stored?
How does one install it/set it up?
What is the workflow/dataflow? (pseudocode/flowcharts)
What are the features?
What are the known bugs?
What work is planned?

Working notes (separate page)

Anything your colleagues would need to know, should you win the lottery and never be seen again!
An area to save ongoing development notes to share with your colleagues

Will Sheppard, BBC, 2008

Requirements checklist

Sprint planning and Estimation checklist

Design checklist

Maintainability

Modularisation

On paper

Investigation

think about testing, during design

Agreement

Construction checklist

Database checklist

Monitoring checklist

Maintenance checklist

To cover any changes made to code / configuration: Adding new features or debugging issues

Agile checklist

I'm finished checklist

Deployment checklist

Documentation checklist

Template for documentation

For each product/project (landing page)

For each component (separate page)

Component development information (separate page)

Working notes (separate page)

Search this blog

Highlights

Quick reference

Links

Blog Archive

Subscribe To

Tags