jq cheat sheet

jq manual

Debugging Perl data structures

Use Data::Tersify like this:

    print Dumper( tersify($complicated_data_structure) );

YAML tricks to DRY

Use YAML anchor to copy & paste, or deduplicate data using DRY principle for YAML:

credentials::app-name: &app_credentials_anchor # <-- amp="" anchor="" b="" copies="">
     db:
         username: foo
         password: bar

credentials::app-name-worker: *app_credentials_anchor # <-- alias="" b="" pastes="">

Reserved keywords - Do Not Use these in your code

Do not use these names, you will get confusing errors...

Perl:

• sub import

Mojolicious:

• stash('data', ...)

Best Data::Dumper configuration for debugging Perl

This:

  local $Data::Dumper::Indent=0;
  local $Data::Dumper::Varname='';
  local $Data::Dumper::Terse=1;
  local $Data::Dumper::Pair='=>';
  local $Data::Dumper::Sortkeys=1;
  local $Data::Dumper::Quotekeys=0;

(source)

Or:

use Mojo::Util 'dumper';

Wrap those magic data structures with a class

You understand the the dangers of using magic numbers in your code. And magic strings are another face of the same issue. When you use a lot of strings that have special meanings, it makes your code smell bad, i.e. it's an indication of low quality. These are not user messages or log messages, but rather a fixed string that if mis-typed will break the systems functionality. But it's not just the basic variable types that are magic. Arrays and hashes can also be magic, in the worst possible way.

If you find yourself using a hash in a lot of different places, this can be thought of as a magic hash. It may be a simple hash or it may contain many nested arrays and other sub-hashes. Every operation on the hash has to be done in exactly the right way or it won't work. It's very easy to perform a operation wrong and get unexpected results that won't be detected immediately. You're writing a significant amount of code to read and write the data within the hash, and to catch errors, and you're likely to be creating bugs too. The more code you write, the more bugs you create. This similar, duplicated and boring boilerplate code is spread around all over the application wherever the hash is used. It's also likely that you will need to use magic strings for the hash keys, with all the problems they bring. Even if you use an array at the top level, there may be hashes within it.

The solution is to put a class around the hash, so that you only need to write the hash manipulation code once, and can thoroughly unit test it. All the code related to this data is encapsulated in one place. All calling code will interact with the class interface instead of the hash directly. This is an example of object-oriented development, where objects are passed around and manipulated instead of raw data structures.

P.S. Even without using a class, replacing magic strings with constants would be a serious improvement. Maintainers will be unable to accidentally get a string wrong without seeing an error message that makes it very obvious what is wrong. It's a more foolproof way to develop.

Use Data::Compare to compare data outside a test

Need to compare data outside of a test, without generating "ok" / "not ok" TAP output? Use Data::Compare.

Create a lot of random data files on linux

# Create 10Mb random data

dd bs=1024 count=10240 < /dev/urandom > data

# Split into files of 10K each

split -a 5 -b 10240 data 'split.'

Use Data::Dumper with expensive data in perl

$logger->debug(sub { "large data structure is: ".Dumper( expensive_calculation($foo) ) });

This ensures that expensive_calculation() is not performed unless the log level is DEBUG or below.

Which data file format?

List:

• CSV
• TSV
• INI
• XML
• YAML
• JSON
• RDB
• custom

Make "x" use Data::Dumper in the Perl debugger

tldr:

$DB::alias{x} = 's/^x\s+(.*)/p Data::Dumper::Dumper($1)/';

Thanks Jim

Full article:

In the guts of the script that runs the Perl command-line debugger (accessible whenever you start a perl script with the '-d' command-line argument), there lives a hash, accessible globally as %DB::alias.  Whenever you enter a command at the debugger command line, the first word in the command is looked up in %DB::alias, and if it is found, the corresponding value is used as a substitution pattern against the entire command line.  The substituted line is then processed as usual.  The examples and explanation in the perldebug man page don't make this terribly clear.  The actual code that is executed by the debugger (after stuffing the first word in the command line into the variable $i) is:

eval "\$cmd =~ $alias{$i}";

This can be used to completely customize the debugger.  For example, the ordinary output of the debugger's built-in "x" command is pretty ugly.  Substituting a different formatter (such as Data::Dumper::Dumper) can be accomplished with the following alias:

$DB::alias{x} = 's/^x\s+(.*)/p Data::Dumper::Dumper($1)/';

This converts the line "x $something" into the line "p Data::Dumper::Dumper($something)".  Incidentally, solving this particular problem was the impetus for this blog post.

More extensive customization is possible.  For instance, alias substitution occurs once in the stock debugger.  Armed with the knowledge of how alias substitution works, you can create an alias that expands aliases iteratively like this:

my $in_exp = 0;

sub expand_db_aliases

{

     die "Recursively expanding the expansion alias" if $in_exp++;

     my $cmdref = shift;

     my $exp_cnt = 0;

     while($cmdref =~ /^(\S+)/ and $DB::alias{$1})

     {

          my $i = $1;

          die "Alias expansion exceeds 100 iterations" if $exp_cnt++ > 100;

          my $cmd = $$cmdref;

          package DB;

          eval "\$cmd = $alias{$i}";

          die $@ if $@;

          $$cmdref = $cmd;

     }

     $in_exp--;

} ;

$DB::alias{exp} = '//; expand_db_aliases(\$cmd);';

The heavy lifting is done in the expand_db_aliases subroutine, which matches the actual instructions used by the debugger to expand aliases as closely as possible.

Loading your customizations automatically

If there is a file named ".perldb" in your home directory, the Perl debugger will load it and interpret its contents as Perl code after it initializes itself.  You can put any code-based customizations you desire into this file.  You can (and should) also use this file to load any dependencies required by your customizations.  For the replacement "x" command, I added the following to my .perldb file:

use Data::Dumper;

$DB::alias{x} = 's/xx\s+(.*)/p Data::Dumper::Dumper($1)/';

Make Data::Printer only display internal data, not methods

use Data::Printer { class => { show_methods => 'none', parents => 0, linear_isa => 0 } };

(source)

Parameter checking Perl

use MooseX::Params::Validate;
use MooseX::Types::Moose qw{ ArrayRef Int };
use MooseX::Types::Structured qw{ Dict };
use MooseX::Types::Common::Numeric qw{ PositiveInt PositiveOrZeroInt };
use MooseX::Types::Common::String qw{ NonEmptyStr };

sub some_method {
    my ($self, $channel_id, $some_array_of_structs) = validated_list(
        \@_,
        channel_id   => { isa => PositiveInt },
        channel_name => { isa => NonEmptyStr },
        some_array_of_structs => {
            isa => ArrayRef[ Dict[
                some_id    => PositiveInt,
                quantity   => PositiveOrZeroInt,
                cruciality => Int
            ] ]
        }
    );
    # ...
}

Get unique values from an array in Perl

# Unique an array

@unique_list = keys %{{ map { $_ => 1 } @big_list }}; # aka dedupe a list

# Unique a hash on a specified key 'id'

@unique_list = values %{{ map { $_->{id} => $_ } %big_hash }};

Explanation

Dedupe lines in place

in vim, select a list of values with shift-V (visual mode), and then type :sort u to sort them and remove duplicates.

Different ways to set up a database test in Perl

• Use a DBI wrapper subroutine to code up each SQL statement:
• pros: all the data is kept inside each test, and you can put Perl comments next to the SQL statements
• cons: you have to translate the SQL statements into Perl and back for testing/debugging

subtest 'a' {

    clear_db;
    db_wrapper( db => 'db1' sql =>'INSERT INTO table_a SET x = ?, y = ?', bind_values => [1, 2]);
    # do something
}

subtest 'b' {

    clear_db;
    db_wrapper( db => 'db1' sql =>'INSERT INTO table_a SET x = ?, y = ?', bind_values => [1, 3]);

    # do something

}

• Put all the statements inline, in the DATA section, and use a special subroutine to read them all:
• pros: the database statements stay in SQL format
• cons: you have to use different ID numbers for each test, 

setup_db_once( data => \*DATA );
...

subtest 'a' { # do something with first row }

subtest 'b' { # do something with second row }

...
__DATA__

INSERT INTO table_a SET x = 1, y = 2

INSERT INTO table_a SET x = 5, y = 6

• Use a DBI wrapper subroutine to read SQL statements from an inline heredoc:
• all the pros and none of the cons

subtest 'a' {

    clear_db;
    setup_db_per_test( data => <<EOM

INSERT INTO table_a SET x = 1, y = 2

EOM;
    # do something
}

subtest 'b' {

    clear_db;

    setup_db_per_test( data => <<EOM

INSERT INTO table_a SET x = 1, y = 3

EOM;

    # do something

}

Pretty print Data::Dumper, with variable names

use Data::Dumper;

$Data::Dumper::Indent = 1; # pretty print

Data::Dumper->Dump([\@debug_log],['debug_log']); # name the variable

XSL array/hash/lookup table

The lookup file:

<?xml version="1.0"?>

<lookup>

<location id="1769" guide="TT00379a" average="TT123456"/>

<location id="1230" guide="TT003999" average="TT000001"/>

</lookup>

The code: NOTE: $id is a local variable:

<!-- look up the country guide ID -->

<xsl:variable name="guide">

<xsl:for-each select="document('lookup.xml')">

<xsl:value-of select="key('map',$id)/@guide"/>

</xsl:for-each>

</xsl:variable>

Select fields from a hashref in Perl

my @wanted_fields = qw/link id title image/;
my %display_fields;
@display_fields{@wanted_fields} = @{ %{$thing->to_hashref} }{@wanted_fields};

How to convert a tab separated file into a Perl hashref


open(LIST,'<',$datafile) || die "Failed to open ".$datafile;
my @header = split("\t", <list>);
chomp(@header);
my @nodes = ();
while (my $line = <LIST>) {
chomp($line);
my $i=0; my %node = map { $header[$i++] => ($_?$_:'[empty field]') } split("\t", $line);
push @nodes, \%node;
}
close(LIST);

print Dumper(\@nodes);