A piggy bank of commands, fixes, succinct reviews, some mini articles and technical opinions from a (mostly) Perl developer.

Jump to

Quick reference

Set up Joplin

## Install

Install node 12+

Then:
```
NPM_CONFIG_PREFIX=~/.joplin-bin npm install -g joplin

sudo ln -s ~/.joplin-bin/bin/joplin /usr/bin/joplin
```


## Configure

```
joplin help config
joplin config sync.target 7 # dropbox
joplin sync # then follow instructions
```

Quick reference

.

Mac: \r
Unix: \n
Windows: \r\n
vi s/,/,\r/
$CRLF = "\015\012";
perl -le'$q = "\047"; print $q' # single quote

unset PERL5LIB; unset PERL_LOCAL_LIB_ROOT; unset PERL_MB_OPT; unset PERL_MM_OPT; eval $(perl -Mlocal::lib=./local)

perl -MDevel::Cover=+ignore,.*,-select,^lib,-select,^t,-silent,on t/path/to/test.t

python -m SimpleHTTPServer 8080
python3 -m http.server 8080

warn "mojo response = ".$t->tx->res->to_string;
...->or(sub { diag("Result = ".$t->tx->res->to_string) });

use Data::Printer { class => { internals => 0, inherited => 'public' } };

my $r=\@_; $l->debug("entering ".( caller(0) )[3])." with ". sub { $l->dump(args => $r) }); # log calls to subroutines

patch -p0 < file.patch

DBIC_TRACE_PROFILE=console DBIC_TRACE=1
TEST_METHOD=foo

$Data::Dumper::Maxdepth = 1;

$DB::single = 1;

echo -ne "\033]0;"some title"\007" # set putty title

shopt -s checkwinsize; reset # fix terminal display issue

# bash strict mode
set -euo pipefail
IFS=$'\n\t'
# debug mode
set -vx

(s)printf:
printf '<%+d>', 12; # prints "<+12>"
printf '<%6s>', 12; # prints "< 12>"
printf '<%-6s>', 12; # prints "<12 >"

eval $(dircolors|sed 's/di=01;34/di=01;36/'); # light blue directories

if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root"
exit 1
fi

| python -m json.tool # sorts keys
| json_pp # doesn't sort keys (perl JSON module)

Search and replace over two lines:
sed ':a;N;$!ba;s/\n second line: / second line: /g'


Bash quotes: '\'' & '"'"'

print "debug = ".Dumper({%{{$_->get_columns}}{qw/id shipment_id/}}) foreach $shipment_rs->items; # DBIC quick peek

print $result->{_column_data} # DBIC quicker peek

require Carp; Carp::cluck "got here";

telnet host port

openssl s_client -debug -connect host:port

www.tablesgenerator.com

How to really loop over filenames with spaces in

 Don't parse the output of "ls".

Instead:

for file in *

do

  ...

done


(source)

Automating mouse movement and clicks on Linux

Windows has AutoHotkey (AHK), and Linux has xdotool.

But xdotool 3.20160805.1 doesn't work on Wayland at all. It silently fails. Turns out xdotool only works on X11, which is no longer the default for Ubuntu.

In 2024, using Ubuntu 22, this version of xdotool is the latest one available by running `apt install xdotool`.

To find out if you're using Wayland:

$ loginctl show-session 2 -p Type
Type=wayland
$ echo $XDG_SESSION_TYPE
wayland

To switch to X11, edit/etc/gdm3/custom.conf, uncomment the line WaylandEnable=false, and reboot.

Now xdotool will work perfectly.

xdotool README does mention some alternatives:

  • ydotool (available in apt) -- last updated January 2023, also didn't work for me on Wayland or X11
  • dotool -- updated more recently, but not packaged for Ubuntu, and building from source requires Go, which is quite heavy so it's easier to just switch to X11 for xdotool.

Ubuntu Linux setup basics

For username "foo":

$ adduser foo

$ passwd foo

$ sudo usermod -aG sudo foo

mkdir -p /home/foo/.ssh

$ cat the_public_key.pem >> /home/foo/.ssh/authorized_keys

chown -R foo:foo /home/foo/.ssh

$ chmod 700 /home/foo/.ssh

$ chmod 600 /home/foo/.ssh/authorized_keys
Disable default account:
$ usermod -s /usr/sbin/nologin default_username

Notes:
  • Not useradd.
  • Even when logging in with just SSH key, user must have a password. It will only be used for sudo commands.

Sudoers basics

Elasticsearch advanced queries

See also ElasticSearch basics.

DQL to filter by non-zero length: Advert.location_query:* (does not work as a filter)
Or in Lucene: Advert.location_query:?*

Results are limited to 10,000 records, unless you use the scroll API which can paginate and also make parallel requests.

Gist:

MySQL date display format conversion

MySQL date time conversion functions: 

  • UNIX_TIMESTAMP(date) docs
  • FROM_UNIXTIME(epoch) docs

Examples:

select name, from_unixtime(time_added, '%Y-%m-%d %h:%i')

from company

order by date_added desc

limit 10; -- list the most recently added companies

You may also omit the '%Y-%m-%d %h:%i' format string, to get the default format YYYY-MM-DD HH:MM:SS, e.g. `2023-09-07 09:43:51`


Chrome extensions: Download manager reviews


  • DownThemAll: Queues, but doesn't intercept
  • Thunder Download Manager: Intercepts, but doesn't queue
  • Free Download Manager: Just says "Loading..." (on Ubuntu)
  • Chrono Download Manager: Intercepts and queues! And resumes. Perfect!

curl -o / wget -O

curl --output file
curl -o file

wget --output-document file
wget -O file

Best to just always use curl, and know it uses lowercase for common arguments like normal.

It's usually already installed as well.

Video processing for fun

Mac

  • QuickTime Player
    • Edit menu | Add clip after...
    • 40 videos, total 250Mb = (Not Responding) & Pinwheel of doom
    • 34 videos, total 140Mb = several minutes of editing, then (Not Responding) & Pinwheel of doom
  • iMovie
    • To download for an older version of MacOS, open App Store | Purchased | Install

Linux

Windows

Web

Android

iPhone


Command-line data processing 2023

Some developer tools with CLI  for processing XML, XHTML, HTML, JSON, YAML, etc.

XML

  • xsh (perl - Choroba)
    • cpm XML::XSH2
    • xsh -P file.xml
    • ls
    • help ls
    • help | less
    • <TAB> autocompletion
  • xmllint
    • xmllint --xpath "//foo" file.xml
    • xmllint --shell file.xml
  • xmltarlet
  • xq (golang - )
    • apt-get install xq
  • xq (python - jeffbr13)
    • pip install xq

JSON

  • jq
    • cat file.json | jq . # format
    • cat file.json | jq '.[]' # extract array

Search

  • fzf
  • ripgrep
  • ag
  • ack
    • Doesn't search "binary" files by default
  • vim - for searching files

Windows 10 robocopy basics

robocopy c:\temp\source c:\temp\destination /E /DCOPY:DAT /R:10 /W:3

Running Emby on Linux

wget https://github.com/MediaBrowser/Emby.Releases/releases/download/4.5.4.0/emby-server-deb_4.5.4.0_amd64.deb


sudo dpkg -i emby-server-deb_4.5.4.0_amd64.deb


sudo systemctl status emby-server.service


How to maintain software

You have a massive legacy spaghetti mess of code, and you need to add a new feature.

Write tests first.

Put all your new code in a new module, and only insert single lines into the legacy code to call the new code. This allows you to properly encapsulate the new code, wrap it in try/catch blocks, etc. and not break the legacy system.

Pass whole objects into the new code, this is a necessary consequence of encapsulation.


How big is libgen? Answer: 131TiB (= 144Tb)

# First method: 131TiB

$ wget https://phillm.net/libgen-stats.php -O libgen-stats.json


$ cat libgen-stats.json | jq '.[]|.size_bytes' | perl -lne'$x+=$_;END{print$x}'

144116297318150


$ bc -l

144116297318150 / 1024 / 1024 / 1024 / 1024

131.07300884998949186410 Tebibytes


$ bc -l

144116297318150 / 1000 / 1000 / 1000 / 1000

144.11629731815000000000 Terabytes



New software cheat sheet

 

## Languages

- Perl: best for fast prototyping
- Ruby: best for having fun
- Node: best for public projects
- Python: best for experiments

## Frameworks

### Perl

- DBIx::Class 💯
- Mojolicious 💯

### Node

- Express (popular)
- React?
- Koa?
- NestJS?
-....? for testing: Qunit?
-....? for web framework

### Python

- Django? (this one)
- Flask?

### Ruby

- Rails?
- Active Record?

## Libraries

### Perl

- just everything

### Node

- minimist reads command line arguments
- cheerio parses HTML
- daysjs manages dates/times
  - customParseFormat plugin parses dates
- knex does CRUD on databases
- fs perfoms operations on the file system
- import process from 'node:process'; // process.exit(1);

Books about software design theory

Ludo: Here's the online slide deck from Mark Jason Dominus’ 2002 YAPC lightning talk “Design Patterns Aren’t”, on Professor Christoper Alexander, his book A Pattern Language, the ‘Gang of Four’ and design patterns.

APL was book two of a trilogy also incorporating The Timeless Way of Building (volume 1) and The Oregon Experiment (volume 3).

Alexander’s earlier work Notes on the Synthesis of Form was also a big hit with computer scientists in the 1960s, even though it, too, was primarily aimed at designers and architects. (And if you still have any empty shelves on your bookcase, Alexander’s 2002-2004 four-volume work The Nature of Order: An Essay on the Art of Building and the Nature of the Universe is apparently considered his magum opus; and he also published The Battle for the Life and Beauty of the Earth: A Struggle between Two World-Systems, about how his team designed & constructed a Japanese school, in 2012.)

Will: There's another book with a possibly contrasting view: Semantic Software Design. It says the main work in software is coming up with the concepts/paradigm/metaphor which explains what the software is doing. And also we shouldn't think of developers as manufacturers, builders or architects. But rather creative artists. Mainly because we never produce the same piece of work twice.