A piggy bank of commands, fixes, succinct reviews, some mini articles and technical opinions from a (mostly) Perl developer.
Jump to
Set up Joplin
Quick reference
Windows: \r\n
vi s/,/,\r/
$CRLF = "\015\012";
perl -le'$q = "\047"; print $q' # single quote
unset PERL5LIB; unset PERL_LOCAL_LIB_ROOT; unset PERL_MB_OPT; unset PERL_MM_OPT; eval $(perl -Mlocal::lib=./local)
perl -MDevel::Cover=+ignore,.*,-select,^lib,-select,^t,-silent,on t/path/to/test.t
python -m SimpleHTTPServer 8080
python3 -m http.server 8080
warn "mojo response = ".$t->tx->res->to_string;
...->or(sub { diag("Result = ".$t->tx->res->to_string) });
use Data::Printer { class => { internals => 0, inherited => 'public' } };
my $r=\@_; $l->debug("entering ".( caller(0) )[3])." with ". sub { $l->dump(args => $r) }); # log calls to subroutines
patch -p0 < file.patch
DBIC_TRACE_PROFILE=console DBIC_TRACE=1
TEST_METHOD=foo
$Data::Dumper::Maxdepth = 1;
$DB::single = 1;
echo -ne "\033]0;"some title"\007" # set putty title
shopt -s checkwinsize; reset # fix terminal display issue
# bash strict mode
set -euo pipefail
IFS=$'\n\t'
# debug mode
set -vx
(s)printf:
printf '<%+d>', 12; # prints "<+12>"
printf '<%6s>', 12; # prints "< 12>"
printf '<%-6s>', 12; # prints "<12 >"
eval $(dircolors|sed 's/di=01;34/di=01;36/'); # light blue directories
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root"
exit 1
fi
| python -m json.tool # sorts keys
| json_pp # doesn't sort keys (perl JSON module)
Search and replace over two lines:
sed ':a;N;$!ba;s/\n second line: / second line: /g'
Bash quotes: '\'' & '"'"'
print "debug = ".Dumper({%{{$_->get_columns}}{qw/id shipment_id/}}) foreach $shipment_rs->items; # DBIC quick peek
print $result->{_column_data} # DBIC quicker peek
require Carp; Carp::cluck "got here";
telnet host port
openssl s_client -debug -connect host:port
www.tablesgenerator.com
Automating mouse movement and clicks on Linux
Windows has AutoHotkey (AHK), and Linux has xdotool.
But xdotool 3.20160805.1 doesn't work on Wayland at all. It silently fails. Turns out xdotool only works on X11, which is no longer the default for Ubuntu.
In 2024, using Ubuntu 22, this version of xdotool is the latest one available by running `apt install xdotool`.
To find out if you're using Wayland:
$ loginctl show-session 2 -p Type
Type=wayland
$ echo $XDG_SESSION_TYPE
wayland
To switch to X11, edit/etc/gdm3/custom.conf, uncomment the line WaylandEnable=false, and reboot.
Now xdotool will work perfectly.
xdotool README does mention some alternatives:
- ydotool (available in apt) -- last updated January 2023, also didn't work for me on Wayland or X11
- dotool -- updated more recently, but not packaged for Ubuntu, and building from source requires Go, which is quite heavy so it's easier to just switch to X11 for xdotool.
Ubuntu Linux setup basics
For username "foo":
$ adduser foo
$ passwd foo
$ sudo usermod -aG sudo foo
$ mkdir -p /home/foo/.ssh
$ cat the_public_key.pem >> /home/foo/.ssh/authorized_keys
$ chown -R foo:foo /home/foo/.ssh
$ chmod 700 /home/foo/.ssh
$ chmod 600 /home/foo/.ssh/authorized_keysDisable default account:$ usermod -s /usr/sbin/nologin default_username
Notes:- Not useradd.
- Even when logging in with just SSH key, user must have a password. It will only be used for sudo commands.
Elasticsearch advanced queries
MySQL date display format conversion
MySQL date time conversion functions:
Examples:
select name, from_unixtime(time_added, '%Y-%m-%d %h:%i')
from company
order by date_added desc
limit 10; -- list the most recently added companies
You may also omit the '%Y-%m-%d %h:%i' format string, to get the default format YYYY-MM-DD HH:MM:SS, e.g. `2023-09-07 09:43:51`
Chrome extensions: Download manager reviews
- DownThemAll: Queues, but doesn't intercept
- Thunder Download Manager: Intercepts, but doesn't queue
- Free Download Manager: Just says "Loading..." (on Ubuntu)
- Chrono Download Manager: Intercepts and queues! And resumes. Perfect!
curl -o / wget -O
wget -O file
Video processing for fun
Mac
- QuickTime Player
- Edit menu | Add clip after...
- 40 videos, total 250Mb = (Not Responding) & Pinwheel of doom
- 34 videos, total 140Mb = several minutes of editing, then (Not Responding) & Pinwheel of doom
- iMovie
- To download for an older version of MacOS, open App Store | Purchased | Install
Linux
Windows
Web
- wave.video - Really good. Limited to 15 mins.
- More
Android
iPhone
Command-line data processing 2023
Some developer tools with CLI for processing XML, XHTML, HTML, JSON, YAML, etc.
XML
- xsh (perl - Choroba)
- cpm XML::XSH2
- xsh -P file.xml
- ls
- help ls
- help | less
- <TAB> autocompletion
- xmllint
- xmllint --xpath "//foo" file.xml
- xmllint --shell file.xml
- xmltarlet
- xq (golang - sibprogrammer)
- apt-get install xq
- xq (python - jeffbr13)
- pip install xq
JSON
- jq
- cat file.json | jq . # format
- cat file.json | jq '.[]' # extract array
Search
- fzf
- ripgrep
- ag
- ack
- Doesn't search "binary" files by default
- vim - for searching files
Running Emby on Linux
wget https://github.com/MediaBrowser/Emby.Releases/releases/download/4.5.4.0/emby-server-deb_4.5.4.0_amd64.deb
sudo dpkg -i emby-server-deb_4.5.4.0_amd64.deb
sudo systemctl status emby-server.service
How to maintain software
You have a massive legacy spaghetti mess of code, and you need to add a new feature.
Write tests first.
Put all your new code in a new module, and only insert single lines into the legacy code to call the new code. This allows you to properly encapsulate the new code, wrap it in try/catch blocks, etc. and not break the legacy system.
Pass whole objects into the new code, this is a necessary consequence of encapsulation.
How big is libgen? Answer: 131TiB (= 144Tb)
# First method: 131TiB
$ wget https://phillm.net/libgen-stats.php -O libgen-stats.json
$ cat libgen-stats.json | jq '.[]|.size_bytes' | perl -lne'$x+=$_;END{print$x}'
144116297318150
$ bc -l
144116297318150 / 1024 / 1024 / 1024 / 1024
131.07300884998949186410 Tebibytes
$ bc -l
144116297318150 / 1000 / 1000 / 1000 / 1000
144.11629731815000000000 Terabytes
New software cheat sheet
## Languages
- Perl: best for fast prototyping
- Ruby: best for having fun
- Node: best for public projects
- Python: best for experiments
## Frameworks
### Perl
- DBIx::Class 💯
- Mojolicious 💯
### Node
- Express (popular)
- React?
- Koa?
- NestJS?
-....? for testing: Qunit?
-....? for web framework
### Python
- Django? (this one)
- Flask?
### Ruby
- Rails?
- Active Record?
## Libraries
### Perl
- just everything
### Node
- minimist reads command line arguments
- cheerio parses HTML
- daysjs manages dates/times
- customParseFormat plugin parses dates
- knex does CRUD on databases
- fs perfoms operations on the file system
- import process from 'node:process'; // process.exit(1);
Books about software design theory
Ludo: Here's the online slide deck from Mark Jason Dominus’ 2002 YAPC lightning talk “Design Patterns Aren’t”, on Professor Christoper Alexander, his book A Pattern Language, the ‘Gang of Four’ and design patterns.
APL was book two of a trilogy also incorporating The Timeless Way of Building (volume 1) and The Oregon Experiment (volume 3).
Alexander’s earlier work Notes on the Synthesis of Form was also a big hit with computer scientists in the 1960s, even though it, too, was primarily aimed at designers and architects. (And if you still have any empty shelves on your bookcase, Alexander’s 2002-2004 four-volume work The Nature of Order: An Essay on the Art of Building and the Nature of the Universe is apparently considered his magum opus; and he also published The Battle for the Life and Beauty of the Earth: A Struggle between Two World-Systems, about how his team designed & constructed a Japanese school, in 2012.)
Will: There's another book with a possibly contrasting view: Semantic Software Design. It says the main work in software is coming up with the concepts/paradigm/metaphor which explains what the software is doing. And also we shouldn't think of developers as manufacturers, builders or architects. But rather creative artists. Mainly because we never produce the same piece of work twice.