Command-line data processing 2023

Some developer tools with CLI  for processing XML, XHTML, HTML, JSON, YAML, etc.

XML

• xsh (perl - Choroba)
• cpm XML::XSH2
• xsh -P file.xml
• ls
• help ls
• help | less
• <TAB> autocompletion
• xmllint
• xmllint --xpath "//foo" file.xml
• xmllint --shell file.xml
• xmltarlet
• xq (golang - sibprogrammer)
• apt-get install xq
• xq (python - jeffbr13)
• pip install xq

JSON

• jq
• cat file.json | jq . # format
• cat file.json | jq '.[]' # extract array

Search

• fzf
• ripgrep
• ag
• ack
• Doesn't search "binary" files by default
• vim - for searching files

Docker basics

# Download an image and spin up a container, run it, connect to it

docker pull [name pf image] # download an image

docker run -d -p 80:80 --name pintail-whoami pintailai/pintail-whoami:0.0.1 # download + run

docker run --rm -v /path/on/machine:/app/out image-name:stable params to app

# List running containers

docker ps # show running containers

docker ps | cut -c-$(tput cols) # show running containers without wrapping to the next line

docker ps -q # show just the IDs of the running containers

docker ps -q | head -1 # show ID of most recently started container (how to sort)

# Run a shell on a container

docker run -i -t [Container ID] /bin/bash

# Connect to a shell on an already-runnning docker container.
# (Shell: Use /bin/bash for ubuntu or /bin/ash for alpine)

docker exec -it [Container ID] [shell]

docker exec -it `docker ps -q | head -1` /bin/bash # run shell on the most recently started container

# Copy a file off

docker cp <container>:<src-path> <local-dest-path>

# Delete all stopped containers and images

docker system prune -a

# List all images

docker image ls

Query installed packages on Ubuntu

Ubuntu uses debian packages, not RPMs so yum isn't relevant.

List all packages:
dpkg-query -l

Display summary info for a package:
dpkg-query -s package_name

Display files installed by a package:
dpkg-query -L package_name

Git setup

git config --global user.name "Your Name Comes Here"
git config --global user.email you@yourdomain.example.com

git config --global color.diff auto
git config --global color.status auto
git config --global color.branch auto

git config --global core.editor vim

Using vim's viewports

• :sp [filename] will split the Vim window horizontally
• Can be written out entirely as :split
• [filename] is optional, if it's omitted then the current file is loaded in the new viewport
• :vsp [filename] will split the Vim window vertically
• Can be written out as :vsplit
• Ctrl-e [filename] loads a new file into the current viewport
• Ctrl-w Ctrl-w moves between Vim viewports.
• Ctrl-w j moves one viewport down.
• Ctrl-w k moves one viewport up.
• Ctrl-w h moves one viewport to the left.
• Ctrl-w l moves one viewport to the right.
• Can also use the arrow keys instead of h/j/k/l
• Ctrl-w = tells Vim to resize viewports to be of equal size.
• Ctrl-w - reduce active viewport by one line.
• Ctrl-w 10- reduce active viewport by ten lines.
• Ctrl-w + increase active viewport by one line.
• Ctrl-w 10+ increase active viewport by ten lines.
• Ctrl-w q will close the active window.
• Ctrl-w r will rotate windows to the right.
• Ctrl-w R will rotate windows to the left.

Full article

Run bash in a viewport, with this plugin (easy installation).

How to checkout hosted subversion project

Google code:

svn checkout https://example.googlecode.com/svn/trunk/ example_directory --username example@gmail.com

Assembla:

svn checkout https://subversion.assembla.com/svn/example/ example_directory --username your_username

How to talk to memcached

1. telnet to the server (or localhost) on the right port
2. type "flush_all" to invalidate the whole cache
3. type "delete [key]" to invalidate a particular item

See also the memcache protocol