How to really loop over filenames with spaces in

 Don't parse the output of "ls".

Instead:

for file in *

do

  ...

done

(source)

Delete all files except one on linux

Glob

• shopt -s extglob
• rm -v !("filename")
• rm -v !("filename1"|"filename2")
• rm -v !(*.zip|*.odt)
• shopt -u extglob

Find

• find /directory/ -type f -not -name 'PATTERN' -delete
• find /directory/ -type f -not -name 'PATTERN' -print0 | xargs -0 -I {} rm {}
• find /directory/ -type f -not -name 'PATTERN' -print0 | xargs -0 -I {} rm [options] {}
• find . -type f -not \(-name '*gz' -or -name '*odt' -or -name '*.jpg' \) -delete

Glob ignore

• cd test
• GLOBIGNORE=*.odt:*.iso:*.txt
• rm -v *
• unset GLOBIGNORE

(source)

How to draw a histogram on linux

Generate a bar chart with Perl:

perl -lane 'print $F[0], "\t", "=" x ($F[1] / 5)' file

Adjust the number 5 to your desired width.

(source)

Linux renaming examples, when filenames have spaces

Do your filenames have spaces? Do you curse bash for making it difficult to loop over them?

Fear not! (and wash that potty mouth out with soap). This loop is all you need:

find . -type f -name "*.mp3" -print0 | while IFS= read -r -d '' file; do ls "$file"; done

# Rename "foo.mp3.mp3" to "foo.mp3"

find . -type f -name "*.mp3" -print0 | while IFS= read -r -d '' file; do file=`basename "$file"`; newfile=$(perl -le'$ARGV[0]=~s/\.mp3//;print$ARGV[0]' "$file"); echo mv "$file" "$newfile"; done


# Add a leading zero to numbered filenames

find . -type f -name "064*" -print0 | while IFS= read -r -d '' file; do file=`basename "$file"`; echo mv "$file" "0$file"; done

# The above commands will print out what they're about to do. Check it is what you expect! Then make this change to run them for real:

Change: echo mv foo bar
To: `mv foo bar`

# ^ Those ` are backticks

# Note the order of files is not sorted

(source)

Web scraping for fun

I wanted to know the relative popularity of different locations in which Hindi movies are filmed.

I achieved this in about 15 minutes, with as little coding as I could manage.

Resources used: Linux, bash, wget, grep, uniq, sort, Chrome, XPath helper extension, a text editor, regexes.

Docker basics

# Download an image and spin up a container, run it, connect to it

docker pull [name pf image] # download an image

docker run -d -p 80:80 --name pintail-whoami pintailai/pintail-whoami:0.0.1 # download + run

docker run --rm -v /path/on/machine:/app/out image-name:stable params to app

# List running containers

docker ps # show running containers

docker ps | cut -c-$(tput cols) # show running containers without wrapping to the next line

docker ps -q # show just the IDs of the running containers

docker ps -q | head -1 # show ID of most recently started container (how to sort)

# Run a shell on a container

docker run -i -t [Container ID] /bin/bash

# Connect to a shell on an already-runnning docker container.
# (Shell: Use /bin/bash for ubuntu or /bin/ash for alpine)

docker exec -it [Container ID] [shell]

docker exec -it `docker ps -q | head -1` /bin/bash # run shell on the most recently started container

# Copy a file off

docker cp <container>:<src-path> <local-dest-path>

# Delete all stopped containers and images

docker system prune -a

# List all images

docker image ls

How to apply a stash by name in git

1) In git it's easy to retrieve stashes by number like this:

git stash apply stash@{5}

But that number changes as you add more stashes to the list.

2) Instead you could perform some jiggery-pokery in bash to look up the stash by name:

git stash apply $(git stash list | grep "$NAME" | cut -d: -f1)

You can apply multiple stashes as long as they don't overlap.

3) Or you could save the commit and refer to it by tag:

git commit -m'The usual changes to aid debugging'
git tag foo
git reset --hard HEAD^
git cherry-pick -n foo

(-n is --no-commit)

This way if they overlap you can use git's merging/conflict mechanism.

(source)

Bash shortcuts for faster editing

To open the command line in vi:

set -o vi
# type or navigate to a long command
# press [escape]
# navigate the line using vi keyboard commands
# press 'v' to open the command in vi

bash default parameters

# Set dir to $1, or "foo" if $1 isn't set

DIR=${1:-foo}

Useful bash flags for scripts verbose trace expand error exit

Useful flags:
set -v # "verbose" - echo commands

set -x # "xtrace" - like verbose but expands commands

set -e # "errexit" - abort script at first error

set -u # "nounset" - strict mode: undefined variable causes an exit

Long format:

set -o verbose

Bash "strict mode":
#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

(source, source)

Ctrl-C doesn't interrupt loop in Bash script

If you want Ctrl-C to be able to stop your loop in Bash, put this inside the loop:

trap "echo Exited!; exit;" SIGINT SIGTERM

(source)

Terminal does not wrap properly in linux

Sometimes lines don't wrap correctly in a bash terminal in linux - long lines overlap and overwrite the beginning of the line instead of continuing onto the next line.

To fix it:
shopt -s checkwinsize

To check it:
shopt  | grep checkwinsize
You should see:
checkwinsize    on

If it still isn't working, try:
reset

(source)

See hidden characters in a string

perl -e '$HOSTNAME = `hostname -s`; print $HOSTNAME;' | od -c

Easily add colours to bash scripts

RED="$(tput setaf 1)"
GREEN="$(tput setaf 2)"
YELLOW="$(tput setaf 3)"
WHITE="$(tput setaf 7)"
RESET=$WHITE

echo "${RED}An error occurred${RESET}"
echo "${YELLOW}Warning: Be careful${RESET}"
echo "${GREEN}Everything is groovy${RESET}"

(source)

Old, naive way:

GREEN="\[\033[32m\]"
YELLOW="\[\033[33m\]"
RESET="\[\033[0m\]"

Bash script must run as root

# This script must be run as root

if [ "$(id -u)" != "0" ]; then
    echo "This script must be run as root" 1>&2
    exit 1
fi

sudo doesn't list files properly

Problem:

You can list a directory using sudo:
$ sudo ls /var/log/apache/
access_log  error_log

But you can't list files with a wildcard using sudo:

$ sudo ls /var/log/apache/*_log
ls: cannot access /var/log/apache/*_log: No such file or directory

Solution:

Ensure parameter expansion happens in the shell run by sudo, not in the user's shell:
$ sudo bash -c 'ls /var/log/apache/*_log'

(source)

How to capture linux "time" output

Pipe time output into a file like this:

{ time ls; } 2>time.output

Don't forget the semicolon!

(source)

Shuffle lines in bash

#!/bin/bash
# Print out the lines from a file, in a random order
FILE=$1
if [[ -z $FILE ]]
then
    echo "usage: shuffle [filename]"
    exit 1
fi
for i in `cat $FILE`; do echo "$RANDOM $i"; done | sort | sed -r 's/^[0-9]+ //'

Thanks

Bash parameter expansion

Have you ever seen ## (hash hash/pound pound) or %% (percent percent) inside a bash script and wondered what it means? It's a form of parameter expansion that allows you to manipulate your strings using regexes.

All the following commands work with a variable called $string.
'pattern' means bash pattern, or can also be an ordinary string:

• String length: ${#string}
• Extract a substring: ${string:position}
• Extract a substring, specifying length: ${string:position:length}
• Delete shortest match of substring from the beginning of string: ${string#substring}
• Delete shortest match of substring from the end of string: ${string%substring}
• Delete longest match of substring from the beginning of string: ${string##substring}
• Delete longest match of substring from the end of string: ${string%%substring}
• Find and replace first substring: ${string/pattern/replacement}
• Find and replace all substrings: ${string//pattern/replacement}

Thanks to TheGeekStuff

My .bashrc config

# general:
export PATH="~/scripts:$PATH"

alias ls="ls -F --color"
alias grep="grep --color=auto"

# git aliases
function gitdiff() {    git diff --no-ext-diff -w "$@" | vim -R -
}

function gitlog() {
    git log --name-only "$@"
}

function gitcommit() {
    git --no-pager diff --no-ext-diff
    echo
    read -p "Are you sure you want to commit these changes? " yn
    case $yn in
        [Yy]* ) git commit "$@"; break;;
    esac
}

# git prompt
host_colour="01;34"
export PS1="\[\033[${host_colour}m\]\h\[\033[00m\]/\u \t \w \$ "

function __git_commits_behind {
    if [ -d .git ]
    then
        git st | perl -ne'm{Your branch is behind.+by (\d+) commit} && print "behind $1< "'
    fi
}

function __git_commits_ahead {
    if [ -d .git ]
    then
        git st | perl -ne'm{Your branch is ahead of.+ by (\d+) commit} && print "ahead $1> "'
    fi
}

function update_prompt {
    local branch=$(git branch --no-color 2>/dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1) /')
    export PS1="\[\033[${host_colour}m\]\h\[\033[00m\]/\u \[\033[${branch_colour}m\]${branch}$(__git_commits_behind)$(__git_commits_ahead)\[\033[00m\]\t \w \$ "
}
export PROMPT_COMMAND=update_prompt