How big is libgen? Answer: 131TiB (= 144Tb)

# First method: 131TiB

$ wget https://phillm.net/libgen-stats.php -O libgen-stats.json

$ cat libgen-stats.json | jq '.[]|.size_bytes' | perl -lne'$x+=$_;END{print$x}'

144116297318150

$ bc -l

144116297318150 / 1024 / 1024 / 1024 / 1024

131.07300884998949186410 Tebibytes

$ bc -l

144116297318150 / 1000 / 1000 / 1000 / 1000

144.11629731815000000000 Terabytes

Create a lot of random data files on linux

# Create 10Mb random data

dd bs=1024 count=10240 < /dev/urandom > data

# Split into files of 10K each

split -a 5 -b 10240 data 'split.'

Shuffle lines in bash

#!/bin/bash
# Print out the lines from a file, in a random order
FILE=$1
if [[ -z $FILE ]]
then
    echo "usage: shuffle [filename]"
    exit 1
fi
for i in `cat $FILE`; do echo "$RANDOM $i"; done | sort | sed -r 's/^[0-9]+ //'

Thanks