Analyzing HAR files with jq

See gist:

Web scraping for fun

I wanted to know the relative popularity of different locations in which Hindi movies are filmed.

I achieved this in about 15 minutes, with as little coding as I could manage.

Resources used: Linux, bash, wget, grep, uniq, sort, Chrome, XPath helper extension, a text editor, regexes.

How to copy off images from Chrome's cache (on Mac)

How to find Chrome cache on Mac

export YOU=your_username
export PROFILE=Default

mkdir ~/chrome_images
cp /Users/$YOU/Library/Caches/Google/Chrome/$PROFILE/Cache/* ~/chrome_images/
for k in `/bin/ls`; do mv $k $k.jpg; done # rename all files to jpg

open ~/chrome_images

# Now browse the image folder and find the images you want

# More sophisticated renaming example:

if [[ `file $filename | grep JPEG` ]]; then mv $filename{,.jpg} ; fi

Easily save music playlists from Youtube on Ubuntu

git clone https://github.com/rg3/youtube-dl.git
cd youtube-dl
./youtube-dl 'https://www.youtube.com/playlist?list=PL517964F9D64C8D0B' --ignore-errors --extract-audio --audio-quality 0 --audio-format mp3 --prefer-ffmpeg --output '%(playlist_index)s - %(title)s.%(ext)s'

(source, source)

...but then the mp3 files have the wrong length and can't be fast forwarded properly in many players (youtube-dl 2014.04.30.1)

To fix them:

sudo apt-get install vbrfix
cd /path/to/mp3s
find . -type f -iname '*.mp3' -exec vbrfix {} {} \;
rm vbrfix.log vbrfix.tmp

(source, source)

Parse HTML with Ruby and Nokogiri

doc = Nokogiri::HTML(open(filename, 'r'))
puts doc.xpath('//title')[0].inner_html
puts doc.xpath('//h2')

Web scraping with Ruby

In order of goodness:

• Nokogiri
• Hpricot
• Scrapi

Use Firefox as if it were WWW::Mechanize::Shell

Depends on plugin Mozrepl: https://github.com/bard/mozrepl/wiki
Use WWW::Mechanize::Firefox

Scanning HTML

Screen scraping with HTML::TreeBuilder:

my $real_h1 = $tree->look_down(
    '_tag', 'h1',
    sub {
        my $link = $_[0]->look_down('_tag','a');
        return 1 unless $link; # no link means it's fine
        return 0 if $link->attr('href') =~ m{/dyna/}; # a link to there is bad
        return 1; # otherwise okay
    }
);