Recording HTTP responses with Perl

There are two main ways. If your HTTP requests are being done in a different way, you will need to build your own solution.

The LWP way:

use LWP::UserAgent::Mockable;

BEGIN {
    # Options: record|playback|passthrough
    $ENV{LWP_UA_MOCK}      ||= 'playback';
    $ENV{LWP_UA_MOCK_FILE} ||= "$0-lwp-mock.out";
}

END {
    LWP::UserAgent::Mockable->finished();
}

# You might be able to normalise the requests using the callbacks

The Mojo way:

use Mojo::UserAgent::Mockable;

my $mock_ua = Mojo::UserAgent::Mockable->new(
    mode        => $ENV{MOJO_UA_MOCK},
    file        => "$0-mojo-ua-mock.out",
#    ignore_body => 1,
# or:
    request_normalizer => sub {
        my ( $req, $recorded_req ) = @_; 

        my ($time) = $recorded_req->body =~ m{(.*)};
        my ($hash) = $recorded_req->body =~ m{(.*)};

        my $body = $req->body;
        $body =~ s{.*}{$time};
        $body =~ s{.*}{$hash};

        $req->body($body);
    },  

);

END {
    $mock_ua->save if $ENV{MOJO_UA_MOCK} eq 'record';
}

ok my $t = Test::Mojo->new("My::App");

$t->app->mock( useragent => sub { $mock_ua } );

# Configure the proxy in record/passthrough mode only

if ( $MODE =~ /record|passthrough/ ) { 

    local $ENV{HTTPS_PROXY} = $t->app->config->{http_proxy};

    local $ENV{HTTP_PROXY}  = $t->app->config->{http_proxy};

    my $proxy = Mojo::UserAgent::Proxy->new;

    $proxy->detect;

    $mock_ua->proxy( $proxy );

}

HTTPS in Perl

After Crypt::SSLeay or IO::Socket::SSL is installed, LWP will automatically work for URLs starting https://

Simple Perl CGI and LWP

#!/usr/bin/perl

use CGI;

my $q = CGI->new;
print $q->header();

my $url = $q->param('url');

###

require LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->proxy('http', 'http://your_proxy_here:80');

my $response = $ua->get($url);

if ($response->is_success) {
print $response->content; # or do fancy stuff here
}
else {
die $response->status_line;
}