Add headers or cookies to request in Mojolicious tests

When using Test::Mojo, sometimes you want to modify the request to add a cookie or custom headers.

.

nginx basics

How to serve up a directory on the web

• sudo apt-get install nginx
• vi /etc/nginx/nginx.conf # observe how the html { } section has: include /etc/nginx/sites-enabled/*;
• vi /etc/nginx/sites-available/foo # build the server { } section

server {
    listen 80 default_server;
    root /var/www/html/foo;
    index index.html
    location / {
        autoindex on;
        auth_basic "Restricted";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }
}

• # build .htpasswd file
• ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/foo
• service nginx restart

Webserver experiment 2019

Rough draft notes:

DNS

• http://freedns.afraid.org

Platform

• Digital Ocean
• Ubuntu 16.04

Application container

• UWSGI
• https://uwsgi-docs.readthedocs.io/en/latest/Install.html
• https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html
• wget https://projects.unbit.it/downloads/uwsgi-2.0.17.1.tar.gz
• apt-get install libperl-dev
• python uwsgiconfig.py --build
• ./uwsgi --plugins psgi -s :3031 -M -p 4 --wsgi-file /path/to/foobar.py -m # for python. Works
• But it breaks if you do "python uwsgiconfig.py --plugin plugins/psgi core" afterwards?
• ./uwsgi: unrecognized option '--wsgi-file'
• touch buildconf/python.ini
• python uwsgiconfig.py --build python
• core/plugins.c:153:35: error: ‘UWSGI_PLUGIN_DIR’ undeclared (first use in this function)
• plugin_filename = uwsgi_concat3(UWSGI_PLUGIN_DIR, "/", plugin_name);
• # Rebuilt from scratch and problem went away
• apt-get install libperl-dev
• python uwsgiconfig.py --build psgi core
• python uwsgiconfig.py --plugin plugins/psgi
• python uwsgiconfig.py --build http
• python uwsgiconfig.py --plugin plugins/http
• ./uwsgi --plugins psgi -s :3031 -M -p 4 --psgi /path/to/hello.pl -m # for perl. Fails with...
• "no python application found"
• tail -f /var/log/nginx/error.log
• ./uwsgi --plugins psgi --http-socket 127.0.0.1:3031 --http-socket-modifier1 5 --psgi /path/to/hello.pl
• Fails with 502 bad gateway after timeout 
• cat buildconf/http.ini
• [uwsgi]
• plugin_dir = .
• # Rebuild
• python uwsgiconfig.py --build http
• ...
• AttributeError: 'NoneType' object has no attribute 'find'
• # Try again
• ./uwsgi --plugins http,psgi --http-socket 127.0.0.1:3031 --http-socket-modifier1 5 --psgi /srv/web1/hello.pl

Webserver Nginx

• https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html
• /etc/nginx/sites-available/default
• location / {
• uwsgi_pass 127.0.0.1:3031;
• include uwsgi_params;
• }
• nginx -s reload
• # App should now be available on port 80, via Nginx
• # Python - works well
• # Perl - couldn't get it to work
• # Ruby - to try

Webserver Lighttpd

• /etc/lighttpd/lighttpd.conf
• CGI
• server.modules += ( "mod_cgi" )
• $HTTP["url"] =~ "/cgi-bin/" {
• cgi.assign = ( ".pl" => "/usr/bin/perl" )
• }
• # Put CGI script in server.document-root + /cgi-bin/
• # Works
• PSGI
• server.modules += ( "mod_scgi" )
• scgi.protocol = "uwsgi"
• scgi.server = (
• "/" => (( "host" => "127.0.0.1", "port" => 3031, "check-local" => "disable" )),
• )
• # Doesn't work.
• Python only?

FastCGI

• # Try with nginx or lighttpd
• https://nginxlibrary.com/perl-fastcgi/

Webserver Apache 2

• # This might be a better choice for Perl

Perl one-liner to serve a directory over HTTP

plackup -e 'use Plack::App::Directory; Plack::App::Directory->new({ root => "/opt/dweb/packages" })->to_app;'

(source)

Simplest easiest quickest web server script

cd ~
python -m SimpleHTTPServer 8080

(source)

Simple perl webserver

HTTP::Server::Simple

(contrast with python one-liner)

#!/usr/bin/perl

# https://metacpan.org/pod/HTTP::Server::Simple
# /versions/perl-5.8.7/bin/perl -MHTTP::Server::Simple test2.pl

use strict;
use warnings;

{
package MyWebServer;

use HTTP::Server::Simple::CGI;
use base qw(HTTP::Server::Simple::CGI);

my %dispatch = (
    '/hello' => \&resp_hello,
    '/' => \&resp_index,
);

sub handle_request {
    my $self = shift;
    my $cgi  = shift;

    my $path = $cgi->path_info();
    my $handler = $dispatch{$path};

    if (ref($handler) eq "CODE") {
        print "HTTP/1.0 200 OK\r\n";
        $handler->($cgi);

    } else {
        print "HTTP/1.0 404 Not found\r\n";
        print $cgi->header,
              $cgi->start_html('Not found'),
              $cgi->h1('Not found'),
              $cgi->end_html;
    }
}

sub resp_hello {
    my $cgi  = shift;   # CGI.pm object
    return if !ref $cgi;

    my $who = $cgi->param('name');

    print $cgi->header,
          $cgi->start_html("Hello"),
          $cgi->h1("Hello $who!"),
          $cgi->end_html;
}

sub resp_index {
    my $cgi = shift;
    print $cgi->header;
    open(my $fh,'<','index.html') or die "can't open index.html";
    local $/ = undef;
    my $out = <$fh>;
    close($fh);
    print $out;
}

}

# start the server on port 8080
my $pid = MyWebServer->new(8085)->background();
print "Use 'kill $pid' to stop server.\n";

(source)

Quickly use a webserver to serve a local directory

Steps:

• sudo yum install lighttpd
• # Create a file containing:
  
  

  server.document-root = "/path/to/your/directory/here/" 
  
  server.port = 3000
  
  mimetype.assign = (
    ".html" => "text/html", 
    ".txt" => "text/plain",
    ".jpg" => "image/jpeg",
    ".png" => "image/png" 
  )
  
  

• lighttpd -t -f lighttpd.conf # check config
• lighttpd -D -f lighttpd.conf # run webserver

(source)

Simple small Perl webservers

Quick start, if it's just a proof-of-concept and won't be running for long:

• HTTP::Daemon - can't get simpler than this. It even picks a port for you
• HTTP::Server::Simple - it works

More complicated to set up, but much more robust:

• PSGI/Plack - the future
• lighttpd - it's pronounced lighty

Use lighttpd as a proxy-pass server

$HTTP["host"] =~ "made.up.hostname" { # hostname you want to create
    $SERVER["socket"] == ":1234" { # port you want to create

        server.errorlog      = "/var/logs/madeup/lighttpd-error.log"
        accesslog.filename   = "/var/logs/madeup/lighttpd-access.log"

        proxy.server = ( "" =>
                           ( ( 
                               "host" => "123.456.789.012", # where you want to proxy to
                               "port" => 5678, # port you want to proxy to
                             ) ) 
                         )   
    }   
}

# Now go to http://made.up.hostname:1234 and you will get directed through to http://123.456.789.012:5678

Developing over FTP

Using Eclipse and RSE plugin:

• Logs in okay
• But seems to always be closing directory sub-trees I've opened in 'Remote Systems' panel
• It wants to do an 'ls' on every parent directory in the hierarchy, every time I change files. Very slow.

Using CurlFtpFS:

• Install in Ubuntu from the software centre
• to mount: curlftpfs -v username:password@server.example.com local_dir/
• to unmount: fusermount -u local_dir/
• Performance is unusably slow... Eclipse wants to know about every single file on the remote server, not just the ones I'm editing.
• Saving the file through Eclipse using CurlFtpFS takes even longer than it would to upload it separately.

Using FileZilla and gedit:

• Browsing is fast
• Editing is fast
• But you have to navigate to every file manually each session, it doesn't remember what you had open
• The system of holding temp files locally is not ideal, but FileZilla detects changes in them very nicely
• You still can't grep over the files as they're all held remotely, and it's a bit fiddly to navigate to a specific file

Editing locally with Eclipse, and uploading after changes are made:

• Write an Ant script to upload any modified files?
• ...to investigate

Password protect a directory in Apache

Put a .htaccess file in the directory you want to protect, containing:

AuthUserFile /full/path/to/your/htpasswd_file
AuthGroupFile /dev/null
AuthName "Password Protected Area"
AuthType Basic



require valid-user

Then use an online htpasswd generator to create the htpasswd_file at the location you specified for AuthUserFile above. Make sure this is outside your public webspace.

How to install subversion on your linux shared hosting account

1. Find out which linux distro your web host is running
2. Download a set of subversion binaries for that linux distro
1. or install that linux distro on a VM and compile into a PREFIX directory
3. Copy bin/svn and require libs onto your web host (see list below for svn 1.6.16)
4. Voila!

List of libs required for svn client:
svntest/lib/libsvn_ra-1.so.0
svntest/lib/libssl.so.0.9.8
svntest/lib/libsvn_diff-1.so.0
svntest/lib/libneon.so.27
svntest/lib/libsvn_ra_neon-1.so.0
svntest/lib/libsvn_subr-1.so.0
svntest/lib/libsvn_ra_local-1.so.0
svntest/lib/libsvn_wc-1.so.0
svntest/lib/libsvn_ra_svn-1.so.0
svntest/lib/liblber-2.4.so.2
svntest/lib/libsvn_fs_util-1.so.0
svntest/lib/libsvn_delta-1.so.0
svntest/lib/libldap-2.4.so.2
svntest/lib/libcrypto.so.0.9.8
svntest/lib/libsvn_fs_fs-1.so.0
svntest/lib/libsvn_fs-1.so.0
svntest/lib/libapr-1.so.0
svntest/lib/libserf-0.so.0
svntest/lib/libaprutil-1.so.0
svntest/lib/libgdbm.so.3
svntest/lib/libsvn_ra_serf-1.so.0
svntest/lib/libsvn_client-1.so.0
svntest/lib/libsvn_repos-1.so.0

Nginx high-performance webserver

Nginx is a high-performance HTTP server. http://wiki.nginx.org/Main

Seems to be used by quite a lot of sites and a few people have noticed nginx 404 error pages by facebook and youtube indicating they may also be using it? http://wiki.nginx.org/NginxWhyUseIt

Comes also with a memcached module to talk with memcached directly: http://www.igvita.com/2008/02/11/nginx-and-memcached-a-400-boost/

And has embedded perl in it as standard: http://wiki.nginx.org/NginxEmbeddedPerlModule

And call perl directly from SSI
<!- # perl sub="module::function" arg="parameter1" arg="parameter2"... >

For XSLT fans: http://wiki.nginx.org/NginxHttpXsltModule

Read an internal Apache variable

...such as %{MY_MOD_OUTPUT_NOTE}n -- that would be available for logging, but not visible by mod_include.

RewriteEngine On
RewriteRule ^(.*)$ $1 [E=MY_MOD_ENV_VAR:%{ENV:MY_MOD_OUTPUT_NOTE}]

Reloading mod_perl modules automatically when they change

Beware of using Apache::StatINC or Apache::Reload, especially is the module in question is a module acting as a configuration file. A severe shortage of fun can occur.