STDERR and STDOUT appear in the wrong order when piping to a file

a Perl script - test.pl:

print "OUT 1\n";
print STDERR "ERR 2\n";
print "OUT 3\n";
print STDERR "ERR 4\n";
print "OUT 5\n";

Run the script, piping all output into a file:

perl test.pl &> file.log

cat file.log

ERR 2

ERR 4

OUT 1

OUT 3

OUT 5

Output is in the wrong order :(

Run it again using unbuffer:

unbuffer perl test.pl &> file.log

cat file.log

OUT 1

ERR 2

OUT 3

ERR 4

OUT 5

Output is in the correct order :)

(source)

Another way:

script -c 'perl ~/temp/stream.pl' file.log

cat file.log

(source)

How to capture linux "time" output

Pipe time output into a file like this:

{ time ls; } 2>time.output

Don't forget the semicolon!

(source)

Run an external command from Perl, capturing both the output and the exit value

# This is how to run an external command from Perl,
# capturing both the output and the exit value

sub run_external_command {
    my ($self, $command) = @_;
    $self->logger->debug("Running command: $command");

    my $output = '';
    $command .= ' 2>&1 | '; # capture all output
    open(my $capture, $command);
    {
        local $/ = undef;
        $output = <$capture>;
    }
    close($capture);
    chomp($output);

    my $exit_value = ${^CHILD_ERROR_NATIVE} >> 8;
    $self->logger->debug("Exit value of command: '$exit_value'");
    $self->logger->debug("Output of command: '".($output || '[no output]')."'");

    return ($exit_value, $output);
}

This is another way:

use IPC::Open3;

sub run_external_command_2 {
  local (*my_in, *my_out, *my_err);

  my $command = "some command here that might error";
  my $pid = open3(\*my_in, \*my_out, \*my_err, $command) or die($!);

  close(my_in);
  my $std_out = do { local $/; };close(my_out);
  my $std_err = do { local $/; };close(my_err);

  waitpid($pid, 0);
  my $exit_code = $? >> 8;

  return { exit_code => $exit_code, std_out => $std_out, std_err => $std_err };
}

Run an SQL script

• (a) pipe it into the program:  mysql < script.sql
• (b) source it from within mysql:  source script.sql

See also batch commands

Combine STDOUT and STDERR

It's supposed to be done by adding 2>&1 to the end of the command, but that doesn't work for certain commands like time.

Instead make a script combine.sh containing this:

$1 $2 $3 $4 $5 $6 $7 $8 $9 ${10} ${11} ${12} ${13} ${14} ${15} ${16} ${17} ${18} ${19} ${20} 2>&1

Call it like this:

./combine.sh command_to_capture with parameters 2>&1 > outputfile 

Have unattended scripts look after themselves

# Send all output to a logfile and supress input
typeset LOG="/tmp/${0##*/}.out"
mv $LOG ${LOG}.old >/dev/null 2>&1
[[ -t 1 ]] && echo "Writing to logfile '$LOG'."
exec > $LOG 2>&1
exec < /dev/null 2<&1

Coloured log output

Tail the log and pipe it through this:

#!/usr/bin/perl

use warnings;
use strict;
use Term::ANSIColor qw(color);

while( <> ) {
chomp;
s/\t/ /g;
(/ERROR|failed/) && do {
print( color( 'red' ). $_ .color( 'reset' )."\n" );
next;
};
(/WARN/) && do {
print( color( 'magenta' ). $_ .color( 'reset' )."\n" );
next;
};
(/succeeded/) && do {
print( color( 'green' ). $_ .color( 'reset' )."\n" );
next;
};
(/INFO/) && do {
print( color( 'yellow' ). $_ .color( 'reset' )."\n" );
next;
};
(/DEBUG/) && do {
print( color( 'blue' ). $_ .color( 'reset' )."\n" );
next;
};
print $_."\n";
}

Combine STDOUT and STDERR

It's supposed to be done by adding 2>&1 to the end of the command, but that doesn't work for certain commands like time.

Instead make a script combine.sh containing this:

$@ 2>&1

Call it like this:

./combine.sh command_to_capture with parameters 2>&1 > outputfile