Use Perl-like regular expression in vi

You have to escape some special characters and not others:

s/"\(.\+\)\$/"\1"} = "\1"/

Escape special pluses, capturing parentheses and literal $

use \1 for the captured strings.

is reserved

While attempting to read the filehandle that you think points to a file, you get this error:

Modification of a read-only value attempted

Maybe this means the script is reading from the inline __DATA__ section, and you're trying to write to it?

Make IMAP Gmail work on blackberry

If you enter your Google email address when prompted, there is no way to configure advanced settings to make IMAP accounts work. In fact, I called Orange and was told that their Blackberries do not support IMAP with Gmail, and it does not work.

But they are wrong. It does work - you need to enter an email address that is not one of the major commercial providers, but does host email accounts. I used a fake email alias @ a domain I know has a working mail record: siteaboutnothing.com

Then, as you follow the prompts, choose "Advanced settings" or similar, and you are able to set the server to imap.gmail.com and change the email address back to your gmail account.

I now have it set up so messages marked as read on my phone are sync'ed with my main gmail account and I don't have to manage two accounts at once.

Email output from the crontab

method 1 - for all commands

Add this as the first line of crontab:

MAILTO=address1@host.com,address2@host.co.uk

method 2 - for specific commands

Add this line anywhere near the top:

NOTIFY_LIST="address1@host.com,address2@host.co.uk"

Use this for the command:

00 09 * * * run_command | mail -s "command has run" $NOTIFY_LIST

Template Toolkit troubleshooting

If a module upon which your code is dependent does not compile, then Template Toolkit 2 under mod_perl 1.3 can give two misleading error messages:

1. can't locate that module in the @INC path, although it is really there
2. plugin error - load() returned a false value

The solution to both error messages is to syntax check all the dependent modules.

Reloading mod_perl modules automatically when they change

Beware of using Apache::StatINC or Apache::Reload, especially is the module in question is a module acting as a configuration file. A severe shortage of fun can occur.

MySQL stored procedure gotchas

END and END IF statements must always end with a semicolon.

Debugging Perl DBI

You can set the tracelevel from 1-9 to see diagnostic messages in the logs!

Putting MySQL to sleep

Use

select sleep(30);

You can call this from a script and then run

show full processlist;

on the server to verify that the script is hitting it.

See hex dump of file on Linux or Mac

hexdump -C [file]

or

cat [file] | xxd

Errors in crontab

When your clever crontab entry doesn't work:

/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'

/bin/sh: -c: line 1: syntax error: unexpected end of file

You might need to escape some characters, like the percent sign: \%

Also consider that the backslash may not have the desired effect if it's inside double quotes.

Allow different users to access MySQL

GRANT ALL PRIVILEGES ON *.* TO 'username'@'hostname';

The single quotes around username and hostname are required.

Vim 'unset' command

Instead of an "unset" command, vi prefixes the option with "no".

So to turn off visible tabs and newlines, use:

:set nolist

Find files in workspace that perforce doesn't know about

for file in `find . -type f`; do p4 diff -f ./$file; done | grep "not on client" | sed 's/on client/in depot/'

But the sed part doesn't work (wrong stream?)

How to change the colours of directories and files

To change the colour of directories from dark blue to light blue (cyan):

    export LS_COLORS=$(echo $LS_COLORS|sed 's/di=01;34/di=01;36/')

See also

    man dircolors

and

    dircolors --print-database

...for a list of the colours.

System maintenance

There is a reason we have a process of change control. It has become a dirty word in some departments, but the bare minimum is:

• making a change in the development environment
• testing it, preferably by more than one person/team
• packaging up the release and labeling it properly
• making a formal request to deploy it, including roll-back instructions

In addition, a precedent should never be set to fix something that another department has broken!

Log4perl

    use Log::Log4perl qw(:easy);
    Log::Log4perl->easy_init($ERROR); 

    DEBUG "This doesn't go anywhere";
    ERROR "This gets logged";

See stored procedures in MySQL

List them

mysql> show procedure status;

See the code of one

mysql> show create procedure [name];

Perl sprintf

        # Format number with up to 8 leading zeroes
        $result = sprintf("%08d", $number);

        # Round number to 3 digits after decimal point
        $rounded = sprintf("%.3f", $number);

Perl's sprintf permits the following universally-known conversions:

   %% a percent sign
 %c a character with the given number
 %s a string
 %d a signed integer, in decimal
 %u an unsigned integer, in decimal
 %o an unsigned integer, in octal
 %x an unsigned integer, in hexadecimal
 %e a floating-point number, in scientific notation
 %f a floating-point number, in fixed decimal notation
 %g a floating-point number, in %e or %f notation

Perl test comparison methods

Test::More

use Test::More tests => 23;
# or
use Test::More qw(no_plan); 

ok($got eq $expected, $test_name);

like ($got, qr/expected/, $test_name);

is_deeply($got_complex_structure, $expected_complex_structure, $test_name);

Test::Differences

Compare data structures

eq_or_diff $got, $expected, "description"

Test::Lazy

No descriptions required

# Will evaluate the code and check it:
try('qw/a/' => eq => 'a');

# Don't evaluate, but still compare:
check(1 => is => 1);

Everybody do the DBI

  use DBI;

@driver_names = DBI->available_drivers;
%drivers      = DBI->installed_drivers;
@data_sources = DBI->data_sources($driver_name, \%attr);

$dbh = DBI->connect($data_source, $username, $auth, \%attr);

$rv  = $dbh->do($statement);
$rv  = $dbh->do($statement, \%attr);
$rv  = $dbh->do($statement, \%attr, @bind_values);

$ary_ref  = $dbh->selectall_arrayref($statement);
$hash_ref = $dbh->selectall_hashref($statement, $key_field);

$ary_ref  = $dbh->selectcol_arrayref($statement);
$ary_ref  = $dbh->selectcol_arrayref($statement, \%attr);

@row_ary  = $dbh->selectrow_array($statement);
$ary_ref  = $dbh->selectrow_arrayref($statement);
$hash_ref = $dbh->selectrow_hashref($statement);

$sth = $dbh->prepare($statement);
$sth = $dbh->prepare_cached($statement);

$rc = $sth->bind_param($p_num, $bind_value);
$rc = $sth->bind_param($p_num, $bind_value, $bind_type);
$rc = $sth->bind_param($p_num, $bind_value, \%attr);

$rv = $sth->execute;
$rv = $sth->execute(@bind_values);
$rv = $sth->execute_array(\%attr, ...);

$rc = $sth->bind_col($col_num, \$col_variable);
$rc = $sth->bind_columns(@list_of_refs_to_vars_to_bind);

@row_ary  = $sth->fetchrow_array;
$ary_ref  = $sth->fetchrow_arrayref;
$hash_ref = $sth->fetchrow_hashref;

$ary_ref  = $sth->fetchall_arrayref;
$ary_ref  = $sth->fetchall_arrayref( $slice, $max_rows );

$hash_ref = $sth->fetchall_hashref( $key_field );

$rv  = $sth->rows;

$rc  = $dbh->begin_work;
$rc  = $dbh->commit;
$rc  = $dbh->rollback;

$quoted_string = $dbh->quote($string);

$rc  = $h->err;
$str = $h->errstr;
$rv  = $h->state;

$rc  = $dbh->disconnect;

Bash menu

Try this menu script for bash scripts:

while getopts ":u:a:s:v" options; do

  case $options in

    u ) uname=$OPTARG;;

    a ) attrs=$OPTARG;;

    s ) searchattr=$OPTARG;;

    v ) att=ALL;;

    h ) echo $USAGE;;

    \? ) echo $USAGE

         exit 1;;

    * ) echo $USAGE

          exit 1;;

  esac

done

echo uname = $uname

Test scripts

Even a set of test scripts often benefit from modularised separate libraries, configuration files and dedicated data files. And logging.

Importing a module's namespace in Perl

Ever get this error for a module you've written?

Use of inherited AUTOLOAD for non-method X() is deprecated

Put this at the top of your module:

package My::Module;

use Exporter;

our @ISA = 'Exporter';

our @EXPORT = qw(put the methods here);

And then in the main program:

use My::Module qw( the methods to import );

keywords: export import exporting

Bash variable comparisons

...bash logic...

Note that integer and string comparison use a different set of operators.

integer comparison

-eq

is equal to

if [ "$a" -eq "$b" ]

-ne

is not equal to

if [ "$a" -ne "$b" ]

-gt

is greater than

if [ "$a" -gt "$b" ]

-ge

is greater than or equal to

if [ "$a" -ge "$b" ]

-lt

is less than

if [ "$a" -lt "$b" ]

-le

is less than or equal to

if [ "$a" -le "$b" ]

<

is less than (within double parentheses)

(("$a" < "$b"))

<=

is less than or equal to (within double parentheses)

(("$a" <= "$b"))

>

is greater than (within double parentheses)

(("$a" > "$b"))

>=

is greater than or equal to (within double parentheses)

(("$a" >= "$b"))

string comparison

=

is equal to

if [ "$a" = "$b" ]

==

is equal to

if [ "$a" == "$b" ]

This is a synonym for =.

[original page]

Set file modes in Perforce

Perforce.

Edit a file and:

• explicitly set RCS keywords to be expanded (k)
• have the executable bit set (x)
• always be writeable on the client (w)

p4 edit -t text+kxw

or if it's already being edited:

p4 reopen -t text+kxw

See also

p4 help filetypes

+w always writable on client
+x exec bit set on client

+k $Keyword$ expansion of Id, Header, Author, Date, DateTime, Change, File, Revision
+ko $Keyword$ expansion of Id, Header only
+l exclusive open: disallow multiple opens

[full article]

Shortcuts in Firefox 2

You can set up a shortcut keyword by

1. going to the properties of a bookmark in 'organize bookmarks',
2. putting a %s as the wildcard in the URL
3. setting a keyword for the shortcut

Then you can type the keyword and a wildcard in the address bar to load that url.

[full article]

Make Firefox 3 look like Firefox 2

From the neo-luddite department.

Perforce commands

p4 sync -f ./...

p4 diff -f [file]

p4 filelog -l -i -t [file] // complete history (-l = long text, -i = follow branches, -t = include times)

p4 annotate [file] // show file with changes marked

Tagging

// NOTE: If no revision specified, the head revision will be tagged

p4 tag -l ProjectName_R01 [filename]#revision // tag a specific revision of a file.

p4 files @ProjectName_R01 // list files on a tag

vi command

When tabs are automatically converted to spaces:

:set tabstop=4

:set expandtab

To insert a hard tab:

Ctrl-V, [tab]

How to use encoding

Encoding is necessary only for transmission of data.

Any data stored in files or a database should never be encoded.

Get the transmission right (decode 'automatically' as part of the receiving process, as soon as possible after receiving the data) and you'll never need to worry about decoding issues.

Regular expressions in vi

Keep the first character of a line, but then insert an 'A':

s/^\(.\)/\1A/

NOTE: parentheses and the plus sign must be escaped.

Freeing memory in C

If you allocate memory in a subroutine for a variable that you then return, you’ve got to free the variable you return:

 
char * allocstring(char * text)
{
    char * s = malloc(s, strlen(text)+1);
    strcpy(s, text);
    return s;
}
 
void main()
{
    char * s = allocstring(“Hello World”);
    free(s);
}

Regular expressions in C: Advice

If you can avoid using regular expressions in C, then *do not use them!!*

Instead try the strstr (find substring) and strtok (like perl's split) routines.

These warnings:

warning: initialization makes pointer from integer without a cast

warning: assignment makes pointer from integer without a cast

...for a line containing malloc could mean that you forgot to #include "stdlib.h".

Functions are assumed to return an integer type by default if not declared.

C printf and scanf format specifiers

Format Specifier	Type
%d (or %i)	int
%c	char
%f	float
%lf	double
%s	string
%x	hexadecimal

Comparing strings in C

In c it is not possible to directly compare two strings so a statement like if (string1==string2) is not valid. Most c libraries contain a function called the strcmp().This is used to compare two strings in the following manner.

if(strcmp(name1,name2)==0)
 puts("The names are the same");
else
 puts("The names are not the same.");

Thanks johnt

debugging technique

When programming C, it can be difficult to track down a segmentation fault.

It is often caused by attempting to read uninitialised memory.

Try skipping parts of the program until you locate the statement causing it.

Then replace each part of the data by hard-coded values, one by one, until you find the uninitialised variable/pointer.

c logging

if your fprintf statements don't reach the apache error log, you may have to flush the output buffer:

fprintf(stderr, "error statement\n");

fflush(stderr);

Logging in an apache handler

ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, "debug message");

C pointer error

This error:

    request for member `xxx' in something not a structure or union

...could mean you need to use structure->member instead of structure.member, as sturcture is a pointer.

Burning DVDs

2008: Windows

• DVD Flick is perfect for combining .AVI video with .SRT subtitles to produce DVD files.
• DVD Shrink can reduce a commercial DVD in size to fit onto a standard 4.7Gb blank DVD. There is no noticable loss of quality.
• Leawo DVD to MP4 Converter works well.
• Frontends using cdrtools are supposed to burn .VOB files, but I couldn't get this to work. Nothing beats Nero 6 if you can find an old copy. The latest version is 185Mb, and you have to pay for it!

Sep 2012:

• Free trial of Aimersoft DVD Creator easily burns MP4 files to DVD. It's easy to create chapters. But it writes a watermark in the middle of the screen.

c function errors

Errors like these:

previous implicit declaration of `function'

`function' was previously implicitly declared to return `int'

...point to a missing function declaration at the top of the file.

Concatenate C strings, and return them from a function

Don't do this. Just use strcat() and strncpy() instead.

#include <stdlib.h>

char* concat(char*, char*);

int main (int argc, char** argv)
{
char *x = "something";
char *y = " completely different.";
  char *combined = concat(x, y);
printf("And now for %s\n", combined);
}

char* concat(char *a, char *b)
{
  char *target = malloc( strlen(a) + strlen(b) + 1 );
  strcpy(target,a);
strcat(target,b);
return target;
}

C static

The keyword 'static' before a function name means that the function will only be available to callers in the file where it is declared.

Regular expressions in C

Using glibc's regex.h

• Make sure the number of expected matches (nmatch) is high enough, or there will be garbage at the end of the matchptr array
  

• If regexec() succeeds [returns zero], matchptr contains:
• 0: the whole regex
• 1: first parenthesis match
• if regexec() fails [returns a value], don't even look in matchptr

• Characters that must be escaped:
• parentheses
• plus signs
• If you're matching the end of line, it must be inside any parenthesis!
• 
  

location of C libs on nolsearch09

according to gcc -print-search-dirs

/usr/lib/gcc-lib/i386-redhat-linux/3.2.3/../../../bcc/include

or

/usr/lib/bcc/include

Deeply copy a perl hash

use Storable;

$bad_recipe = dclone($recipe);

Vim: Turn tabs into spaces

:set tabstop=4

:set expandtab

MySQL variables - caching

http://dev.mysql.com/tech-resources/articles/mysql-query-cache.html

Show cache status

> show variables like 'query%';

Show cache counters

mysql> show status like 'qc%';

Bash regular expressions

Log4perl log levels

OFF

FATAL

ERROR

WARN

INFO

DEBUG

TRACE

ALL

vi tab length

:set tabstop=4

Delete documents from IDOL

http://host:indexport/DREDELETEDOC?docs=[doc IDs]&DREDBname=[DB name]

[DB name] is mandatory

See also DREDELETEREF

Bash arithmetic/command expansion

Command expansion, I use a lot

echo something $( date +%s )

Backticks also work for command expansion

echo something `date +%s`

...but do they work any differently to $( )?

Conditionals, like an if statement, use square brackets

if [ "$end_hour" = "24" ]; then ....; fi

Arithmetic, like 1 + 2, uses this:

i=$(( i + 1 ))
let i+=1

See http://www.softpanorama.org/Scripting/Shellorama/arithmetic_expressions.shtml

Temporarily suspend a process

At times, you may find it necessary to temporarily suspend a process, and then resume its execution at a later time. The following two commands will suspend a process, and the resume it, respectively:

# kill -STOP 945
# kill -CONT 945

Bash getopts

Use named command line parameters in Bash scripts.

Evidence Based Scheduling

Currently only available in Fogbugz 6.

This is a system that learns from your work estimates, decides how good you are at estimating, and automatically adjusts as you go along to give a more realistic estimate. It works on the principle that people tend to consistently either over-estimate, under-estimate by the same factor. It will also automatically factor in and balance out any delays that occur.

Test Coverage Vim Mappings

From the vi text editor, see which tests cover the file you're working on, or which modules are covered by the test you're in.

Features on the way: Being able to know which tests cover a particular line of a file!

http://search.cpan.org/dist/Devel-CoverX-Covered/lib/Devel/CoverX/Covered.pm

Perforce: Diff -f only on files that have changed

p4 diff -f ./...  | egrep -B 2 '^<|^>'  | less

MySQL: Log to a file

tee [filename]

[commands to log]

notee

Show size of MySQL tables

show table status

Displays size of tables and indexes, in bytes.

Profiling MySQL

SET profiling = 1;

SET profiling_history_size = 100; -- maximum is 100

SHOW PROFILES; -- shows a list of recent commands

SHOW PROFILE; -- shows a very detailed breakdown of time spent on each database task

For example, if a lot of time is spent 'Sending data', try improving your indexes.

http://dev.mysql.com/doc/refman/5.0/en/show-profiles.html

MySQL auto increment tip

TRUNCATE a table to reset its auto_increment number

MySQL JOIN syntax

(remember the brackets):

SELECT COUNT(*) FROM search_fact JOIN (date_dm) USING (date_id) WHERE day = 22 AND month = 4 AND year = 2008;

or equivalently:

SELECT COUNT(*) from search_fact, date_dm WHERE search_fact.date_id = date_dm.date_id AND day = 22 AND month = 4 AND year = 2008;

Select all the search_fact rows that have a date_id that corresponds to 22/04/2008.

MySQL DELETE syntax

DELETE FROM result_set USING search, result_set WHERE search.id = result_set.id AND occurred >= '2008-04-26 00:00:00' AND occurred < '2008-04-27 00:00:00';

Delete from result_set, using a range of dates in search, matching on id.

SELECT COUNT(*) FROM result_set JOIN search WHERE search.id = result_set.id AND occurred >= '2008-04-26 00:00:00' AND occurred < '2008-04-27 00:00:00';

Select the rows that would be deleted, as above.

MySQL LEFT JOIN syntax

SELECT count(*) FROM search LEFT JOIN click ON search.id = click.id WHERE click.id IS NOT NULL;

or

SELECT count(*) FROM search, click WHERE search.id = click.id AND click.id IS NOT NULL;

or

SELECT count(*) FROM search, click WHERE search.id = click.id;


Count the rows in search that have a corresponding matching id row in click.

Five Ways to Improve Your Perl Programming

Article

MySQL tools

MySQL Workbench

Free, works on Windows only at time of writing.

MySQL Administrator, MySQL Query Browser and MySQL Migration Toolkit

All free, and work on Windows, Mac and Linux.
Source code also available.

Installing MySQL on linux

Download these files:

MySQL-server-community-5.0.45-0.rhel3.i386.rpm
MySQL-client-community-5.0.45-0.rhel3.i386.rpm
MySQL-shared-community-5.0.45-0.rhel3.i386.rpm
MySQL-devel-community-5.0.45-0.rhel3.i386.rpm

Run this command:

rpm -i MySQL*

NOTE: The default data directory is /var/lib/mysql
Make sure the /var partition has enough space for your database!

You can get the files from redhat.

validating xml files from the command line

It may need to be installed, but it really good at checking encodings, xml schemas etc..

http://xmlsoft.org/xmllint.html

xmllint --valid --noout test.xml

- worked for Rob H.

Hanging commands

If a command seems to be taking a very long time, e.g. an SQL query:

Check the diskspace: df -h

It could be that the /var parition is full (this is the default location for MySQL tables).

XSL regular expressions

NOTE: Variables must be declared in a root <xsl:template> node.

<!-- extract the ID -->
<xsl:variable name="id">
<xsl:analyze-string select="doc:entry/doc:id" regex="id/(\d+)">
<xsl:matching-substring>
<xsl:value-of select="regex-group(1)"/>
</xsl:matching-substring>
</xsl:analyze-string>
</xsl:variable>

XSL: generate the current date

For stylesheets processed using Java:

<xsl:stylesheet version="2.0" xmlns:java="java">

<today-date>
<xsl:value-of select="java:util.Date.new()" />
</today-date>

XSL: use a variable

<xsl:value-of select="$location"/>

XSL lookup table

The lookup file:

<?xml version="1.0"?>
<lookup>
<location id="1769" guide="TT00379a" average="TT123456"/>
<location id="1230" guide="TT003999" average="TT000001"/>
</lookup>

The code: NOTE: $id is a local variable.

<!-- look up the country guide ID -->
<xsl:variable name="guide">
<xsl:for-each
select="document('lookup.xml')">
<xsl:value-of select="key('map',$id)/@guide"/>
</xsl:for-each>
</xsl:variable>

XSL: http requests

Getting the hostname given in the current http request can be done using XSP.

GNU screen

Type 'screen' after you log into a server.

Or type byobu for an enhancement to GNU screen!

"Allows users to quickly switch their .screenrc to any of the available profiles"
See also http://launchpad.net/byobu
byobu version 3.5
Screen version 4.00.03jw4 (FAU) 2-May-06

Byobu features:

• Visible at bottom of screen at all times:
• All screen tabs
• Server welcome message
• Uptime
• Load
• Hardware spec
• [configurable to display anything you want]
• Tab names show special characters to provide more info:
• - (dash) means that window will be switched to with Ctrl-A, A
• @ (at) means that tab has seen activity since you last saw it

Screen keyboard commands:

• Ctrl-A, c -- to create a new terminal session
• Ctrl-A, Ctrl-A -- to alt-tab
• Ctrl-A, " -- to list the terminals
• Ctrl-A, A -- to rename the terminal
• Ctrl-A M -- (monitor for activity) Toggles monitoring of the current window
• Ctrl-A _ -- (monitor for silence) Toggle

Best screen feature:

1. Log into the server.
2. Type screen to start a session.
3. Work as normal.
4. When finished, just close/kill the terminal program, don't log off.
5. Later, on any (other) PC, log into the server again.
6. Type screen -D -R to retrieve your session, just as you left it.

I don't know of a way to change the order of the tabs.

vi syntax higlighting

Use

:set syn=perl

:set syn=tt2

and

:highlight Comment ctermfg=Green guifg=Green

Combine STDOUT and STDERR

It's supposed to be done by adding 2>&1 to the end of the command, but that doesn't work for certain commands like time.

Instead make a script combine.sh containing this:

$@ 2>&1

Call it like this:

./combine.sh command_to_capture with parameters 2>&1 > outputfile

mount a directory from another host

in /etc/fstab:
//example.com/inetpub2 /nolmcs02/inetpub smbfs credentials=/root/nolmcs2-auth.txt,domain=NATIONAL,uid=noladmin,gid=noladmin,directio,hard 0 0

where /root/nolmcs2-auth.txt (or whatever you name it) is a file containing the username and password to use:

[root@nolappsdc01 ~]# cat /root/nolmcs2-auth.txt
username=
password=

You can mount this manually to test using:

mount -t smbfs //example.com/inetpub2 /nolmcs02/inetpub2 -o credentials=/root/nolmcs2-auth.txt,domain=NATIONAL,uid=noladmin,gid=noladmin,directio,hard

The credentials file is used so it will work on restart automatically. Instead of credentials=, you can use username=, etc. temporarily.

restart mysql

Usage: /etc/init.d/mysql {start|stop|restart|reload|force-reload|status} [ MySQL server options ]