How to install libxmlsec1 via Alien::LibXMLSec on Mac OSX

1. Make sure openssl is installed
2. Download Alien::LibXMLSec (it's not on CPAN)
3. Unzip it & change to dir
4. perl Makefile.PL
5. PKG_CONFIG_PATH=/usr/local/Cellar/openssl@1.1/1.1.1k/lib/pkgconfig make
6. make
7. make install

Note: To get the Makefile.PL to run (i.e. step 3½), you may also need to:

• git clone Alien::Build from its git repo
• do a `dzil build` on that
• then set the PERL5LIB to point to the built instance of Alien::Build
• ...and also make sure its deps are in PERL5LIB.

Perl modules for SOAP

Options/notes:

• W3C::Soap - not good
• SOAP::Lite - not as good as XML::Compile
• XML::Compile - recommended
• XML::Compile::SOAP
• XML::Compile::SOAP11
• XML::Compile::WSDL11

...but generally SOAP isn't much fun.
Expect the WSDL to be wrong, you may need to store a "fixed" copy locally.

Which data file format?

List:

• CSV
• TSV
• INI
• XML
• YAML
• JSON
• RDB
• custom

Parse XML in PHP

http://php.net/manual/en/function.file-get-contents.php - fetch URL
http://php.net/manual/en/simplexmlelement.xpath.php - parse XML
http://www.php.net/manual/en/simplexml.examples-basic.php - register namespaces, etc.

Example for Google calendar. Most important parts are the namespaces and (string) casting.

    $contents = file_get_contents($calendar_url); // should end with 'full', not 'basic'
    $xml = new SimpleXMLElement($contents);

    $namespaces = $xml->getDocNamespaces();
    $xml->registerXPathNamespace('__empty_ns', $namespaces['']);
    $xml->registerXPathNamespace('gd', 'http://schemas.google.com/g/2005'); # xmlns:gd='http://schemas.google.com/g/2005'

    $entries = $xml->xpath("//__empty_ns:entry");
    while(list( , $node) = each($entries)) {
        $time_nodes = $node->xpath("gd:when/@startTime");
        while ($time_node = each($time_nodes)) {
            # <gd:when endTime="2011-08-19T17:00:00.000+01:00" startTime="2011-08-19T14:00:00.000+01:00"></gd:when>

            $time_text = (string)$time_node[1]->startTime;

            die "time = $time_text\n";

Web scraping with Ruby

In order of goodness:

• Nokogiri
• Hpricot
• Scrapi

Testing XML with Java

Testing

• XMLUnit
• JUnit - most popular

XML

• org.w3c.dom
• javax.xml.xpath (Parsing an XML Document with XPath, The Java XPath API)
• org.jdom.xpath (JDom and Xpath, required libs)

with Java

• Selenium
• Eclipse
• Maven + Jetty

oXygen Eclipse plugin

oXygen XML/XSLT: http://www.oxygenxml.com/download_oxygenxml_editor.html#Eclipse

• it's not all that, but step-through-debugging does work
• have to manually set the file association defaults to Oxygen types
• unreadable default colour scheme for Oxygen filetypes. It is difficult to import syntax-highlighting preferences in Eclipse. You're better off setting them yourself.
• to use step-through debugging, you need to:
• 'Configure Transformation Scenario' (under XML/XSL menu - it appears when you open an XML or XSL file)
• open the 'Debug Scenario' perspective in the XML/XSL menu
• setting the input parameters for debugging is fiddly
• the XML file being used as input must be "part of the project"

For-each over static values in XSL

When you want to do this:

<xsl:for-each select="$some_value in ('a','b','c','d')"><!-- invalid? -->

You have to do this:

<xsl:variable name="items">
    <item>a</item>
    <item>b</item>
    <item>c</item>
    <item>d</item>
</xsl:variable>

<xsl:for-each select="$items/item">
    <xsl:choose>
    <xsl:when test="$some_value = .">

Eclipse plugins

Eclipse 3.5

• Target Management (Remote file browsing, via SSH): http://www.eclipse.org/dsdp/tm/
• works great (but sshfs is simpler)
• Instead of that, see also http://wills-tech-notes.blogspot.com/2010/07/remote-file-browsing.html
• XSLT: http://www.eclipse.org/webtools/
• can't download through update manager, it's too slow
• supposed to offer XML/XSL features, but I can't see any
• Oxygen XML/XSLT: http://www.oxygenxml.com/download_oxygenxml_editor.html#Eclipse
• Works. To use step-through debugging,you need to open the 'Debug Scenario' in the XML or XSL menu (it appears when you open an XML or XSL file)
• vim -- why not use vim editor inside Eclipse? No good solution, apparently (none of these were tested):
• http://www.viplugin.com/viplugin/ (2009 - costs a few pounds)
• http://eclim.org/vim/ (recent)
• http://vrapper.sourceforge.net/ (recent)
• http://sourceforge.net/projects/vimplugin/ (2007, but states it supports syntax files)
• Full Screen: http://code.google.com/p/eclipse-fullscreen/ (just works) 
• P4WSAD: Perforce plugin: http://www.perforce.com/perforce/products/p4wsad.html (had to download zip file from FTP site and install manually, as automatic install URL was too slow)
• Plugin can be slow to use if you browse the repository directly, but is faster if you check out, then do right-click | Import as project, and use the standard Eclipse Navigator or Project tab to work with the files.

How to use Saxon XSL 2 processor

java -jar saxon9he.jar -xsl:/path/to/file.xsl -s:/path/to/file.xml -o:/path/to/output.xml parameter1='value1' parameter2='value2'

parser error : Start tag expected, '<' not found

What this error means:
You're not giving the parser the XML you think you are.

XSL tips

• In XSL templates, when generating new static elements with attributes, instead of <xsl:element name="foo"><xsl:attribute name="bar"><xsl:value-of select="'baz'"> etc, you can just do <foo bar="baz">! The main use for <xsl:element> and <xsl:attribute> is for dynamically named elements and attributes.

• Print debug statements like this: <xsl:message><xsl:text>min_score = </xsl:text><xsl:value-of select="$min_score"/></xsl:message>
  

Stop XML::LibXSLT and XML::LibXML connecting to w3.org

If you have this line in your XML/XSL:

<!DOCTYPE xml PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

Then your parser may connect to w3.org for validation. If w3 blocks you for hammering their server, then you may get this error message: http error : Operation in progress

http error : Operation in progress

If you don't want/need validation against the DTD (and you probably don't), you can turn off that feature in XML::LibXML:

1.70
$parser = XML::LibXML->new(load_ext_dtd => 0);

< 1.70
$parser = XML::LibXML->new();
$parser->load_ext_dtd(0);

XSLT1 processing, using Perl

Example taken from XML::LibXSLT docs, code commented and replaced because of old versions of libs:
$ perl -MXML::LibXML -le'print $XML::LibXML::VERSION'

1.69

$ perl -MXML::LibXSLT -le'print $XML::LibXSLT::VERSION'

1.59


Code:

#!/usr/bin/perl

use strict;
use warnings;
use XML::LibXSLT;
use XML::LibXML;

my $xslt = XML::LibXSLT->new();

#my $source = XML::LibXML->load_xml(location => 'file.xml');
my $parser = XML::LibXML-> new();
my $source = $parser->parse_file('file.xml');

#my $style_doc = XML::LibXML->load_xml(location=>'file.xsl', no_cdata=>1);
my $parser2 = XML::LibXML-> new();
my $style_doc = $parser2->parse_file('file.xsl');

my $stylesheet = $xslt->parse_stylesheet($style_doc);

my $results = $stylesheet->transform($source);

#print $stylesheet->output_as_bytes($results);
print $results->toString;

Using an XML configuration file for Log::Log4perl

XML configuration file:

<!--
* General configuration for log4perl
* LOGFILE.filename may be set by the application
* All logging goes to STDOUT (CONSOLE)
-->
<log4perl>
<log4perl.rootlogger>WARN, CONSOLE, LOGFILE</log4perl.rootLogger>

<log4perl.appender.console>Log::Log4perl::Appender::Screen</log4perl.appender.CONSOLE>
<log4perl.appender.console.layout>PatternLayout</log4perl.appender.CONSOLE.layout>
<log4perl.appender.console.layout.conversionpattern>[%d] [%p %c] - %m%n</log4perl.appender.CONSOLE.layout.ConversionPattern>

<log4perl.appender.logfile>Log::Log4perl::Appender::File</log4perl.appender.LOGFILE>
<!-- Let logging come to STDERR on the console, and *pipe* into a log so that unexpected errors are caught
<log4perl.appender.logfile.filename>/optional/absolute/path/to/logfile.log</log4perl.appender.LOGFILE.filename>
-->
<log4perl.appender.logfile.mode>append</log4perl.appender.LOGFILE.mode>
<log4perl.appender.logfile.layout>PatternLayout</log4perl.appender.LOGFILE.layout>
<log4perl.appender.logfile.layout.conversionpattern>[%d] [%p %c] - %m%n</log4perl.appender.LOGFILE.layout.ConversionPattern>

<!-- Add different log levels for specified modules
<log4perl.logger.clickthrough.transform>INFO</log4perl.logger.ClickThrough.Transform>
-->
</log4perl>

How to read the configuration:

my $log4perl = XML::Simple->new(ForceArray=>0, KeyAttr=>[])->XMLin( $log4perl_config_filename );
my $logname = $0;
$logname =~ s/\.pl$/.log/;
$log4perl->{'log4perl.appender.LOGFILE.filename'} = $logname;
Log::Log4perl::init($log4perl);
my $logger = Log::Log4perl->get_logger(__PACKAGE__);

XML parsing fun with Perl

• XML::Simple - No XPath, just simple nested hashrefs. Good options for collapsing keys. Useful for configuration files.

• XML::Twig - Designed to handle sub-trees of very large documents, fast.

• XML::XPath - find() method produces a nodelist, which is just an arrayref (see XML::XPath::XMLParser). It's dumb to have to "re-parse" nodes in order to look further down. Maybe I'm missing something?

• XML::TreeBuilder or HTML::TreeBuilder - No XPath, but a splendid look_down() method. Returns objects on which you can also call look_down().

• XML::LibXML and XML::LibXML::XPathContext - The best.
• But HTML source often requires tidying into XML
• And there is fun to be has converting entities

use XML::LibXML;
use XML::LibXML::XPathContext;
$parser = XML::LibXML->new();
$xml = $parser->parse_string($self->raw_xml);
$xpath = XML::LibXML::XPathContext->new($xml->documentElement);
$xpath->registerNs('atom', 'http://www.w3.org/2005/Atom');
print $xpath->findvalue("/atom:feed/atom:title");


Nothing seems good for testing XML paths, tags and attributes.

Display output of XSL transformation using PHP

Use XAMPP or similar to serve (reportedly works on PHP 5):

<?
$query = $_GET['q'];
$filePath = 'http://source-server:8889/page/'.$query;

$xslDoc = new DOMDocument();
$xslDoc->load("http://localhost/xslt/page_render.xsl");
$xmlDoc = new DOMDocument();
$xmlDoc->load( $filePath );

$proc = new XSLTProcessor();

$proc->importStylesheet($xslDoc);
echo $proc->transformToXML($xmlDoc);
?>

XML::Validator::Schema

#!/usr/bin/perl

use XML::SAX::ParserFactory;
use XML::Validator::Schema;
# create a new validator object, using foo.xsd
$validator = XML::Validator::Schema->new(file => 'foo.xsd');
# create a SAX parser and assign the validator as a Handler
$parser = XML::SAX::ParserFactory->parser(Handler => $validator);
# validate foo.xml against foo.xsd
eval { $parser->parse_uri('foo.xml') };
die "File failed validation: $@" if $@;

The problem is:

• XML::Validator::Schema ignores missing tags
• and attribute fixing doesn't work, at all

Validating with XML::XPath

use XML::XPath; # update: use XML::LibXML and XML::LibXML::XPathContext instead
use Test::More qw(no_plan);

my $id = 298;
my $xml = &get_url("/$id")->{xml};
my $xp = XML::XPath->new( xml => $xml );
my $r = '/feed'; # root
ok( $xp->exists($r), "there should be a feed" );
ok( $xp->exists("$r/id"), "there should be a feed ID" );
ok( $xp->find("$r/id") =~ m{/$id}, "feed ID should be correct" );
ok( $xp->exists("$r/link[\@rel=\"self\"]"), "rel=self attr is correct" );
ok( ($xp->find("$r/link")->get_nodelist)[0]->getAttribute('href'), "href link exists" ); # this is ugly
check( $xp->find("$r/updated") => like => qr/^\d{4}\-\d\d\-\d\dT\d\d:\d\d:\d\d\.\d+Z$/ );

my $entries = $xp->find("/feed/entry"); # find all entry sections
ok( $entries, "there are entries" );
if ($entries->size) {
my $e = "/entry";
foreach my $node ($entries->get_nodelist) {
# it's bloody stupid transforming the XML::XPath object to a string and then parsing back to an object
# instead maybe loop through each node in the list using xpath [0] [1], etc?
# But why should I have to hack around this....
my $entry = XML::XPath->new( xml => $node->toString );
ok( $entry->exists("$e/title"), "title exists" );
}
}

# Conclusion: XML::XPath is bad for writing tests.

Nested comments

Are your comments disappearing when you wrap them with a <xsl:comment> tag?

They may already be inside a comment.