XML Collector

From OpenNMS
Jump to: navigation, search

Warning.png IMPORTANT

This feature is available only from OpenNMS 1.10 onward.


Installation

Note.png Plugin Installation

The XML Collector is not part of the core packages. For this reason, an additional package named opennms-plugin-protocol-xml must be installed separately in order to properly use this collector.


The stock sample configuration files were created for 3GPP Data Collection, which is a very special case of parsing XML Data that requires some advanced features of the XML Collector that probably can be ignored for almost all standard cases.

So, for standard usage, it is recommended to create the configuration files from scratch and use the current content as a configuration example.

Configuration Basics

This collector has been created to extract data from an XML document in order to store it on RRDs/JRBs and for threshold processing. This XML can be retrieved in many ways including HTTP and SFTP and the local filesystem.

Similar to the SNMP collector, there are two kinds of data:

  1. Single (or scalar) metrics, usually related with a node resource.
  2. Tabular metrics, usually related with Generic Index Resources (similar to SNMP tables).


Like any other collector, it requires the creation of an XML collection inside a file named xml-datacollection-config.xml, and associate it with a service/package on collectd-configuration.xml, in order to enable and use this collector.

For tabular metrics, the resource types used must be defined inside some file created in the etc/datacollection/ directory.

Single Metrics

The whole idea of this collector is to define the proper XPaths to access the relevant information from the XML document.

For example, suppose that the XML document has the following structure:

<data>
  <server name=”srv01”>
    <cpu>30</cpu>
    <mem value=”85”/>
    <filesystem size=”100Mb”>
        <usage type=”percentage”>80</usage>
    </filesystem>
  </server>
</data>

In order to store the value of the cpu, memory and file system usage on a RRD, it is required a xml-source like the following:

<xml-source url="http://{ipaddr}/stats">
    <xml-group name="srv-stats" resource-type="node" resource-xpath="/data/server">
      <xml-object name="cpu" type="GAUGE" xpath="cpu" />
      <xml-object name="mem" type="GAUGE" xpath="mem/@value” />
      <xml-object name="fsUsage" type="GAUGE" xpath="filesystem/usage" />
      <xml-object name="fsSize" type="String" xpath="filesystem/@size" />
    </xml-group>
</xml-source>
  • resource-type=node is mandatory for node level (scalar) statistics.
  • resource-xpath is the absolute XPath to the resource entry.
  • xpath on each xml-object is a relative XPath calculated from the resource’s Xpath to the metric value.

The above example assume that the XML document will be retrieved using HTTP. The place holder {ipaddr} will be replaced at runtime with the IP Address of the node. For example, if the address is 192.168.0.1, the URL used at runtime will be http://192.168.0.1/stats

In order to create charts for the above configuration, the graph templates should look like this:

reports=sample.cpu, sample.mem, sample.fs

report.sample.cpu.name=Sample CPU
report.sample.cpu.columns=cpu
report.sample.cpu.type=nodeSnmp
report.sample.cpu.command=--title="Sample CPU" \
 DEF:v1={rrd1}:cpu:AVERAGE \
 LINE2:v1#ff0000:"CPU" \
 GPRINT:v1:AVERAGE:"    Avg\\: %8.2lf %s" \
 GPRINT:v1:MIN:"Min\\: %8.2lf %s" \
 GPRINT:v1:MAX:"Max\\: %8.2lf %s\\n"

report.sample.mem.name=Sample Memory
report.sample.mem.columns=mem
report.sample.mem.type=nodeSnmp
report.sample.mem.command=--title="Sample Memory" \
 DEF:v1={rrd1}:mem:AVERAGE \
 LINE2:v1#ff0000:"Memory" \
 GPRINT:v1:AVERAGE:"    Avg\\: %8.2lf %s" \
 GPRINT:v1:MIN:"Min\\: %8.2lf %s" \
 GPRINT:v1:MAX:"Max\\: %8.2lf %s\\n"

report.sample.fs.name=Sample FileSystem Usage
report.sample.fs.columns=fsUsage,fsSize
report.sample.fs.type=nodeSnmp
report.sample.fs.command=--title="Sample FileSystem Usage %" \
 DEF:used={rrd1}:fsUsage:AVERAGE \
 DEF:size={rrd1}:fsSize:AVERAGE \
 CDEF:usage=used,size,/,100,*
 LINE2:usage#ff0000:"% Usage" \
 GPRINT:usage:AVERAGE:"    Avg\\: %8.2lf %s" \
 GPRINT:usage:MIN:"Min\\: %8.2lf %s" \
 GPRINT:usage:MAX:"Max\\: %8.2lf %s\\n"

The name attribute of the xml-object tag must be used for the columns attribute on the template and for the DEF name as well.

Because these are single metrics, nodeSnmp must be used for the type.

Tabular Metrics

Warning.png IMPORTANT

The following information is for educational purposes only. Solaris does not provide the statistics used on the examples in XML format by default.


The difference here is that the XML document is in a tabular form similar to SNMP Table statistics.

This requires to define a custom resource type. This can be added directly on datacollection-config.xml or added indirectly using an external reference. The second method is preferred in order to simplify the configuration and avoid confusions with the current SNMP configuration.

Suppose that we want to process Solaris Zones statistics, which comes with the following structure:

<zones>
	<zone id="0" name="global" timestamp="1299258742">
		<parameter key="nproc" value="245" />
		<parameter key="nlwp" value="1455" />
		<parameter key="pr_size" value="2646864" />
		<parameter key="pr_rssize" value="1851072" />
		<parameter key="pctmem" value="0.7" />
		<parameter key="pctcpu" value="0.24" />
	</zone>
	<zone id="871" name="zone1" timestamp="1299258742">
		<parameter key="nproc" value="24" />
		<parameter key="nlwp" value="328" />
		<parameter key="pr_size" value="1671128" />
		<parameter key="pr_rssize" value="1193240" />
		<parameter key="pctmem" value="0.4" />
		<parameter key="pctcpu" value="0.07" />
	</zone>
	<zone id="871" name="zone2" timestamp="1299258742">
		<parameter key="nproc" value="24" />
		<parameter key="nlwp" value="328" />
		<parameter key="pr_size" value="1671128" />
		<parameter key="pr_rssize" value="1193240" />
		<parameter key="pctmem" value="0.4" />
		<parameter key="pctcpu" value="0.07" />
	</zone>
</zones>

It is clear that this is a tabular data and each resource is a Solaris Zone.

First, a resource type must be defined. For this requirement, a new file named datacollection/solaris-zones.xml will be created with the following content:

<datacollection-group name="Solaris Zones">
  <resourceType name="solarisZone" label="Solaris Zone"
   resourceLabel="Zone ${zoneName}" >
      <persistenceSelectorStrategy class="org.opennms.netmgt.collectd.PersistAllSelectorStrategy"/>
      <storageStrategy class="org.opennms.netmgt.dao.support.IndexStorageStrategy"/>
  </resourceType>
</datacollection-group>

Then the XML source should look like this:

<xml-source url="http://{ipaddr}/zone-stats">
    <xml-group name="solaris-zone-stats" resource-type="solarisZone"
                resource-xpath="/zones/zone"
                key-xpath="@name">
      <xml-object name="zoneName" type="string" xpath="@name" />
      <xml-object name="nproc" type="GAUGE" xpath="parameter[@key='nproc']/@value" />
      <xml-object name="nlwp" type="GAUGE" xpath="parameter[@key='nlwp']/@value" />
      <xml-object name="pr_size" type="GAUGE" xpath="parameter[@key='pr_size']/@value" />
      <xml-object name="pr_rssize" type="GAUGE" xpath="parameter[@key='pr_rssize']/@value" />
      <xml-object name="pctmem" type="GAUGE" xpath="parameter[@key='pctmem']/@value" />
      <xml-object name="pctcpu" type="GAUGE" xpath="parameter[@key='pctcpu']/@value" />
    </xml-group>
</xml-source>
  • resource-type must match the name of the resource type created in datacollection/solaris-zones.xml
  • resource-xpath is an absolute XPath to the resource entries.
  • key-xpath is the unique resource identifier, like the ifIndex for SNMP Interface statistics. The idea is to use something that is unique won’t change from one XML to another.
  • xpath on each xml-object is a relative XPath calculated from the resource’s Xpath to the metric value.

In order to create charts for the above configuration, the graph templates should look like this:

reports=sample.zonecpu, sample.zonemem

report.sample.zonecpu.name=Sample Zone CPU
report.sample.zonecpu.columns=pctcpu
report.sample.zonecpu.type=solarisZone
report.sample.zonecpu.propertiesValues=zoneName
report.sample.zonecpu.command=--title="Zone CPU {zoneName}" \
 DEF:v1={rrd1}:pctcpu:AVERAGE \
 LINE2:v1#ff0000:"CPU" \
 GPRINT:v1:AVERAGE:"    Avg\\: %8.2lf %s" \
 GPRINT:v1:MIN:"Min\\: %8.2lf %s" \
 GPRINT:v1:MAX:"Max\\: %8.2lf %s\\n"

report.sample.zonemem.name=Sample Zone Memory
report.sample.zonemem.columns=pctmem
report.sample.zonemem.type=solarisZone
report.sample.zonemem.propertiesValues=zoneName
report.sample.zonemem.command=--title="Zone Memory {zoneName}" \
 DEF:v1={rrd1}:pctmem:AVERAGE \
 LINE2:v1#ff0000:"Memory" \
 GPRINT:v1:AVERAGE:"    Avg\\: %8.2lf %s" \
 GPRINT:v1:MIN:"Min\\: %8.2lf %s" \
 GPRINT:v1:MAX:"Max\\: %8.2lf %s\\n"

Note that the templates were added only for two of the 6 numeric metrics.

The name attribute of the xml-object tag must be used for the columns attribute on the template and for the DEF name as well.

Because these are tabular metrics, solarisZone (which is the resource type for this particular metrics) must be used for the type.

A Note on Namespaces

If your source XML uses namespaces, you will need to use some special XPath syntax to essentially ignore the namespace elements since the XPath processor is not namespace aware (at least not as of 2014-10-10). Here's an example:

<h:html xmlns:xdc="http://www.xml.com/books"
        xmlns:h="http://www.w3.org/HTML/1998/html4">
 <h:head><h:title>Book Review</h:title></h:head>
 <h:body>
  <xdc:bookreview>
   <xdc:title h:style="font-family: sans-serif;">
     XML: A Primer</xdc:title>
   <h:table>
    <h:tr align="center">
     <h:td>Author</h:td><h:td>Price</h:td>
     <h:td>Pages</h:td><h:td>Date</h:td></h:tr>
    <h:tr align="left">
     <h:td><xdc:author>Simon St. Laurent</xdc:author></h:td>
     <h:td><xdc:price>31.98</xdc:price></h:td>
     <h:td><xdc:pages>352</xdc:pages></h:td>
     <h:td><xdc:date>1998/01</xdc:date></h:td>
    </h:tr>
   </h:table>
  </xdc:bookreview>
 </h:body>
</h:html>

Say you want the h:table element to be your resource. Your XPath expression would have to be something like:

//*[local-name()='table']

If you have local name conflicts, you'll have to resort to using some preprocessing as discussed later instead.

XML Collections

First, all required collections must be included inside xml-datacollection-config.xml. The structure of each collection looks like this:

<xml-datacollection-config rrdRepository="/opt/opennms/share/rrd/snmp/"
    xmlns="http://xmlns.opennms.org/xsd/config/xml-datacollection">
    <xml-collection name="Collection 1">
        <rrd step="300">
            <rra>RRA:AVERAGE:0.5:1:2016</rra>
            <rra>RRA:AVERAGE:0.5:12:1488</rra>
            <rra>RRA:AVERAGE:0.5:288:366</rra>
            <rra>RRA:MAX:0.5:288:366</rra>
            <rra>RRA:MIN:0.5:288:366</rra>
        </rrd>
        <xml-source url="http://{ipaddr}/src_1">
        ...
        </xml-source>
        <xml-source url="http://{ipaddr}/src_n">
        ...
        </xml-source>
    </xml-collection>
    ...
    <xml-collection name="Collection n">
        <rrd step="300">
            <rra>RRA:AVERAGE:0.5:1:2016</rra>
            <rra>RRA:AVERAGE:0.5:12:1488</rra>
            <rra>RRA:AVERAGE:0.5:288:366</rra>
            <rra>RRA:MAX:0.5:288:366</rra>
            <rra>RRA:MIN:0.5:288:366</rra>
        </rrd>
        <xml-source url="http://{ipaddr}/src_1">
        ...
        </xml-source>
        <xml-source url="http://{ipaddr}/src_n">
        ...
        </xml-source>
    </xml-collection>
 </xml-datacollection-config>

Inside a collection package on collectd-configuration.xml, in a similar way used for all other collectors, the service entry must have a collection parameter that points to a specific XML collection.

Then, the content of each xml-source must be defined. There are two ways to do this:

  1. Explicitly define the xml-groups inside each xml-source tag (as explained in earlier examples)
  2. Use an external reference, in order to share the same xml-group list across many xml-sources or just simplify the content of xml-datacollection-config.xml


Let’s take the Solaris Zones as an example:

<xml-datacollection-config rrdRepository="/opt/opennms/share/rrd/snmp/"
    xmlns="http://xmlns.opennms.org/xsd/config/xml-datacollection">
    <xml-collection name="Solaris-Zones">
        <rrd step="300">
            <rra>RRA:AVERAGE:0.5:1:2016</rra>
            <rra>RRA:AVERAGE:0.5:12:1488</rra>
            <rra>RRA:AVERAGE:0.5:288:366</rra>
            <rra>RRA:MAX:0.5:288:366</rra>
            <rra>RRA:MIN:0.5:288:366</rra>
        </rrd>
        <xml-source url="http://{ipaddr}/zone-stats">
            <import-groups>xml-datacollection/solaris-zones.xml</import-groups>
        </xml-source>
    </xml-collection>
 </xml-datacollection-config>

That means, all the XML groups defined inside the external file named xml-datacollection/solaris-zones.xml will be added to this xml-source.

The content of this external file is:

<xml-groups>
    <xml-group name="solaris-zone-stats" resource-type="solarisZone"
                resource-xpath="/zones/zone"
                key-xpath="@name">
      <xml-object name="zoneName" type="string" xpath="@name" />
      <xml-object name="nproc" type="GAUGE" xpath="parameter[@key='nproc']/@value" />
      <xml-object name="nlwp" type="GAUGE" xpath="parameter[@key='nlwp']/@value" />
      <xml-object name="pr_size" type="GAUGE" xpath="parameter[@key='pr_size']/@value" />
      <xml-object name="pr_rssize" type="GAUGE" xpath="parameter[@key='pr_rssize']/@value" />
      <xml-object name="pctmem" type="GAUGE" xpath="parameter[@key='pctmem']/@value" />
      <xml-object name="pctcpu" type="GAUGE" xpath="parameter[@key='pctcpu']/@value" />
    </xml-group>
</xml-groups>

XML Sources URLs

There are several ways to define the URL used to retrieve the XML Data. One important placeholder, {ipaddr}, has already been introduced, but there are other placeholders that can be used inside the URL like any asset record and the core elements of the node like: nodeId, nodeLabel, foreignSource, foreignId.

The idea is to be able to parameterize the URL as much as possible and put nodes' specific information in their asset records or any core elements.

The XML can be retrieved using basically any protocol, but two of them have been modified in order to use basic authentication. They are HTTP and SFTP, here are some examples using placeholders:

http://{username}:{password}@{ipaddr}/statistics/data.htm?serverId={foreignId}

Where {username} and {password} are asset records and the runtime URL should look like this:

http://admin:admin@192.168.0.1/statistics/data.htm?serverId=1234

On a similar way the following URL is also valid:

sftp://{username}:{password}@{ipaddr}/statistics/data.xml

3GPP is an example of a custom URL handling, because the target file is different on each collection interval, so that's why the sample configuration files uses a slightly different sftp protocol named sftp.3gpp.

In addition you can use cron to collect files from somewhere else and output to a local file with the remote node's IP address in the file path.

file:///var/tmp/customstats/server-{ipaddr}.xml

XML Collector configuration

This is an example of how to create a collection package with a service that will use the XML collector. This must be included inside collectd-configuration.xml:

<package name="XML Collection">
    <filter>IPADDR != '0.0.0.0'</filter>
    <include-range begin="1.1.1.1" end="254.254.254.254"/>
    <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
    <service name="SolarisZones" interval="300000" user-defined="false" status="on">
        <parameter key="collection" value="Solaris-Zones"/>
    </service>
</package>

<collector service="SolarisZones" class-name="org.opennms.protocols.xml.collector.XmlCollector"/>

There is a service named SolarisZones. The XML Collector will be associated with this service and applied to all nodes according with the filters defined on the package “XML Collection” using the collection “Solaris Zones” (the collection parameter must match the name of any xml-collection defined on xml-datacollection-config.xml).

Graph Templates

On a similar way used for all other collectors, a set of graph templates should be defined either by directly modifying snmp-graph.properties or by adding them on an external file, for example, snmp-graph.properties.d/xml-graphs.properties.

Advanced Features

It is important to know that the XML collector for tabular metrics can use a custom StorageStrategy and/or PersistSelectorStrategy, defined with the resource type, to customize the way you store the data and which resources should be taken in consideration (and skip unwanted data).

For the Solaris Zone example, suppose that the globa zone must be omitted. This case requires the usage of the PersistRegexSelectorStrategy like follows:

<resourceType name="solarisZone" label="Solaris Zone” resourceLabel="Zone ${zoneName}" >
    <persistenceSelectorStrategy class="org.opennms.netmgt.collectd.PersistRegexSelectorStrategy">
      <parameter key="match-expression" value="#zoneName != 'global'" />
    </persistenceSelectorStrategy>
    <storageStrategy class="org.opennms.netmgt.dao.support.IndexStorageStrategy"/>
</resourceType>

The match-expression is a valid SPEL (i.e., a valid expression written in Spring Expression Language). All attributes defined as String inside the xml-group can be used inside the expressions.

Examples

Reading XML status from HP Blade Enclosure

Reading XML status from http://xml.buienradar.nl

Use a Groovlet to make your own custom data collector

New in 1.12

Warning.png IMPORTANT

The following features are available only from OpenNMS 1.12 onward.


The documentation of the following features is a work in progress, so it still needs work ;)

Handle HTTPS

Now HTTP is supported as a valid protocol on the XmlSource's URL. If you can skip the SSL validation, you can configure the XML Source like the following:

  <xml-source url="https://{ipaddr}/secure-get-example">
    <request method="GET">
        <parameter name="timeout" value="6000"/>
        <parameter name="retries" value="2"/>
        <parameter name="disable-ssl-verification" value="true"/>
    </request>
     <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>

Handle GET Requests

Besides standard GET requests, you can now customize header and control some parameters like retries and timeout, for example:

  <xml-source url="http://{ipaddr}/get-example">
    <request method="GET">
        <parameter name="timeout" value="6000"/>
        <parameter name="retries" value="2"/>
        <header name="User-Agent" value="{comment}"/>
    </request>
     <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>

In the above example, there are two fixed parameters and one dynamic header. It is dynamic because the value is retrieved from the asset field named comment from the node.


Parsing not well formatted HTML with XPath

HTML is not required to be a strict XML in terms of the syntax. Most browsers are going to be able to process not well formatted HTML, for example:

   <p>This is one paragraph<br><p>This is another paragraph<br>

The above HTML is not valid in terms of XML syntax, it should be written like the following to be valid

   <p>This is one paragraph</p><br/><p>This is another paragraph</p><br/>

BTW, both definitions are equivalent.

XPath expects a well formatted XML, so if you want to process an HTML with XPath and the XML Collector, the HTML must be well formatted. If you are not sure if the HTML is well formatted or not, you can configure your collector like the following:

  <xml-source url="http://{ipaddr}:8080/status.html">
    <request>
      <parameter name="pre-parse-html" value="true"/>
    </request>
    <xml-group name="count-stats" resource-type="node" resource-xpath="/html/body">
      <xml-object name="count" type="GAUGE" xpath="p[contains(.,'Document Count:')]" />
    </xml-group>
   </xml-source>

This will be able to process the following HTML:

  <html>
    <head>
      <title>collector test</title>
    </head>
    <body>
      <p>Document Count: 5<br>
    </body>
  </html>

Keep in mind that any non-numeric characters will be removed from the "selected text", so "Document Count: 5" will be parsed as just "5". For floating point numbers, it should use the period character as a separator, for example: "45.56".

That means 45,56 is not supported, actually it will be parsed as 4556; so something like 4,563.33 will be 4563.33

Configuring HTML parsing with CSS Selectors instead of XPath

Dealing with HTML with XPath is probably not the best solution in modern Web Applications. It is much better to use CSS selectors instead, for example, let's take the following HTML as an example:

  <html>
  <head>
    <title>Sample Stock Market Data</title>
  <body>
  <div id="cnnBody">
    <div class="cnnBody_Left wsodContent">
      <div id="wsod_marketsOverview">
        <div class="clearfix">
          <div class="clearfix wsod_StockHeader" style="margin-bottom:10px;">
            <div class="wsod_fLeft" style="width:215px;">
              <h2>Markets Overview</h2>
            </div>
            <div class="wsod_fRight" id="wsod_tickerRoll">
              <li class="tickerNasdaq">
                <a href="/data/markets/nasdaq" class="wsod_symbol">Nasdaq</a>
                <span class="quoteChange" stream="change_579435"><span class="posChange">+57.54</span></span>
                <div class="clearFloat"><br clear="all" /></div>
                <div class="bannerQuote">
                  <span stream="last_579435"><span class="posLast">3,578.30</span></span>
                </div>
                <span class="quotePctChange" stream="changePct_579435"><span class="posChangePct">+1.63%</span</span>
              </li>
            </div>
          </div>
        </div>
      </div>
    </div>
  </div>
  </body>
  </html>

Now, configure the XML Collector in collectd-configuration.xml like the following:

  <package name="Sample CSS Selector">
    <filter>IPADDR != '0.0.0.0'</filter>
    <include-range begin="1.1.1.1" end="254.254.254.254"/>
    <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
    <service name="HTTP_CSS" interval="300000" user-defined="false" status="on">
        <parameter key="handler-class" value="org.opennms.protocols.http.collector.HttpCollectionHandler"/>
        <parameter key="collection" value="http-css"/>
    </service>
  </package>

Then configure the XML source like:

  <xml-collection name="http-css">
    <rrd step="300">
      <rra>RRA:AVERAGE:0.5:1:8928</rra>
      <rra>RRA:AVERAGE:0.5:12:8784</rra>
      <rra>RRA:MIN:0.5:12:8784</rra>
      <rra>RRA:MAX:0.5:12:8784</rra>
    </rrd>
    <xml-source url="http://{ipaddr}:8080/market.html">
       <xml-group name="market" resource-type="node" resource-xpath="div#cnnBody">
         <xml-object name="nasdaq" type="GAUGE" xpath="li.tickerNasdaq span.posLast" />
       </xml-group>
    </xml-source>
  </xml-collection>

Pre-process XML data with XSLT 1.0

For some use cases, the raw XML data could be complicated to parse or it is just too big. Now you can create a XSLT 1.0 file to applied over the raw XML data prior applying the XML-Source to facilitate the configuration (or reduce its complexity).

Let's say the XSLT lives on /opt/opennms/etc/pre-process.xslt, the configuration to use that file to pre-process the XML data should be:

  <xml-source url="http://{ipaddr}/get-example">
    <request method="GET">
        <parameter name="xslt-source-file" value="/opt/opennms/etc/pre-process.xslt"/>
    </request>
     <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>

The above example uses the GET method, but it also works with POST as well.

JSON Collector

There are occasions where the data required to be collected is not exposed as a XML document. JSON is becoming very popular these days, and it is more common than XML specially for ReST interfaces.

Now, thanks to Apache Commons JXPath, it is possible to declare the XML-Source configuration using XPath to parse JSON documents.

Let's say the example for the Solaris Zones explained above (once again, that is not enabled on Solaris, it is just an example) returns a JSON data instead of XML. The configuration for the XML-Source remains the same, assuming the sample JSON data looks like the following:

{
   "zones": {
     "zone" : {
       "id" : 0,
       "name" : "global",
       "timestamp" : 1299258742,
       "parameter" : [
         { "key" : "nproc", "value" : "245" },
         { "key" : "nlwp", "value" : "1455" },
         { "key" : "pr_size", "value" : "2646864" },
         { "key" : "pr_rssize", "value" : "1851072" },
         { "key" : "pctmem", "value" : "0.7" },
         { "key" : "pctcpu", "value" : "0.24" }
       ]
     },
     "zone" : {
       "id" : 871,
       "name" : "zone1",
       "timestamp" : 1299258742,
       "parameter" : [
         { "key" : "nproc", "value" : "24" },
         { "key" : "nlwp", "value" : "328" },
         { "key" : "pr_size", "value" : "1671128" },
         { "key" : "pr_rssize", "value" : "1193240" },
         { "key" : "pctmem", "value" : "0.4" },
         { "key" : "pctcpu", "value" : "0.07" }
       ]
     }
   }
}

The only change required to use the JSON Collector is using a special handler on collectd-configuration.xml:

  <package name="Solaris JSON Collector">
    <filter>IPADDR != '0.0.0.0'</filter>
    <include-range begin="1.1.1.1" end="254.254.254.254"/>
    <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
    <service name="HTTP_JSON" interval="300000" user-defined="false" status="on">
        <parameter key="handler-class" value="org.opennms.protocols.json.collector.DefaultJsonCollectionHandler"/>
        <parameter key="collection" value="Solaris"/>
    </service>
  </package>

New in 1.13 and 14.0

Warning.png IMPORTANT

The following features are available only from OpenNMS 1.13.2 onward.


The documentation of the following features is a work in progress, so it still needs work ;)

Extremely fast XML Parser

A new set of XML Collection Handlers have been implemented a GPL Library called VTD-XML.

This library can be at least 60 times faster than the DOM Parser in Java and even other parsers.

These handlers could be very useful when parsing very large XMLs.

In order to use these handlers you need an additional package called opennms-plugin-collector-vtdxml-handler

Configuration examples for standard XML processing

 <package name="XML Collection">
     <filter>IPADDR != '0.0.0.0'</filter>
     <include-range begin="1.1.1.1" end="254.254.254.254"/>
     <service name="VTD_XML" interval="300000" user-defined="false"
   status="on">
       <parameter key="collection" value="vtd-xml-collection"/>
       <parameter key="handler-class"
   value="org.opennms.protocols.xml.vtdxml.DefaultVTDXmlCollectionHandler"/>
     </service>
   </package>
   
   <collector service="VTD_XML"
   class-name="org.opennms.protocols.xml.collector.XmlCollector"/>

Configuration examples for processing 3GPP over SFTP

 <package name="XML Collection">
     <filter>IPADDR != '0.0.0.0'</filter>
     <include-range begin="1.1.1.1" end="254.254.254.254"/>
     <service name="3GPP" interval="300000" user-defined="false"
   status="on">
       <parameter key="collection" value="vtd-xml-collection"/>
       <parameter key="handler-class"
   value="org.opennms.protocols.xml.vtdxml.Sftp3gppVTDXmlCollectionHandler"/>
     </service>
   </package>
   
   <collector service="3GPP"
   class-name="org.opennms.protocols.xml.collector.XmlCollector"/>


New in 16.0.0

Warning.png IMPORTANT

Because of NMS-7650 this feature is only available from OpenNMS 16 onward.


Handle POST Requests

Besides standard GET requests, you can now send customized POST requests and process the output.

application/form-urlencoded

  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/x-www-form-urlencoded'><![CDATA[
        <form-fields>
           <form-field name='firstName'>Alejandro</form-field>
           <form-field name='lastName'>Galue</form-field>
         </form-fields>
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>

The XML format of the form is fixed and should always be configured like that. Pay special attention to the CDATA block. This is required if you're going to put any XML inside the content tag.

Of course, you can make the content dynamic, for example:

  <content type='application/x-www-form-urlencoded'><![CDATA[
    <form-fields>
      <form-field name='nodeLabel'>{nodeLabel}</form-field>
      <form-field name='foreignId'>{foreignId}</form-field>
    </form-fields>
  ]]></content>

Both fields are generated dynamically from the standard node's parameters.

application/xml

  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/xml'><![CDATA[
        <person>
          <firstName>Alejandro</firstName>
          <lastName>Galue</lastName>
        </person>
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>

Similar with the form fields, the content can by dynamic.

application/json

  <xml-source url="http://{ipaddr}/post-example">
    <request method="POST">
      <content type='application/json'><![CDATA[
         {
           person: {
             firstName: 'Alejandro',
             lastName: 'Galue'
           }
         }
      ]]></content>
    </request>
    <import-groups>xml-datacollection/my-groups.xml</import-groups>
  </xml-source>

Similar with the form fields, the content can by dynamic, but it is extremely important to pay attention to the indentation of the "{" and "}" in the JSON format to avoid confusions with the placeholders, for example:

  <content type='application/json'><![CDATA[
    {
      node: {
        nodeLabel: '{nodeLabel}',
        foreignId: '{foreignId}'
      }
    }
  ]]></content>