Creating a xmp collector

From OpenNMS
Jump to navigation Jump to search

Overview

I've created a collector for a new network management protocol (XMP) I'm developing. XMP is a connection-oriented management protocol that uses XML for data definition and transfer over a TCP connection encapsulated within SSL. It uses a relatively simple data model based on that used by the Internet Management Framework (SNMP).

I glued XMP into OpenNMS in order to take advantage of data collection and trending, events, etc. Developing a collector is conceptually fairly simple but obfuscated by a plethora of obstacles including object-oriented concepts and software engineering. OpenNMS uses a data-model tightly integrated with the SNMP. Fortunately, XMP uses a similar data model so it integrates fairly easily.

The process, for creating a collector is:

  1. create entries into the various OpenNMS config files that collectd and capsd use.
  2. create XSDs for your protocol-specific config files. You'll want a protocol config file (e.g. xmp-config.xml) and a data collection config file (xmp-datacollection.xml)
  3. automatically create XML parsers for your config files via Castor
  4. write your factories for parsing the protocol config and data-collection config files.
  5. write your collector code
  6. re-build OpenNMS
  7. Jar up your collector code and drop into OpenNMS
  8. Oh yeah, test!

Config Files

  • Edit existing OpenNMS config files to add support for your new collector
    • capsd-configuration.xml - I added an entry in this file so that the capabilities daemon would probe systems for XMP support. My entry, for now, used the existing TCP plugin to test for the availability of my protocol.
    <protocol-plugin protocol="XMP" class-name="org.opennms.netmgt.capsd.plugins.TcpPlugin" scan="on" user-defined="false">
        <property key="banner" value="*" />
        <property key="port" value="5270" />
        <property key="timeout" value="3000" />
        <property key="retry" value="1" />
    </protocol-plugin>
    • collectd-configuration.xml - I added entries in this file so that the collect daemon would know how to glue in my collector. One key parameter is collection whose value defines which set of MIB variables should be collected. The value default in this configuration file matches up with an xmp-collection tag in the xmp-datacollection.xml config file.
    <service name="XMP" interval="300000" user-defined="false" status="on"> 
           <parameter key="collection" value="default"/>
    </service>
    <collector service="XMP" class-name="org.opennms.netmgt.protocols.xmp.collector.XmpCollector"/>
  • poller-configuration.xml - I added entries in this file so that the XMP service would be monitored to ensure that it is up and running. I used the simple TCPMonitor.
    <service name="XMP" interval="300000" user-defined="false" status="on">
      <parameter key="retry" value="1" />
      <parameter key="port" value="5270" />
      <parameter key="timeout" value="2" />
    </service>
  <monitor service="XMP" class-name="org.opennms.netmgt.poller.monitors.TcpMonitor" />
  • Add/Create protocol-specific config files
    • xmp-config.xml
    • xmp-datacollection-config.xml
  • The protocol-specific config files go into the etc runtime directory

XSDs XML Schema Definitions for Your Config Files

Create XSDs for your protocol-specific config files. OpenNMS uses Castor for automatically creating java classes to parse config files.

  • xmp-config.xsd - this config file gives protocol specific parameters including port, retries, timeout (milliseconds), and which user profile to use (called authenUser). For the first release of our collector, I kept it fairly simple and patterned it after the SNMP configuration file (snmp-config.xml). The SNMP configuration file, however, allows per-node and per-IP address range configuration while the XMP configuration file does not.
  <?xml version="1.0"?>
  <xmp-config port="5270" retry="1" timeout="2000" authenUser="public">
  </xmp-config>
  • xmp-datacollection.xsd - this config file essentially specifies which MIB objects the data collector should collect. It can be fairly complex due to the immense diversity of SNMP agents, MIBs, and implementations. For the first release of our collector, we kept it fairly simple. The data collection config file should specify RRD parameters, and groups of MIB objects, and system definitions (which groups to collect for which agents.)

Creating XSD Parsers

OpenNMS uses Castor to automatically generate Java classes to parse the XSD files we created in the previous section.

  • Place XSDs in /path-to-source/opennms-config/src/main/castor/
  • Build OpenNMS by running build from the top-level source directory via
./compile.pl -Dopennms.home=/opt/opennms
./assemble.pl
  • Resulting in several automatically generated Java classes (plus some internal classes). The resulting java classes are placed in /path/to/opennms/opennms-config/target/generated-sources/castor/

For the xmp-config.xml file:

  • XmpConfig.java/class - top level element for the xmp-config.xml config file
  • XmpConfigDescriptor.java/class - I have no idea what/why this class exists.

For the xmp-datacollection.xml:

  • XmpDatacollectionConfig.java/class - Top-level element for the xmp-datacollection-config.xml configuration file.
  • XmpCollection.java/class - a grouping of XMP related RRD parms, MIB object groups and sysoid based system definitions
  • XmpCollectionDescriptor.java/class - I have no idea what/why this class is generated.
  • XmpDatacollectionConfigDescriptor.java/class - I have no idea what/why this class is generated.

Write your Config File Factories

You need to write a factory to kick off parsing/instantiation of your protocol config file (xmp-config.xml) and your data collection config file (xmp-datacollection.xml).

I wrote two classes:

  1. XmpConfigFactory.java that handles the xmp-config.xml
  2. XmpCollectionFactory.java that handles the xmp-datacollection.xml

Collector Code

You must then develop your collector code in Java. Colectors implement the ServiceCollector interface. My collector consists of the following files/classes:

  • XmpCollector.java - main class used by OpenNMS to collect data via XMP. In my implementation, this class actually establishes XMP sessions with XMP agents and queries them for MIB objects. The collector is invoked once per agent. One collector instance exists inside OpenNMS.
  • XmpCollectionSet.java - A CollectionSet holds the results of an actual XMP query or queries. However, the results are not stored directly in the CollectionSet but are encapsulated in a series of classes. A CollectionSet contains CollectionResource objects which each contain a set of Attribute groups. AttributeGroups correspond to those in the data collection config file.
  • XmpCollection.java - class that eats our data collection config file and returns a set of MIB objects that to be collected. Right now, XMP is monolithic and implemented by one provider so we do not have to worry about different XMP agents implementing different MIBs and we dont have to worry about the nuances of different MIB implementations. In SNMP, there is an additional step that a default collection might contain lots of different groups but different systems support different groups of MIB objects.
  • XmpCollectionResource.java - A CollectionSet contains a set of CollectionResources. These CollectionResources, in turn, contain AttributeGroups which contain actual CollectionAttributes or XmpVars. Attribute groups closely mirror the data collection config file. A single collection resource can be used to contain all the scalar MIB objects that are collected. However, each row in a particular table should be contained in its own collection resource object because each row is stored in a separate RRD repository.
  • XmpCollectionAttribute.java - is an actual data point collected via XMP; what this means in English is that we've finally arrived at an actual Xmp variable -- something that has a MIB object name, type, and value.
  • XmpCollectionAttributeType.java - is an object that essentially specifies the data type of the collection attribute. Some data collectors allow administrators to override the mib object data type in the data collection config file. That type is returned by this object. ONMS has a set of well known (aka semantically understood) set of types based on the SNMP SMI.

Build It

Copy your code into the double-secret OpenNMS build locations.

  • XmpCollector.java -> /path/to/src/opennms-services/src/main/java/org/opennms/netmgt/collectd
  • XmpCollectionSet.java -> /path/to/src/opennms-services/src/main/java/org/opennms/netmgt/collectd
  • XmpCollectionAttribute.java -> /path/to/src/opennms-services/src/main/java/org/opennms/netmgt/collectd
  • XmpCollectionResource.java -> /path/to/src/opennms-services/src/main/java/org/opennms/netmgt/collectd
  • XmpCollectionAttributeType.java -> /path/to/src/opennms-services/src/main/java/org/opennms/netmgt/collectd
  • XmpCollectionFactory.java /path/to/src/opennms-services/src/main/java/org/opennms/netmgt/config
  • XmpConfigFactory.java /path/to/src/opennms-services/src/main/java/org/opennms/netmgt/config

I also needed to copy my generic XMP protocol code into this directory. Long-term, I need to find out how to better integrate Xmp.jar package into OpenNMS.

Re-build OpenNMS.

Seeing XMP Data

Seeing is believing. A picture is worth a thousand words.

One must edit snmp-graph.properties in order to see your data using the various RRD graphing tools built into ONMS. How this file is edited will determine how the graphs appear in the WebUI. Errors in this file can cause the WebUI to quit working (worse case) or not display the graphs properly.

Here is an example of a graph command that will display the number of processes and threads (scalar data or node data), over time, for the target system. We name our graphs xmp.graphname; each graph states a name, columns (MIB object data to use for the graph), type, and command. Type corresponds to nodeSnmp for scalar data while command is the actual JRobin command that gets invoked to generate the graph. Documentation for the fields can be found on the RRDtool website. In the graph below, numProcesses and numThreads correspond to MIB objects that are collected using XMP.

report.xmp.procs.name=Processes/Threads
report.xmp.procs.columns=numProcesses, numThreads
report.xmp.procs.type=nodeSnmp
report.xmp.procs.command=--title="Processes/Threads" \
 DEF:p={rrd1}:numProcesses:AVERAGE \
 DEF:t={rrd2}:numThreads:AVERAGE \
 LINE2:p#0000ff:"Procs" \
 GPRINT:p:AVERAGE:"Avg \\: %10.2lf %s" \
 GPRINT:p:MIN:"Min \\: %10.2lf %s" \
 GPRINT:p:MAX:"Max \\: %10.2lf %s\\n" \
 LINE2:t#00ff00:"Thrds" \
 GPRINT:t:AVERAGE:"Avg \\: %10.2lf %s" \
 GPRINT:t:MIN:"Min \\: %10.2lf %s" \
 GPRINT:t:MAX:"Max \\: %10.2lf %s\\n"

Here is an example of a graph command that will graph filesystem utilization (tabular data), over time, for the target system. nodeTtype in this case is xmpFilesys which corresponds to a resourceType statement that I placed in the datacollection-config.xml file.

report.xmp.filesys.name=Filesystem Utilization
report.xmp.filesys.columns=usedBlocks,freeBlocks,availBlocks
report.xmp.filesys.type=xmpFilesys
report.xmp.filesys.command=--title="Filesystem Utilization" \
 DEF:u={rrd1}:usedBlocks:AVERAGE \
 DEF:f={rrd2}:freeBlocks:AVERAGE \
 DEF:a={rrd3}:availBlocks:AVERAGE \
 LINE2:u#0000ff:"Used Blocks" \
 GPRINT:u:AVERAGE:"Avg \\: %10.2lf %s" \
 GPRINT:u:MIN:"Min \\: %10.2lf %s" \
 GPRINT:u:MAX:"Max \\: %10.2lf %s\\n" \
 LINE2:f#00ff00:"Free Blocks" \
 GPRINT:f:AVERAGE:"Avg \\: %10.2lf %s" \
 GPRINT:f:MIN:"Min \\: %10.2lf %s" \
 GPRINT:f:MAX:"Max \\: %10.2lf %s\\n" \
 LINE2:a#ff0000:"Avail Blocks" \
 GPRINT:a:AVERAGE:"Avg \\: %10.2lf %s" \
 GPRINT:a:MIN:"Min \\: %10.2lf %s" \
 GPRINT:a:MAX:"Max \\: %10.2lf %s\\n"

The following resourceType definition, declared in datacollection-config.xml serves several purposes.

  1. The name xmpFilesys is used in snmp-graph.properties to tell ONMS where to find our data. Our collection resource class (XmpCollectionResource) uses this resourceType declaration to know where to store RRD files.
  2. It tells the WebUI how display the dialog for selecting which instances to graph.
  3. And, it tells ONMS how to persist the data.
     <resourceType name="xmpFilesys" label="Filesystems" resourceLabel="${index}" >
        <persistenceSelectorStrategy class="org.opennms.netmgt.collectd.PersistAllSelectorStrategy"/>
        <storageStrategy class="org.opennms.netmgt.dao.support.IndexStorageStrategy"/>
     </resourceType>

Once the graphs are defined and configured, you need to glue them into the reports clause of the snmp-graph.properties file.

reports=mib2.HCbits, mib2.bits, mib2.percentdiscards, mib2.percenterrors, \
blah blah blah blah,\
xmp.filesys,xmp.procs

Notes

The relationship between what you want collected (the xmp-datacollection-config.xml/xsd) and what you have actually collected (CollectionSet) is conceptually very simple but in practice is quite obtuse due to all the objected-oriented concepts and software engineering that get in the way. In the table below, I have paired up the java classes that correspond with each other. All are either generated by Castor or written by me with the exception of AttributeGroup. I am unaware of a parallel to the Groups construct that exists in xmp-datacollection-config.xml.

What we Collect (Java) What We Want to Collect (XSD/XML) What Is Generated (Java)
XmpCollectionResource.java <xmp-collection> XmpCollection.java
<groups> Groups.java
AttributeGroup.java <group> Group.java
XmpCollectionAttribute.java <MibOjb> MibObj.java, MibOjbDescriptor.java


  • Collector is instantiated only once.
  • Initialize method called and then getRrdRepository called for each agent.
  • Be sure to check whether a string is null or simply one of length 0; strings parsed from the various config files will usually be length=0 instead of null.
  • The collector may be multi-threaded; that is, collectd may invoke your collect() method several times in parallel via multiple threads. Do not block.
  • Should we use a different subdir under share/rrd ? That is, what if there is both snmp and XMP? Should we use share/rrd/xmp ? Right now, no because ONMS really overloads SNMP as being "performance" data regardless of where/how it is collected. Most likely, the path "rrd/snmp" is too hardcoded into java classes.
  • Dont know if tabular data is being properly handled and stored need to verify the node type and attribute group type; it is not we should have sub-dirs under each node for tabular data
    • We needed to override getResourceDir() in XmpCollectionResource so that we could properly specify the right sub-dir under
    • We are using the algorithm of tableName-instance for sub-dirs under a node's data directory
  • For tabular data, we use instance for each row; if we ask for instance/key 'xmpd', we will get back processes with xmpd as strstr() in the process table. Returned key is "pid:command-line" which will not be good since pid may change.
    • What should we use for RRD file name and instance so as to be unique but also be consistent across changes in underlying keys?
      • What we are trying out is thus: if datacollection-config.xml has a targetInstance, use that for naming the resource; if not, use the returned key/instance from the query as the instance for the collection resource
  • Relationship between datacollection-config.xml resourceTypes, WebUI, etc.
    • datacollection-config.xml has xml nodes called resourceTypes; the WebUI uses these types to display possible graphs. Our group nodes in xmp-datacollection-config.xml now have a parameter called resourceType which, for tabular objects, refers to a resourceType in datacollection-config.xml.
    • resourceType, when non-null, is used for RRD file naming; RRD filenames are now
      • share/rrd/snmp/nodeId/resourceType/instance/rrdFile.jrb for tabular objects
      • share/rrd/snmp/nodeID/rrdFile.jrb for scalar objects
    • instances are screened for characters slash, back-slash, and colon as they will mess up file names and graphing functions. An undescore character is substituted for them.

Errors

Some fun errors along the way and fixes.

  • Collector daemon fails to load my protocol collector.
WARN  [Main] Collectd: instantiateCollectors: Failed to load collector XmpCollector for service XMP:
java.lang.IllegalAccessException: Class org.opennms.netmgt.collectd.Collectd can not access a member of class 
XmpCollector with modifiers ""

Ooops! I did not explicitly declare my constructor public! Remember, I'm a programmer, not a software engineer.

  • Data collection failed!
INFO  [CollectdScheduler-50 Pool-fiber8] OneToOnePersister: Persisting data for resource XmpCollectionResource@56a557
DEBUG [CollectdScheduler-50 Pool-fiber8] CollectableService: sendEvent: Sent event uei.opennms.org/nodes/dataCollectionFailed
for 27/192.24.251.126/XMP
  • Nothing running on port 8443: Forgot to copy over my opennms.properties config file from my previous installation to my built-from-source installation.
  • I was failing to load my own keystore despite including it in the various JAR files. Fixed by using whatever class loader my thread inherits rather than using the system class loader.
  • Use getHostAddress() method of class InetAddress to get a string representation of agent's IP address. Otherwise, toString returns something of form hostname/ipaddress which cannot be used with socket classes.
  • Collector collect() finishes OK but data does not show up in RRD files. A host of problems can cause this. Check to make sure your CollectionResource class is returning the appropriate RRD filename (if tabular). If scalar, your collection resource need not override GetResourceDir().

See Also