Monitoring the Java VM

From OpenNMS
Jump to navigation Jump to search


Contents

This is a general description how to monitor metrics and collect data from a Java 5 Virtual Machine. With Java 5 there are two general ways to manage a running VM so:

  • The built-in SNMP agent
  • The built-in JMX interface

These two techniques can be applied to any Java application but we will focus on the JBoss Application Server and on Tomcat here. There are a few documents that already focus on one special use-case (Tomcat 5.5 JMX How-To, JMX_configuration) but this should be a general purpose description.

Enabling the JVM Monitoring Agent

The desired Monitoring Agent (either SNMP or JMX) is enabled by setting system properties for the JVM. Actually we can activate both agents at the same time.

Enabling the JMX Agent

To enable the JMX Agent we need to set the following properties. As we most likely want to remote monitor the JVM we only focus on this scenario here. Monitoring a production environment locally is a bad idea as the monitoring application itself sometimes uses quite some memory and thus influences the running application.

For a simple setup we are using the following properties and pass them on the command-line for the Java Application:

# Enable the jmx agent remotely on port 9003
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=9003"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false"

Starting the application with these parameters we can connect with a JMX Management tool (e.g. jconsole or the OpenNMS jsr160 Plugin) and monitor or get/set properties of the VM.

Note that per default, when enabling JMX for remote management authentication is required and it's encrypted through SSL. For the ease of use we've disabled security in this scenario.

For a production use I'd recommend to set these two parameters as well to enable simple password/role access:

JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.password.file=jmxremote.password"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.access.file=jmxremote.access"

More about securing the access can be found here.

Note that, your /etc/hosts file might prevent the jmx agent from working properly if, for example, you have a line that resolves your openNMS server's hostname to a loopback IP address (127.0.0.0/8).

Enabling the SNMP Agent

Enabling the SNMP Agent is as simple as above, by setting the following properties:

# Enable the snmp agent of the JavaVM on port 1610
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.snmp.port=1610"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.snmp.acl.file=ACLFilePath"

Additionally we must provide an ACL File that defines the permissions (SNMP communities) for accessing the Agent. If you don't set the com.sun.management.snmp.acl.file property a global file is taken from the JRE home directory JAVA_HOME/lib/management/snmp.acl.

A sample ACL that allows read-only access from our management station (manager) would look like this:

acl = {
       {
         communities = public, private
         access = read-only
         managers = manager
       }
     }
# Traps are sent to localhost only
trap = {
         {
           trap-community = public
           hosts = manager
         }
       }

JBoss and Tomcat

Applying the above steps to Tomcat or JBoss is a straight forward Process. Just set these Properties in your Tomcat's start script (or the tomcat user properties) or in the file run.conf in JBoss' bin directory.

Integrating JBoss MBeans in the VM's MbeanServer

If we are using the JMX agent it is possible to register JBoss' MBeans with the platform MBeanServer and providing more information about the applications running in JBoss. This however requires at least JBoss 4.0.3, although there is a Bug at least in 4.2.2 which results in the JBoss not starting with this setting and throwing an Exception. In this case you can still collect data with the JBossCollector.

To register the JBoss MBeans in the platform MBeanServer we need to add these properties in the JBoss' run.conf

JAVA_OPTS="$JAVA_OPTS -Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl"
JAVA_OPTS="$JAVA_OPTS -Djboss.platform.mbeanserver"

Testing the Agents

All steps described above are prerequisites necessary for providing information for management applications like OpenNMS. Before we are going on with configuring OpenNMS we can test the functionality of the JMX or SNMP Agent.

SNMP Agent

We do a simple snmpwalk on the MIB tree exported by the JVM (the Java VM Management MIB can be downloaded here). We query the MIB tree at .iso.org.dod.internet.private.enterprises.sun.products.jmgt.standard.jsr163.jvmMgtMIB or in numbers .1.3.6.1.4.1.42.2.145.3.163.1.1.4.2.0

snmpwalk -c public -v 2c 127.0.0.1:1610 .1.3.6.1.4.1.42.2.145.3.163.1.1.4.2.0

Of course you can use your favourite SNMP Browser for that as well.

JMX Agent

For querying the JMX Agent we will use jconsole, a graphical Tool that comes with the JDK. To connect to the Agent against the configuration above we just start jconsole like this:

jconsole <hostname or ip of jboss>:9003

If you get an error from JConsole

Exception in thread "JConsole.addHost"
  java.lang.IllegalArgumentException: java.lang:type=Runtime not found in the connection.
...

You might have run into this problem. I didn't test the setup with jconsole first and it took me a few hours to find out why OpenNMS wasn't working then. In this case just go to the jmx-console of your JBoss istallation and inspect the MBean java.lang:type=Runtime once. After that connection with jconsole should work fine.

Configuring OpenNMS

Now that we have configured the JavaVM that is running our favourite application it's time to get some of it's provided information into OpenNMS. This is a rather straight-forward process and doesn't differ much from other sources - at least for SNMP. Querying JMX is a bit more complex as there are actually two ways to do it in OpenNMS (I got only one working).

SNMP

JMX

As I mentioned above there are two ways to get data from a JMX agent. There is the JMX or JBoss4 Plugin and the JSR160 Plugin. I tried both but didn't succeed with the JBoss4 Plugin. So I'm using the JSR160 Plugin here.

Configuring capsd

Make sure there is a protocol-plugin configuration in your capsd-configuration.xml so capsd can discover the service:

<protocol-plugin protocol="JVM" class-name="org.opennms.netmgt.capsd.plugins.Jsr160Plugin"
  scan="on" user-defined="false">
  <property key="port" value="9003"/>
  <property key="type" value="default"/>
  <property key="factory" value="PASSWORD-CLEAR"/>
  <property key="protocol" value="rmi"/>
  <property key="urlPath" value="/jmxrmi"/>
  <property key="retry" value="2"/>
  <property key="timeout" value="2000"/>
</protocol-plugin>

The username and password parameters are as configured above. Here we just use non-authenticated access.

Configuring collectd and jmx-datacollection

I've also added a new package definition to collectd.configuration.xml:

 <package name="jboss">
   <filter>IPADDR IPLIKE *.*.*.*</filter>
   <include-range begin="192.168.1.101" end="192.168.1.106"/>
   <service name="JVM" interval="300000" user-defined="false" status="on">
     <parameter key="port" value="9003"/>
     <parameter key="retry" value="2"/>
     <parameter key="timeout" value="3000"/>
     <!--
      <property key="username" value="<username>"/>
      <property key="password" value="<password>"/>
     -->
     <parameter key="protocol" value="rmi"/>
     <parameter key="urlPath" value="/jmxrmi"/>
     <parameter key="ds-name" value="jmx"/>
     <parameter key="friendly-name" value="jboss"/>
     <!-- This must match the collection name in the jmx-datacollection.xml that defines the set of mbeans you want -->
     <parameter key="collection" value="jsr160"/>
   </service>
 </package>
 <!-- this links the collector to your service defined in the jboss package. 
 <collector service="JVM"          class-name="org.opennms.netmgt.collectd.Jsr160Collector"/>

In jmx-datacollection.xml we define the MBeans we want to collect data from. There are many MBeans exposed by the JMX Agent. Depending on your application you are running ther might be additional MBeans with attributes to query. I configured those MBeans below. To get the right objectname for an MBean you can either use jconsole first to get the name or - in case of JBoss - just connect to the jmx-console and get the MBean name there.

 <jmx-collection name="jsr160"
    maxVarsPerPdu = "50">
    <rrd step = "300">
        <rra>RRA:AVERAGE:0.5:1:8928</rra>
        <rra>RRA:AVERAGE:0.5:12:8784</rra>
        <rra>RRA:MIN:0.5:12:8784</rra>
        <rra>RRA:MAX:0.5:12:8784</rra>
    </rrd>
  
    <mbeans>
      <mbean name="SystemInfo" objectname="jboss.system:type=ServerInfo">
          <attrib name="FreeMemory"   alias="FreeMemory"       type="gauge"/>
          <attrib name="TotalMemory"  alias="TotalMemory"      type="gauge"/>
      </mbean>
 
      <mbean name="DefaultDSManagedConnectionPool" objectname="jboss.jca:service=ManagedConnectionPool,name=DefaultDS">
          <attrib name="AvailableConnectionCount"   alias="DefDS_AvailConns"    type="gauge"/>
          <attrib name="ConnectionCount"            alias="DefDS_Conns"         type="gauge"/>
          <attrib name="InUseConnectionCount"       alias="DefDS_InUseConns"    type="gauge"/>
          <attrib name="ConnectionCreatedCount"     alias="DefDS_CreatedConns"  type="counter"/>
          <attrib name="ConnectionDestroyedCount"   alias="DefDS_DestroyConns"  type="counter"/>
      </mbean>
 
      <!-- Global Request Processor HTTP
      <mbean name="GlobalRequestProcessor" objectname="jboss.web:type=GlobalRequestProcessor,name=http-0.0.0.0-8080">
          <attrib name="requestCount"     alias="GRP_requests"     type="counter"/>
          <attrib name="maxTime"          alias="GRP_maxTime"      type="gauge"/>
          <attrib name="bytesSent"        alias="GRP_bytesSent"    type="counter"/>
          <attrib name="bytesReceived"    alias="GRP_bytesRec"     type="counter"/>
          <attrib name="processingTime"   alias="GRP_procTime"     type="counter"/>
          <attrib name="errorCount"       alias="GRP_errors"       type="counter"/>
      </mbean -->
 
      <!-- Thread Pool HTTP
      <mbean name="ThreadPool" objectname="jboss.web:type=ThreadPool,name=http-0.0.0.0-8080">
          <attrib name="currentThreadsBusy"   alias="BusyThreads"      type="gauge"/>
          <attrib name="currentThreadCount"   alias="Threads"          type="gauge"/>
          <attrib name="minSpareThreads"      alias="MinSpareThreads"  type="gauge"/>
          <attrib name="maxSpareThreads"      alias="MaxSpareThreads"  type="gauge"/>
          <attrib name="maxThreads"           alias="MaxThreads"       type="gauge"/>
      </mbean -->
 
      <!-- Global Request Processor JK -->
      <mbean name="GlobalRequestProcessor"
             objectname="jboss.web:type=GlobalRequestProcessor,name=jk-8009">
          <attrib name="requestCount"     alias="GRP_requests"     type="counter"/>
          <attrib name="maxTime"          alias="GRP_maxTime"      type="gauge"/>
          <attrib name="bytesSent"        alias="GRP_bytesSent"    type="counter"/>
          <attrib name="bytesReceived"    alias="GRP_bytesRec"     type="counter"/>
          <attrib name="processingTime"   alias="GRP_procTime"     type="counter"/>
          <attrib name="errorCount"       alias="GRP_errors"       type="counter"/>
      </mbean>
 
      <!-- Thread Pool JK-->
      <mbean name="ThreadPool" objectname="jboss.web:type=ThreadPool,name=jk-8009">
          <attrib name="currentThreadsBusy"   alias="BusyThreads"      type="gauge"/>
          <attrib name="currentThreadCount"   alias="Threads"          type="gauge"/>
          <attrib name="minSpareThreads"      alias="MinSpareThreads"  type="gauge"/>
          <attrib name="maxSpareThreads"      alias="MaxSpareThreads"  type="gauge"/>
          <attrib name="maxThreads"           alias="MaxThreads"       type="gauge"/>
      </mbean>
   </mbeans>
 </jmx-collection>

Besides the SystemInfo MBean we also query the default connection pool and some other MBeans. In my case I've commented out the HTTP Global Request Processor and the HTTP Thread Pool of JBoss, as I'm only interested in the JK aequivalents of these (If you are using JBoss without an Apache frontend server you might change this).

Configuring the Graphs

Even though we are querying via JMX the configuration of the generated graphs is located in snmp-graph.properties. The default file shipped with OpenNMS contains all settings in a section called JBoss. I've "abused" the existing report template for HTTP Reports for JK Reports and only changed a fe strings (title etc.):

report.jboss.grp.tp.name=GRP_throughput
report.jboss.grp.tp.columns=GRP_bytesSent, GRP_bytesRec
report.jboss.grp.tp.type=interfaceSnmp
report.jboss.grp.tp.command=--title="JK Global Request Processor - Throughput" \
 DEF:sent={rrd1}:GRP_bytesSent:AVERAGE \
 DEF:rec={rrd2}:GRP_bytesRec:AVERAGE \
 LINE2:sent#0000ff:"BytesSent" \
 GPRINT:sent:AVERAGE:" Avg  \\: %7.2lf %s" \
 GPRINT:sent:MIN:"Min  \\: %7.2lf %s" \
 GPRINT:sent:MAX:"Max  \\: %7.2lf %s\\n" \
 LINE2:rec#00ff00:"BytesRec " \
 GPRINT:rec:AVERAGE:" Avg  \\: %7.2lf %s" \
 GPRINT:rec:MIN:"Min  \\: %7.2lf %s" \
 GPRINT:rec:MAX:"Max  \\: %7.2lf %s\\n"

report.jboss.grp.cnt.name=GRP_counts
report.jboss.grp.cnt.columns=GRP_requests, GRP_errors
report.jboss.grp.cnt.type=interfaceSnmp
report.jboss.grp.cnt.command=--title="JK Global Request Processor - Counts" \
 DEF:req={rrd1}:GRP_requests:AVERAGE \
 DEF:errors={rrd2}:GRP_errors:AVERAGE \
 LINE2:req#0000ff:"Requests" \
 GPRINT:req:AVERAGE:" Avg  \\: %7.2lf %s" \
 GPRINT:req:MIN:"Min  \\: %7.2lf %s" \
 GPRINT:req:MAX:"Max  \\: %7.2lf %s\\n" \
 LINE2:errors#ff0000:"Errors  " \
 GPRINT:errors:AVERAGE:" Avg  \\: %7.2lf %s" \
 GPRINT:errors:MIN:"Min  \\: %7.2lf %s" \
 GPRINT:errors:MAX:"Max  \\: %7.2lf %s\\n" 

report.jboss.grp.time.name=GRP_time
report.jboss.grp.time.columns=GRP_procTime
report.jboss.grp.time.type=interfaceSnmp
report.jboss.grp.time.command=--title="JK Global Request Processor - Time" \
 DEF:proc={rrd1}:GRP_procTime:AVERAGE \
 LINE2:proc#0000ff:"ProcessTime" \
 GPRINT:proc:AVERAGE:" Avg  \\: %6.2lf %s" \
 GPRINT:proc:MIN:"Min  \\: %6.2lf %s" \
 GPRINT:proc:MAX:"Max  \\: %6.2lf %s\\n" \ 

report.jboss.http.tp.name=Http Thread Pool
report.jboss.http.tp.columns=BusyThreads, Threads
report.jboss.http.tp.type=interfaceSnmp
report.jboss.http.tp.command=--title="JK Thread Pool" \
 DEF:busy={rrd1}:BusyThreads:AVERAGE \
 DEF:threads={rrd2}:Threads:AVERAGE \
 LINE2:busy#00ff00:"BusyThreads" \
 GPRINT:busy:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:busy:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:busy:MAX:"Max  \\: %5.2lf %s\\n" \
 LINE2:threads#0000ff:"ThreadLimit" \
 GPRINT:threads:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:threads:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:threads:MAX:"Max  \\: %5.2lf %s\\n"


And that's about it. After the JVM Service is discovered on your node you should see the new graphs in the Resource Graphs section!

References