Syslogd

From OpenNMS
Jump to navigation Jump to search

Functionality

Syslogd allows OpenNMS to receive syslog datagrams and turn them into events, and then into alarms and/or notifications, much as Trapd does for SNMP traps.

Availability

This feature is available in stable releases since 1.6.0, and first appeared in the 1.3.2 unstable release.

Functionality with substring-based event UEI matching and message hiding (for messages containing sensitive data) was introduced in release 1.3.7.

Additional enhancements to enable regular expression-based matching and extraction of regex match-groups as event parameters are available from the 1.3.8 release.

As of version 1.10, matching on the sending process' name as been implmented as well as parameters assignment.

Cascading

Syslogd up until 1.3.7 supported BSD style forwarding. The 1.3.8 release will attempt to match a nodename to the database, do hostname lookups via the resolver, and is designed to work better with syslog-ng.

  • The regular expression used to extract the originating hostname and other message metadata is defined in the syslogd-configuration.xml file in the forwarding-regexp attribute of the <configuration> element.
  • The default setup is for a BSD style syslog. Depending on how messages are cascaded, this can easily be changed by altering the forwarding-regexp.
  • The example configuration below is from a cascaded syslog-ng + stunnel setup.

Configuration

<?xml version="1.0"?>
<syslogd-configuration>
    <configuration
            syslog-port="10514"
            new-suspect-on-message="true"
            forwarding-regexp="^((.+?) (.*))\r?\n?$"
            matching-group-host="2"
            matching-group-message="3"
            />


    <ueiList>
        <ueiMatch>
            <match type="substr" expression="CRISCO"/>
            <uei>uei.opennms.org/tests/syslogd/substrUeiRewriteTest</uei>
        </ueiMatch>
        <ueiMatch>
            <match type="regex" expression=".*?foo: (\d+) out of (\d+) tests failed for (\S+)$"/>
            <uei>uei.opennms.org/tests/syslogd/regexUeiRewriteTest</uei>
        </ueiMatch>
        <ueiMatch>
            <process-match expression="^sshd$" />
            <match type="regex" expression="^Failed (.*?) for invalid user (\S+) from (.*?) port (\d+) ssh(\d)$" />
            <uei>uei.opennms.org/vendor/openssh/syslog/sshd/invalidUser</uei>
                <parameter-assignment matching-group="1" parameter-name="authMethod" />
                <parameter-assignment matching-group="2" parameter-name="user" />
                <parameter-assignment matching-group="3" parameter-name="remoteHost" />
                <parameter-assignment matching-group="4" parameter-name="remotePort" />
                <parameter-assignment matching-group="5" parameter-name="protocolVersion" />
        </ueiMatch>
    </ueiList>

    <hideMessage>
        <hideMatch>
            <match type="substr" expression="TEST"/>
        </hideMatch>
        <hideMatch>
            <match type="regex" expression="[Dd]ouble secret"/>
        </hideMatch>
    </hideMessage>

</syslogd-configuration>

The <configuration> element specifies global parameters that specify how Syslogd will receive and process syslog messages. This element has the following attributes:

  • Port, default syslogd port to listen on
  • New suspect on messages specifies whether Syslogd will inject a newSuspect event when it receives a syslog message originating from a host that cannot be resolved to an existing node managed by OpenNMS. This functionality is equivalent to the same option in Trapd. New suspects will be queued for discovery by Provisiond.
  • The 'Cascading' (forwarding) magic - these parameters define how cascaded or forwarded syslog messages are found and deconstructed
    • forwarding-regexp specifies a regular expression used to identify forwarded syslog messages
    • matching-group-host specifies which match-group from the forwarding-regexp contains the originating hostname
    • matching-group-message specifies which match-group from the forwarding regexp contains the body of the syslog message
  • A description of regular expressions you can find at http://java.sun.com/j2se/1.5.0/docs/api/java/util/regex/Pattern.html

Matching syslog messages

As of version 1.10, matching syslog messages can be done against the following data:

  • Content of the syslog message (<match> tag): Required
  • Name of the process sending the syslog message (<process-match> tag): A regular expression.
  • syslog message's facility (<facility> tag): One of kern,kernel,user,mail,daemon,auth,syslog,lpr,news,uucp,cron,authpriv,ftp,ntp,audit,alert,clock,local0,local1,local2,local3,local4,local5,local6,local7. This tag may be specified several time.
  • syslog message's severity (<severity> tag): One of emergency, alert, critical, error, warning, notice, info, debug. This tag may be specified several time.
  • sender's hostname (<hostname-match> tag): A regular expression.
  • sender's host address (<hostaddr-match> tag): A regular expression.

A syslog message must match all criteria for an event to be fired.

Parameters assignment

As of version 1.10, if you define groups in the expression attribute of a <match> tag, you can assign these groups' value to a named parameter that can later be used within the event generated.

Example:

With the following ueiMatch definition, %parm[bindPort]%, %parm[bindAddress]%, and %parm[errorMessage]% can be defined in the corresponding event's description to add more information:

       <ueiMatch>
           <process-match expression="^sshd$" />
           <match type="regex" expression="^Bind to port (\d+) on (.*?) failed: (.*)\.$" />
           <uei>uei.opennms.org/vendor/openssh/syslog/sshd/bindFailure</uei>
               <parameter-assignment matching-group="1" parameter-name="bindPort" />
               <parameter-assignment matching-group="2" parameter-name="bindAddress" />
               <parameter-assignment matching-group="3" parameter-name="errorMessage" />
       </ueiMatch>

Discarding Messages

Since stable release 1.6.3 and development release 1.7.1, it is possible to configure Syslogd to discard certain messages without ever creating events. To configure this behavior, simply change the value of the <uei> element inside a <ueiMatch> element to match the value of the discard-uei attribute in the <configuration> element of syslogd-configuration.xml. The default value of this attribute is DISCARD-MATCHING-MESSAGES.

Integration with syslog-ng

To integrate with syslog-ng, you have two options to add your syslog-ng configuration:

parser="org.opennms.netmgt.syslogd.SyslogNGParser"

destination opennms {udp("127.0.0.1" port(10514));};
log {source(s_all);destination(opennms);};


OR

parser="org.opennms.netmgt.syslogd.Rfc5424SyslogParser"

destination opennms {syslog("127.0.0.1" transport("udp") port(10514));};
log {source(s_all);destination(opennms);};

Correcting Hostnames in OpenNMS

If you have problems getting the correct hostname into the opennms events (by example all syslog events are marked as being from "localhost") you might try to add the following option to your syslog-ng configuration:

options { chain_hostnames(no) };

This will create hostnames in the log entry like "routername" instead of "routername/routername". Beware if you have other applications depending on the syslog format!

Another possibility would be to change the OpenNMS syslogd-configuration from the default

...
  matching-group-host="1"
  matching-group-message="2"
...

(which will match the format of the "normal" syslog daemons) to

...
  matching-group-host="2"
  matching-group-message="3"
...

Check the value of your regexp - for syslog-ng, it is recommended to use a basic regexp as follows:

...
            forwarding-regexp="^((.+?) (.*))\n?$"
...

If you still have problems go to Problem solving

Integration with rsyslog

The following is an example of how to use rsyslog to act as a syslog relay between a device and OpenNMS.

The following sample configuration file in /etc/rsyslog.d/00-custom.conf will receive syslog messages and, if the source device's IP starts with 10, forward the syslog messages in RFC5424 format via UDP to OpenNMS (which is listening on port 10514).

# direct log messages from the specified source IP addresses to another local syslog host on port 10514 (OpenNMS)
if $fromhost-ip startswith '10.' then @127.0.0.1:10514;RSYSLOG_SyslogProtocol23Format
# and then stop processing further syslog rules for syslog
& stop

OpenNMS is configured to receive syslog messages using the RFC5424 parser in syslogd-configuration.xml:

...
    <configuration
            syslog-port="10514"
            new-suspect-on-message="false"
            parser="org.opennms.netmgt.syslogd.Rfc5424SyslogParser"
            discard-uei="DISCARD-MATCHING-MESSAGES"
            />
...

Example Custom Event

    <event>00-custom.conf
        <uei>uei.mycompany.net/generic/smartd/DiskLost</uei>
        <event-label>Smartd Disk could not be inserted</event-label>
        <descr>
                An event indicating a lost disk
        </descr>
        <logmsg dest='logndisplay'>Smartd, disk insert failure:%id% (%id%) args(%parm[##]%):%parm[all]%</logmsg>
        <severity>Critical</severity>
        <alarm-data reduction-key="%uei%:%dpname%:%nodeid%" alarm-type="1" />
    </event>

Activation

To activate this, uncomment the parts in service-configuration.xml where applicable.

        <service>
                <name>OpenNMS:Name=Syslogd</name>
                <class-name>org.opennms.netmgt.syslogd.jmx.Syslogd</class-name>
                <invoke at="start" pass="0" method="init"/>
                <invoke at="start" pass="1" method="start"/>
                <invoke at="status" pass="0" method="status"/>
                <invoke at="stop" pass="0" method="stop"/>
        </service>

Local Syslog

If you need to run a local syslog as syslogd uses by default port 514 - syslog-ng is a good replacement. It can generate events without having to listen to port 514

Technical description

The service is built around traditional UDP based syslog messages, and the service is a receiver of messages created by remote syslog services.

Scalability

In the implementation I tried to look at scalability and as it's a UDP based service the minimization of message loss. The syslogd is implemented as a multithreaded fifo-queue and seems to scale quite well in test scenario

Status

All RFC Applicable Facilities and Priorities are implemented.

How does it work

Question - What will happen as a syslog event arrives?

Answer - The message is parsed, the sender compared to known nodes. If the sender is an unknown node, an event is sent to discover the node. (This is configurable but it's the default behaviour). Otherwise the message is tagged to the correct node, broadcast to eventd, and from there on essentially is an openNMS event.

Note: The matching process stops at the first regular expression that matches the syslog message (you can thus insert a "discard all" expression at the bottom of your <ueiList> to filter out all syslog messages that you don't care about).

Question - Event how?

Answer - We convert the syslog message to an openNMS Event, the matching event as usual is found in the etc/events directory.

I.e. for each message received, it will get prioritized, categorized and submitted to the eventprocessor for notification or other configured actions.

Question - What if I need to match syslog messages with embedded newlines?

Answer - You can enable DOTALL mode (aka single-line mode) by adding (?s) to the very beginning of your forwarding-regexp, so ^((.+?) (.*)\n?$ becomes (?s)^((.+?) (.*)\n?$. Note that if you add (?s) to your forwarding-regexp, you should also add it to the expression attribute for each <match> element in each of your <ueiMatch> elements.

Problem solving

Most problems with syslog integration are due to different syslog formats received from different syslog daemons. This results in parsing errors like ip address is not found, ip address contains garbage, host name is not found, hostname seems to be always localhost, hostname contains garbage etc.. In this case you should try to forward all your syslog messages to a syslog file and look out for the different formats you receive like

Nov 19 00:13:47 host1 2540680: Nov 19 00:13:47.123: this is the message from host 1
Nov 19 00:13:48 host2/host3 2540681: Nov 19 00:13:48.523: this is the message from host x forwarded by host y
Nov 19 00:13:53 192.168.1.2 2540682: Nov 19 00:13:53.856: this is the message from ip 192.168.1.2
Nov 19 00:13:55 192.168.1.2/10.2.3.4 2540683: Nov 19 00:13:55.529: this is the message from ip x forwarded by ip y

You should try to get always the same format from the different syslogs, else you have to change the syslogd-configuration and try to match the different formats with your regexp's. Especially take care to match the hostname part correct adjusting

            matching-group-host="..."
            matching-group-message="..."

Note: In version 14+, set the key syslogd to have the value TRACE as the log level in the file log4j2.xml to get DEBUG-like logging in the log file /var/log/opennms/syslogd.log.

To limit trial-and-error time you might take a look at the postgres-db of opennms:

Connect to the db:

psql -U userid pwd

(Standard: user=opennms pwd=opennms)

Search for entries coming from syslogd

opennms=# select * from events where eventuei like '%syslog%';

If you have a lot of syslog entries you might want to narrow the search using standard sql parameters.

Now you should see something like

eventid                 | 12159902
eventuei                | uei.opennms.org/syslogd/local7/Error
nodeid                  | 2
eventtime               | 2009-12-26 03:35:33+01
eventhost               | mgtserv1
eventsource             | syslogd
ipaddr                  | 10.99.99.99
eventdpname             | undefined
eventsnmphost           |
serviceid               |
eventsnmp               |
string,text);processid=0(string,text)
eventcreatetime         | 2009-12-26 03:35:33.821+01
ocess: 2664138 <br> PID: 0 </p>
eventloggroup           |
eventlogmsg             |  <p>An OpenNMS Event has been received as a Syslog Message </p>
                           Message: Dec 26 03:35:31.813: Late collision message from port 8 <br>
eventseverity           | 5
eventpathoutage         |
eventcorrelation        |
eventsuppressedcount    |
eventoperinstruct       |
eventautoaction         |
eventoperaction         |
eventoperactionmenutext |
eventnotification       |
eventtticket            |
eventtticketstate       |
eventforward            |
eventmouseovertext      |
eventlog                | Y
eventdisplay            | Y
eventackuser            |
eventacktime            |
alarmid                 |

opennms=#

If the eventhost field contains some part of your syslog message or something else recognizable this might give you a clue what you have to change in your regexp or parameter numbering to get the correct values for hostname and messages.

More examples

Automating alarms and reducing amount of messages Syslogd_Automations

Integration with Hyperic Hyperic_HQ_Syslog_Integration (functionally obsolete, but may contain useful information and examples)