SNMP spams my log

From OpenNMS
Jump to: navigation, search

Default Behaviour

On Linux systems SNMP usually protocols incoming SNMP connections in your syslog file. Depending on your SNMP polling cycle and your amount of monitoring systems the entries will multiplied. Also monitors which uses SNMP like HostResourceSwRunMonitor to monitor processes generate them.

Example:

Mar 4 21:12:48 hostname snmpd[31684]: Connection from UDP: [127.0.0.1]:45788
Mar 4 21:12:48 hostname snmpd[31684]: Received SNMP packet(s) from UDP: [127.0.0.1]:45788

As you can imagine you can get a big amount of this entries, even in small and middle sized environments. And just when you have to search in your logs for some issues the SNMP log entries are really annoying.

To get only warnings or errors into your syslog file you can use the following configurations.


Modify the line

SNMPDOPTS='-Lsd -Lf /dev/null -p /var/run/snmpd.pid'

in /etc/default/snmpd.


SNMP log level:

id short description
0 Emergencies System is unusable
1 Alerts Immediate action needed
2 Critical Critical conditions
3 Errors Error conditions
4 Warnings Warning conditions
5 Notifications Informational messages
6 Informational Normal but significant conditions
7 Debugging Debugging messages

From notifications onwards:

SNMPDOPTS='-LS0-5d -Lf /dev/null -u snmp -g snmp -I -smux,mteTrigger,mteTriggerConf -p /var/run/snmpd.pid'

From warnings onwards:

SNMPDOPTS='-LS0-4d -Lf /dev/null -u snmp -g snmp -I -smux,mteTrigger,mteTriggerConf -p /var/run/snmpd.pid'

If you want, read the SNMP man page for more information about logging option.

SNMP needs to be restarted!

Statfs

If you got angry about the default SNMP log behavior, you doubtless came across above entries like this:


... snmpd[1234] Cannot statfs /var/lib/docker/containers/: Permission denied ...
... snmpd[1234] Cannot statfs /var/lib/docker/aufs/mnt/: Permission denied ...
... snmpd[1234] Cannot statfs /run/docker/netns/: Permission denied ...
... snmpd[1234] Cannot statfs /run/user/1000/gvfs: Permission denied ...
... snmpd[1234] Cannot statfs /sys/kernel/debug/tracing: Permission denied ...


Short and precise: There is no way to configure SNMP to supress these log entries!

Redhat Bugzilla says about it:

Because as I wrote in comment #2, snmpd reads /proc/mounts and runs statfs on each entry there. If any statfs call fails it logs an error. So, either stafs must not fail (i.e. no „net:[4026532288]“ entries in /proc/mounts) or snmpd must be fixed to log something more useful and only once.

Source: https://bugzilla.redhat.com/show_bug.cgi?id=1314610#c10


So the solution is very simple. We have to ignore these logs on (r)syslog side.

Here is a rsyslog example:

/etc/rsyslog.d/040-snmp-statfs.conf

if $programname == 'snmpd' and $msg contains 'statfs' then {
   stop
}