Decrease snmpd logging level on Ubuntu 18.04

I recently updated some servers from Ubuntu 16.04 to 18.04, and found that the snmpd daemon was generating way too many log entries in /var/log/syslog - one for every SNMP query coming from our monitoring system.

In older Ubuntus I had edited /etc/defaults/snmpd to change the SNMPDOPTS line to have a different -Ls parameter, but it seems that on the new Ubuntu, the systemd service for snmp doesn't use that defaults file at all. A comment on this serverfault question gave me a clue on how to fix it in systemd - I thought I'd elaborate here.

If you run

systemctl cat snmpd.service

You see the current service file:

# /lib/systemd/system/snmpd.service
[Unit]
Description=Simple Network Management Protocol (SNMP) Daemon.
After=network.target
ConditionPathExists=/etc/snmp/snmpd.conf

[Service]
Environment="MIBSDIR=/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf:/usr/share/mibs/site:/usr/share/snmp/mibs:/usr/share/mibs/iana:/usr/share/mibs/ietf:/usr/share/mibs/netsnmp"
Environment="MIBS="
Type=simple
ExecStartPre=/bin/mkdir -p /var/run/agentx
ExecStart=/usr/sbin/snmpd -Lsd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux,mteTrigger,mteTriggerConf -f
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

I wanted to override the ExecStart line with something different. To do that, run

systemctl edit snmpd.service

This brings up your default editor with a blank file. I entered these new lines:

# Override default "-Lsd" paramter to "-LSwd" to decrease logging level
[Service]
ExecStart=
ExecStart=/usr/sbin/snmpd -LSwd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux,mteTrigger,mteTriggerConf -f

The first ExecStart= line is a bit odd, without it you get an error:

snmpd.service: Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. Refusing.

so the first line 'clears' the setting before processing your own version.

Save your file (systemctl edit stores it as /etc/systemd/system/snmpd.service.d/override.conf), and then run service snmpd restart to have take effect. If you re-run systemctl cat snmpd.service you should now see:

# /lib/systemd/system/snmpd.service
[Unit]
Description=Simple Network Management Protocol (SNMP) Daemon.
After=network.target
ConditionPathExists=/etc/snmp/snmpd.conf

[Service]
Environment="MIBSDIR=/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf:/usr/share/mibs/site:/usr/share/snmp/mibs:/usr/share/mibs/iana:/usr/share/mibs/ietf:/usr/share/mibs/netsnmp"
Environment="MIBS="
Type=simple
ExecStartPre=/bin/mkdir -p /var/run/agentx
ExecStart=/usr/sbin/snmpd -Lsd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux,mteTrigger,mteTriggerConf -f
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/snmpd.service.d/override.conf
# Override default "-Lsd" paramter to "-LSwd" to decrease logging level
[Service]
ExecStart=
ExecStart=/usr/sbin/snmpd -LSwd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux,mteTrigger,mteTriggerConf -f

Which is a combination of the default service along with your override.

If you have other servers you want to copy your /etc/systemd/system/snmpd.service.d/override.conf file to, you need to run

systemctl daemon-reload
service snmpd restart

to have it take effect.

KVM Networking

Still playing with KVM (Kernel-based Virtual Machine), this time checking out some networking features. I've been running Ubuntu 8.04 LTS Server (Hardy Heron), both as the host and as a VM on that host. Networking is setup to use a bridge.

KVM offers different emulated NICs, I took a quick look at running iperf between the VM and the host, and got these speeds for a few select NIC models:

  • RTL-8139C+ (the default): ~210 Mb/sec
  • Intel e1000 (somewhat recommended here): ~ 330Mb/sec
  • virtio: ~ 700Mb/sec

The thing about virtio though is that it doesn't work when the VMs RAM is set to 4GB. So I guess you can have fast networking, or lots of memory, but not both.

Playing with KVM and LVM on Linux

I'm still experimenting with Ubuntu 8.04 Server (Hardy Heron), and have switched from Xen to KVM (Kernel-based Virtual Machine). Xen worked well on a little test machine I had, but when I tried it on a brand-new Supermicro server, it turned out to have a problem with the Intel NIC. Since it seems Ubuntu is recommending KVM over Xen, and the server supports hardware virtualization, I figured I'd give it a try.

One big difference is that KVM does full emulation, which means any disk space you give it from LVM (Logical Volume Manager), will be a full virtual disk, with a partition table. It's a little more complicated to access filesystems within the virtual disk that it was with Xen, I wanted to jot some notes down here mostly for myself on how to do that.

If I've created a logical volume named /dev/myvg/test_vm and installed another linux on it with a single ext3 filesystem (/dev/sda1 from the point of view of the VM) and some swap space (/dev/sda5), it can be accessed when the VM isn't running with the help of the kpartx utility...

kpartx -av /dev/myvg/test_vm

would read the partition table on the virtual disk and create:

/dev/mapper/myvg-test_vm1 
/dev/mapper/myvg-test_vm2 
/dev/mapper/myvg-test_vm5

Then you can

mount /dev/mapper/myvg-test_vm1 /mnt

to mess with the VMs /dev/sda1. To clean things up when finished, run:

umount /mnt
kpartx -d /dev/myvg/test_vm

Snapshots

If you want to look at the contents of a running VM's disks (perhaps for backing it up) you can use LVM snapshots. For example:

lvcreate --snapshot --size 1G --name test_snap /dev/myvg/test_vm
kpartx -av /dev/myvg/test_snap
mount /dev/mapper/myvg-test_snap1 /mnt
   .
   (play with VM's /dev/sda1 in /mnt)
   .
umount /mnt
kpartx -dv /dev/myvg/test_snap
lvremove /dev/myvg/test_snap

Ubuntu server locale errors

This is mostly a note to myself... After setting up a minimal Ubuntu server install (in Xen), following these instructions using debootstrap I saw lots of errors like this:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

Checking with locale -a would show

C
POSIX

While a full Ubuntu server install (off a CD) would show:

C
en_US.utf8
POSIX

This command seems to have generated the missing locale and made everybody happy.

localedef --no-archive -i en_US -c -f UTF-8 en_US.UTF-8

Xen and UFW on Ubuntu

I've been experimenting with setting up Ubuntu Server 8.04 (Hardy Heron) to run Xen, and had a minor problem with UFW (Uncomplicated Firewall) running in the dom0 blocking network access to a domU running in bridged mode. It seems the fix is just to edit /etc/defaults/ufw and make this change to enable forwarding:

--- a/default/ufw       Thu Oct 23 10:00:33 2008 -0500
+++ b/default/ufw       Thu Oct 23 10:34:36 2008 -0500
@@ -16,7 +16,7 @@ DEFAULT_OUTPUT_POLICY="ACCEPT"

 # set the default forward policy to ACCEPT or DROP.  Please note that if you
 # change this you will most likely want to adjust your rules
-DEFAULT_FORWARD_POLICY="DROP"
+DEFAULT_FORWARD_POLICY="ACCEPT"

 #
 # IPT backend

and then run ufw disable; ufw enable.

I believe dom0 is now protected, and it'll be up the the domU to protect itself. I can't say I'm entirely comfortable with Linux IPTables, sure wish PF was available as an alternative.