SNMPV3 polling failures and OOM messages in system log

1. When SNMPV3 is configured, authentication fails with messages similar to below in the snmpd.log

2023-04-27 04:00:59 Authentication failed for test-snmp -profile

2. On PA-7000 and PA-5450: These platforms have a limit (1GB) set on snmpd virtual memory by default. 
After the virtual memory snmpd hits the limit, the snmpd process is restarted. Following system log is generated upon snmpd restart:

2023/06/30 02:34:56 critical general general 0 snmpd - virtual memory limit exceeded, restarting

Usually there is no other impact other than few missed SNMP polls when snmpd is restarting.

3. Other PA NGFW platforms: Other platforms do not have a similar virtual memory limit set hence snmpd memory could grow till the system runs out of memory and causes OOM.

var/log/messages:
May 23 16:06:32 800 kernel: <any process> invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=352
May 23 16:06:33 800 kernel: Out of memory: Kill process 8888 (snmpd) score 588 or sacrifice child
May 23 16:06:33 800 kernel: Killed process 8888 (snmpd) total-vm:6336588kB, anon-rss:1253872kB, file-rss:0kB

• Palo Alto Firewalls other than PA-7000 and PA-5450
• PAN-OS version 10.1.9.x below 10.1.11, 10.1.10-h4, 10.1.9-h9
• SNMPv3 configured

• The snmpd process memory grows continuously on the device.
• This causes OOM (out of memory) crash and subsequent authentication failures.

1. The issue is fixed under PAN-217208 in PAN-OS 10.1.11.
2. Upgrade to this or higher versions will resolve the issue

1. Use SNMPv2
2. Restart SNMP process, the issue is seen again after few days.

> debug software restart process snmpd

> show system state | match md.apps.s1.mp.prc.snmpd.script.runtime
md.apps.s1.mp.prc.snmpd.script.runtime: { 'actions': [ { 'action': timer-create, 'event': hbScript, 'interval': 300, 'name': hb-script, }, ], 'cgroup-enable': True, 'count': 1, 'display': , 'external-restart-ok': True, 'group': { }, 'hb-enable': True, 'heapprofile-enable': False, 'limits': { 'enable-fd-limit': False, 'enable-phy-limit': False, 'enable-virt-limit': False, 'enable-vmrss-limit': False, 'fd-limit': 1024, 'phy-limit': 2097152, 'virt-limit': 2097152, 'vmrss-limit': 33554432, }, 'logon': True, 'pprof-enable': False, 'process': { 'pid': 15014, }, 'restart-enable': True, 'state-machine': { 'count': 1, 'event': hbScript, 'state': running, 'timer': hb-script, }, 'sysd-namespaces': [ ], 'sysd-notifiers': { }, 'telem-enable': False, }

> debug software virt-limit service snmpd limit 1000000

Success 

> show system state | match md.apps.s1.mp.prc.snmpd.script.runtime
md.apps.s1.mp.prc.snmpd.script.runtime: { 'actions': [ ], 'cgroup-enable': True, 'count': 1, 'display': , 'external-restart-ok': True, 'group': { }, 'hb-enable': True, 'heapprofile-enable': False, 'limits': { 'enable-fd-limit': False, 'enable-phy-limit': False, 'enable-virt-limit': True, 'enable-vmrss-limit': False, 'fd-limit': 1024, 'phy-limit': 2097152, 'virt-limit': 1000000, 'vmrss-limit': 33554432, }, 'logon': True, 'pprof-enable': False, 'process': { 'pid': 15014, }, 'restart-enable': True, 'state-machine': { 'count': 1, 'event': virtLimitEnable 1000000, 'state': running, }, 'sysd-namespaces': [ ], 'sysd-notifiers': { }, 'telem-enable': False, }