Palo Alto Networks Knowledgebase: The useridd process high cpu utilization and/or useridd restart Virtual memory limit exceeded

The useridd process high cpu utilization and/or useridd restart Virtual memory limit exceeded

7361
Created On 02/08/19 00:01 AM - Last Updated 02/08/19 00:01 AM
User-ID
Symptom

Symptoms

Symptoms can range from the following but the list is not exhaustive;

- slow or unable to commit successfully due to mainly to memory depletion
- useridd process repeated crash files
- useridd process is always %CPU top utilization on cli command 'show
system resources'
- GlobalProtect Users unable to connect
- reboot regains responsiveness and functionality
- Group-mapping truncation observed on cli command 'debug user-id dump
idmgr type user-group all'
- HA sync failing
- Traffic hitting incorrect rules
- Repeated system logs similar to below"
2016/06/15 06:47:40 critical general general 0 User-ID manager was
reset. Commit is required to reinitialize User-ID
2016/06/15 06:47:40 critical general general 0 User-ID manager was
reset. Commit is required to reinitialize User-ID
2016/06/15 06:47:39 critical general general 0 User-ID manager was
reset. Commit is required to reinitialize User-ID
2016/06/15 06:47:39 critical general general 0 User-ID manager was
reset. Commit is required to reinitialize User-ID
2016/06/15 06:47:39 critical general general 0 User-ID manager was
reset. Commit is required to reinitialize User-ID
-Repeated masterd.log similar to below"
2016-06-04 03:38:54.278 +0100 INFO: useridd: User restart reason - Virtual
memory limit exceeded (3979520 > 2560000)
2016-06-04 03:39:34.378 +0100 INFO: useridd: received user restart
2016-06-04 03:39:34.392 +0100 INFO: useridd: User restart reason - System
swap
memory usage too high (86024 free)

Diagnosis

Group mapping pulled by firewall from AD/LDAP Servers very high.

show user group-mapping state all

Group Mapping(vsys1, type: active-directory): Company AD Group Information
(job
13)
Bind DN : serviceaccount@company.com
Base : DC=company,DC=com
Group Filter: (None) <<<<<===== no gorup filter
User Filter: (None)
Servers : configured 1 servers
10.129.80.232(389)
Last Action Time: 19577 secs ago(took 4972 secs) <<<<<===== It took
1.38hrs to
pull the data from AD
Next Action Time: Now (started 15977 secs ago)
Number of Groups: 118412 <<<<<<====== very high
<output cut for brevity>



Resolution

Configure group include list for firewall to query specific groups only on
the list.
On WebUI, Device > User Identification > Group Mapping Settings > {select
Group-Mapping instance} > Group Include List > {select from left specific
groups used
by firewall policies}

Limitation on this regards will either come from two which ever hits the
ceiling first:
1) In all platform whether hardware or VM, the maximum number of
user-groups is 99,999 as this is the max number of user groups supported
in idmgr.

#system logs
User-ID manager was reset. Commit is required to reinitialize User-ID
#useridd.log
2016-06-15 06:49:14.512 +0100 Error:
_pan_idmgr_find_avail_id(pan_idmgr.c:464): no id avail for type 22 last id
2147583647

Here 2147583647 is special, because
1024*1024*1024*2 = 2147483648
2147583647-2147483648 = 99999
99,999 is the maximum number of groups we support in idmgr.

In this case this problem may or may not go with high memory. High Memory
doesn't need to necessarily happen.

or

2) Useridd process virtual memory limit which depends per platform. To
give an example, see cli command below for a PA-5050 show virtual limit is
2097152 or ~2GB.
This means that useridd process can anytime restart if exceeding this
value, the system logs can be something similar to 'User restart reason -
Virtual memory limit exceeded'.
show system state | match snmpd.script.runtime
md.apps.s0.mp.prc.snmpd.script.runtime: { 'actions': [ { 'action':
timer-create, 'event': hbScript, 'interval': 300, 'name': hb-script, }, ],
'count': 1, 'display': , 'external-restart-ok': True, 'group': { },
'hb-enable': True, 'limits': { 'enable-fd-limit': False,
'enable-virt-limit': False, 'enable-vmrss-limit': False, 'fd-limit': 1024,
'virt-limit': 2097152, 'vmrss-limit': 33554432, }, 'process': {
'core-dumped': False, 'exit-code': 0, 'pid': 3575, }, 'restart-enable':
True, 'state-machine': { 'count': 1, 'event': hbScript, 'state': running,
'timer': hb-script, }, 'sysd-namespaces': [ ], 'sysd-notifiers': { }, }

 

Note: Maximum number of groups that can be defined in policies per vsys is
640.



Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CldjCAC&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments
Choose Language