AIOps Alert "Process Memory Depletion

AIOps Alert "Process Memory Depletion - User Id"

2822

Created On 02/15/22 22:38 PM - Last Modified 06/07/23 22:53 PM

User-ID

PAN-OS

AIOps for NGFW

Symptom

Alert from AIOps regarding process memory depletion for "useridd"

Environment

PAN-OS
AIOps

Cause

AIOps has detected a memory depletion in the useridd process.

Resolution

If you receive this Alert, it is recommended to collect the following Troubleshooting Data below and open a Support Case. After data is collected, considering following the Mitigation Steps to bring down the memory usage of the useridd process till Support can analyze the data.

Troubleshooting Data

Collect Tech Support File (GUI: Device > Support Click Generate Tech Support File)

Generate a trace file: collect the output of the following CLI commands:

set cli pager off
show clock
show system software status | match useridd
debug software trace user-id

Generate a core file: collect the output of the following command

show clock
debug software core user-id
show system files
set cli pager on

Note: Wait few minutes between generating the core file and checking "show system files", see if a core file with name similar to the following "useridd-20230405152400-10.1.5-h1.tar.gz" gets generated under /var/cores directory (initially the core file will show up with extension .xml as it is still getting formed then with extension .core before it gets finally zipped and gets the extension .tar.gz).
If your firewall is a small platform then a core file might not get generated due to the lack of disk-space.

Export the core file (HOW TO EXPORT CORE FILES FROM A PALO ALTO NETWORKS DEVICE) and (HOW TO UPLOAD CORE FILES DIRECTLY TO SUPPORT ).
```
scp export core-file management-plane from useridd-20230405152400-10.1.5-h1.tar.gz to username@host:path
```
1. Note: Core file can also be downloaded from UI under DEVICE > Support click "Download Core files"
Collect the Device State (GUI: Device>Setup>Operations- Export: Export device state)
Gather data below from AIOps
1. Check the date and timestamp of when the memory depletion started.
2. Check the overall health of your firewall.
From your firewall System and Configuration logs (MONITOR > Logs) check if there were any configuration change, PANOS upgrades/downgrades, or any other changes performed around the time of the start of the issue that might have triggered this behavior.
(Optional) If performing Mitigation Steps below, collect another Tech Support File after completing steps
Open a case with the above data.

Mitigation Steps

Till the issue is resolved, perform one of the following operations to bring down the memory usage of user-id:

NOTE: Recommended to be performed in a maintenance window.

Potential Impact of restart the process:

User ip mapping, group mapping, useridd xmlAPI request, Cloud Directories will not be available during process restart.

Option 1 - Restart the process that is consuming excessive memory (recommended to be performed in a maintenance window)

Restart the user-id process using below command
```
debug software restart process user-id
```

Option 2 - In HA environments (recommended to be performed in a maintenance window),

Disable "Preemptive" mode (GUI: Device > High Availability > General > Election Settings: Uncheck Preemptive ) on both Active and Passive device and commit your configuration change.
Failover to the passive device (From Active Device: Device > High Availability > Operations > Click Suspend local device)
Restart the user-id process on the previously Active device (i.e. the current suspended device).
```
debug software restart process user-id
```
From CLI run show management-clients to ensure that all processes have started successfully.

> show management-clients
              Client PRI    State Progress
-------------------------------------------------------------------------
            ha_agent  25     init        0
              sslmgr  10     init        0
               authd  10     init        0
             cryptod  10     init        0
              dagger  10     init        0    (op cmds only)
                cord  10     init        0
                logd  10     init        0    (op cmds only)
             reportd  10     init        0    (op cmds only)
             useridd  10     init        0
        distributord  10     init        0
                iotd  10     init        0
Overall status: init. Progress: 0
Warnings:
Errors:

Note: Restarting the process will temporarily mitigate the issue and the same issue may come back sometimes later.