AIOps Alert "Process Memory Depletion - User Id"

• Alert from AIOps regarding process memory depletion for "useridd"

• PAN-OS
• AIOps

If you receive this Alert, it is recommended to collect the following Troubleshooting Data below and open a Support Case. After data is collected, considering following the Mitigation Steps to bring down the memory usage of the useridd process till Support can analyze the data. 

Troubleshooting Data

1. Collect Tech Support File  (GUI: Device > Support  Click Generate Tech Support File)

2. Generate a trace file: collect the output of the following CLI commands:

   set cli pager off
   show clock
   show system software status | match useridd
   debug software trace user-id
   

3. Generate a core file: collect the output of the following command

   show clock
   debug software core user-id
   show system files
   set cli pager on
   

4. Export the core file  (HOW TO EXPORT CORE FILES FROM A PALO ALTO NETWORKS DEVICE) and (HOW TO UPLOAD CORE FILES DIRECTLY TO SUPPORT ).

   scp export core-file management-plane from useridd-20230405152400-10.1.5-h1.tar.gz to username@host:path

   1. Note: Core file can also be downloaded from UI under DEVICE > Support click "Download Core files"
5. Collect the Device State (GUI: Device>Setup>Operations- Export: Export device state)
6. Gather data below from AIOps
   1. Check the date and timestamp of when the memory depletion started.
   2. Check the overall health of your firewall.
7. From your firewall System and Configuration logs (MONITOR > Logs) check if there were any configuration change, PANOS upgrades/downgrades, or any other changes performed around the time of the start of the issue that might have triggered this behavior.
8. (Optional) If performing Mitigation Steps below, collect another Tech Support File after completing steps
9. Open a case with the above data.
    

Mitigation Steps

Till the issue is resolved, perform one of the following operations to bring down the memory usage of user-id:

Potential Impact of restart the process:

•  User ip mapping, group mapping, useridd xmlAPI request, Cloud Directories will not be available during process restart.
   

Option 1 - Restart the process that is consuming excessive memory (recommended to be performed in a maintenance window)

1. Restart the user-id process using below command

   debug software restart process user-id

Option 2 - In HA environments (recommended to be performed in a maintenance window),

1. Disable "Preemptive" mode (GUI: Device > High Availability > General > Election Settings: Uncheck Preemptive ) on both Active and Passive device and commit your configuration change.
2. Failover to the passive device (From Active Device: Device > High Availability > Operations > Click Suspend local device)
3. Restart the user-id process on the previously Active device (i.e. the current suspended device).

   debug software restart process user-id

4. From CLI run show management-clients to ensure that all processes have started successfully.

> show management-clients
              Client PRI    State Progress
-------------------------------------------------------------------------
            ha_agent  25     init        0
              sslmgr  10     init        0
               authd  10     init        0
             cryptod  10     init        0
              dagger  10     init        0    (op cmds only)
                cord  10     init        0
                logd  10     init        0    (op cmds only)
             reportd  10     init        0    (op cmds only)
             useridd  10     init        0
        distributord  10     init        0
                iotd  10     init        0
Overall status: init. Progress: 0
Warnings:
Errors: