AIOps Alert "Process Memory Depletion - Device Server"

• Alert from AIOps regarding process memory depletion for devsrvr

• PAN-OS
• AIOps

If you receive this Alert, it is recommended to collect the following Troubleshooting Data below and open a Support Case. After data is collected, considering following the Mitigation Steps to bring down the memory usage of the devsrvr process till Support can analyze the data. 

Troubleshooting Data

1. Collect Tech Support File  (GUI: Device > Support  Click Generate Tech Support File)

2. Generate a trace file using the following CLI command

debug software trace device-server

Note: debug info will be generated in the next few seconds to a minute depending the available resource of the firewall. Also remember to collect the session logs to be generated by the above command.

3. Generate a core file using following command

debug software core device-server

4. Collect the Device State (GUI: Device>Setup>Operations- Export: Export device state)
5. Export the core file  (HOW TO EXPORT CORE FILES FROM A PALO ALTO NETWORKS DEVICE)
6. Gather data below from AIOps
   1. Check the dates indicated by AIOps as to when the memory depletion started.
   2. Check if there were any config modifications, PANOS upgrades/downgrades, or any other changes performed around the time that might have triggered this behavior.
7. (Optional) If performing Mitigation Steps below, collect another Tech Support File after completing steps
8. Open a case with the above data.
    

Mitigation Steps

Potential Impact of restart the process:

• Following functionality handled by devsrvr daemon will not be available during process restart
  • Config push to dataplane
  • URL filtering request response
  • Other miscellaneous communication with dataplane

Option 1 (Standalone Device)

Option 2 (Device in Active/Passive HA)

1. Disable "Preemptive" mode (GUI: Device > High Availability > General > Election Settings: Uncheck Preemptive )
2. Failover to the passive device (From Active Device: Device > High Availability > Operations > Click Suspend local device)
3. Restart device-server from the CLI from the now Suspended device (debug software restart process device-server)
4. From CLI run show management-clients to ensure that all processes have started succesfully.

> show management-clients
              Client PRI    State Progress
-------------------------------------------------------------------------
            ha_agent  25     init        0
              sslmgr  10     init        0
               authd  10     init        0
             cryptod  10     init        0
              dagger  10     init        0    (op cmds only)
                cord  10     init        0
                logd  10     init        0    (op cmds only)
             reportd  10     init        0    (op cmds only)
             useridd  10     init        0
        distributord  10     init        0
                iotd  10     init        0
Overall status: init. Progress: 0
Warnings:
Errors: