AIOps Alert "Process Memory Depletion - Management Server"

• Alert from AIOps regarding process memory depletion for "mgmtsrvr"

• PAN-OS
• AIOps Alert

A memory depletion in the "mgmtsrvr" process was found.

If you receive this Alert, it is recommended to collect the following Troubleshooting Data below and open a Support Case. After data is collected, considering following the Mitigation Steps to bring down the memory usage of the mgmtsrvr process till Support can analyze the data. 

Troubleshooting Data

1. Collect Tech Support File  (GUI: Device > Support  Click Generate Tech Support File)

2. Generate a trace file using the following CLI command

debug software trace management-server

3. Generate a core file using following command

debug software core management-server

4. Collect the Device State (GUI: Device>Setup>Operations- Export: Export device state)
5. Export the core file  (HOW TO EXPORT CORE FILES FROM A PALO ALTO NETWORKS DEVICE)
6. Gather data below from AIOps
   1. Check the dates indicated by AIOps as to when the memory depletion started.
   2. Check if there were any config modifications, PANOS upgrades/downgrades, or any other changes performed around the time that might have triggered this behavior.
7. (Optional) If performing Mitigation Steps below, collect another Tech Support File after completing steps
8. Open a case with the above data.
    

Mitigation Steps

Potential Impact of restart the process:

• After PAN-OS 10.1, mgmtsrvr on firewall serves as a message dispatcher(Similar to panorama), Backend daemons such as configd, distributord, iotd, logrcvr, pl-dlp_agent, reportd, and useridd will be restarted.
• Prior to PAN-OS 10.1, following functionality on firewall will not be available during the process restart, 
  • UI and CLI access
  • HA Sync
  • Panorama push
  • Dynamic Updates
• The mgmtsrvr on Panorama serves as a message dispatcher, All backend daemons are dependent on mgmtsrvr, when mgmtsrvr is restarted, all backend daemons will be restarted. Single access point to all the firewalls via proxy session on GUI.

Option 1 (Standalone Device)

1. Save and export the candidate config.
2. Save and export the current configuration.
3. Perform a full commit
4. Restart the management-server process using below command

debug software restart process management-server

5. (For PAN-OS 10.0.X or 10.1.X) Restart the device-server to ensure that the commits go through without a problem. 

debug software restart process device-server

Option 2 (Device in Active/Passive HA)

1. Disable "Preemptive" mode (GUI: Device > High Availability > General > Election Settings: Uncheck Preemptive )
2. Failover to the passive device (From Active Device: Device > High Availability > Operations > Click Suspend local device)
3. Restart the management-server from the CLI from the now Suspended device (debug software restart process management-server)
4.  (For PAN-OS 10.0.X or 10.1.X) Restart the device-server to ensure that the commits go through without a problem. (debug software restart process device-server)
5. From CLI run show management-clients to ensure that all processes have started successfully.

> show management-clients
              Client PRI    State Progress
-------------------------------------------------------------------------
            ha_agent  25     init        0
              sslmgr  10     init        0
               authd  10     init        0
             cryptod  10     init        0
              dagger  10     init        0    (op cmds only)
                cord  10     init        0
                logd  10     init        0    (op cmds only)
             reportd  10     init        0    (op cmds only)
             useridd  10     init        0
        distributord  10     init        0
                iotd  10     init        0
Overall status: init. Progress: 0
Warnings:
Errors: