AIOps Alert "Process Memory Depletion

AIOps Alert "Process Memory Depletion - Configd"

8136

Created On 02/15/22 00:51 AM - Last Modified 04/12/24 21:45 PM

PAN-OS

Panorama

Symptom

Alert from AIOps regarding process memory depletion for "configd"

Environment

Palo Alto Firewalls
Supported PAN-OS
AIOps

Cause

A memory depletion in the configd process has been detected.

Resolution

Collect the following Troubleshooting Data and open a Support Case.
After data is collected, Follow the Mitigation Steps to bring down the memory usage of configd till Support can analyze the data.

Troubleshooting Data:

Collect Tech Support File (GUI: Device > Support Click Generate Tech Support File)
Generate a trace file using the following CLI command

debug software trace configd

Generate a core file using following command

debug software core configd

Collect the Device State (GUI: Device>Setup>Operations- Export: Export device state)
Export the core file (How To Export Core Files From A Palo Alto Networks Device)
Gather data below from AIOps
1. Check the dates indicated by AIOps as to when the memory depletion started.
2. Check if there were any config modifications, PANOS upgrades/downgrades, or any other changes performed around the time that might have triggered this behavior.
(Optional) If performing Mitigation Steps below, collect another Tech Support File after completing steps
Open a case with the above data.

Mitigation Steps:

Till the issue is resolved, you can restart the configd process that is consuming excessive memory.
Prior to PAN-OS 10.1, configd is only available on Panorama.
Starting from 10.1, configd is also available on firewall as well

NOTE: Recommended to be performed in a maintenance window. Configuration related operation will not be available during daemon restart, including but not limited to commit, revert config, push config to firewall, save config to disk and policy optimizer will also not be available.

Option 1 (Standalone Device)

Save and export the candidate config.
Save and export the current configuration
Perform a full commit
From CLI to restart the process run:
```
debug software restart process configd 
```
Note: This will cause the loss of access to CLI and GUI for few minutes.
(For devices on 10.0.X or 10.1.X) Restart the device-server debug software restart process device-server

Option 2 (Device in Active/Passive HA)

Disable "Preemptive" mode (GUI: Device > High Availability > General > Election Settings: Uncheck Preemptive )
Failover to the passive device (From Active Device: Device > High Availability > Operations > Click Suspend local device)
Restart configd from the CLI from the now Suspended device (debug software restart process configd)
From CLI run show management-clients to ensure that all processes have started succesfully.

> show management-clients
              Client PRI    State Progress
-------------------------------------------------------------------------
            ha_agent  25     init        0
              sslmgr  10     init        0
               authd  10     init        0
             cryptod  10     init        0
              dagger  10     init        0    (op cmds only)
                cord  10     init        0
                logd  10     init        0    (op cmds only)
             reportd  10     init        0    (op cmds only)
             useridd  10     init        0
        distributord  10     init        0
                iotd  10     init        0
Overall status: init. Progress: 0
Warnings:
Errors:

NOTE: restarting the process will temporarily mitigate the issue and the same issue may come back sometimes later.

Additional Information

PAN-169173 - Fixed an issue where, if you continuously performed partial commits of a configuration with a high number of Dynamic Address Groups, Panorama became unresponsive and commits were slower than expected. (Fixed on 9.1.12)
PAN-171159 - Fixed a memory leak on the configd process on Panorama caused during multi-clone operations for rules. (Fixed on 9.1.12, 10.0.9)