How To Troubleshoot High Management Plane Memory Usage

How To Troubleshoot High Management Plane Memory Usage

35359
Created On 02/17/22 23:09 PM - Last Modified 11/29/23 17:42 PM


Objective


Below is general guidance on troubleshooting a PAN-OS device that is hitting high Management Plane memory usage.

Environment


  • PAN-OS
  • AIOps


Procedure


Finding possible causes for peaks in MP Memory Usage

If the memory growth peaks and then falls, check if the peaks in memory usage align with any of the following events:

  1. Commit operations.
  2. High concurrent rate of REST API calls.
  3. Ad-hoc reports or scheduled reports being run, or predefined reports being enabled. If scheduled reports or predefined reports are configured, then memory usage can increase every hour on the hour or at 2:02 AM device time. 

Steps to reduce MP Memory Usage using general recommendation

If the memory has an upwards trend and is growing continuously then this could be a software defect where memory is not being freed. 
  1. Check your device available memory using:
    > show system resources
    top - 13:55:32 up 16 days,  1:58,  1 user,  load average: 4.69, 4.52, 4.73
    Tasks: 169 total,   5 running, 164 sleeping,   0 stopped,   0 zombie
    %Cpu(s): 89.3 us,  9.5 sy,  0.0 ni,  1.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    MiB Mem :   5030.6 total,    118.0 free,   2443.5 used,   2469.1 buff/cache
    MiB Swap:   4001.9 total,   4001.9 free,      0.0 used.   2006.7 avail Mem
    Look for avail Mem to know how much MP memory is still available on your device.
  2. If High memory usage is aligned with a Commit operation, 
    1. Consider reducing the size of the config to reduce memory footprint. 
    2. Reduce the frequency of Commit operations. 
  3. If there is a high concurrent rate of REST API calls, consider spacing out the REST API calls.
  4. Report generation can introduce memory load on the device. 
    1. Avoid running ad-hoc reports at the time of high memory usage.
    2. Avoid scheduled custom reports that cover a long duration period. Whenever possible, generate the reports on Panorama instead of the NGFW.
    3. For ACC reports, avoid querying over a long duration period.
    4. By default, all pre-defined reports are enabled, while many of them may not be useful to the organization. Disable unnecessary predefined reports on the device from Device > Setup > Logging and Reporting Settings.
    5. Reduce the number of custom reports. On Panoramas, specify “Panorama Data” as the source of the report. 
    6. During times of high load, do not keep ACC or log monitoring UI tab open and set to auto refresh. This queries the log database and recompiles the output periodically.

Steps to reduce MP Memory Usage using recommendation specific to the contributing factor

  1. Check which process is currently using the highest percentage of memory:
    > show system resources follow
    Press shift+M to sort by memory usage.
  2. If suspecting that one the following processes is having a memory depletion then refer to the relative article:
    1. Management-server : Process Memory Depletion - Management Server.
    2. Configd: Process Memory Depletion - Configd.
    3. User-ID: Process Memory Depletion - User Id.
    4. Device-server: Process Memory Depletion - Device Server.
  3. If it's not possible to reduce the activities on the device that are known to use high MP memory, then consider an upgrade to a device with a larger memory capacity.
Note: As last resort and during off hours or a maintenance window a reboot of your device (FW or Panorama) will release the used memory and reduce the MP memory usage. If the high memory usage issue reoccurs frequently even after following the above suggestions then provide the following information and open a support case: 
  • Tech Support File
  • CLI output from the command show system resources (run 3 times at 30 seconds interval). 


Additional Information


Check How To Troubleshoot High Management Plane CPU Usage for more information on how to reduce the load on Management Plane.
If your device is a Panorama VM then check how to Increase CPUs and Memory on the Panorama Virtual Appliance.
If your device is a PA-VM then make sure that you have allocated the necessary resources to it, refer to VM-Series System Requirements and the option to increase the memory (RAM) of your FW.
For VM-Flex post 10.2 refer to Maximum Limits Based on Tier and Memory
For VM-Flex pre 10.2 refer to Maximum Limits Based on Tier and Memory.



Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000oNDmCAM&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Choose Language