How To Troubleshoot High Management Plane Memory Usage

• PAN-OS
• AIOps

Finding possible causes for peaks in MP Memory Usage

If the memory growth peaks and then falls, check if the peaks in memory usage align with any of the following events:

1. Commit operations.
2. High concurrent rate of REST API calls.
3. Ad-hoc reports or scheduled reports being run, or predefined reports being enabled. If scheduled reports or predefined reports are configured, then memory usage can increase every hour on the hour or at 2:02 AM device time. 

Steps to reduce MP Memory Usage using general recommendation

1. Check your device available memory using:

   > show system resources
   top - 13:55:32 up 16 days,  1:58,  1 user,  load average: 4.69, 4.52, 4.73
   Tasks: 169 total,   5 running, 164 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 89.3 us,  9.5 sy,  0.0 ni,  1.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
   MiB Mem :   5030.6 total,    118.0 free,   2443.5 used,   2469.1 buff/cache
   MiB Swap:   4001.9 total,   4001.9 free,      0.0 used.   2006.7 avail Mem

   Look for avail Mem to know how much MP memory is still available on your device.
2. If High memory usage is aligned with a Commit operation, 
   1. Consider reducing the size of the config to reduce memory footprint. 
   2. Reduce the frequency of Commit operations. 
3. If there is a high concurrent rate of REST API calls, consider spacing out the REST API calls.
4. Report generation can introduce memory load on the device. 
   1. Avoid running ad-hoc reports at the time of high memory usage.
   2. Avoid scheduled custom reports that cover a long duration period. Whenever possible, generate the reports on Panorama instead of the NGFW.
   3. For ACC reports, avoid querying over a long duration period.
   4. By default, all pre-defined reports are enabled, while many of them may not be useful to the organization. Disable unnecessary predefined reports on the device from Device > Setup > Logging and Reporting Settings.
   5. Reduce the number of custom reports. On Panoramas, specify “Panorama Data” as the source of the report. 
   6. During times of high load, do not keep ACC or log monitoring UI tab open and set to auto refresh. This queries the log database and recompiles the output periodically.

Steps to reduce MP Memory Usage using recommendation specific to the contributing factor

1. Check which process is currently using the highest percentage of memory:

   > show system resources follow

   Press shift+M to sort by memory usage.
2. If suspecting that one the following processes is having a memory depletion then refer to the relative article:
   1. Management-server : Process Memory Depletion - Management Server.
   2. Configd: Process Memory Depletion - Configd.
   3. User-ID: Process Memory Depletion - User Id.
   4. Device-server: Process Memory Depletion - Device Server.
3. If it's not possible to reduce the activities on the device that are known to use high MP memory, then consider an upgrade to a device with a larger memory capacity.

• Tech Support File
• CLI output from the command show system resources (run 3 times at 30 seconds interval). 

Check How To Troubleshoot High Management Plane CPU Usage for more information on how to reduce the load on Management Plane.
If your device is a Panorama VM then check how to Increase CPUs and Memory on the Panorama Virtual Appliance.
If your device is a PA-VM then make sure that you have allocated the necessary resources to it, refer to VM-Series System Requirements and the option to increase the memory (RAM) of your FW.
For VM-Flex post 10.2 refer to Maximum Limits Based on Tier and Memory
For VM-Flex pre 10.2 refer to Maximum Limits Based on Tier and Memory.