PA-500 High Management CPU and poor performance with high logging

PA-500 High Management CPU and poor performance with high logging

5062
Created On 01/16/19 09:58 AM - Last Modified 06/24/20 03:17 AM


Symptom
PA 500 runs very slow and it was possible to  notice that Management CPU is between 98-100% almost all the time. 

MP CPU Spikes up very frequently and "logrcvr", "mgmtsrvr" and "genindex.sh" appears to be the offending processes:
 
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
3265 20 0 1095m 282m 4780 S 35.4 14.2 93995:05 logrcvr 
14961 20 0 5260 1560 1232 R 16.9 0.1 0:51.34 genindex.sh 

3265 20 0 1095m 205m 3264 S 76.7 10.4 94000:37 logrcvr 
3217 20 0 1198m 315m 7280 S 14.7 15.9 38133:24 mgmtsrvr

High IOWait process was seen as well, however disk space utilization was fine:
 
PU USAGE DISTRIBUTION Avg Max Max Date
---------------------------------------------
User 54% 54.3% 01-08-06:15
System 13% 13.9% 01-05-12:11
Nice 2% 2.6% 01-09-02:36
Idle 25% 26.6% 01-04-11:29
IOWait 1% 1.1% 01-04-11:29

Due to poor management plane performance commits and other tasks may be affected and get stuck, and using commit force should help in order to go through.
 


Environment
  • Palo Alto PA-500 Firewall.
  • PAN-OS 8.0 and 8.1,
  • High rate of logging configured.


Cause
When average IOWait is greater than 1%, it is likely causing processes to run slower than normal and it is important to reduce the amount of logging.

Resolution
  1. Set the Max days for Logs, In this case the max days was configured to value '90' using this article, which automatically purged all the logs older than 90 days.
  2. In this case device was also upgraded to latest stable release. These both actions improved the performance remarkably.


Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CmcSCAS&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments