PA-500 High Management CPU and poor performance with high logging
20043
Created On 01/16/19 09:58 AM - Last Modified 06/24/20 03:17 AM
Symptom
PA 500 runs very slow and it was possible to notice that Management CPU is between 98-100% almost all the time.
MP CPU Spikes up very frequently and "logrcvr", "mgmtsrvr" and "genindex.sh" appears to be the offending processes:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3265 20 0 1095m 282m 4780 S 35.4 14.2 93995:05 logrcvr
14961 20 0 5260 1560 1232 R 16.9 0.1 0:51.34 genindex.sh
3265 20 0 1095m 205m 3264 S 76.7 10.4 94000:37 logrcvr
3217 20 0 1198m 315m 7280 S 14.7 15.9 38133:24 mgmtsrvr
High IOWait process was seen as well, however disk space utilization was fine:
PU USAGE DISTRIBUTION Avg Max Max Date
---------------------------------------------
User 54% 54.3% 01-08-06:15
System 13% 13.9% 01-05-12:11
Nice 2% 2.6% 01-09-02:36
Idle 25% 26.6% 01-04-11:29
IOWait 1% 1.1% 01-04-11:29
Due to poor management plane performance commits and other tasks may be affected and get stuck, and using commit force should help in order to go through.
Environment
- Palo Alto PA-500 Firewall.
- PAN-OS 8.0 and 8.1,
- High rate of logging configured.
Cause
When average IOWait is greater than 1%, it is likely causing processes to run slower than normal and it is important to reduce the amount of logging.
Resolution
- Set the Max days for Logs, In this case the max days was configured to value '90' using this article, which automatically purged all the logs older than 90 days.
- In this case device was also upgraded to latest stable release. These both actions improved the performance remarkably.