Out of Memory (OOM) condition due to many HTTPD processes

• High memory utilization
• Some processes are killed due to "out of memory" conditions.
• There are a lot of HTTPD processes running.
• HTTPD processes are the highest swap memory consumers.

• Palo Alto Firewalls
• Supported PAN-OS
• API Calls

• Due to a high API call rate, the web_backend process opens a lot of HTTPD child processes.
• Although Palo Alto Firewall is able to manage until 25 HTTPD child processes, the amount of memory consumption per HTTPD can impact the behavior.
• The suggestion is to run less than 5 API calls per second.

1. Reduce the number of API queries.
2. Restart the web-backend process to kill every child HTTPD process and free swap memory:                                  

    > debug software restart process web-backend 

Check the memory swap available and swap consumption in mp-monitor.log to confirm HTTPD are the highest consumer and the swap is low or near 0, for example:

2022-08-11 13:13:55.011 +0000 --- swapusage Name PID VmSwap VmRSS 
httpd 29367 189504 kB 1360 kB 
httpd 9974 186332 kB 1512 kB 
httpd 9822 185948 kB 1636 kB 
httpd 27996 185580 kB 1460 kB 
httpd 30251 184332 kB 1388 kB 
--- Truncated output ---- 
2022-08-11 13:14:14.269 +0000 --- memory Last 180 seconds 
Type Free (kB) min (kB) Total (kB) 
Mem 170452 66476 3849876 
Swap 0 0 3056660 

- Check nginx/access.log to confirm the API queries is at a higher rate than recommended:

::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"

After restarting the web-backend process,  check the swap memory and HTTPD to confirm back to normal:

2022-08-11 13:15:55.022 +0000 --- swapusage
Name PID VmSwap VmRSS
useridd 5059 69200 kB 250552 kB
authd 5436 54964 kB 12844 kB
sslmgr 5431 49300 kB 8936 kB
dnsproxyd 5433 44892 kB 18136 kB
routed 5435 25204 kB 15144 kB
varrcvr 5427 24300 kB 25660 kB
---- Truncated output ----

2022-08-11 13:20:14.310 +0000 --- memory
Last 180 seconds
Type Free (kB) min (kB) Total (kB)
Mem 139156 76032 3849876
Swap 2514208 2514200 3056660