OOM 由于许多HTTPD流程

4068

Created On 08/12/22 12:47 PM - Last Modified 06/09/25 23:01 PM

Symptom

内存利用率高
某些进程由于“内存不足”情况而被终止。
有很多的HTTPD进程运行。
HTTPD 进程是最高的交换内存消费者。

Environment

模型： PA-3020
PANOS: 9.1.12-h3

Cause

由于高API调用率，web_backend进程开启了很多HTTPD子进程。虽然NGFW能够管理到 25HTTPD子进程，每个进程的内存消耗量HTTPD可以影响行为。
建议运行少于 5API每秒调用。

Resolution

减少数量API查询。
重新启动网络后端进程以杀死每个孩子HTTPD进程和空闲交换内存：
```
 > debug software restart process web-backend 
```

Additional Information

- 检查 mp-monitor.log 中可用的内存交换和交换消耗以确认HTTPD是最高消费者并且掉期很低或接近于 0，例如：

2022-08-11 13:13:55.011 +0000 --- swapusage Name PID VmSwap VmRSS 
httpd 29367 189504 kB 1360 kB 
httpd 9974 186332 kB 1512 kB 
httpd 9822 185948 kB 1636 kB 
httpd 27996 185580 kB 1460 kB 
httpd 30251 184332 kB 1388 kB 
--- Truncated output ---- 
2022-08-11 13:14:14.269 +0000 --- memory Last 180 seconds 
Type Free (kB) min (kB) Total (kB) 
Mem 170452 66476 3849876 
Swap 0 0 3056660

- 检查 nginx/access.log 以确认API查询速度高于建议的速度：

::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"
::ffff:10.26.129.53 - - [10/Aug/2022:02:54:13 +0000] "POST /api/index.php" 400 148 "RestSharp/104.4.0.0"

- 重启 web-backend 进程后，检查交换内存和HTTPD确认恢复正常：