High memory consumption in kubernetes over time (OOM )
3940
Created On 10/24/22 06:52 AM - Last Modified 10/31/24 14:26 PM
Symptom
Over time the defenders will cause Out of Memory errors in Kubernetes due to memory leak
of certain ubuntu kernel.
Environment
- Self hosted kubernetes version 1.24.3
- Docker version 20.10.7
- Defenders v22.06.197
Cause
Reproduce the issue in Ubuntu 18.04.06 running kernel 4.15.0-177-generic
Steps to reproduce:
-
Enable CNNF for Hosts
-
Run the following in a host with a running Defender
while true; do wget -q -O- http://127.0.0.1; done -
View TCP slab size increase in the host
watch -n 0.1 "cat /proc/slabinfo | grep TCP"
Resolution
Upgrading kernel to the recommended HWE kernel version (5.4.0-124-generic) as described here Kernel/LTSEnablementStack - Ubuntu Wiki resolved the issue.