Prisma Cloud Compute: How to troubleshoot OOM killer container restarts for Docker containers?

• Review the output for the following Docker commands:
  - docker ps
  - docker inspect <name-of-the-container>
  - docker stats <name-of-the-container>

The objective is to determine if the Docker container has restarted, if yes, is OOM killer the cause of container restarts.

• Linux/macOS machine
• Docker runtime

• docker ps
  
  1. This command will display the running containers in your environment.
  2. The STATUS column must be checked to determine for how long has the container 
       been healthy and running.
  
     
  3. The running time will prove that the container was maybe restarted/recreated 2 hours      (in this case)
  
  
• docker inspect <name-of-the-container>
  
  1. The snippet shown below has been snipped from the output of this command, it                confirms that OOM killer was the cause of the container being taken down/killed.
      
      The container was last killed on FinishedAt": "2025-05-19T13:42:31.566652718Z       by the OOM killer process.
  
     
  
  2. There is one more snippet or rather a single line in the output of this command which      mentions that "RestartCount": 41.
  
  
• docker stats <name-of-the-container>
  
  1. This command will display the cgroup limit set for the container, which is most likely          being exceeded causing the restarts.
  
  

Resolution: 
The docker update command is a powerful utility that allows you to modify certain resource limits and configuration options for running containers without needing to stop and recreate them. This can be very useful for dynamically adjusting resource allocation based on workload.

Here's how docker update works, especially in relation to resource limits:

• Memory Limits:

  • --memory or -m: Sets the maximum amount of memory the container can use (e.g., 512m, 2g). If a container exceeds this limit, it may be OOMKilled (Out Of Memory Killed) by the system.
  • --memory-swap: The total amount of memory and swap the container can use. If you set --memory but not --memory-swap, the container can use swap space up to twice the memory limit (if host swap is enabled). To prevent a container from using any swap, set --memory-swap to the same value as --memory
• In this use case, 4Gi memory is not enough for the container hence, container restarts are observed. This can be changed by using the following command:

docker update --memory=6g <name-of-the-container>

Please note that:

• Memory Swap (--memory-swap): Allows you to update the total memory the container can use (RAM + Swap).
  • If --memory-swap is set to the same value as --memory, the container cannot use swap.
  • If --memory-swap is -1, the container can use unlimited swap (up to the host's available swap).
  • If --memory is set and --memory-swap is not, the container can use swap up to the same amount as the memory limit (total 2x the memory limit)

The OOM messages can be checked by using the following command:

sudo dmesg -T | grep -i "oom-killer" or sudo journalctl -k | grep -i "oom-killer"