The NGFW pod status shows frequent container restarts

4094

Created On 02/01/23 01:25 AM - Last Modified 07/12/23 02:05 AM

10.2

10.1

11.0

CN-Series

Symptom

The restart count in the kubectl get pods -n kube-system -l app=ngfw output increases very frequently.

$ kubectl get pods -n kube-system -l app=pan-ngfw
NAME                                       READY   STATUS    RESTARTS   AGE
pan-ngfw-dep-6454cff886-7k75f              1/1     Running   88         12d
pan-ngfw-dep-6454cff886-cm6x5              1/1     Running   234        12d

Events from the kubectl describe pod for the NGFW pod shows container being killed and re-created

$ kubectl describe pods pan-ngfw-dep-6454cff886-cm6x5 -n kube-system
...
Events:
  Type     Reason          Age                    From     Message
  ----     ------          ----                   ----     -------
  Warning  Unhealthy       40m (x86 over 4d18h)   kubelet  Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: process_linux.go:130: executing setns process caused: exit status 1: unknown
  Normal   Pulled          40m                    kubelet  Successfully pulled image "kuberepo.firstcdn.com/mirantis/panos_cn_ngfw:10.1.5-h2"" in 92.318889ms
  Warning  Unhealthy       40m                    kubelet  Readiness probe errored: rpc error: code = Unknown desc = container not running (9abcdda5811ecd1c054d2f36894233aa051ba97c861cdf25d21483af0dc317c7)
  Normal   Created         40m (x230 over 4d18h)  kubelet  Created container pan-ngfw-container
  Normal   Started         40m (x230 over 4d18h)  kubelet  Started container pan-ngfw-container
  Normal   SandboxChanged  19m (x231 over 4d18h)  kubelet  Pod sandbox changed, it will be killed and re-created.
  Normal   Killing         19m (x231 over 4d18h)  kubelet  Stopping container pan-ngfw-container
  Normal   Pulling         19m (x231 over 4d18h)  kubelet  Pulling image "kuberepo.firstcdn.com/mirantis/panos_cn_ngfw:10.1.5-h2""
  Warning  Unhealthy       19m                    kubelet  Readiness probe errored: rpc error: code = Unknown desc = container not running (38cb9a5f382645872f8ca68ad59a4698c13b6f520451b3d408a5695c97634af4)

Environment

Platform	CN-Series
Deployment	CNv2
PAN-OS Versions	10.1 +

Cause

The above event logs could be due to the lack of resources (CPU/memory) allocated to the NGFW pod.

Resolution

Check the CPU and memory allocation for the NGFW pods and ensure that it meets the minimum requirements as per the below table.

$ kubectl describe pods pan-ngfw-dep-6454cff886-cm6x5 -n kube-system
...  
Containers:
  pan-ngfw-container:
    ...
    Ready:          True
    Restart Count:  234
    Limits:
      cpu:     1
      memory:  4Gi
    Requests:
      cpu:      1
      memory:   4Gi

Additional Information