Prisma Cloud Compute : OpenShift Nodes transitioning to 'NotReady' state with the Block Rule in Policy
10140
Created On 04/04/23 02:33 AM - Last Modified 04/07/23 07:18 AM
Symptom
-
OpenShift Nodes transitioning to 'NotReady' state with the Block Rule in Policy
Environment
- Prisma Cloud Compute (Self-Hosted and SaaS)
- OpenShift
Cause
Container Runtime (CRIO)
- When the Policy with a Block Rule is applied or removed, a Container Runtime (CRIO) restart may occur because the Defenders in scope swap the *runc* binary on the nodes where they are deployed to reload the runtime configuration
- It may take a few minutes for CRIO (and Kublet) services to restart after reloading the configuration
- During this time, Node may switch to the 'NotReady' state when CRIO or Kubelet is unavailable
- Meanwhile, the worker node cannot run/schedule workloads till it comes back to the Ready state
- NotReady state is expected when Prisma Cloud performs runc swap in the following scenarios:
- Defender stop
- Defender start
- The first blocking rule is added
- The last blocking rule is removed
- Defender upgrade (as it causes defender restart)
Important to Note
- This NotReady state is noticed ONLY with the CRIO Runtime Environment because of the restriction with OpenShift in intercepting the calls to runc
- This is the current design with CRIO runtime
Resolution
- A restart of the CRIO runtime may be needed
- As the 'NotReady' state transition happens during certain scenarios, this issue can be avoided by refraining from making following changes:
- Adding the first blocking rule
- Removing the last blocking rule
- Restarting / Upgrading Defender
- Toggling the Secret Injection option
Workaround
- Create a dummy Block Rule in the Policy and apply it to the Cluster to transition the Nodes to the 'NotReady' state once.
- Post this, make the necessary changes in the Policy Rules as and when required
Additional Information
Blocking Rule Policies
This includes the following:
- Defend => Compliance => Trusted images - any blocking rule
- Defend => Compliance => Containers and images => Deployed - any blocking rule
- Defend => Compliance => Hosts => Hosts - any blocking rule
- Defend => Access => Secrets - any rule that exists
- Defend => Vulnerabilities => Images => Deployed
- Any rule that has a blocking effect
- Any rule has a CVE set in exceptions with a block effect
- Any rule has a tag set in exceptions with a block effect
- Any rule has a block threshold