Prisma Cloud Compute - K8 version upgrade on AKS clusters fail when defenders are installed
Symptom
With Twistlock defenders deployed to AKS clusters, the Kubernetes version upgrade tasks fail with below or similar errors.
Multiple VM extensions failed to be provisioned on the VM. The Extensions failed due to the errors: [Extension Name: 'AKSLinuxExtension' Error Message: Provisioning of VM extension AKSLinuxExtension has timed out. Extension provisioning has taken too long to complete. The extension last reported "Plugin enabled".
More information on troubleshooting is available at https://aka.ms/vmextensionlinuxtroubleshoot]
[Extension Name: 'vmssCSE' Error Message: Provisioning of VM extension vmssCSE has timed out. Extension provisioning has taken too long to complete. The extension last reported "Plugin enabled".
Environment
- Prisma Cloud Runtime Security
- Prisma Cloud Compute Edition
- Azure AKS clusters with Prisma Cloud Compute Defenders
Cause
The Radar > Radar Settings > Container Network Monitoring feature redirects the traffic to the pods to a particular queue called NFqueue.
When there are operations such as K8 version upgrade, it creates a sudden burst of Network traffic and that creates a bottle neck in this queue.
As a results there is high latency and packet loss which results in communication issues in the traffic.
Thus as a result the operations fail.
Resolution
The "Container Network Monitoring" is a deprecated feature which is not used for any Security Enforcement on Prisma Cloud Twistlock Defenders.
The only use for it as of now is to get the connection lines seen under Radar > Containers where connectivity between the pods are represented graphically.
Since this has no other realtime use, it is advised to disable the Container Network Monitoring and Host Network Monitoring.
This will stop the packets getting routed to the NFqueue on the cluster nodes and thus the bottleneck will not be created.