Cortex Cloud: KSPM connector - No jobs are getting created due to taints
Symptom
- Run command "kubectl get all" to check if any pod is in pending state.
- Then run "kubectl describe pods -n <namespace>" to check the reason why the pods are in a pending state.
The following error should be seen under events section of the output from above command:
[FailedScheduling] 0/5 nodes are available: 1 node(s) had untolerated taint {schedule: documenthub}, 1 node(s) had untolerated taint {schedule: smartview}, 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
Environment
- Cortex Cloud
- Cortex XSIAM
- KSPM
- Kubernetes Connector
Cause
This issue could be related to node scheduling restrictions in the cluster.
Based on the behavior, the Cortex pods are unable to be placed on the available Linux nodes because the node requirements and pod scheduling rules do not currently align. As a result, the pods remain in a Pending state, which causes the cluster to appear with a warning status in the console.
Resolution
It would be worth reviewing the node configuration and pod scheduling settings to ensure that the Cortex pods are allowed to run on the available worker nodes.
Like the realtime agent we wish to apply our workload tolerations for multiple taints. The easiest way to do it is to update the client's values.yaml that they installed from the UI Modify it once, and then the taints issue should be resolved.
They need to add a tolerations section under the global section:
global:
distribution:
...
namespace: panw
tolerations:
- operator: "Exists"
effect: "NoSchedule"
- operator: "Exists"
effect: "PreferNoSchedule"
- operator: "Exists"
effect: "NoExecute"
...
konnector-template:
For Example: