Prisma Cloud Compute: Validating webhook/Admission webhook failing with error "failed to verify certificate: x509: certificate has expired or is not yet valid" and causing pods to fail
Symptom
While making use of the validating webhook in Prisma Cloud, the following symptoms are experienced:
- Example Error (in CloudWatch logs for example):
-
W0307 16:22:43.029125 11 dispatcher.go:205] Failed calling webhook, failing open validating-webhook.twistlock.com: failed calling webhook "validating-webhook.twistlock.com": failed to call webhook: Post "https://defender.prismacloud.svc:443/l2gojlonml280tsnaja9cmb9kg2z?timeout=10s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-03-07T16:22:43Z is after 2025-02-26T08:16:00Z
-
- Defender pods are going down due to a CrashLoopBackOff error
- When examining the defender-ca.pem file in a fresh defender daemonset downloaded from the console, it is already expired.
- To examine the file, run the following command replacing "ENCODED-VALUE" with the string in the YAML file after defender-ca.pem:
-
echo "<ENCODED-VALUE>" | base64 -d | openssl x509 -text -noout
- take note of the "Not After" value
-
- To examine the file, run the following command replacing "ENCODED-VALUE" with the string in the YAML file after defender-ca.pem:
Environment
-
Prisma Cloud Enterprise (SaaS) v34
-
Prisma Cloud Compute (Self host) v34
Cause
This issue is caused by an expired defender certificate in the SaaS tenant. Although SaaS does not make use of the defender certificate for defender-to-console communication, this certificate is used when implementing the admission webhook in kubernetes.
This means that, although there will be no impact on defender-to-console communication due to this expired certificate, the validating webhook functionality will be affected.
Resolution
The defender certificate will need to be updated from the SaaS backend. Please reach out to support to work towards getting that certificate properly updated.
Additional Information
Documentation on the validating webhook here
Documentation on certificates here
- Note that only Compute edition requires the use of defender certificates for console-to-defender communication. However, admission control requires the cert regardless of whether SaaS or self-hosted is being used.