Guide to configuring Kubernetes auditing on Prisma SaaS.

• Refer to the Kubernates Audting Documentation
• Navigate to the "Configure the API server" to refer to the steps provided configure audit webhook backend
• On the Compute Console, navigate to Defend > Access > Kubernetes > Go to settings
• Notice that on the Console, after clicking on the "Go to settings" option to find the webhook url required for the configuration, the settings are empty and with no webhook url present there. Refer to the image below to relate to explanation:

• Prisma Cloud Compute (SaaS deployment)
• Kubernetes (Google Kubernetes Engine)
• Kubernetes (Elastic Kubernetes Service and Azure Kubernetes Service)

• According to the release notes for Prisma Cloud Compute version 21.08.514
• Removes support for Kubernetes dynamic audit configuration, which was deprecated in Kubernetes 1.19. Dynamic audit configuration was previously used in Prisma Cloud Kubernetes auditing. Prisma Cloud Kubernetes auditing now uses the regular webhook backend .

• Path to Console’ URL (Compute / Manage / System / Utilities)
• export console= API Token (Compute / Manage / System / Utilities)
• export token= <API Token>
• The following utilities need to be installed:
  • curl
  • jq
  • openssl
  • kubectl
  • Firefox
  • A Supported method for updating kube-apiserver manifests. This is typically done by editing /etc/kubernetes/manifests/kube-apiserver.manifest on master nodes but may be Linux/K8s distribution dependent.

TASK FLOW:

1. Verify Kubernetes Auditing is enabled in Prisma Cloud Compute
2. Generate Webhook URL
3. Download Certificate Chain and convert to base64
4. Generate audit-webhook.yaml
5. Add webhook and policy configuration to all master nodes
6. Update kube-apiserver manifests on all master nodes
7. Testing to verify if the setup is working properly

Step 1: Verify Kubernetes Auditing is enabled in Prisma Cloud Compute.
Browse to Compute / Access / Kubernetes and verify that Kubernetes auditing is enabled and has a rule that you’ll be testing (Exec or attach to a pod is easiest)
 

 
Step 2: Generate Webhook URL

In order to generate the webhook URL, you need to generate a webhook suffix and append it to the '/api/v1/kubernetes/webhook/' endpoint

Run the following command to create a webhook variable:

 

Step 3: Download Certificate Chain and convert to base64 

Download the certificate chain and export it to base64

• Open Firefox and browse to the 'Path to Console' URL
• Click the Certificate icon followed by the Connection Secure arrow in the Toolbar:

• Click More Information

• Click View Certificate

• Scroll down to Miscellaneous and download the PEM Chain

• In your terminal, cd to the folder the PEM chain was downloaded to.
• Run the following command to save the PEM chain as a base64 encoded variable:

export cacerts=$(openssl base64 -in cloud-twistlock-com-chain.pem -A)

Step 4: Generate audit-webhook.yaml

Generate audit-webhook.yaml and audit-policy.yaml files required to send audit events to Prisma Cloud

• Run the following command to generate audit-webhook.yaml:

cat <<EOF > audit-webhook.yaml

apiVersion: v1
kind: Config
preferences: {}
clusters:
- name: audit-cluster
  cluster:
    server: $webhookurl
    certificate-authority-data: $cacerts
contexts:
- name: webhook
  context:
    cluster: audit-cluster
    user: kube-apiserver
current-context: webhook
EOF

• Run the following command to generate the default audit policy:

cat <<EOF > audit-policy.yaml

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Generate audit events only for ResponseComplete or panic stages of a request.
omitStages:
  - "RequestReceived"
  - "ResponseStarted"
rules:
  # Audit on pod exec/attach events
  - level: Request
    resources:
    - group: ""
      resources: ["pods/exec", "pods/attach"]

  # Audit on pod creation events
  - level: Request
    resources:
    - group: ""
      resources: ["pods"]
    verbs: ["create"]

  # Audit on changes to the twistlock namespace (defender daemonset)
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    namespaces: ["twistlock"]

  # Default catch all rule
  - level: None
EOF

Step 5: Add webhook and policy configuration to all master nodes

You then need to add the audit files to all Master nodes in a file that’s accessible to kube-apiserver.

• Run the following command to view host volumes that are accessible to kube-apiserver:

kubectl get pods -n kube-system kube-apiserver-<rest of pod name> -o jsonpath='{range .spec.volumes[*]}{.hostPath.path}{"\n"}{end}'

•  Make a note of a path that makes the most sense for you to place the YAML files that were generated earlier. For example, this returned the following paths for me:

/var/log/kube-apiserver.log
/etc/ssl
/etc/pki/tls
/etc/pki/ca-trust
/usr/share/ssl
/usr/ssl
/usr/lib/ssl
/usr/local/openssl
/var/ssl
/etc/openssl
/etc/kubernetes/in-tree-cloud.config
/srv/kubernetes/ca.crt
/srv/kubernetes/kube-apiserver
/srv/sshproxy
/etc/kubernetes/kube-apiserver-healthcheck/secrets

I determined the /srv/kubernetes/kube-apiserver path made the most sense for me.

• SCP audit-webhook.yaml and audit-policy.yaml to each master node in the path you have selected (/srv/kubernetes/kube-apiserver in my case).

Step 6: Update kube-apiserver manifests on all master nodes

This step will vary depending on the k8s distribution so confirm with the vendor/docs that this is the proper way before moving forward.

• Locate the kube-apiserver.manifest file
• Open kube-apiserver.manifest in a text editor and locate a section that looks similar to this:

spec:

  containers:
  - command:
    - kube-apiserver
    - --admission-control-config-file=/etc/kubernetes/extra-config/admission-control-config.yaml
    - --advertise-address=X.X.X.X
    - --allow-privileged=true
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    - --audit-log-path=/var/log/kubernetes/kube-apiserver.log

• Add the following options to the command list under ‘- kube-apiserver’ and make sure to update the path in red to the path you selected earlier.

--audit-policy-file=/srv/kubernetes/kube-apiserver/audit-policy.yaml
--audit-webhook-config-file=/srv/kubernetes/kube-apiserver/audit-webhook.yaml

• The update kube-apiserver.manifest should look like this when you’re done:

spec:

  containers:
  - command:
    - kube-apiserver
    - --admission-control-config-file=/etc/kubernetes/extra-config/admission-control-config.yaml
    - --advertise-address=X.X.X.X
    - --allow-privileged=true
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    - --audit-policy-file=/srv/kubernetes/kube-apiserver/audit-policy.yaml
    - --audit-webhook-config-file=/srv/kubernetes/kube-apiserver/audit-webhook.yaml
    - --audit-log-path=/var/log/kubernetes/kube-apiserver.log

• Note: Your kube-apiserver manifest file MUST have an audit-log-path option enabled in order to turn on audit logging.
• Once you save the manifest file it should cause the kube-apiserver to restart - this can take about 30 seconds to complete and you may temporarily lose access to the API.

• Perform a test against one of the default rules to verify it’s working.

• Run an exec within a container to test one of the default rules:

kubectl exec <pod> -it -- /bin/sh

• Check the Prisma Cloud Console in Compute / Monitor / Events / Kubernetes Audits

Troubleshooting

If the kube-apiserver pod fails to start back up, typically that means there’s an issue with your manifest file. You’ll need to troubleshoot this however the process is kubernetes distribution dependent. A few possible methods are:

• journalctl --unit kubelet
• docker logs <container>
• crictl logs <container>

If the Prisma Cloud Console is failing to show audit logs, typically the kube-apiserver logs will provide a clue:

• kubectl logs kube-apiserver-<rest of pod name>

• Kubernetes Auditing
• https://kubernetes.io/docs/tasks/debug-application-cluster/audit/