Prisma Cloud Compute: Kubernetes Audits are showing inconsistent results for GCP/GKE

• When testing, Kubernetes Audits for GKE clusters only appear for certain tests and not for others
  • (i.e. when testing the privileged pod rule, sometimes the audit appears after creating a privileged pod and other times it does not)
• Kubernetes Audits may not appear at all for certain GKE clusters
• After filtering based on project or cluster, more but not all of the audits may begin to come through

• The following log lines should appear within the console logs:

ERRO [DATE/TIME] scheduler.go:96 Failed to run Kubernetes auditing service: failed to init GKE logging client: googleapi: Error 429: Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'logging.googleapis.com' for consumer 'project_number:<PROJECT-NUMBER>'.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/<PROJECT-NUMBER>",
      "quota_limit": "ReadRequestsPerMinutePerProject",
      "quota_limit_value": "60",
      "quota_location": "global",
      "quota_metric": "logging.googleapis.com/read_requests",
      "service": "logging.googleapis.com"
    },
    "reason": "RATE_LIMIT_EXCEEDED"
  },
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Request a higher quota limit.",
        "url": "https://cloud.google.com/docs/quotas/help/request_increase"
      }
    ]
  }
]
, rateLimitExceeded

• Prisma Cloud Compute (Enterprise/Self-hosted) v33.03.138
• Google Cloud Platform/Google Kubernetes Engine (GCP/GKE)

The cause for this issue is low GCP quota limits. When these limits are too low, the large amount of read requests per minute may result in throttling by GCP

These limits can be seen by navigating to IAM & Admin > Quotas & System Limits.

The specific quota for this issue is the ReadRequestsPerMinute Quota which can be searched for in the field. 

After clicking on this field, the quotas are able to be altered through the below page.

To resolve the issue, configure the Limits and Quotas for the ReadRequestsPerMinute API by editing the quota.