Error querying OCSP responder" as certificate revocation status checks fail on Panorama"

Error querying OCSP responder" as certificate revocation status checks fail on Panorama"

40020
Created On 09/25/18 15:19 PM - Last Modified 08/03/20 22:20 PM


Symptom


Panorama had been displaying the cloud logging service service logs, and then suddenly stops displaying the logs due to its inability to contact the OCSP to complete the certificate revocation checks.
 

Diagnosis

  • less mp-log lcaas_agent.log

2018-08-27 15:16:47,108 lcaas_agent INFO Server-cert revocation check status: unavailable
This indicates that Panorama could not complete the certificate revocation checks.
 
2018-08-27 15:16:47,279 lcaas_agent INFO Resp from cloud service : [{"query":"8ad748e7-edbb-413a-b2c8-89c36750a859.api2-lc-prod-us.gpcloudservice.com:444","CustomerID":"117789002","region":"americas","region-display":"americas","ingest":"8ad748e7-edbb-413a-b2c8-89c36750a859.in2-lc-prod-us.gpcloudservice.com"}]
It is important to note that the above 'Resp from cloud service' does not necessarily mean that the service is responding to Panorama.
"request plugins cloud_services logging-service status" does not return any output and remains stuck until you kill the task using "CTRL+C"
 
  • less mp-log plugin_cloud_services.log
2018-08-27 16:27:36.712 -0500 INFO: [update-device-cert] OCSP/CRL check status:
('unavailable', 'Output to be sent to /tmp/ocspoutput_911171761.data.\nOCSP URL from the certificate http://ocsp.paloaltonetworks.com/ocsp.\nOCSP cert status check is hosted atocsp.paloaltonetworks.com.\nTrying connection to Host ocsp.paloaltonetworks.com for checking cert status.\nError querying OCSP responsder\n

2018-08-27 16:27:36.712 -0500 ERROR: [update-device-cert] No cert/key found. Probably trusted channel is not setup. Cannot continue.

You can see from the plugin_cloud_services.log that Panorama could not get a response from ocsp.paloaltonetworks.com where the OCSP cert status check is hosted due to a security rule on the edge firewall blocking access to this destination.

How was Panorama able to display the logs from the cloud service successfully before it stopped displaying them?
The reason is that at the time of onboarding logging-service, the user might have configured allow 'any' destination rule on the perimeter firewall and later modified the rule to allow granular access to Logging Service FQDNs only based on the Region(US or EU) from Panorama as source skipping the other destinations listed in the solution below:-

 

Resolution


Panorama needs to access these FQDNs for the initial setup and one-time password, and for ongoing certificate revocation checks.


Note: 
For OCSP, you must also allow the firewalls to access ocsp.paloaltonetworks.com on port 80


Open the security policy to Logging service to add the the above FQDNS as 'Destination addresses' and services(444, 443, 80)

These are listed in the following document as well and should be allowed access to before fine tuning the security policies for Panorama access to Logging Service (internet bound). Cortex Data Lake Getting Started, TCP Ports and FQDNs Required for Cortex Data Lake

After successfully configuring the rule, Panorama should begin rendering logs and you can check connectivity to logging service using:

> request plugins cloud_services logging-service status

pass

{"@status": "success", "result": {"PODamericas": {"name": "americas", "Status": {"type": "status", "value": "OK", "tooltip": "OK"}, "@num_instances": 1, "Storage Us
ed (TB)": {"type": "number", "value": "0.516887", "limit": 1}, "Estimated Log Retention (Days)": 132, "entry": [{"name": "Americas", "Status": {"type": "status", "v
alue": "OK", "tooltip": "OK"}, "infra-audit-utilization": {"header": ["Infrastructure and Audit Logs", "Utilization"], "type": "number", "value": 1.94, "limit": 20.
48, "unit": "GB"}, "infra-audit-retention": {"header": ["Infrastructure and Audit Logs", "Retention"], "type": "number", "value": 151, "unit": "Days"}, "detail-util
ization": {"header": ["Detailed Logs", "Utilization"], "type": "number", "value": 509.06, "limit": 819.2, "unit": "GB"}, "detail-retention": {"header": ["Detailed L
ogs", "Retention"], "type": "number", "value": 132, "unit": "Days"}, "summary-utilization": {"header": ["Summary Logs", "Utilization"], "type": "number", "value": 1
8.29, "limit": 184.32, "unit": "GB"}, "summary-retention": {"header": ["Summary Logs", "Retention"], "type": "number", "value": 141, "unit": "Days"}, "@quota_info":
 {"quota_details": "{\"log-disk-quota\":{\"detailed\":80,\"infra-audit\":2,\"summary\":18},\"log-expiration-period\":{\"detailed\":395,\"infra-audit\":395,\"summary
\":395},\"min-retention-warning-period\":{\"detailed\":14,\"infra-audit\":14,\"summary\":14},\"@name\":\"americas\",\"theater-quota\":{\"quota_count\":1}}", "quota_
count": 1}}]}}}
 
See the certificate revocation status as follows in lcaas_agent.log  
2018-08-27 15:25:47,108 lcaas_agent INFO Server-cert revocation check status: good


If the revocation status still shows 'unavailable', delete and re-fetch the Panorama-certificate using OTP.

Additional Information


For help to delete and re-fetch certificates on Panorama, please see The SSL certificate error" causing Panorama to not Display Logs from the logging-service"
Other users also viewed:
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/kcSArticleDetail?id=kA10g000000ClDi&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FkcSArticleDetail

Choose Language