Panorama_managed firewall is not able connect to CDL-with CSR certificate error

• The firewall is managed by Panorama.
• The Panorama is running PAN-OS 9.0.9  and in management-only mode. The managed firewall(s) are a mix of PAN-OS 9.0 and 8.1 code.
• All logs are sent to Cortex Data Lake (CDL) and then forwarded to the syslog-server.
• The Panorama managed firewall is not able to connect to CDL for logging service and the "logging Service status" shown there is a CSR certificate error.

From Firewall Web UI:

• Firewall shows all the licenses are up to date.

From Firewall Command line:

• Verify with lcaas_agent log that there is the same error message as what is observed form web UI:

"CSR signing error to Panorama: "Error sending CSR signing request to Panorama, 'status': failure"

The firewall has no connectivity issues with the cloud server: pings to cloud FQDN (see listed below FQDN(s) all ping through successful.

• firewall-prd1.us.cdl.paloaltonetworks.com (TCP 3978)
• pcl-prd1.us.cdl.paloaltonetworks.com (TCP 444)
• fei-prd1.us.cdl.paloaltonetworks.com (TCP port 443)
• br-prd1.us.cdl.paloaltonetworks.com (TCP port 443)
• lic.lc.prod.us.cs.paloaltonetworks.com (TCP port 444)

If intending to fix the issue with the logging service follow this troubleshooting document.

 From the firewall command line:

• The commands for "delete license" and then "re-fetch license" goes fine.
• When intending to "delete logging service certificate" then "re-fetch certificate", the jobs for "certificate fetch" would fail.
• >"request logging-service-forwarding certificate fetch" ---> ** the fetch will create a job id and the job will fail.

• Palo Alto Firewall managed by Panorama
• PAN-OS 8.1 and 9.0
• Logging sent to Cortex Data Lake.

1. All license(s) are up to date with a good valid expiration date
2. Deleting and re-fetching license(s) has no issue
3. Firewall and Panorama have same NTP server and clocks are in sync
4. Pings from firewall and Panorama to cloud service FQDNs are all good, indicating there is no blockage on upstream devices.

> debug management-server conn
> request logging-service-forwarding certificate delete
> show NTP
> request logging-service-forwarding status >>> which will show logging-service license as "yes", but no logging service 
                                                customer info, nor is showing the logging service agent data.

> request logging-service-forwarding status
        Logging Service Licensed: Yes                                --------------> Firewall logging license is good
        Logging Service forwarding enabled: Yes
        Logging Service Certificate information:
        Info: Error sending CSR signing request to Panorama           --------------> Error message
        Status: failure
        Logging Service Customer file information:
                   Info: Logging Service license is not provisioned.  -------------> no customer data
                   Status: failure

primary-active)> request plugins cloud_services logging-service status

fail
{"@status": "fail", "result": {"PODamericas": {"message": "Unable to connect to API gateway. (35, 'error:14094415:SSL routines:SSL3_READ_BYTES:                 
sslv3 alert certificate expired')", "result": "fail"}}}
fail
Logging service certificate expired           ----------------------------> Expired logging service Certificate message
Failed to fetch ingest/query FQDN for customer (curl failed)
failure
0c22ed23-456c-43ae-8e96-d314cd341ff5.in2-lc-prod-us.gpcloudservice.com
e8870173-8fd2-4114-99d3-6f450fdfb093.api2-lc-prod-us.gpcloudservice.com:444

> ping host 0c22ed23-456c-43ae-8e96-d314cd341ff5.in2-lc-prod-us.gpcloudservice.com
   PING firewall-prd1.us.cdl.paloaltonetworks.com (34.69.208.173) 56(84) bytes of data.
    64 bytes from 34.69.208.173: icmp_seq=1 ttl=100 time=69.3 ms
    64 bytes from 34.69.208.173: icmp_seq=2 ttl=100 time=69.1 ms
    64 bytes from 34.69.208.173: icmp_seq=3 ttl=100 time=69.1 ms

> ping host e8870173-8fd2-4114-99d3-6f450fdfb093.api2-lc-prod-us.gpcloudservice.com
   64 bytes from 35.184.126.116: icmp_seq=1 ttl=100 time=69.1 
   64 bytes from 35.184.126.116: icmp_seq=1 ttl=100 time=69.1 ms

1. From Panorama command line:

> request plugins cloud_services panorama-certificate delete
Pass

2.  Follow this document to generate Panorama OTP for use in the following step
3. Use the request plugin command

> request plugins cloud_services panorama-certificate fetch OTP <OTP obtained by following the document>

1. From Panorama command line:

> show plugins clouds_service panorama-certificate status ----->the logging service will now show a new date with no error.

2. From Firewall CLI and re-fetch certificate

> request logging-service-forwarding certificate fetch

3. The fetch now shows there is no error for the logging-service certificate. The firewall now shows in good status connects to CDL.