How to troubleshoot a connection failure between the firewall and SLS (aka. CDL)

Troubleshoot a connection failure between the firewall and Strata Logging Service (SLS).

• Firewall
• Strata Logging Service (SLS)

1. General status check. CLI below will show information and potential issues related to licensing, customer info (tenant ID, ingest/query fqdns) and actual logging status.
   1. For logging-service setting Enable Cortex Data Lake forwarding only use:

      > request logging-service-forwarding status

   2. For logging-service setting Enable Duplicate Logging (Cloud and On-Premise) use:

      > debug log-receiver log-forwarding-connections status

2. Confirm license is valid from the output of the CLI in step1 or from:

> request license info

3. Confirm Certificate status is successful. Look for errors with certificates and OCSP from CLI output in step 1 or from: 
   1. For firewall running 10.0 or earlier 

      > request logging-service-forwarding certificate info

   2. For firewall running 10.1 or later

      > show device-certificate info
      > show device-certificate status

4. Check Customer info is correct and not missing (e.g. region, API FQDNs) from CLI output in 1 or from:

> request logging-service-forwarding customerinfo show

5. Check Logging status is active and connected. If not connected look at individual checks which are shown in the logging status (DNS, Registration, SSL, TCP) from CLI output in step 1 or from: 
   1. For logging-service setting Enable Cortex Data Lake forwarding only use:

      > show logging-status

      For the Cortex Data Lake, the agent is > Log Collection Service
      'Log Collection log forwarding agent' is active and connected to <IP_address>.
   2. For logging-service setting Enable Duplicate Logging (Cloud and On-Premise) use:

      > debug log-receiver rawlog_fwd_trial stats global show
      > debug log-receiver rawlog_fwd_trial connmgr

6. For connectivity problems, verify the required ports are allowed from firewall to logging services. Check TCP Ports and FQDNs Required for Strata Logging Service.
7. For SSL handshake problems, collect a packet capture on either the management interface or the DP Interface that is being used to connect to CDL and check to see whether the SSL handshake is completing. 

Output when licensing is valid

> request logging-service-forwarding status 
Logging Service Licensed: Yes
Logging Service forwarding enabled: Yes
Duplicate logging enabled: No
Enhanced application logging enabled: No

Output when certificate status is valid 10.0 and earlier

Logging Service Certificate information: 
        Info: Successfully fetched Logging Service certificate
        Not Valid after: 2022-05-03 15:23:49
        Not Valid before: 2022-02-02 15:23:49
        Status: success
        Last fetched: 2022/02/08 15:44:43

Output when certificate status is valid 10.1 and later

Device Certificate information:
        Current device certificate status: Valid
        Not valid before: 2022/09/12 04:19:11 PDT
        Not valid after: 2022/12/11 03:19:11 PST
        Last fetched timestamp: 2022/09/12 04:29:12 PDT
        Last fetched status: success
        Last fetched info: Successfully fetched Device Certificate

Output when Customer info is correct. It should not be missing fields such as region. API, FQDNs

Logging Service Customer file information: 
        Customer ID: 123456789
        EAL Ingest FQDN: 9bf092b2-861f-4268-ad29-0e7d52930f9e.fei-lc-prod-eu.gpcloudservice.com
        Ingest FQDN: 9bf092b2-861f-4268-ad29-0e7d52930f9e.in2-lc-prod-eu.gpcloudservice.com
        Info: Successfully fetched Logging Service customer info
        Query FQDN: 9bf092b2-861f-4268-ad29-0e7d52930f9e.api2-lc-prod-eu.gpcloudservice.com:444
        Status: success
        Last Fetched: 2022/02/08 15:01:08

Output for a working connection to the logging service.

>Log Collection Service 
'Log Collection log forwarding agent' is active and connected to 192.x.x.x
================================================
connid: 192.1.1.1
================================================
DNS :
    Successfully resolved FQDN (9bf092b2-861f-4268-ad29-0e7d52930f9e.in2-lc-prod-eu.gpcloudservice.com), IP (192.x.x.x)
                           success
               2022/02/08 15:44:43
Registration :
         registration request sent
                           success
               2022/02/08 15:44:45
SSL :
    ssl channel established to (192.x.x.x)
                           success
              2022/02/08 15:44:45
Status :
             Connection successful
                           success
               2022/02/08 15:44:45
TCP :
        tcp connection established
                           success
               2022/02/08 15:44:43
Connect-Agent-Status :
    connect succeeded for FQDN 9bf092b2-861f-4268-ad29-0e7d52930f9e.in2-lc-prod-eu.gpcloudservice.com (IP: 192.x.x.x)
                           success

Note 1: Documentation only calls for the paloalto-logging-service and paloalto-shared-services app-id to which you must allow traffic to ensure that the firewall can successfully connect to Strata Logging Service (Cortex Data Lake) but you will also need:

• web-browsing
• SSL
• OCSP

Note 2: For further information on how to troubleshoot firewall connectivity with CDL refer to Troubleshooting Firewall Connectivity
Note 3: If Palo Alto Networks Firewall is a VM-series and

> request logging-service-forwarding status
Logging Service Licensed: No

check How to fetch Cortex Data Lake license for PA-VM.
Note 4: For 7k with LFC card after the upgrade to 10.1 make sure that the installation of the device certificate is done before enabling Duplicate logging.
Note 5: The impact of expired CDL license on FW log forwarding to SLS (CDL) and log storage in SLS can be found here.
Note 6: After the loss of the connection between FW and SLS, FW logs will be queued and sent once the connection is restored. If queue is full before connection establishment, some logs will be lost. Use below CLI to check that:

> show counter global filter delta yes | match queue_full