How to troubleshoot the issue of the Strata Cloud Manager no longer receiving device telemetry data.

• Fix the issue of Strata Cloud Manager no longer receiving telemetry data from the firewall or Panorama.

• Strata Cloud Manager (AIOps for NGFW)
• PANW Firewall 
• Panorama
• Delayed Telemetry Data

1. Check if the device telemetry is enabled using CLI command:

   show device-telemetry details

2. Check if aiops plugin is installed on your device (This is specific to Panorama):

   show plugins installed

   Otherwise, refer to the document "Cloud Connector Plugin" for guidance.
3. Verify that the device has an active Cortex Data Lake (CDL) license (This is specific to AIOps for NGFW Premium):

   request license info

   1. For FW UI check under DEVICE > Licenses.
   2. For Panorama UI check under PANORAMA > Licenses.
   3. Note: If you have recently renewed your Logging Service license (aka. CDL license) and the Customer Support Portal (CSP) is showing them as valid you might need to re-fetch the licenses and click on "Retrieve license keys from license server" form UI, which is equivalent to CLI:

      request license fetch

4. Ensure that the region matches between the one configured in the device telemetry and the CDL region: This check is necessary when attempting to send telemetry data for a device onboarded under either the AIOps Premium instance or the Free AIOps instance with Cortex Data Lake (logging service) license.
   1. The device telemetry region is configured under the UI: DEVICE > Setup >Telemetry and can also be checked from the CLI using:

      show device-telemetry settings

   2. The Cortex Data Lake region is configured under the UI: DEVICE > Setup > Cortex Data Lake and can be checked from the CLI using (This is specific to Firewall): 

      request logging-service-forwarding customerinfo show

   3. If the command in b doesn't show the expected result even after fixing the CDL region configuration on the firewall, then refer to the "Troubleshooting Firewall Connectivity" document.
   4. To verify the CDL region for the Panorama issue the CLI command:

      request plugins cloud_services logging-service status

      The last printed line should show the CDL region.
5. Ensure that the device certificate is installed and still valid:

   show device-certificate status

   refer to Install a Device Certificate if needed.
6. Check that the firewall/Panorama can resolve the FQDN of the device-telemetry endpoint:
   1. To check the FQDN of the device-telemetry endpoint using the CLI command:

      show device-telemetry details

   2. Use the CLI command:

      ping host <FQDN endpoint>

      to check whether firewall can resolve that FQDN to an IP address. If not then check both DNS and NTP server configuration of the firewall under UI DEVICE > Setup > Services. Also confirm that your network allows access to the FQDN and App-IDs for telemetry.
7. Another way to troubleshoot the problem would be to issue the CLI command:

   show device-telemetry stats all

   
   Since its output will help indicate the cause of the firewall's failure to send the device telemetry bundle:
   1. For reason equal to "DNS lookup failed", check step 6 in this document.
   2. For reason equal to "Client Certificate issue", check step 5 in this document.
   3. For reason equal to "Send File to CDL Receiver Failed", if you see the following error "dt INFO S1: CDL: RSP KEY RESPONSE: ['Error: dosys timed out']" in the device_telemtry_curl.log
       then check PAN-210331 fixed in 10.1.9, 10.2.4, and 11.01 find release notes here. Otherwise check document1 and document2.
   4. For reason equal to "CDL Receiver Key Empty", ensure that the TCP ports and FQDN are allowed for the CDL tenant to communicate with the firewall/Panorama, check document3 and document4.
   5. Note be patient when addressing each issue or reason for failure before issuing again the CLI command "show device-telemetry stats all" as it takes some time before it would reflect the updated status. You can also track the change in the behavior in the output of the device_telemetry_curl.log and device_telemetry.log

      tail follow yes mp-log device_telemetry_curl.log
      tail follow yes mp-log device_telemetry.log
      

8. Check the last attempt and last success of collecting telemetry data using the CLI command:

   show device-telemetry collect-now

   1. In case you want to trigger the collection of the telemetry bundle then issue the CLI command:

      request device-telemetry collect-now

9. Check that the device has connection to the CDL is up:
   1. For the firewall use the CLI command:

      request logging-service-forwarding status

      otherwise, refer to the "Troubleshooting Firewall Connectivity " document.
   2. For Panorama use the CLI command:

      request plugins cloud_services logging-service status

      refer to Configure Panorama for Cortex Data Lake (10.0 or Earlier) and Configure Panorama for Cortex Data Lake (10.1 or Later).

If multiple firewalls are reporting delayed telemetry data, check status.paloaltonetworks.com to determine whether there is a backend issue on the Strata Cloud Manager or AIOps for NGFW side.

Preliminary checks, before proceeding with the troubleshooting steps outlined above, are as follows:

1. Verify that the firewall is running PAN-OS 10.0.x or a higher version to support the transmission of telemetry data to Strata Cloud Manager/AIOps for NGFW.
2. Confirm that the device is associated with the same CSP account as the AIOps instance.
3. Ensure that you add the device to the correct tenant by navigating to the hub  (https://apps.paloaltonetworks.com/hub) and selecting Common Services > Device Associations > Add Device under the appropriate CSP account.

Note 1:
For Panorama ensure that both NTP and DNS servers are configured.
If after renewing the device-certificate and CDL license and triggering the collection of the device-telemetry data using CLI command:

request device-telemetry collect-now

show device-telemetry collect-now

debug software restart process reportd

wait couple of minutes then trigger the collection of telemetry data again you should then see the "Last Attempt" of sending telemetry data showing a recent date which indicates that Panorama is sending telemetry data.

Note 2: 
Refer to How to troubleshoot the Firewall onboarding and connection issues with Strata Cloud Manager if you intend to utilize Strata Cloud Manager not only for monitoring your NGFW but also for managing and configuring it.
Below are the titled of some referenced documents in this knowledge base article:
document1: Device Telemetry fails with the error - "Failed to send: file" seen in the System Logs.
document2: Device Telemetry "Failed to connect to storage.googleapis.com port 443: No route to host"
document3: Failed to send telemetry data with error: CDL Receiver Key Empty
document4: Error message "CDL Receiver Key Empty" seen when viewing telemetry stats