How to Troubleshoot gRPC Connection Failure from the Firewall’s Dataplane to the Advanced URL Filtering Service

How to troubleshoot connection failure from DataPlane of the firewall to Advanced URL filtering service?

• Palo Alto Firewalls
• Supported PAN-OS
• Advanced URL Filtering
• Inline Deep Learning Analysis for Advanced URL Filtering

1. Check that a device certificate is valid and present on the Firewall.

   show device-certificate status

2.  Check if the Advanced URL license is present and valid.

   request license info

3. Check that the Data Services service route is properly configured. (Default is management).
4. Check if the upstream firewall is allowing application paloalto-aurl-idl and oscp (for certificate validation).
5. Troubleshoot the connection between Firewall Data Plane (DP) and Advanced URL filtering (filemanager):
   1. Check the cloud connection status to the Firewall DP:

      show ctd-agent status security-client

      • Under the section "Security Client UrlCat(2)" the cloud connection should show connected and Pool state should show Ready (2).
   2. If showing Pool state: Invalid License (6) then go back to step 2
   3. If showing Pool state: Invalid Config (7) then check if you have configured a URL filtering profile where you have enabled "enable the cloud inline categorization" under the Inline Categorization tab as mentioned in the Inline Deep Learning Analysis for Advanced URL Filtering.  
   4. If showing Pool state: Closed (4) then 
      • Check the network connection between the Firewall Data Services service route, source IP, and the Advanced URL filtering (filemanager) server, destination FQDN:

        traceroute host urlcat.hawkeye.services-edge.paloaltonetworks.com

      • Note: This command is valid in case Management is used as Data Services service route and the urlcat.hawkeye.services-edge.paloaltonetworks.com is the FQDN of the Advanced URL filtering server found in the output of 5.a.
      • Otherwise add "source" to the command followed by the IP address of the dataplane interface used as service route and the proper FQDN depending on your region.
      • Ping the server to confirm the IP address being used (Add the "source" interface if required):

        ping host urlcat.hawkeye.services-edge.paloaltonetworks.com

      • Check if the connection is established on port 443 between the Firewall and the Advanced URL filtering (filemanager) server.

        show netstat numeric-hosts yes numeric-ports yes | match 34.111.222.75 

      • Where 34.11.222.75 would be the IP address resolved by ping.
   5. Check Firewall system logs related to this DP connection:

      show log system subtype equal ctd-agent-connection direction equal backward

6. If the issue is not yet resolved or the logs does not provide enough information, contact Palo Alto Support.

For configuration assistance check Inline Deep Learning Analysis for Advanced URL Filtering.
Example of the output of show ctd-agent status security-client when the connection is up between FW DP and the advanced URL filtering filemanager server in the cloud.

Security Client UrlCat(2)
        Current cloud server:   urlcat.hawkeye.services-edge.paloaltonetworks.com:443
        Cloud connection:       connected
        Config:
                Number of gRPC connections: 1, Number of workers: 5
                Debug level: 2, Insecure connection: false, Cert valid: true, Key valid: true, CA count: 383
                Maximum number of workers: 10
                Maximum number of sessions a worker should process before reconnect: 256
                Maximum number of messages per worker: 0
                Skip cert verify: false
        Grpc Connection Status:
                State Ready (3), last err <nil>
                Pool state: Ready (2)
                     last update: 2023-04-11 13:54:15.504881805 -0700 PDT m=+613070.685846885
                     last connection retry: 2023-04-11 13:52:20.099382218 -0700 PDT m=+612955.280347535
                     last pool close: 2023-04-11 11:43:27.36857076 -0700 PDT m=+605222.549535906