How to troubleshoot gRPC connections failure between Firewall and the user-id Edge service

How to troubleshoot gRPC connections failure between Firewall and the user-id Edge service

13358
Created On 08/06/24 21:29 PM - Last Modified 11/25/25 23:56 PM


Objective



How to troubleshoot connections failure between Firewall and the user-id Edge service.

Environment


  • Next-Generation Firewalls (NGFW)
  • PAN-OS 11.0 and above.
  • ICD (Identity Client Daemon)
  • Edge User-ID service
  • CIE (Cloud Identity Engine)

Procedure


  1. Check that a device certificate is valid and present on the FW: 
    show device-certificate status
  2. Check the configuration of the firewall. Navigate to Device > Setup > Management, and under the PAN-OS Edge Service Settings, ensure that the "Enable User Context Cloud Service" checkbox is checked if you want to use the cloud User-ID redistribution feature.
    1. User context cloud service
  3. Check the configuration of the service route of the cloud user-id feature. Navigate to Device > Setup > Services, and under Service Route Configuration look for "iot". The cloud user-id redistribution feature uses the IoT service route.
    1. cloud user-id service route
       
  4. Troubleshoot the connection between firewall Management Plane (MP) and Edge User-ID service:
    1. Use the CLI command: 
      > show cloud-userid config-details
      a sample of that output is shown below: 
      > show cloud-userid config-details
Cloud UserID: enabled
Connection: disconnected
    2. For more details on the connection to the Edge User-ID server, use the CLI command: 
      > show cloud-userid statistics
      a sample of that output is shown below: 
      > show cloud-userid statistics
Summary of CUID gRPC client:
number of connection reset:       38700
number of connection failed:      8
number of connection established: 38702
number of connection attempts:    38711
number of connection released:    38702
number of connection selected:    38700
number of selections failed:      43891
number of bytes sent:             1122249
number of bytes received:         0
Last gRPC connection Attempt:     2024-07-31 06:50:03 -0400 EDT
Last successful gRPC connection:  2024-07-31 06:49:58 -0400 EDT
Cloud user-id URL:                identity.services-dge.paloaltonetworks.com
Source Address:                   192.168.5.3
Destination Address:              34.136.155.117:443
Device cert status: Installed
	Validity: 
		Notbefore: 2024-07-29 01:09:23 +0000 UTC 
		Notafter: 2024-10-27 01:09:22 +0000 UTC
max gRPC connections: 1, ongoing: 1, max alive time: unlimited, max bytes sent: unlimited
Cloud user-id connection: disconnected
UPLOAD:
IP User Mappings:         0
IP Port User Mappings:    0
IP Tags:                  0
User Tags:                0
IP Host(Quarantine List): 0
DOWNLOAD:
IP User Mappings:         0
IP Port User Mappings:    0
IP Tags:                  0
User Tags:                0
IP Host(Quarantine List): 0
    3. Check the network connection between the Firewall IoT service route, source IP, and the Edge user-id server, destination FQDN: 
      > traceroute host identity.services-edge.paloaltonetworks.com
       
      1. Note: This command is valid in case Management or Default is used as IoT service route , otherwise add source to the command followed by the IP address of the dataplane interface used as service route.  
        > traceroute source <IP address of the IoT service route dataplane interface> host identity.services-edge.paloaltonetworks.com
    4. Check the netstat to the Edge user-id server: 
      show netstat numeric-ports yes numeric-hosts yes | match <IP address of the Edge user-id server>
    5. Ensure that no device or firewall rule is blocking the connection to the Edge user-id server on port 443.
    6. Check Firewall system logs related to this connection: 
      > show log system direction equal backward subtype equal cuid-connection
      1. Following messages in the logs are typically seen: 
        "gRPC connection to identity.services-edge.paloaltonetworks.com:443 is broken, error: rpc error: code = Unknown desc = [UploadCUID]"
"gRPC connection to identity.services-edge.paloaltonetworks.com:443 is broken, error: Feature is not enabled or device cert isn't available for CUID"
"gRPC connection to identity.services-edge.paloaltonetworks.com:443 is broken, error: dial tcp: lookup identity.services-edge.paloaltonetworks.com on"
"gRPC connection to identity.services-edge.paloaltonetworks.com:443 is broken, error: connection error: desc = "transport: authentication handshake failed: read tcp"
      2. error: rpc error: code = Unknown desc = [UploadCUID]: Ensure that you have properly onboarded the Cloud Identity Engine Instance, that you have activated sharing and mapping, configured a default segment, etc... Refer to Cloud Identity Engine User Context.
      3. error: Feature is not enabled or device cert isn't available for CUID: This message does not indicate that the the Cloud identity engine user context is not enabled on the firewall. Ensure that you have activated the Cloud Identity Engine License on the tenant. Refer to Cloud Identity Engine User Context.
      4. error: dial tcp: lookup identity.services-edge.paloaltonetworks.com on: Ensure that the TCP connection between the interface used for IOT service and the identity.services-edge.paloaltonetworks.com has been established. Refer to step 4.d in this document.
      5. error: connection error: desc = "transport: authentication handshake failed: read tcp: Ensure that you have configured Your Network to Allow Cloud Identity Agent Traffic.
    7. As last resort and if needing to restart the connection between FW and Edge user-id server use the CLI command: 
      > debug cloud-userid reset-connection


        
       
Other users also viewed:
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000HDq6CAG&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Choose Language