How to troubleshoot connection failure between Firewall MP and Application Cloud Engine (ACE)
Objective
- To troubleshoot connection failure between Firewall MP and Application Cloud Engine (ACE).
- The information in this article applies only to PAN-OS versions on 12.1.x or higher
- For the versions below 12.1, Refer to "How to Troubleshoot GRPC Connections failure between Firewall and ACE Application Cloud Engine"
Environment
- Palo Alto Firewalls
- PAN-OS 12.1 and above.
- App-ID Cloud Engine (ACE)
- Knowledge Centered Service Cloud
Procedure
-
Check that a device certificate is valid and present on the FW.
show device-certificate status -
Check that the SaaS Security Inline license is present and Valid.
request license info -
Check that the Data Services service route is properly configured. (Default is management)
-
Check if the upstream firewall is allowing application paloalto-ace, paloalto-ace-kcs, paloalto-dlp-service and OCSP (for certificate validation).
-
The following FQDNs need to be allowed in a security policy for certificate verification:
-
-
-
- ocsp.paloaltonetworks.com
- crl.paloaltonetworks.com
- ocsp.digicert.com
- crl.digicert.com
- crl3.digicert.com
- crl4.digicert.com
- ocsp.godaddy.com
- crl.godaddy.com
-
-
Make sure that the App-ID cloud engine is enabled on the Firewall (it is enabled by default on the FW).
-
Troubleshoot the connection between Firewall Management Plane (MP) and App-ID Cloud Engine (ACE):
-
Check the cloud connection status to the Firewall MP.
show cloud-appid connection-to-cloud
-
Note: In addition to the connection status, this output will help guide your troubleshooting by indicating whether the problem is the device certificate or the missing license as explained here.
-
-
Check the network connection between the Firewall Data Services service route, source IP, and the ACE server, destination FQDN:
traceroute host kcs.ace.tpcloud.paloaltonetworks.com
-
Note: This command is valid in case Management is used as Data Services service route and the kcs.ace.tpcloud.paloaltonetworks.com is the FQDN of the ACE server found in the output of 6.a. Otherwise add source to the command followed by the IP address of the dataplane interface used as service route.
-
-
Check if connection is established on port 443 between the Firewall and the ACE server:
show netstat numeric-hosts yes numeric-ports yes | match x.y.z.q
-
Here replace x.y.z.q by the IP address of the ACE server resolved by the DNS server in 6.b
-
-
Check Firewall system logs related to this MP connection:
show log system subtype equal app-cloud-engine direction equal backward
-
-
-
As last resort and if needing to restart the connection between FW MP and ACE server use:
debug cloud-appid reset connection-to-cloud
-
Additional Information
Prepare to Deploy APP-ID Cloud Engine
Troubleshoot APP-ID Cloud Engine
The Firewall maintains two connections to the cloud: One connection from Firewall MP to ACE server and another connection from Firewall DP to Content Cloud server. This document focuses on the MP connection.
Below is an example of the output of a good connection between FW and ACE.
> show cloud-appid connection-to-cloud
ACE Cloud server: kcs.ace.tpcloud.paloaltonetworks.com:443
Cloud connection: connected
Summary of ACE gRPC client:
number of connection reset: 0
number of connection failed: 110
number of connection established: 1685
number of connection attempts: 1795
number of connection released: 1684
number of connection selected: 3248
number of selections failed: 8
number of bytes sent: 183674
number of bytes received: 207531676
Last gRPC connection Attempt: 2025-08-01 14:01:39 -0700 PDT
Last successful gRPC connection: 2025-08-01 14:01:39 -0700 PDT
Summary of gRPC connections [configured source IP: ]
Device cert status: Installed
Validity:
Notbefore: 2025-07-17 07:41:13 +0000 UTC
Notafter: 2025-10-15 07:41:12 +0000 UTC
max gRPC connections: 1, ongoing: 0, max alive time: unlimited, max bytes sent: unlimited
[0]gRPC conn[10.6.158.66:42730 -> 34.102.229.209:443], state true, selected 0, backup false,0, device cert
send: wire 0, app 0, num 0; receive: wire 0, app 0, num 0
Current Time: 2025-08-01 14:02:39.424805538 -0700 PDT m=+1383066.393353224