How to troubleshoot gRPC connections failure between Firewall and ACE Application Cloud Engine/Content Cloud

21599

Created On 04/03/23 22:40 PM - Last Modified 05/24/23 03:41 AM

App-ID

10.2

10.1

11.0

Data Security

Objective

How to troubleshoot connections failure between Firewall MP/DP and ACE Application Cloud Engine/Content Cloud.

Environment

Palo Alto Firewalls
PAN-OS 10.1 and above.
App-ID Cloud Engine (ACE)
Content Cloud (filemanager)

Procedure

Check that a device certificate is valid and present on the FW.
```
show device-certificate status
```
Check that the SaaS Security Inline license is present and Valid.
```
request license info
```
Check that the Data Services service route is properly configured. (Default is management)
Check if the upstream firewall is allowing application paloalto-ace, paloalto-ace-kcs and oscp (for certificate validation).
Make sure that the App-ID cloud engine is enabled on the Firewall (it is enabled by default on the FW).
Troubleshoot the connection between Firewall Management Plane (MP) and App-ID Cloud Engine (ACE):
1. Check the cloud connection status to the Firewall MP.
```
show cloud-appid connection-to-cloud   
```
  1. Note: In addition to the connection status, this output will help guide your troubleshooting by indicating whether the problem is the device certificate or the missing license as explained here.
2. Check the network connection between the Firewall Data Services service route, source IP, and the ACE server, destination FQDN:
```
traceroute host kcs.ace.tpcloud.paloaltonetworks.com
```
  1. Note: This command is valid in case Management is used as Data Services service route and the kcs.ace.tpcloud.paloaltonetworks.com is the FQDN of the ACE server found in the output of 5.a. Otherwise add source to the command followed by the IP address of the dataplane interface used as service route.
3. Check if connection is established on port 443 between the Firewall and the ACE server:
```
 show netstat numeric-hosts yes numeric-ports yes | match 34.120.110.215
```
  1. Where 34.120.110.215 would be the IP address of the ACE server resolved by the DNS server in 5.b
4. Check Firewall system logs related to this MP connection:
```
show log system subtype equal app-cloud-engine direction equal backward
```
5. As last resort and if needing to restart the connection between FW MP and ACE server use:
```
debug cloud-appid reset connection-to-cloud
```
Troubleshoot the connection between Firewall Data Plane (DP) and Content Cloud (filemanager):
1. Check the cloud connection status to the Firewall DP.
```
show ctd-agent status security-client
```
  1. Note: under the section "Security Client Ace" the cloud connection should show connected and Pool state should show Ready (2).
2. Check the network connection between the Firewall Data Services service route, source IP, and the Content Cloud (filemanager) server, destination FQDN:
```
traceroute host ace.hawkeye.services-edge.paloaltonetworks.com
```
  1. Note: This command is valid in case Management is used as Data Services service route and the ace.hawkeye.services-edge.paloaltonetworks.com is the FQDN of the Content Cloud server found in the output of 6.a. Otherwise add source to the command followed by the IP address of the dataplane interface used as service route and the proper FQDN depending on your region.
3. Check if connection is established on port 443 between the Firewall and the Content Cloud (filemanager) server:
```
 show netstat numeric-hosts yes numeric-ports yes | match 34.111.222.75
```
  1. Where 34.11.222.75 would be the IP address of the Content Cloud server resolved by the DNS server in 6.b
4. Check Firewall system logs related to this DP connection:
```
show log system subtype equal ctd-agent-connection direction equal backward
```
5. As last resort and if needing to restart the connection between FW DP and Content Cloud server use the following CLI with great caution knowing that it is very disruptive as it also affects other Firewall's inline cloud analysis services (IOT, enterprise DLP, advanced URL filtering).
```
debug software restart process ctd-agent
```
  1. Note: Restarting ctd-agent will reset the connection between Firewall DP and the Content Cloud server.

Additional Information

Prepare to Deploy APP-ID Cloud Engine
Troubleshoot APP-ID Cloud Engine

The Firewall maintains two connections to the cloud: One connection from Firewall MP to ACE server and another connection from Firewall DP to Content Cloud server.
Below is an example of the output of a good connection between FW and ACE.

> show ctd-agent status security-client

Security Client Ace(1)
        Current cloud server:   ace.hawkeye.services-edge.paloaltonetworks.com:443
        Cloud connection:       connected
        Config:
                Number of gRPC connections: 2, Number of workers: 5
                Debug level: 2, Insecure connection: false, Cert valid: true, Key valid: true, CA count: 383
                Maximum number of workers: 10
                Maximum number of sessions a worker should process before reconnect: 1024
                Maximum number of messages per worker: 0
                Skip cert verify: false
        Grpc Connection Status:
                State Ready (3), last err <nil>
                Pool state: Ready (2)
                     last update: 2023-04-10 11:29:18.888023715 -0700 PDT m=+330284.693222939
                     last connection retry: 2023-04-10 09:27:50.049422541 -0700 PDT m=+322995.854622294
                     last pool close: 2023-04-10 09:27:32.141236266 -0700 PDT m=+322977.946435573

How to troubleshoot gRPC connections failure between Firewall and ACE Application Cloud Engine/Content Cloud

Objective

Environment

Procedure

Additional Information

Other users also viewed: