How to troubleshoot the Firewall onboarding and connection issues with Strata Cloud Manager

• Firewall
• Strata Cloud Manager

1. First make sure that you have followed all the steps in the document Onboard a Firewall.
   1. Allow the communication between Palo Alto Networks Next-Gen Firewall (NGFW) and Strata Cloud Manager on the TCP ports and Fully Qualified Domain Names (FQDN).
2. Check the output of the CLI command:

   > show cloud-management-status
   
   Managed by Cloud Management Service
   Endpoint : cyzf2994-f01f-48f7-ab8c-d1cd4b439200.prod.us.ngfw.cloudmgmt.paloaltonetworks.com
   Connected : yes
   DNS:
   msg: Successfully resolved FQDN
   status: success
   timestamp: 2023/11/07 12:43:15
   TCP:
   msg: TCP channel established
   status: success
   timestamp: 2023/11/07 12:43:15
   SSL:
   msg: SSL channel established
   status: success
   timestamp: 2023/11/07 12:43:15

    
   1. If the Endpoint FQDN doesn't appear in the output of the command above, it most likely means that you haven't configured the firewall to be managed by the cloud service. Navigate to Device > Setup > Management and edit the Panorama Settings.

2. If the DNS doesn't show that it successfully has resolved FQDN in the command above, verify that both ntp and dns are configured properly under Device > Setup > Services 

   > show ntp
   > traceroute host <Endpoint fqdn that is in the output of step 2 >

3. If the TCP channel is not showing as established in the command above, then check the ms.log and look for message similar to below:

   > less mp-log ms.log:
   2023-10-09 13:24:27.060 -0700 Error:  pan_comm_get_tcp_conn_gen(comm_utils.c:702): COMM: cannot connect. remote ip=35.69.247.210 port=3978 err=Connection timed out(110) sock=18
   2023-10-09 13:26:58.612 -0700 Error:  pan_comm_get_tcp_conn_gen(comm_utils.c:702): COMM: cannot connect. remote ip=35.69.247.210 port=3978 err=Connection timed out(110) sock=18
   2023-10-09 13:29:17.876 -0700 Error:  pan_comm_get_tcp_conn_gen(comm_utils.c:702): COMM: cannot connect. remote ip=35.69.247.210 port=3978 err=Connection timed out(110) sock=18
   2023-10-09 13:31:37.140 -0700 Error:  pan_comm_get_tcp_conn_gen(comm_utils.c:702): COMM: cannot connect. remote ip=35.69.247.210 port=3978 err=Connection timed out(110) sock=18
   

   You can also use the output of the CLI command:

   > show netstat numeric-hosts yes numeric-ports yes | match 3978
   tcp        0      1 192.168.34.137:52324      35.69.247.210:3978      SYN_SENT  <<<<<
   

   which indicates that the firewall is sending the SYN but there is a device in between the firewall and the Virtus server / endpoint of the Strata Cloud Manager that is blocking the SYN/ACK. Note: 35.69.247.210 is the IP address that the FQDN of the Virtus server "cyzf2994-f01f-48f7-ab8c-d1cd4b439200.prod.us.ngfw.cloudmgmt.paloaltonetworks.com" (shown in the output of the CLI command in step 2) resolves.
4. If TCP connection is established but the output for the SSL is empty as shown below:

    TCP:
          msg: TCP channel established
          status: success
          timestamp: 2023/12/19 13:37:25
       SSL:
          msg:
          status:
          timestamp:
   
   

   Then this most likely indicates a problem with the SSL handshake or a device blocking the SSL connection. To verify this, perform a tcpdump on the management interface with the host being the IP address to which the Endpoint FQDN resolves:

   > tcpdump filter "host 35.69.247.210"

   Export the packet capture to an SCP server so you can check which device is resetting the connection.

   > scp export mgmt-pcap from mgmt.pcap to user1@192.168.222.88:/home/user1/

   Also, since this traffic is identified as SSL on port 3978, make sure that your gateway FW's security policy rule, if applied to this traffic, is not configured with service "application-default". Otherwise it will not allow this traffic.
    

Palo Alto Networks NGFW (Managed by Strata Cloud Manager) uses the dedicated non-standard port 3978 to communicate with Strata Cloud Manager by default. In PAN-OS 11.2, you can instead configure NGFW (Managed by Strata Cloud Manager) onboarding to Strata Cloud Manager to use destination port 443 instead of port 3978. You can review the new feature here:
https://docs.paloaltonetworks.com/pan-os/11-2/pan-os-release-notes/features-introduced-in-pan-os/management-features?otp=concept-nk2_nxh_2bc#concept-nk2_nxh_2bc

After the upgrade, navigate the UI to Device > Setup > Management > Panorama Settings. Click on the gear icon. Change the TCP port from the default to 443.