Error:
An unexpected error occurred. Please click Reload to try again.
Error:
An unexpected error occurred. Please click Reload to try again.
Managed Firewalls showing disconnected from the Panorama even t... - Knowledge Base - Palo Alto Networks

Managed Firewalls showing disconnected from the Panorama even though network connectivity is good.

151185
Created On 06/20/22 07:17 AM - Last Modified 10/24/23 17:39 PM


Symptom


  • Managed Firewalls are disconnected from Panorama which is on PAN-OS 10.1.
    • > show panorama-status
  • Pings between the Firewalls and Panorama are working.
  • Netstat output on the Firewalls show connnections are Established to the Panorama on port 3978.
    • > show netstat all yes numeric-hosts yes numeric-ports yes
      
      Example:
      admin@hio-awsgwlbine1e2pafwA-1> show netstat all yes numeric-hosts yes numeric-ports yes | match 172.31.17.171
      tcp        0      0 10.59.142.180:56872     172.31.17.171:3978      ESTABLISHED
      tcp        0      0 10.59.142.180:33018     172.31.17.171:3978      ESTABLISHED
  • ms.log on the Firewall shows the following errors :-
    • > less mp-log ms.log
       -0700 cmsa: agent index=0
       -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
       -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN
       -0700 [Secure conn] Secure channel for Firewall to panorama communication not enabled for secure conn.
       -0700 Warning:  pan_cmsa_mgmt_assign_ssl_ctx(src_panos/cms_agent.c:2348): client using default (legacy) context
       -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
       -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN
       -0700 COMM: connection established. sock=27 remote ip=XXXX port=3978 local port=60392
       -0700 cms agent: Pre. send buffer limit=46080. s=27
       -0700 cms agent: Post. send buffer limit=2097152. s=27
       -0700 Error:  cs_load_certs_ex(cs_common.c:655): keyfile not exists
       -0700 Error:  pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:883): cms agent: cs_load_certs_ex failed
       -0700 cmsa: client will use default context
       -0700 Error:  sc3_ca_exists(sc3_certs.c:221): SC3: Failed to get the current CA name.
       -0700 Warning:  sc3_init_sc3(sc3_utils.c:351): SC3: Failed to get the Current CC name
       -0700 SC3: CA: '', CC/CSR: 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX'
       -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
       -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN
       -0700 Warning:  sc3_init_sctx(sc3_ctx.c:323): SC3: not set, skip cert loading
       -0700 SC3A: using SNI (from AK): XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX
       -0700 SC3A: using sc3 ctx with no cert
       -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
      ......
      

       


Environment


  • Firewalls managed by Panorama
  • Panorama is on PAN-OS 10.1 and above
  • Managed Firewall PAN-OS 10.1
  • Secured Onboarding 


Cause


  • To strengthen your security posture, PAN-OS 10.1 introduces improved mutual authentication between a new device and Panorama on first connection.
  • Authentication Key for Secure Onboarding has been introduced.


Resolution


Remove the firewall from panorama then re-add it back​​​​​​:
  1. Check if a valid Authentication Key exists on the Panorama :
    • Panorama > Device Registration Auth Key
  2. Create a new one if a valid one does not already exist :
  3. Remove the Firewall from Managed Devices on the Panorama and perform a local Panorama Commit.
    • Panorama > Managed Devices > Summary.
    • Note: You will need to first disassociate the device from the Device Group and Template Stack. If the device is included in the Collector Group's preference list you will need to remove it from here as well.
  4. Remove the Panorama IP address from the Firewall and perform a local Firewall Commit.
    • Device > Setup > Panorama Settings
  5. Add the Firewall as a managed device again on Panorama and perform a local Panorama Commit.
  6. Add the Panorama IP address on the Firewall along with the Authentication Key and perform a local Commit.
  7. Authentication Key can be found at Panorama > Device Registration Auth Key
Reset the secure communication between the firewall and panorama​​​​​​:
    If the firewall is still showing as disconnected, then try to reset secure communication between firewall and panorama.

    How to reset secure communication between firewall and panorama

    Take a management interface packet  capture and investigate​​​​​​:
      After doing the above steps and the firewall is still showing as disconnected, then try to take a management interface pcap and investigate.

      Sample command below has the IP 20.20.20.20 as the IP address of the panorama.

      admin@Lab70-205-PA-460> tcpdump filter "host 20.20.20.20" snaplen 0
      Press Ctrl-C to stop capturing
      dropped privs to tcpdump
      tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes


      Export the results as per the link below:
      How To Packet Capture (tcpdump) On Management Interface

      In this sample capture below, it is seen that 3-way handshake has been established, this would explain why the 'show netstat all yes numeric-hosts yes' command also has the established status.

      The first indication of a problem is seen in packets 66 and 67, where the firewall (.53 IP) sends some large packets (2896 len) to panorama (.20 IP). Going through the capture, we can tell that panorama did not acknowledge these packets. The firewall then retransmitted packets 72 to 89 that also went unacknowledged.

      A device in between the firewall and panorama is most likely dropping large packets of a certain size causing the panorama not to receive these packets.


      mtu.issue.png

      Resolution to this is to try to lower the management interface MTU of the firewall to 1400 or 1300 and see if the condition improves.

      Can the MTU be Changed on the Management Interface?


       


      Actions
      • Print
      • Copy Link

        https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000wkjSCAQ&lang=en_US%E2%80%A9&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

      Choose Language