Managed Firewalls showing disconnected from the Panorama even though network connectivity is good.

100862

Created On 06/20/22 07:17 AM - Last Modified 10/24/23 17:39 PM

Deployment

Device Management

10.2

10.1

PAN-OS

Panorama

Symptom

Managed Firewalls are disconnected from Panorama which is on PAN-OS 10.1.
- ```
> show panorama-status
```
Pings between the Firewalls and Panorama are working.

Netstat output on the Firewalls show connnections are Established to the Panorama on port 3978.

> show netstat all yes numeric-hosts yes numeric-ports yes

Example:
admin@hio-awsgwlbine1e2pafwA-1> show netstat all yes numeric-hosts yes numeric-ports yes | match 172.31.17.171
tcp        0      0 10.59.142.180:56872     172.31.17.171:3978      ESTABLISHED
tcp        0      0 10.59.142.180:33018     172.31.17.171:3978      ESTABLISHED

ms.log on the Firewall shows the following errors :-

> less mp-log ms.log
 -0700 cmsa: agent index=0
 -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
 -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN
 -0700 [Secure conn] Secure channel for Firewall to panorama communication not enabled for secure conn.
 -0700 Warning:  pan_cmsa_mgmt_assign_ssl_ctx(src_panos/cms_agent.c:2348): client using default (legacy) context
 -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
 -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN
 -0700 COMM: connection established. sock=27 remote ip=XXXX port=3978 local port=60392
 -0700 cms agent: Pre. send buffer limit=46080. s=27
 -0700 cms agent: Post. send buffer limit=2097152. s=27
 -0700 Error:  cs_load_certs_ex(cs_common.c:655): keyfile not exists
 -0700 Error:  pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:883): cms agent: cs_load_certs_ex failed
 -0700 cmsa: client will use default context
 -0700 Error:  sc3_ca_exists(sc3_certs.c:221): SC3: Failed to get the current CA name.
 -0700 Warning:  sc3_init_sc3(sc3_utils.c:351): SC3: Failed to get the Current CC name
 -0700 SC3: CA: '', CC/CSR: 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX'
 -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
 -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN
 -0700 Warning:  sc3_init_sctx(sc3_ctx.c:323): SC3: not set, skip cert loading
 -0700 SC3A: using SNI (from AK): XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX
 -0700 SC3A: using sc3 ctx with no cert
 -0700 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
......

Environment

Firewalls managed by Panorama
Panorama is on PAN-OS 10.1 and above
Managed Firewall PAN-OS 10.1
Secured Onboarding

Cause

To strengthen your security posture, PAN-OS 10.1 introduces improved mutual authentication between a new device and Panorama on first connection.
Authentication Key for Secure Onboarding has been introduced.

Resolution

Remove the firewall from panorama then re-add it back:

Check if a valid Authentication Key exists on the Panorama :
- Panorama > Device Registration Auth Key
Create a new one if a valid one does not already exist :
- Authentication Key for Secure Firewall Onboarding
- Panorama Device Registration Auth Key.
Remove the Firewall from Managed Devices on the Panorama and perform a local Panorama Commit.
- Panorama > Managed Devices > Summary.
- Note: You will need to first disassociate the device from the Device Group and Template Stack. If the device is included in the Collector Group's preference list you will need to remove it from here as well.
Remove the Panorama IP address from the Firewall and perform a local Firewall Commit.
- Device > Setup > Panorama Settings
Add the Firewall as a managed device again on Panorama and perform a local Panorama Commit.
Add the Panorama IP address on the Firewall along with the Authentication Key and perform a local Commit.
Authentication Key can be found at Panorama > Device Registration Auth Key

Reset the secure communication between the firewall and panorama:

If the firewall is still showing as disconnected, then try to reset secure communication between firewall and panorama.

How to reset secure communication between firewall and panorama

Take a management interface packet capture and investigate:

After doing the above steps and the firewall is still showing as disconnected, then try to take a management interface pcap and investigate.

Sample command below has the IP 20.20.20.20 as the IP address of the panorama.

admin@Lab70-205-PA-460> tcpdump filter "host 20.20.20.20" snaplen 0
Press Ctrl-C to stop capturing
dropped privs to tcpdump
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

Export the results as per the link below:
How To Packet Capture (tcpdump) On Management Interface

In this sample capture below, it is seen that 3-way handshake has been established, this would explain why the 'show netstat all yes numeric-hosts yes' command also has the established status.

The first indication of a problem is seen in packets 66 and 67, where the firewall (.53 IP) sends some large packets (2896 len) to panorama (.20 IP). Going through the capture, we can tell that panorama did not acknowledge these packets. The firewall then retransmitted packets 72 to 89 that also went unacknowledged.

A device in between the firewall and panorama is most likely dropping large packets of a certain size causing the panorama not to receive these packets.

mtu.issue.png

Resolution to this is to try to lower the management interface MTU of the firewall to 1400 or 1300 and see if the condition improves.

Can the MTU be Changed on the Management Interface?

Managed Firewalls showing disconnected from the Panorama even though network connectivity is good.

Symptom

Environment

Cause

Resolution

Other users also viewed: