Firewalls are disconnected randomly on Panorama

• Firewalls recently upgraded from 10.0 to 10.1 or 10.2 are disconnected from Panorama randomly.
• The firewalls are securely onboarded to Panorama using the authentication key after the upgrade and is in connected state.
• However, these devices are seen disconnected intermittently or after a firewall restart.
• SC3 reset helps to fix the issue temporarily, but the issue re-surfaces.

Firewall > less mp-log ms.log
11:26:27.607 +1000 [Secure conn] Secure channel for Firewall to panorama communication not enabled for secure conn.
pan_cmsa_mgmt_assign_ssl_ctx(src_panos/cms_agent.c:2364): client using default (legacy) context

Panorama > less mp-log configd.log
11:26:31.816 +1000 Warning: sc3_register(sc3_register.c:211): SC3: connstat for 'xxxxx5678901': 0
11:26:31.816 +1000 Warning: pan_cfg_handle_mgt_reg(pan_cfg_mgt_handler.c:4837): SC3: device 'xxxxx5678901' is not SC3 capable

• Panorama managed Next Gen Firewalls
• PAN-OS 10.1.x, 10.2.x and 11.1.x

Due to the fact that the firewalls were already connected to Panorama before the PAN-OS upgrade, it is using the legacy certificate to connect to Panorama, configuring SC3 on top of this will create a conflict and can cause disconnections.

• Maintain only one connection method to the Panorama - either legacy or SC3. Since SC3 is the preferred method, follow the given steps to switch from legacy to SC3.
  1. Login to the firewall CLI.
     1. To verify whether the existing connection is using legacy method, run the command show system state  | match 'cfg.ms' 
     2. If the output is similar to cfg.ms.csr: 12345678-abcd-efgh-ijkl-123456789012, then it is using a legacy connection.
     3. To use new connection method (SC3), the first step is to remove the legacy method by running the command request legacy reset (Note: This command is available from 10.1.14, 10.2.11, 11.0.3 and 11.1.0 onwards)
     4. Then restart the management server - debug software restart process management-server
  2. Set the authentication key on Panorama, if not created already. Log into the Panorama GUI (Panorama tab > Device Registration Auth Key > Add new)
  3. Login back to the firewall CLI and set the authkey created from the previous step.
     1. request authkey set <auth_key> 
     2. Run the command show panorama-status to confirm the firewall is connected 
• Verify the SC3 registration process is successful by referring to this KB article.