HA1 control link with encryption enabled, keeps flapping after upgrade from 9.1 to 10.0
15059
Created On 07/29/22 17:16 PM - Last Modified 09/25/24 13:47 PM
Symptom
- In an Active/Passive or Active/Active HA setup, after an upgrade from 9.1 to 10.0, HA1 link keeps flapping :
2022/05/21 13:48:57 critical ha connect 0 HA Group 4: Control link running on HA1-Backup connection 2022/05/21 13:48:56 critical ha connect 0 HA Group 4: All HA1 connections down 2022/05/21 13:48:56 critical ha connect 0 HA Group 4: HA1 connection down 2022/05/21 13:48:54 info ha connect 0 HA Group 4: HA1 connection up 2022/05/21 13:48:54 info ha ha1-lin 0 HA1-Backup peer link up 2022/05/21 13:48:54 info ha ha1-lin 0 HA1 peer link up 2022/05/21 13:48:54 info ha connect 0 HA Group 4: Control link running on HA1 connection 2022/05/21 13:48:53 critical ha connect 0 HA Group 4: All HA1 connections down 2022/05/21 13:48:53 high ha connect 0 HA Group 4: HA1-Backup connection down 2022/05/21 13:48:51 info ha connect 0 HA Group 4: HA1-Backup connection up 2022/05/21 13:48:51 info ha ha1-lin 0 HA1-Backup peer link up 2022/05/21 13:48:51 info ha ha1-lin 0 HA1 peer link up 2022/05/21 13:48:51 critical ha connect 0 HA Group 4: Control link running on HA1-Backup connection 2022/05/21 13:48:50 critical ha connect 0 HA Group 4: All HA1 connections down 2022/05/21 13:48:50 critical ha connect 0 HA Group 4: HA1 connection down 2022/05/21 13:48:48 info ha connect 0 HA Group 4: HA1 connection up 2022/05/21 13:48:48 info ha ha1-lin 0 HA1-Backup peer link up 2022/05/21 13:48:48 info ha ha1-lin 0 HA1 peer link up
- Error message seen in the ha_agent logs:
mp ha_agent.log 2022-05-21 13:40:56 2022-05-21 13:40:56.170 +1000 debug: ha_peer_send_error(src/ha_peer.c:1725): Group 4 (HA1-MAIN): Sending errro message mp ha_agent.log 2022-05-21 13:40:56 mp ha_agent.log 2022-05-21 13:40:56 Error Msg mp ha_agent.log 2022-05-21 13:40:56 --------- mp ha_agent.log 2022-05-21 13:40:56 flags : 0x2 (close:) mp ha_agent.log 2022-05-21 13:40:56 err code : SSH Tunnel reset (17) mp ha_agent.log 2022-05-21 13:40:56 num tlvs : 1 mp ha_agent.log 2022-05-21 13:40:56 Printing out 1 tlvs mp ha_agent.log 2022-05-21 13:40:56 TLV[1]: type 5 (ERR_STRING); len 17; value: mp ha_agent.log 2022-05-21 13:40:56 53534820 54756e6e 656c2072 65736574 00 mp ha_agent.log 2022-05-21 13:40:56 mp ha_agent.log 2022-05-21 13:40:56 2022-05-21 13:40:56.170 +1000 Error: ha_peer_disconnect(src/ha_peer.c:1860): Group 4 (HA1-MAIN): peer connection error msg set: SSH Tunnel reset
- Encryption is enabled on HA1 :
> show high-availability all <snip> Jumbo-Frames disabled; MTU 1500 HA1 Control Links Joint Configuration: Link Monitor Interval: 3000 ms Encryption Enabled: yes. <<<<<<<<
Environment
HA Upgrade from 9.1 to 10.0
Cause
This is expected behaviour until the HA upgrade is complete
With 10.0, we use aes128-ctr for SSH while in 9.1, we use aes128-cbc. Due to this difference in algorithm, SSH Tunnel keeps resetting and hence HA1 flaps
Resolution
Due to the root cause mentioned above, it is recommended to disable HA1 encryption
Step 6 : https://docs.paloaltonetworks.com/pan-os/10-0/pan-os-new-features/upgrade-pan-os/upgrade-the-firewall-pan-os/upgrade-an-ha-firewall-pair#id062f1ad5-adb3-4d25-b4a4-529bde5dc96a
1. Select DeviceHigh AvailabilityGeneral and edit the Control Link (HA1) section.
2. Uncheck Encryption Enabled so that it is disabled.
3. Commit the change on the HA both peers.
Note**: Disabling encryption shouldn’t cause any unexpected behavior like reboot
Additional Information
- References :
10.0 upgrade reference for HA
Upgrade an HA Firewall Pair
https://docs.paloaltonetworks.com/pan-os/10-0/pan-os-new-features/upgrade-pan-os/upgrade-the-firewall-pan-os/upgrade-an-ha-firewall-pair#id062f1ad5-adb3-4d25-b4a4-529bde5dc96a