Port flaps observed on previous Active Firewall during HA Failover
31252
Created On 03/26/19 08:50 AM - Last Modified 04/16/19 20:28 PM
Symptom
When there is a HA failover from Active to Passive firewall, we see that some ports on previous active firewall go down and then come up. This article explains the behaviour of port flaps observed.
Environment
The logic mentioned in this article applies to all platforms, but based on hardware implementation, symptoms may differ. The specific example in this article is for PA-5000 devices.
Cause
• During HA failover, firewall has to disable the ports on the firewall which was previously active, so that they do not affect network traffic.
• The system achieves this by disabling the MAC layer (layer 2 protocol) on the ports.
• This is also required because this action explicitly tell the peer switch to refresh MAC-PORT table, so that peer switch can immediately change the traffic flow to the new active.
• Based on certain port properties, hardware model, HA State Change, and HA-Passive-Link-State setting, some ports require a complete port reset to disable the MACs
• So port flapping may be observed on certain kind of ports.
The below is a summary of expected behaviour on PA-5000 series devices when it goes from Active-Passive, Active->Non-functional, Passive->Active
1. All copper and SFP ports which are NOT doing LACP Pre-negotiation will flap as the system needs to disable their MACs during HA state change.
2. All SFP+ Ports (Port 21-24) are also reset internally, but they would not show up as flap on system logs. Our hardware implementation allows to disable their MACs without flapping the ports.
3. All Ports (copper/SFP/SFP+) with LACP Pre-negotiation or LLDP turned on are not disabled as the LACPDUs need to be processed during failovers (Not applicable for HA Initial State)
4. All ports which are not configured will also be disabled.
The above events can be confirmed with a combination of pan_dha and system logs:
(In below example, Port 1,2 are LACP ports; Port 5,6 are standalone 1G ports, Ports 23,24 are standalone SFP+ Ports)
2019-03-13 06:26:32.272 +0530 Dataplane HA agent state change callback invoked: local Active => Non-Functional 2019-03-13 06:26:32.272 +0530 Enable link for pre-negotiation 2019-03-13 06:26:32.272 +0530 set interface link properties: name ethernet1/1 speed auto duplex auto state up disable no <<<<< Not disabled because of pre-negotiation 2019-03-13 06:26:32.272 +0530 Enable link for pre-negotiation 2019-03-13 06:26:32.300 +0530 set interface link properties: name ethernet1/2 speed auto duplex auto state up disable no <<<<< Not disabled because of pre-negotiation 2019-03-13 06:26:32.300 +0530 set interface link properties: name ethernet1/5 speed auto duplex auto state auto disable yes <<<<< Copper Port being disabled 2019-03-13 06:26:32.308 +0530 set interface link properties: name ethernet1/6 speed auto duplex auto state auto disable yes <<<<< Copper Port being disabled 2019-03-13 06:26:32.382 +0530 set interface link properties: name ethernet1/23 speed auto duplex auto state auto disable yes <<<<< SFP+ Port being disabled 2019-03-13 06:26:32.387 +0530 set interface link properties: name ethernet1/24 speed auto duplex auto state auto disable yes <<<<< SFP+ Port being disabled
2019/03/13 06:26:32 info port ethern link-ch 0 Port ethernet1/24: Up 10Gb/s-full duplex 2019/03/13 06:26:32 info port ethern link-ch 0 Port ethernet1/23: Up 10Gb/s-full duplex 2019/03/13 06:26:32 info port ethern link-ch 0 Port ethernet1/6: Down auto duplex 2019/03/13 06:26:32 high ha link-mo 0 HA Group 1: Link group 'SMTP-grp' link 'ethernet1/5' is down 2019/03/13 06:26:32 info port ethern link-ch 0 Port ethernet1/5: Down auto duplex 2019/03/13 06:26:32 critical ha state-c 0 HA Group 1: Moved from state Active to state Non-Functional
Notice above no port down messages are seen for Port 1,2,23,24
Resolution
The above behaviour is expected and does not cause any issues unless the ports come up fine.
Additional Information
Note: If the device is going into initial state, then all ports are disabled. The hardware implementation of ports still dictate which ports will be seen flapping on system logs.