After Enabling Active/Passive HA the Network Connectivity is Lost Through the Firewalls
Resolution
Issue
In an environment where the firewall is present and network connectivity is working through the firewall, if another firewall is brought in to create a High Availability (HA) pair then connectivity may be lost for any connected devices when the HA is enabled even if the new firewall is acting as the passive unit and has its non-HA interfaces disabled or turned off.
Cause
After enabling HA, it is expected that the MAC addresses of both devices on non-HA interfaces change to a completely different common MAC address. If there are any ARP entries with the old MAC addresses on routers or hosts that are in the directly connected networks then they will transmit to the wrong MAC address until the ARP entry is updated on those devices. An example is the next hop router toward the Internet could have the MAC address of the primary firewall interface before the HA was enabled causing the firewall not to receive or to reject those frames since the destination MAC address is not the current assigned one.
The same problem could also occur if HA was disabled in an environment where HA was enabled and working.
Example
Primary Before HA
admin@PA-500-1> show interface all
total configured hardware interfaces: 9
name id speed/duplex/state mac address
--------------------------------------------------------------------------------
ethernet1/1 16 100/full/up 00:1b:17:aa:bb:10
ethernet1/2 17 100/full/up 00:1b:17:aa:bb:11
ethernet1/3 18 unknown/unknown/down 00:1b:17:aa:bb:12
ethernet1/4 19 100/full/up 00:1b:17:aa:bb:13
ethernet1/7 22 1000/full/up 00:1b:17:aa:bb:16
ethernet1/8 23 1000/full/up 00:1b:17:aa:bb:17
vlan 1 [n/a]/[n/a]/up 00:1b:17:aa:bb:01
loopback 3 [n/a]/[n/a]/up 00:1b:17:aa:bb:03
tunnel 4 [n/a]/[n/a]/up 00:1b:17:aa:bb:04
Primary After HA
admin@PA-500-1(active)> show interface all
total configured hardware interfaces: 9
name id speed/duplex/state mac address
--------------------------------------------------------------------------------
ethernet1/1 16 100/full/up 00:1b:17:00:11:10
ethernet1/2 17 100/full/up 00:1b:17:00:11:11
ethernet1/3 18 unknown/unknown/down 00:1b:17:00:11:12
ethernet1/4 19 100/full/up 00:1b:17:00:11:13
ethernet1/7 22 1000/full/up 00:1b:17:aa:bb:16
ethernet1/8 23 1000/full/up 00:1b:17:aa:bb:17
vlan 1 [n/a]/[n/a]/up 00:1b:17:00:11:01
loopback 3 [n/a]/[n/a]/up 00:1b:17:00:11:03
tunnel 4 [n/a]/[n/a]/up 00:1b:17:00:11:04
Secondary Before HA
admin@PA-500-2> show interface all
total configured hardware interfaces: 9
name id speed/duplex/state mac address
--------------------------------------------------------------------------------
ethernet1/1 16 unknown/unknown/down 00:1b:17:cc:dd:10
ethernet1/2 17 unknown/unknown/down 00:1b:17:cc:dd:11
ethernet1/3 18 unknown/unknown/down 00:1b:17:cc:dd:12
ethernet1/4 19 unknown/unknown/down 00:1b:17:cc:dd:13
ethernet1/7 22 1000/full/up 00:1b:17:cc:dd:16
ethernet1/8 23 1000/full/up 00:1b:17:cc:dd:17
vlan 1 [n/a]/[n/a]/up 00:1b:17:cc:dd:01
loopback 3 [n/a]/[n/a]/up 00:1b:17:cc:dd:03
tunnel 4 [n/a]/[n/a]/up 00:1b:17:cc:dd:04
Secondary After HA
admin@PA-500-2(passive)> show interface all
total configured hardware interfaces: 9
name id speed/duplex/state mac address
--------------------------------------------------------------------------------
ethernet1/1 16 unknown/unknown/down 00:1b:17:00:11:10
ethernet1/2 17 unknown/unknown/down 00:1b:17:00:11:11
ethernet1/3 18 unknown/unknown/down 00:1b:17:00:11:12
ethernet1/4 19 unknown/unknown/down 00:1b:17:00:11:13
ethernet1/7 22 1000/full/up 00:1b:17:cc:dd:16
ethernet1/8 23 1000/full/up 00:1b:17:cc:dd:17
vlan 1 [n/a]/[n/a]/up 00:1b:17:00:11:01
loopback 3 [n/a]/[n/a]/up 00:1b:17:00:11:03
tunnel 4 [n/a]/[n/a]/up 00:1b:17:00:11:04
MAC addresses have been obfuscated, but done consistently so the example still shows the change.
Other devices off of the ethernet1/1 network may have 00:1b:17:aa:bb:10 instead of 00:1b:17:00:11:10 after the HA was enabled. Until the router gets the correct MAC address then packets sent towards the firewall from the router won't make it.
Resolution
- Clear the ARP entries on any routers or hosts who have ARP entries with the incorrect MAC address
- Wait for the old ARP entries to time out if the HA is not in a production environment
- Do not disable or enable HA as a troubleshooting step without expecting to account for the change in MAC addresses
owner: astanton