HA Non-Functional Loop behavior when HA clustering is enabled

Both firewalls in HA pair go to Suspended state when HA clustering is enabled and soft failures are detected

• HA pair member of an HA cluster
• Soft failures happening on both members of the HA pair

In an HA pair (no HA clustering enabled), when the active device experiences a path monitoring failure and the other device is in a state allowing it to take over traffic forwarding (e.g., passive state), then the device experiencing path monitoring failures goes into non-functional state and the passive device transitions to active state. However, if the other device is in a state that does not allow it to take over traffic forwarding (e.g., non-functional or suspended state), then despite the path monitoring failure, the active device does not go to non-functional state. Instead, it remains in active state and only logs the path monitoring failure. This is so-called 'last device standing' behavior - path monitoring failures do not take down the 'last device standing' and do not move it out of active state. 

If HA clustering is enabled and a device is in a cluster-active state, then path monitoring failures will be processed and acted upon (not only logged) regardless of the cluster states of the other cluster members. This behavior does not change even if the cluster member is also a member of an HA pair. It implies that even if one of the HA pair members is in non-active state (e.g., non-functional or suspended), the other HA pair member is allowed to transition from active to non-functional state in case of a path monitoring failure. In other words, with HA clustering enabled, path monitoring failures can take the 'last device standing' down.

Acting upon a path monitoring failure results in a "loop" in most cases if the path monitoring continues to be down, particularly for path monitoring in an A/P HA pair. In order to monitor the path, a device that detected a failure must come back to an active state. Upon detecting a failure again, the device goes to non-functional state again. Therefore, an A/P HA pair will loop through active/passive/non-functional states, particularly if path monitoring is enabled and is failing on both peers. The loop is broken by the "flap-max" count, which when reached on one device, puts that device in a suspended state. 

When one of the devices of an A/P HA pair goes into suspended state, then there are two possibilities:

1. In a "pure" HA pair (no clustering enabled), the device that is not suspended remains in active state ('last device standing') and the loop is discontinued despite the continuous path monitoring failure.
   In an HA pair that is a member of an HA cluster, the active device continues the path monitoring and if it keeps failing, the device keeps looping between active and non-functional states (the other device is suspended and not participating in the HA state changes).
2. Once the non-suspended device gets to its flap-max (also in play for HA clustering), it goes to a suspended state.

In case of an A/A HA pair, state transitions are slightly different. With clustering enabled, if both devices experience path-monitoring failure, then one of them will transition (or remain) in Active-Primary HA state and the other will move to Non-Functional HA state. Both devices will enter cluster-non-functional state. The cluster state of the Active-Primary device loops through cluster-active and cluster-non-functional states. The other device loops through Tentative and Non-Functional HA states and through cluster-active and cluster-non-functional cluster states. After "flap-max" transitions to cluster-non-functional state, both devices enter suspended state.

In summary, when a HA clustering is enabled for an HA pair and both HA peers experience path monitoring failures, then both HA peers will end up in suspended state.

In order to bring the devices out of suspended state, manual intervention is required.

Note that the discussion above used path monitoring as an example. Equivalent behavior is seen when the firewalls experience other soft failures (link or slot failure).

The behavior where both HA peers can go to suspended state when they are in an HA cluster helps customers with multiple datacenters housing a separate HA pair in each datacenter who would like to failover the entire datacenter to another datacenter if both devices in an HA pair experience soft failure. In this scenario, if both devices of an HA pair experience a soft failure, it implies that there is a general connectivity issue in that particular datacenter. Keeping the 'last device standing' and not failing the entire HA pair would result in service degradation for the traffic affected by the connectivity issue.

None. This is expected behavior.

• The HA cluster state will generally follow the HA pair state if the device is in both an HA cluster and an HA pair. For an HA pair state of active, active-primary, or active-secondary, the cluster state will be cluster-active if there are no soft failures. In case of soft failures, for an HA pair state of active, active-primary, active-secondary, or tentative, the cluster state will follow its own state machine and can be in any cluster state. Passive and non-functional states map to cluster-non-functional, and all other HA states match to similarly named cluster states.
• The HA pair state of any HA active device can still be put into a non-functional cluster state because all faults considered by cluster state are still in play when a HA pair state is active
  • This means that an HA pair active state member can be in a HA cluster non-functional state particularly when a HA pair active state member is there due to "last device standing"
  • Soft faults like link/path monitoring failures and slot mismatch errors would result in a "last device standing" in an HA pair, but if that HA pair is in an HA cluster, then it continues to honor the faults that are normally ignored when the HA pair peer device is in one of the states that are NOT functional.
  • This is different state behavior between HA pairs and an HA pair within an HA cluster allowing HA cluster pairs to behave more like stand-alone devices in an HA cluster; allowing the full DC to go into failure in the case that soft faults fail as they can with a stand-alone device within an HA cluster.
• Link and path monitoring configuration does not have to match on the HA peers and this may be a potential workaround in certain scenarios if a customer wants to avoid ending up with both devices in suspended state. However, implementing such a configuration necessitates excellent knowledge of the environment and the most common "expected" types of failures. It may not be applicable in all situations.