How To Avoid HA Split-Brain due to Missed Heartbeats
Resolution
Issue
Palo Alto Networks uses a private heartbeat link to monitor the health and status of each node in a high availability cluster. Split-brain occurs when the private link goes down, but the cluster nodes are still up. Each node believes that the other is no longer functioning and attempts to start services that the other is running. In some instances the link may not be down, but due to high load on the dataplane, heartbeats may be missed.
Example logs of heartbeat ping failures are shown below:
Resolution
To prevent split-brain due to missed heartbeats, the Heartbeat Backup option should be selected when configuring HA. By selecting this option, the firewalls will use the management ports to provide a backup path for heartbeat and hello messages. The option is found on the WebUI under Device > High Availability > General > Election Settings
owner: panagent