Unexpected fail over of Active Panorama when using peer IP address for HA path-monitoring
13570
Created On 04/12/19 09:58 AM - Last Modified 04/30/19 22:51 PM
Symptom
- Unexpected fail over from Active when the Passive unit was unresponsive.
- Active unit went into "non-functional" state when the Passive was unresponsive.
- HA path-monitoring configured for HA peer IP address.
- ha_agent.log:
2019-04-09 00:23:58.303 -0400 Error: ha_ping_peer_miss(src/ha_ping.c:756): Missed 1 ping timeouts out of 3 (ha1)
2019-04-09 00:23:59.304 -0400 Error: ha_ping_peer_miss(src/ha_ping.c:756): Missed 2 ping timeouts out of 3 (ha1)
2019-04-09 00:24:00.304 -0400 Error: ha_ping_peer_miss(src/ha_ping.c:756): Missed 3 ping timeouts out of 3 (ha1)
2019-04-09 00:24:00.304 -0400 Error: ha_ping_peer_miss(src/ha_ping.c:763): We have missed 4 pings from the peer for group 0 (ha1), restarting connection
2019-04-09 00:24:00.304 -0400 Warning: ha_event_log(src/ha_event.c:47): HA1 connection down
2019-04-09 00:24:00.305 -0400 debug: ha_peer_start(src/ha_peer.c:246): Group 0 (HA1-MAIN): waiting for ping response before starting connection
2019-04-09 00:24:00.305 -0400 Group 0: Path 'Path_mon' destination ip '192.168.1.2' state is going from up to down
2019-04-09 00:24:00.305 -0400 Warning: ha_event_log(src/ha_event.c:47): Path group 'Path_mon' destination IP '192.168.1.2' is down
2019-04-09 00:24:00.305 -0400 Going to non-functional for reason Path down
2019-04-09 00:23:59.304 -0400 Error: ha_ping_peer_miss(src/ha_ping.c:756): Missed 2 ping timeouts out of 3 (ha1)
2019-04-09 00:24:00.304 -0400 Error: ha_ping_peer_miss(src/ha_ping.c:756): Missed 3 ping timeouts out of 3 (ha1)
2019-04-09 00:24:00.304 -0400 Error: ha_ping_peer_miss(src/ha_ping.c:763): We have missed 4 pings from the peer for group 0 (ha1), restarting connection
2019-04-09 00:24:00.304 -0400 Warning: ha_event_log(src/ha_event.c:47): HA1 connection down
2019-04-09 00:24:00.305 -0400 debug: ha_peer_start(src/ha_peer.c:246): Group 0 (HA1-MAIN): waiting for ping response before starting connection
2019-04-09 00:24:00.305 -0400 Group 0: Path 'Path_mon' destination ip '192.168.1.2' state is going from up to down
2019-04-09 00:24:00.305 -0400 Warning: ha_event_log(src/ha_event.c:47): Path group 'Path_mon' destination IP '192.168.1.2' is down
2019-04-09 00:24:00.305 -0400 Going to non-functional for reason Path down
Environment
- Panorama HA
Cause
- The HA peers use hello messages and heartbeats to verify that the peer is responsive and operational. The heartbeat is an ICMP ping to the HA peer, and the peer responds to the ping to establish that the peers are connected and responsive.
- Path monitoring checks the network connectivity and link state for an IP address or group of IP addresses (path group). The active peer uses ICMP pings to verify that one or more destination IP addresses can be reached.
- Thus, if one puts HA peer IP address for path-monitoring, following will happen on active node when the passive node becomes unresponsive for some reason:
> Heartbeats are missed from peer, HA link will go down.
> Path monitoring will fail which will cause the HA state to transition from active to non-functional.
Resolution
- HA peer IP should not be configured under HA path monitoring.
- Additional Note: This same logic can be applied on PANOS firewalls configured in HA depending on network topology.
Additional Information
References:
https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-admin/high-availability/ha-concepts/failover.html
https://docs.paloaltonetworks.com/panorama/8-1/panorama-admin/panorama-high-availability/failover-triggers/ha-heartbeat-polling-and-hello-messages.html