Azure Health Probes fail when the load balancer is probing through two separate interfaces on the firewall using the same source IP address
12393
Created On 01/17/23 00:35 AM - Last Modified 04/05/24 03:04 AM
Symptom
- Azure Load Balancer (Sandwich Setup) fails to probe the firewall's dataplane interfaces.
- This causes traffic disruption as load balancers do not forward traffic due to unhealthy instances.
Environment
- External and Internal Azure load balancers probing from the same source IP
- Supported PAN-OS versions
- PA-VM Platform
- one Virtual Router setup
Cause
- The health check probes from the load balancers are sourced from the same IP address
- If you have one load balancer for inbound traffic and the other for outbound traffic and both are probing through two different interfaces on the firewall (Example: Inbound interface etherent1/2 and Outbound interface ethernet1/1) and sourced from the same IP, then arp requests will fail on the firewall
- Arp requests will fail as the firewall cannot have the same arp entry for the same source IP on two different interfaces under the same Virtual Router
Resolution
- Confirm load balancer health check probes are failing due to "arp".
- This can be done by setting up the packet capture filters using the load balancers probing IP and running the following command from CLI. It should show drops due to "no arp" as shown below
> show counter global filter delta yes packet-filter yes | match drop
flow_fwd_l3_noarp 2 0 drop flow forward Packets dropped: no ARP
- Once confirmed, The issue can be resolved by creating two Virtual Routers with inbound interface attached to one Virtual Router and outbound interface attached to the other.
- This would allow each Virtual Router to arp for the same source IP address without conflicting with each other
- Ensure a route is added to the Probing IP address on each Virtual Router pointing back out their respective interfaces
Additional Information
Note: When adding an additional Virtual Router, Add internal/external routes as needed on each VR using "next-vr" as next hop to ensure proper routing between them.