GlobalProtect Clients Not Able To Access Resources Intermittently After HA Failover
9384
Created On 11/18/21 04:00 AM - Last Modified 04/07/22 19:13 PM
Symptom
- In High-Availability, after a failover, connected GlobalProtect client loses access to resources.
- Sometimes, users are not able to access resources over GP.
- Global Counters on the CLI shows:
flow_tunnel_encap_err 1 0 drop flow tunnel Packet dropped: tunnel encapsulation error flow_tunnel_encap_resolve 1 0 info flow tunnel tunnel structure lookup resolve
- In packet diag (flow basic, tunnel flow), we see:
- Packet is forwarded to incorrect tunnel interface. It should be tunnel.10 instead of tunnel.11
- Packet dropped, tunnel resolution failure
== 2021-11-17 20:28:10.191 -0600 ==
Packet received at fastpath stage, tag 213075, type ATOMIC
Packet info: len 74 port 72 interface 72 vsys 1
wqe index 551795 packet 0x0x800000037dcd00e6, HA: 0, IC: 0
Packet decoded dump:
L2: d4:78:9b:50:49:7f->b4:0c:25:e0:40:48, type 0x0800
IP: 8.8.8.8->172.16.16.16, protocol 1
version 4, ihl 5, tos 0x00, len 60,
id 20797, frag_off 0x0000, ttl 63, checksum 28257(0x6e61)
ICMP: type 0, code 0, checksum 21586, id 1, seq 265
Flow fastpath, session 213075 s2c (set work 0x800000037740d900 exclude_video 0 from sp 0x8000000193b65700 exclude_video 0)
Forwarding lookup, ingress interface 72
L3 mode, router 1
Route lookup in router 1, IP 172.16.16.16
Route found, interface tunnel.11, zone 18, nexthop 172.16.16.0
Packet enters tunnel encap stage, tunnel interface null
== 2021-11-17 20:28:10.192 -0600 ==
Packet received at forwarding stage, tag 213075, type ATOMIC
Packet info: len 74 port 72 interface 132 vsys 1
wqe index 551795 packet 0x0x800000037dcd00e6, HA: 0, IC: 0
Packet decoded dump:
L2: d4:78:9b:50:49:7f->b4:0c:25:e0:40:48, type 0x0800
IP: 8.8.8.8->172.16.16.16, protocol 1
version 4, ihl 5, tos 0x00, len 60,
id 20797, frag_off 0x0000, ttl 62, checksum 28513(0x6f61)
ICMP: type 0, code 0, checksum 21586, id 1, seq 265
Packet enters tunnel encap stage, tunnel interface null
Resolving tunnel 6
Packet dropped, tunnel resolution failure
- Looking at route table, we see two routes for the GP client IP pool. In the FIB table, unexpected route through tunnel.11 is getting installed.
admin@Lab(active)> show routing fib | match 172.16.16 422 172.16.16.0/24 172.16.16.0 ug tunnel.11 1500 admin@Lab(active)> show routing route | match 172.16.16 172.16.16.0/24 172.16.16.0 10 A S E tunnel.11 172.16.16.0/24 172.16.16.0 10 S E tunnel.10
Environment
- Firewall Configured with multiple Global Protect Gateway Configurations
Cause
- The reason two routes are seen in routing table is because same Client IP Pool subnet is used in multiple GP gateways.
- In this example, both GWs: GP_External_Gateway and GP_External_Gateway_CL are using 172.16.16.0/24 client IP pool:
Resolution
- Do not user overlapping Client IP Pool subnet in multiple GP gateways.