GlobalProtect Dual Stack: IPSec connection failed due to keepalive
18261
Created On 04/20/21 00:00 AM - Last Modified 10/23/21 20:16 PM
Symptom
- IPSec connection failed due to keepalive. See Below for example
(P10688-T8416)Debug( 166): 04/19/21 11:47:32:425 Trying to do ipsec connection to 2001:130:800b:1000:1020:0:2b:8[4501] (P10688-T8416)Debug( 487): 04/19/21 11:47:32:425 socket send buffer old size is 65536 (P10688-T8416)Debug( 511): 04/19/21 11:47:32:425 socket send buffer new size is 3145728 (P10688-T8416)Debug( 563): 04/19/21 11:47:32:425 Network is reachable (P10688-T8416)Info ( 178): 04/19/21 11:47:32:425 Connected to: 2001:130:800b:1000:1020:0:2b:8[4501], Sending keep alive to ipsec socket... (P10688-T8416)Info ( 221): 04/19/21 11:47:38:456 failed to receive keep alive (P10688-T8416)Debug( 229): 04/19/21 11:47:38:456 IPSec anti-replay statistics: outside window count 0, replay count 0 (P10688-T8416)Debug( 231): 04/19/21 11:47:38:456 Disconnect udp socket (P10688-T8416)Info ( 364): 04/19/21 11:47:38:456 Connecting to 2001:130:800b:1000:1020:0:2b:8 failed
- Global Protect connection is then established with SSL Tunnel. See below for example
(P10688-T8416)Debug( 766): 04/19/21 11:47:38:456 IPSec fallback reason is IPSec connection failed (P10688-T8416)Debug( 171): 04/19/21 11:47:38:456 VPN idle timeout is 10800; config timeout is 10800 (P10688-T8416)Debug( 219): 04/19/21 11:47:38:487 EnforceDns is enabled, set 2 GP pushed DNS servers (P10688-T8416)Debug( 65): 04/19/21 11:47:38:487 Trying to do SSL connection to 2001:130:800b:1000:1020:0:2b:8(443) (P10688-T8416)Debug( 788): 04/19/21 11:47:38:487 SSL connecting to 2001:130:800b:1000:1020:0:2b:8 (P10688-T8416)Debug( 487): 04/19/21 11:47:38:487 socket send buffer old size is 65536 (P10688-T8416)Debug( 511): 04/19/21 11:47:38:487 socket send buffer new size is 3145728 (P10688-T8416)Debug( 563): 04/19/21 11:47:38:503 Network is reachable (P10688-T8416)Debug(1274): 04/19/21 11:47:38:518 Failed to X509_LOOKUP_load_file (P10688-T8416)Debug( 374): 04/19/21 11:47:38:518 Open_SSL_connection: subject '/CN=testdualgp.com' (P10688-T8416)Debug( 378): 04/19/21 11:47:38:518 Open_SSL_connection: issuer '/CN=GP-CERT' (P10688-T8416)Info ( 113): 04/19/21 11:47:38:534 Connected ssl tunnel to 2001:130:800b:1000:1020:0:2b:8(443) (P10688-T8416)Info ( 374): 04/19/21 11:47:38:534 tunnel to 2001:130:800b:1000:1020:0:2b:8 connected
Environment
- GlobalProtect Configured with Dual Stack
- Environment:
Client Assigned ip by Gateway: 10.10.10.1
Gateway ipv4 pool subnet: 10.10.10.0/24
Gateway ipv6 pool subnet: Not configured
Client Private ipv4: 172.17.8.10
Client ipv6: 2620:000:800b:1000:feed:000:0:10
Portal/Gateway: testdualgp.com
Portal/Gateway resolves to: 10.46.43.8
2001:000:800b:1000:1020::xx:8
Gateway ipv4 pool subnet: 10.10.10.0/24
Gateway ipv6 pool subnet: Not configured
Client Private ipv4: 172.17.8.10
Client ipv6: 2620:000:800b:1000:feed:000:0:10
Portal/Gateway: testdualgp.com
Portal/Gateway resolves to: 10.46.43.8
2001:000:800b:1000:1020::xx:8
- No ipv6 pool is configured in the gateway
Cause
- Land Attack : Client Assigned IP Address is being NAT'ed to the ipv4 gateway address, which will cause the firewall to see the keepalive icmp packets as land attack and then drop the packets
- There is no security policy to allows the keepalive icmp packets, hence traffic is getting denied by a block rule or by default inter-zone rule
- Routing/PBF: Tunnel Traffic is being routed/forwarded through a different interface
Resolution
- Configure a NO NAT rule for the keepalive icmp packets like the one below: POLICIES>NAT Configure NAT
- Make sure there is a Security Rule that allows traffic between the zones like the one below: POLICIES>Security Create a Security Policy Rule
- Ensure there is proper route/pbf for the traffic
Additional Information
- Keepalive packets is used to check if the gateway is reachable when there is no network activity. It also helps in keeping the tunnel up. Keep alive packet for IPSEC is slightly different from that of SSL. For IPSEC, keep alive uses icmp packets while SSL uses keep-alive message type in a SSL packet header. This is why SSL tunnel is successfully established where IPSEC tunnel fails.
- Keepalive icmp packets is between the Assigned IP and the Gateway IP address. In a dual stack deployment where ipv6 pool is not configured and ipv6 is preferred, the IPSEC Tunnel will be formed between the Client ipv6 address (2620:130:800b:1000:feed:238:0:10) and the Gateway ipv6 address (2001:130:800b:1000:1020::2b:8). While the keepalive icmp packet will be between the ipv4 Assigned IP (10.10.10.1) and the ipv4 Gateway IP(10.46.43.8).
== 2021-04-19 12:21:57.250 -0700 == Packet received at fastpath stage, tag 621015, type ATOMIC Packet info: len 194 port 69 interface 69 vsys 1 wqe index 552134 packet 0x0x800000037dd660e4, HA: 0, IC: 0 Packet decoded dump: L2: 00:50:56:81:c3:ae->c4:24:56:66:69:45, type 0x86dd IP6: 2620:130:800b:1000:feed:238:0:10->2001:130:800b:1000:1020:0:2b:8 version 6, traffic class 0x00, flow label 0xfdbe4 payload length 140, next header 17, hop limit 64 UDP: sport 54245, dport 4501, len 140, checksum 20055 Packet enter NATT tunnel decap stage, session 621015, tunnel 3 Packet entered tunnel (3) deapsulation Packet info: len 194 port 69 interface 69 vsys 1 wqe index 552134 packet 0x0x800000037dd660e4, HA: 0, IC: 0 Packet decoded dump: L2: 00:50:56:81:c3:ae->c4:24:56:66:69:45, type 0x86dd IP6: 2620:130:800b:1000:feed:238:0:10->2001:130:800b:1000:1020:0:2b:8 version 6, traffic class 0x00, flow label 0xfdbe4 payload length 140, next header 17, hop limit 64 UDP: sport 54245, dport 4501, len 140, checksum 20055 Doing to esp-decap, tdp:0x800000016cf0ab00 esp-decap. now: 792346 tp->spec.ipsec_spec. start_time: 741546, hardtime: 2592000, softtime: 2591905, hardbytes:0, softbytes:0 2021-04-19 12:21:57.250 -0700 debug: pan_ipsec_esp_decap(ipsec/src/pan_ipsec.c:3432) : wqe 0x8000000377418200 res 0x800000016e3f20e8 b_exclude_video 0 if_idx_e 69 if_idx_t 129 tid 3; saddr 2620:130:800b:1000:feed:238:0:10 => daddr 2001:130:800b:1000:1020:0:2b:8; tp local addr 2001:130:800b:1000:1020:0:2b:8 local spi 4CEB857D Packet after esp deapsulation Packet info: len 98 port 69 interface 69 vsys 1 wqe index 552134 packet 0x0x800000037dcd50c0, HA: 0, IC: 0 Packet decoded dump: L2: 00:50:56:81:c3:ae->c4:24:56:66:69:45, type 0x0800 IP: 10.10.10.1->10.46.43.8, protocol 1 version 4, ihl 5, tos 0x00, len 84, id 43586, frag_off 0x4000, ttl 64, checksum 18214(0x4726) ICMP: type 8, code 0, checksum 21774, id 43586, seq 5036 2021-04-19 12:21:57.251 -0700 debug: pan_ipsec_esp_decap(ipsec/src/pan_ipsec.c:3818): (IPSEC TUNNEL) set work 0x8000000377418200 exclude_video to 0 Tunnel decap completed, feed to interface 129 Tunnel inbound msg ipproto 1
- In a dual stack global protect deployment, When the firewall receives the UDP ESP packets that encapsulates the keepalive icmp packets, the UDP ESP packets is decapsulated and the inner packet (keepalive icmp packet) is subjected to firewalling which includes policy and route lookup. In an ipv4 only Global Protect deployment, the inner packet (keepalive icmp packet) is not subjected to policy lookup.
- The firewall creates sessions for the keepalive icmp packets in a dual stack deployment. Do note that in an ipv4 only Global Protect deployment, the firewall does not create a session for the keepalive icmp packets
admin@Lab35-8-PA-3250> show session id 640793 Session 640793 c2s flow: source: 10.10.10.1 [Tunnel-Zone] dst: 10.46.43.8 proto: 1 sport: 18437 dport: 5333 state: INIT type: FLOW src user: testdual dst user: unknown s2c flow: source: 10.46.43.8 [L3-Untrust] dst: 10.10.10.1 proto: 1 sport: 5333 dport: 18437 state: INIT type: FLOW src user: unknown dst user: testdual start time : Mon Apr 19 12:21:57 2021 timeout : 6 sec total byte count(c2s) : 98 total byte count(s2c) : 98 layer7 packet count(c2s) : 1 layer7 packet count(s2c) : 1 vsys : vsys1 application : ping rule : Trust-to-Untrust service timeout override(index) : False session to be logged at end : True session in session ager : False session updated by HA peer : False layer7 processing : enabled URL filtering enabled : False session via syn-cookies : False session terminated on host : True session traverses tunnel : True session terminate tunnel : False captive portal session : False ingress interface : tunnel.10 egress interface : ethernet1/3 session QoS rule : N/A (class 4) tracker stage firewall : Aged out end-reason : aged-out
- Traffic logs for the keepalive packets are seen below:
- How to configure Global protect dual stack: Link