GlobalProtect Dual Stack: IPSec connection failed due to keepalive

•  IPSec connection failed due to keepalive. See Below for example

(P10688-T8416)Debug( 166): 04/19/21 11:47:32:425 Trying to do ipsec connection to 2001:130:800b:1000:1020:0:2b:8[4501]
(P10688-T8416)Debug( 487): 04/19/21 11:47:32:425 socket send buffer old size is 65536
(P10688-T8416)Debug( 511): 04/19/21 11:47:32:425 socket send buffer new size is 3145728
(P10688-T8416)Debug( 563): 04/19/21 11:47:32:425 Network is reachable
(P10688-T8416)Info ( 178): 04/19/21 11:47:32:425 Connected to: 2001:130:800b:1000:1020:0:2b:8[4501], Sending keep alive to ipsec socket...
(P10688-T8416)Info ( 221): 04/19/21 11:47:38:456 failed to receive keep alive
(P10688-T8416)Debug( 229): 04/19/21 11:47:38:456 IPSec anti-replay statistics: outside window count 0, replay count 0
(P10688-T8416)Debug( 231): 04/19/21 11:47:38:456 Disconnect udp socket 
(P10688-T8416)Info ( 364): 04/19/21 11:47:38:456 Connecting to 2001:130:800b:1000:1020:0:2b:8 failed

• Global Protect connection is then established with SSL Tunnel. See below for example

(P10688-T8416)Debug( 766): 04/19/21 11:47:38:456 IPSec fallback reason is IPSec connection failed
(P10688-T8416)Debug( 171): 04/19/21 11:47:38:456 VPN idle timeout is 10800; config timeout is 10800
(P10688-T8416)Debug( 219): 04/19/21 11:47:38:487 EnforceDns is enabled, set 2 GP pushed DNS servers
(P10688-T8416)Debug(  65): 04/19/21 11:47:38:487 Trying to do SSL connection to 2001:130:800b:1000:1020:0:2b:8(443)
(P10688-T8416)Debug( 788): 04/19/21 11:47:38:487 SSL connecting to 2001:130:800b:1000:1020:0:2b:8
(P10688-T8416)Debug( 487): 04/19/21 11:47:38:487 socket send buffer old size is 65536
(P10688-T8416)Debug( 511): 04/19/21 11:47:38:487 socket send buffer new size is 3145728
(P10688-T8416)Debug( 563): 04/19/21 11:47:38:503 Network is reachable
(P10688-T8416)Debug(1274): 04/19/21 11:47:38:518 Failed to X509_LOOKUP_load_file
(P10688-T8416)Debug( 374): 04/19/21 11:47:38:518 Open_SSL_connection: subject '/CN=testdualgp.com'
(P10688-T8416)Debug( 378): 04/19/21 11:47:38:518 Open_SSL_connection: issuer '/CN=GP-CERT'
(P10688-T8416)Info ( 113): 04/19/21 11:47:38:534 Connected ssl tunnel to 2001:130:800b:1000:1020:0:2b:8(443)
(P10688-T8416)Info ( 374): 04/19/21 11:47:38:534 tunnel to 2001:130:800b:1000:1020:0:2b:8 connected

• GlobalProtect Configured with Dual Stack
• Environment:

• No ipv6 pool is configured in the gateway

• Land Attack : Client Assigned IP Address is being NAT'ed to the ipv4 gateway address, which will cause the firewall to see the keepalive icmp packets as land attack and then drop the packets

• There is no security policy to allows the keepalive icmp packets, hence traffic is getting denied by a block rule or by default inter-zone rule

• Routing/PBF: Tunnel Traffic is being routed/forwarded through a different interface

• Configure a NO NAT rule for the keepalive icmp packets like the one below: POLICIES>NAT    Configure NAT

• Make sure there is a Security Rule that allows traffic between the zones like the one below: POLICIES>Security     Create a Security Policy Rule

• Ensure there is proper route/pbf for the traffic

•  Keepalive packets is used to check if the gateway is reachable when there is no network activity. It also helps in keeping the tunnel up. Keep alive packet for IPSEC is slightly different from that of SSL. For IPSEC, keep alive uses icmp packets while SSL uses keep-alive message type in a SSL packet header. This is why SSL tunnel is successfully established where IPSEC tunnel fails.
• Keepalive icmp packets is between the Assigned IP and the Gateway IP address. In a dual stack deployment where ipv6 pool is not configured and ipv6 is preferred, the IPSEC Tunnel will be formed between the Client ipv6 address (2620:130:800b:1000:feed:238:0:10) and the Gateway ipv6 address (2001:130:800b:1000:1020::2b:8). While the keepalive icmp packet will be between the ipv4 Assigned IP (10.10.10.1) and the ipv4 Gateway IP(10.46.43.8).

== 2021-04-19 12:21:57.250 -0700 ==
Packet received at fastpath stage, tag 621015, type ATOMIC
Packet info: len 194 port 69 interface 69 vsys 1
  wqe index 552134 packet 0x0x800000037dd660e4, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:81:c3:ae->c4:24:56:66:69:45, type 0x86dd
IP6:     2620:130:800b:1000:feed:238:0:10->2001:130:800b:1000:1020:0:2b:8
        version 6, traffic class 0x00, flow label 0xfdbe4
        payload length 140, next header 17, hop limit 64
UDP:    sport 54245, dport 4501, len 140, checksum 20055
Packet enter NATT tunnel decap stage, session 621015, tunnel 3
Packet entered tunnel (3) deapsulation
Packet info: len 194 port 69 interface 69 vsys 1
  wqe index 552134 packet 0x0x800000037dd660e4, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:81:c3:ae->c4:24:56:66:69:45, type 0x86dd
IP6:     2620:130:800b:1000:feed:238:0:10->2001:130:800b:1000:1020:0:2b:8
        version 6, traffic class 0x00, flow label 0xfdbe4
        payload length 140, next header 17, hop limit 64
UDP:    sport 54245, dport 4501, len 140, checksum 20055
Doing to esp-decap, tdp:0x800000016cf0ab00
esp-decap. now: 792346
tp->spec.ipsec_spec. start_time: 741546, hardtime: 2592000, softtime: 2591905, hardbytes:0, softbytes:0
2021-04-19 12:21:57.250 -0700 debug: pan_ipsec_esp_decap(ipsec/src/pan_ipsec.c:3432)
: wqe 0x8000000377418200 res 0x800000016e3f20e8 b_exclude_video 0 if_idx_e 69 if_idx_t 129 tid 3; 
saddr 2620:130:800b:1000:feed:238:0:10 => daddr 2001:130:800b:1000:1020:0:2b:8; 
tp local addr 2001:130:800b:1000:1020:0:2b:8 local spi 4CEB857D
Packet after esp deapsulation
Packet info: len 98 port 69 interface 69 vsys 1
  wqe index 552134 packet 0x0x800000037dcd50c0, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:81:c3:ae->c4:24:56:66:69:45, type 0x0800
IP:     10.10.10.1->10.46.43.8, protocol 1
        version 4, ihl 5, tos 0x00, len 84,
        id 43586, frag_off 0x4000, ttl 64, checksum 18214(0x4726)
ICMP:   type 8, code 0, checksum 21774, id 43586, seq 5036
2021-04-19 12:21:57.251 -0700 debug: pan_ipsec_esp_decap(ipsec/src/pan_ipsec.c:3818): (IPSEC TUNNEL) set work 0x8000000377418200 exclude_video to 0
Tunnel decap completed, feed to interface 129
Tunnel inbound msg ipproto 1

• In a dual stack global protect deployment, When the firewall receives the UDP ESP packets that encapsulates the keepalive icmp packets, the UDP ESP packets is decapsulated and the inner packet (keepalive icmp packet) is subjected to firewalling which includes policy and route lookup. In an ipv4 only Global Protect deployment, the inner packet (keepalive icmp packet) is not subjected to policy lookup.
• The firewall creates sessions for the keepalive icmp packets in a dual stack deployment. Do note that in an ipv4 only Global Protect deployment, the firewall does not create a session for the keepalive icmp packets

admin@Lab35-8-PA-3250> show session id 640793

Session          640793

        c2s flow:
                source:      10.10.10.1 [Tunnel-Zone]
                dst:         10.46.43.8
                proto:       1
                sport:       18437           dport:      5333
                state:       INIT            type:       FLOW
                src user:    testdual
                dst user:    unknown

        s2c flow:
                source:      10.46.43.8 [L3-Untrust]
                dst:         10.10.10.1
                proto:       1
                sport:       5333            dport:      18437
                state:       INIT            type:       FLOW
                src user:    unknown
                dst user:    testdual

        start time                           : Mon Apr 19 12:21:57 2021
        timeout                              : 6 sec
        total byte count(c2s)                : 98
        total byte count(s2c)                : 98
        layer7 packet count(c2s)             : 1
        layer7 packet count(s2c)             : 1
        vsys                                 : vsys1
        application                          : ping
        rule                                 : Trust-to-Untrust
        service timeout override(index)      : False
        session to be logged at end          : True
        session in session ager              : False
        session updated by HA peer           : False
        layer7 processing                    : enabled
        URL filtering enabled                : False
        session via syn-cookies              : False
        session terminated on host           : True
        session traverses tunnel             : True
        session terminate tunnel             : False
        captive portal session               : False
        ingress interface                    : tunnel.10
        egress interface                     : ethernet1/3
        session QoS rule                     : N/A (class 4)
        tracker stage firewall               : Aged out
        end-reason                           : aged-out

• Traffic logs for the keepalive packets are seen below:

• How to configure Global protect dual stack: Link