Global Protect not connecting on IPSEC but can connect on SSL

• Global protect connection successfully happens using SSL protocol but not on IPSEC.
• All users are affected. 
• When source nat rule is disabled, GP on IPSEC works.
• IPSEC is enabled in the GP gateway configuration.

• In PanGPS.log the reason IPSEC failed is because keep-alive was not received and the agent started SSL connection instead

(T12928)Debug( 559): 05/07/21 09:50:16:624 Network is reachable
(T12928)Info ( 178): 05/07/21 09:50:16:624 Connected to: 100.1.1.1[4501], Sending keep alive to ipsec socket...
(T12928)Debug( 278): 05/07/21 09:50:16:640 IPSec tunnel receive failed with error 10054(An existing connection was forcibly closed by the remote host.)
(T12928)Info ( 199): 05/07/21 09:50:16:640 failed to receive keep alive
(T12928)Info ( 221): 05/07/21 09:50:16:640 failed to receive keep alive
(T12928)Debug( 229): 05/07/21 09:50:16:640 IPSec anti-replay statistics: outside window count 0, replay count 0
(T12928)Debug( 231): 05/07/21 09:50:16:640 Disconnect udp socket 
(T12928)Info ( 353): 05/07/21 09:50:16:640 Connecting to 100.1.1.1 failed
(T12928)Info ( 268): 05/07/21 09:50:16:640 Start vpn do_connect() failed
(T12928)Debug( 325): 05/07/21 09:50:16:640 tunnel statistics: send bytes(0) packets(0) errors(0) drops(0) queue-size(0), recv bytes(0) packets(0) errors(0) drops(0) queue-size(0)
(T12928)Debug( 327): 05/07/21 09:50:16:640 do_disconnect is called in VPN stop
(T12928)Debug( 660): 05/07/21 09:50:16:640 ipsec failed to start
(T12928)Info ( 100): 05/07/21 09:50:16:640 VPN is deleted
(T12928)Debug( 169): 05/07/21 09:50:16:640 VPN idle timeout is 10800; config timeout is 10800
(T12928)Debug( 217): 05/07/21 09:50:16:640 EnforceDns is enabled, set 2 GP pushed DNS servers
(T12928)Debug(  65): 05/07/21 09:50:16:640 Trying to do SSL connection to 100.1.1.1(443)
(T12928)Debug( 780): 05/07/21 09:50:16:640 SSL connecting to 100.1.1.1
(T12928)Debug( 487): 05/07/21 09:50:16:640 socket send buffer old size is 65536
(T12928)Debug( 511): 05/07/21 09:50:16:640 socket send buffer new size is 3145728
(T12928)Debug( 559): 05/07/21 09:50:16:640 Network is reachable
(T12928)Debug(1247): 05/07/21 09:50:16:656 Failed to X509_LOOKUP_load_file
(T12928)Debug( 366): 05/07/21 09:50:16:656 Open_SSL_connection: subject '/CN=1001.1.1'
(T12928)Debug( 370): 05/07/21 09:50:16:656 Open_SSL_connection: issuer '/CN=RootCerti'
(T12928)Info ( 113): 05/07/21 09:50:16:656 Connected ssl tunnel to 100.1.1.1(443)
(T12928)Info ( 363): 05/07/21 09:50:16:656 tunnel to 100.1.1.1 connected

• Client packet capture shows that client is sending packet on UDP-4501 but getting "Destination unreachable (Port unreachable)" ICMP message.

NOTE: If we check the content of this ICMP Type 3 Code 3 packet, we will notice that it was in response to a completely different IP packet (ipid is different). We will see later why this happened.

• Session shows 3 packets sent and 3 packets received:

• Debug/packet_diag when enabled with "flow basic" and "tunnel flow" shows following:

== 2021-05-07 09:50:17.997 +0800 ==
Packet received at slowpath stage, tag 337855482, type ATOMIC
Packet info: len 174 port 18 interface 18 vsys 1
  wqe index 229292 packet 0x0x80000000ed94a0e6, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     100.1.1.70->100.1.1.1, protocol 17
        version 4, ihl 5, tos 0x00, len 160,
        id 21988, frag_off 0x0000, ttl 128, checksum 6688(0x1a20)
UDP:    sport 61248, dport 4501, len 140, checksum 46727
Session setup: vsys 1
Session setup: ingress interface ethernet1/3 egress interface ethernet1/3 (zone 3)
NAT policy lookup, matched rule index 0
Policy lookup, matched rule index 1,
Allocated new session 150255.
set exclude_video in session 150255 0x80000000d010b180 0 from work 0x80000000e69bd580 0
Rule: index=0 name=test, cfg_pool_idx=16 cfg_fallback_pool_idx=0
NAT Rule: name=test, cfg_pool_idx=16; Session: index=150255, nat_pool_idx=16
Packet matched vsys 1 NAT rule 'test' (index 1),
source translation 100.1.1.70/61248 => 200.1.1.1/61248
Created session, enqueue to install. work 0x80000000e69bd580 exclude_video 0,session 150255 0x80000000d010b180 exclude_video 0

== 2021-05-07 09:50:17.999 +0800 ==
Packet received at fastpath stage, tag 150255, type ATOMIC
Packet info: len 174 port 18 interface 18 vsys 1
  wqe index 229292 packet 0x0x80000000ed94a0e6, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     200.1.1.1->100.1.1.1, protocol 17
        version 4, ihl 5, tos 0x00, len 160,
        id 21988, frag_off 0x0000, ttl 128, checksum 46692(0xb664)
UDP:    sport 61248, dport 4501, len 140, checksum 21196
Flow fastpath, session 150255 (set work 0x80000000e69bd580 exclude_video 0 from sp 0x80000000d010b180 exclude_video 0)
2021-05-07 09:50:17.999 +0800  pan_flow_process_fastpath(src/pan_flow_proc.c:3895): SESSION-DSCP: set session DSCP: 0x00
NAT session, run address/port translation
Packet enter NATT tunnel decap stage, session 150255, tunnel 1557
Packet entered tunnel (1557) deapsulation
Packet info: len 174 port 18 interface 18 vsys 1
  wqe index 229292 packet 0x0x80000000ed94a0e6, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     200.1.1.1->100.1.1.1, protocol 17
        version 4, ihl 5, tos 0x00, len 160,
        id 21988, frag_off 0x0000, ttl 128, checksum 46692(0xb664)
UDP:    sport 61248, dport 4501, len 140, checksum 21196
Doing to esp-decap, tdp:0x800000007f092840
esp-decap. now: 6088436
tp->spec.ipsec_spec. start_time: 6088436, hardtime: 2592000, softtime: 2591967, hardbytes:0, softbytes:0
2021-05-07 09:50:17.999 +0800 debug: pan_ipsec_esp_decap(ipsec/src/pan_ipsec.c:3269): wqe 0x80000000e69bd580 res 0x800000007fce1708 b_exclude_video 0 if_idx_e 18 if_idx_t 263 tid 1557; saddr 200.1.1.1 => daddr 100.1.1.1; tp local addr 100.1.1.1 local spi 505D735F
Packet after esp deapsulation
Packet info: len 98 port 18 interface 18 vsys 1
  wqe index 229292 packet 0x0x80000000ed9360c0, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     172.16.107.2->100.1.1.1, protocol 1
        version 4, ihl 5, tos 0x00, len 84,
        id 21345, frag_off 0x4000, ttl 64, checksum 27443(0x6b33)
ICMP:   type 8, code 0, checksum 49050, id 21345, seq 1
2021-05-07 09:50:17.999 +0800 debug: pan_ipsec_esp_decap(ipsec/src/pan_ipsec.c:3643): (IPSEC TUNNEL) set work 0x80000000e69bd580 exclude_video to 0
Tunnel decap completed, feed to interface 263
Tunnel inbound msg

== 2021-05-07 09:50:17.999 +0800 ==
Packet received at ingress stage, tag 0, type ORDERED
Packet info: len 98 port 4 interface 263 vsys 1
  wqe index 229292 packet 0x0x80000000ed9360c0, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     172.16.107.2->100.1.1.1, protocol 1
        version 4, ihl 5, tos 0x00, len 84,
        id 21345, frag_off 0x4000, ttl 64, checksum 27443(0x6b33)
ICMP:   type 8, code 0, checksum 49050, id 21345, seq 1
Tunnel inbound. Protocol is ICMP, intercept it
Received icmp packet seq# 21345 from source: 0-0 to destination: 0-0 , is gps2s:no
Packet enters tunnel encap stage, tunnel interface null
Packet entered tunnel (1557) encapsulation
Packet info: len 98 port 4 interface 263 vsys 1
  wqe index 229292 packet 0x0x80000000ed9360c0, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     100.1.1.1->172.16.107.2, protocol 1
        version 4, ihl 5, tos 0x00, len 84,
        id 17890, frag_off 0x4000, ttl 64, checksum 30898(0x78b2)
ICMP:   type 0, code 0, checksum 51098, id 21345, seq 1
Done esp-encap, tdp:0x800000007f092840, gre_encap:0, natt_encap:1, spi:1701752256, esp seq: 1
NAT session, run address/port translation
Tunnel outbound msg
Forwarding lookup, ingress interface 18
L3 mode, virtual-router 1
Route lookup in virtual-router 1, IP 100.1.1.1
Host route found, forward packet to host
Host service check passed, transmit to control plane

• All Pan-OS firewalls
• Topology used for demonstration in this KB article:

• The issue happens because firewall does not support NAT for peer IP on firewall itself in case of GP IPSEC or site-to-site IPSEC VPN.

• Disable NAT rule for Peer IP (GP client/IPSEC peer).