Global Protect not connecting on IPSEC but can connect on SSL

Global Protect not connecting on IPSEC but can connect on SSL

6737
Created On 05/06/21 09:26 AM - Last Modified 07/14/21 03:22 AM


Symptom
  • Global protect connection successfully happens using SSL protocol but not on IPSEC.
  • All users are affected. 
  • When source nat rule is disabled, GP on IPSEC works.
  • IPSEC is enabled in the GP gateway configuration.
User-added image
  • In PanGPS.log the reason IPSEC failed is because keep-alive was not received and the agent started SSL connection instead
(T12928)Debug( 559): 05/07/21 09:50:16:624 Network is reachable
(T12928)Info ( 178): 05/07/21 09:50:16:624 Connected to: 100.1.1.1[4501], Sending keep alive to ipsec socket...
(T12928)Debug( 278): 05/07/21 09:50:16:640 IPSec tunnel receive failed with error 10054(An existing connection was forcibly closed by the remote host.)
(T12928)Info ( 199): 05/07/21 09:50:16:640 failed to receive keep alive
(T12928)Info ( 221): 05/07/21 09:50:16:640 failed to receive keep alive
(T12928)Debug( 229): 05/07/21 09:50:16:640 IPSec anti-replay statistics: outside window count 0, replay count 0
(T12928)Debug( 231): 05/07/21 09:50:16:640 Disconnect udp socket 
(T12928)Info ( 353): 05/07/21 09:50:16:640 Connecting to 100.1.1.1 failed
(T12928)Info ( 268): 05/07/21 09:50:16:640 Start vpn do_connect() failed
(T12928)Debug( 325): 05/07/21 09:50:16:640 tunnel statistics: send bytes(0) packets(0) errors(0) drops(0) queue-size(0), recv bytes(0) packets(0) errors(0) drops(0) queue-size(0)
(T12928)Debug( 327): 05/07/21 09:50:16:640 do_disconnect is called in VPN stop
(T12928)Debug( 660): 05/07/21 09:50:16:640 ipsec failed to start
(T12928)Info ( 100): 05/07/21 09:50:16:640 VPN is deleted
(T12928)Debug( 169): 05/07/21 09:50:16:640 VPN idle timeout is 10800; config timeout is 10800
(T12928)Debug( 217): 05/07/21 09:50:16:640 EnforceDns is enabled, set 2 GP pushed DNS servers
(T12928)Debug(  65): 05/07/21 09:50:16:640 Trying to do SSL connection to 100.1.1.1(443)
(T12928)Debug( 780): 05/07/21 09:50:16:640 SSL connecting to 100.1.1.1
(T12928)Debug( 487): 05/07/21 09:50:16:640 socket send buffer old size is 65536
(T12928)Debug( 511): 05/07/21 09:50:16:640 socket send buffer new size is 3145728
(T12928)Debug( 559): 05/07/21 09:50:16:640 Network is reachable
(T12928)Debug(1247): 05/07/21 09:50:16:656 Failed to X509_LOOKUP_load_file
(T12928)Debug( 366): 05/07/21 09:50:16:656 Open_SSL_connection: subject '/CN=1001.1.1'
(T12928)Debug( 370): 05/07/21 09:50:16:656 Open_SSL_connection: issuer '/CN=RootCerti'
(T12928)Info ( 113): 05/07/21 09:50:16:656 Connected ssl tunnel to 100.1.1.1(443)
(T12928)Info ( 363): 05/07/21 09:50:16:656 tunnel to 100.1.1.1 connected
 
  • Client packet capture shows that client is sending packet on UDP-4501 but getting "Destination unreachable (Port unreachable)" ICMP message.

NOTE: If we check the content of this ICMP Type 3 Code 3 packet, we will notice that it was in response to a completely different IP packet (ipid is different). We will see later why this happened.

User-added image

  • Session shows 3 packets sent and 3 packets received:
User-added image
  • Debug/packet_diag when enabled with "flow basic" and "tunnel flow" shows following:
SLOWPATH:
== 2021-05-07 09:50:17.997 +0800 ==
Packet received at slowpath stage, tag 337855482, type ATOMIC
Packet info: len 174 port 18 interface 18 vsys 1
  wqe index 229292 packet 0x0x80000000ed94a0e6, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     100.1.1.70->100.1.1.1, protocol 17
        version 4, ihl 5, tos 0x00, len 160,
        id 21988, frag_off 0x0000, ttl 128, checksum 6688(0x1a20)
UDP:    sport 61248, dport 4501, len 140, checksum 46727
Session setup: vsys 1
Session setup: ingress interface ethernet1/3 egress interface ethernet1/3 (zone 3)
NAT policy lookup, matched rule index 0
Policy lookup, matched rule index 1,
Allocated new session 150255.
set exclude_video in session 150255 0x80000000d010b180 0 from work 0x80000000e69bd580 0
Rule: index=0 name=test, cfg_pool_idx=16 cfg_fallback_pool_idx=0
NAT Rule: name=test, cfg_pool_idx=16; Session: index=150255, nat_pool_idx=16
Packet matched vsys 1 NAT rule 'test' (index 1),
source translation 100.1.1.70/61248 => 200.1.1.1/61248
Created session, enqueue to install. work 0x80000000e69bd580 exclude_video 0,session 150255 0x80000000d010b180 exclude_video 0
FASTPATH (Inner packet decapsulated):
== 2021-05-07 09:50:17.999 +0800 ==
Packet received at fastpath stage, tag 150255, type ATOMIC
Packet info: len 174 port 18 interface 18 vsys 1
  wqe index 229292 packet 0x0x80000000ed94a0e6, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     200.1.1.1->100.1.1.1, protocol 17
        version 4, ihl 5, tos 0x00, len 160,
        id 21988, frag_off 0x0000, ttl 128, checksum 46692(0xb664)
UDP:    sport 61248, dport 4501, len 140, checksum 21196
Flow fastpath, session 150255 (set work 0x80000000e69bd580 exclude_video 0 from sp 0x80000000d010b180 exclude_video 0)
2021-05-07 09:50:17.999 +0800  pan_flow_process_fastpath(src/pan_flow_proc.c:3895): SESSION-DSCP: set session DSCP: 0x00
NAT session, run address/port translation
Packet enter NATT tunnel decap stage, session 150255, tunnel 1557
Packet entered tunnel (1557) deapsulation
Packet info: len 174 port 18 interface 18 vsys 1
  wqe index 229292 packet 0x0x80000000ed94a0e6, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     200.1.1.1->100.1.1.1, protocol 17
        version 4, ihl 5, tos 0x00, len 160,
        id 21988, frag_off 0x0000, ttl 128, checksum 46692(0xb664)
UDP:    sport 61248, dport 4501, len 140, checksum 21196
Doing to esp-decap, tdp:0x800000007f092840
esp-decap. now: 6088436
tp->spec.ipsec_spec. start_time: 6088436, hardtime: 2592000, softtime: 2591967, hardbytes:0, softbytes:0
2021-05-07 09:50:17.999 +0800 debug: pan_ipsec_esp_decap(ipsec/src/pan_ipsec.c:3269): wqe 0x80000000e69bd580 res 0x800000007fce1708 b_exclude_video 0 if_idx_e 18 if_idx_t 263 tid 1557; saddr 200.1.1.1 => daddr 100.1.1.1; tp local addr 100.1.1.1 local spi 505D735F
Packet after esp deapsulation
Packet info: len 98 port 18 interface 18 vsys 1
  wqe index 229292 packet 0x0x80000000ed9360c0, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     172.16.107.2->100.1.1.1, protocol 1
        version 4, ihl 5, tos 0x00, len 84,
        id 21345, frag_off 0x4000, ttl 64, checksum 27443(0x6b33)
ICMP:   type 8, code 0, checksum 49050, id 21345, seq 1
2021-05-07 09:50:17.999 +0800 debug: pan_ipsec_esp_decap(ipsec/src/pan_ipsec.c:3643): (IPSEC TUNNEL) set work 0x80000000e69bd580 exclude_video to 0
Tunnel decap completed, feed to interface 263
Tunnel inbound msg
== 2021-05-07 09:50:17.999 +0800 ==
Packet received at ingress stage, tag 0, type ORDERED
Packet info: len 98 port 4 interface 263 vsys 1
  wqe index 229292 packet 0x0x80000000ed9360c0, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     172.16.107.2->100.1.1.1, protocol 1
        version 4, ihl 5, tos 0x00, len 84,
        id 21345, frag_off 0x4000, ttl 64, checksum 27443(0x6b33)
ICMP:   type 8, code 0, checksum 49050, id 21345, seq 1
Tunnel inbound. Protocol is ICMP, intercept it
Received icmp packet seq# 21345 from source: 0-0 to destination: 0-0 , is gps2s:no
Packet enters tunnel encap stage, tunnel interface null
Packet entered tunnel (1557) encapsulation
Packet info: len 98 port 4 interface 263 vsys 1
  wqe index 229292 packet 0x0x80000000ed9360c0, HA: 0, IC: 0
Packet decoded dump:
L2:     00:50:56:9b:8d:8e->d4:f4:be:67:02:12, type 0x0800
IP:     100.1.1.1->172.16.107.2, protocol 1
        version 4, ihl 5, tos 0x00, len 84,
        id 17890, frag_off 0x4000, ttl 64, checksum 30898(0x78b2)
ICMP:   type 0, code 0, checksum 51098, id 21345, seq 1
Done esp-encap, tdp:0x800000007f092840, gre_encap:0, natt_encap:1, spi:1701752256, esp seq: 1
NAT session, run address/port translation
Tunnel outbound msg
Forwarding lookup, ingress interface 18
L3 mode, virtual-router 1
Route lookup in virtual-router 1, IP 100.1.1.1
Host route found, forward packet to host
Host service check passed, transmit to control plane
  • Notice that the response (ICMP type 0, code 0) is generated by the GP gateway IP but the route lookup for this packet happens for the GP gateway IP itself instead of the client public IP (In this case 100.1.1.70)

 
 


Environment
  • All Pan-OS firewalls
  • Topology used for demonstration in this KB article:
User-added image
 


Cause
  • The issue happens because firewall does not support NAT for peer IP on firewall itself in case of GP IPSEC or site-to-site IPSEC VPN.


Resolution
  • Disable NAT rule for Peer IP (GP client/IPSEC peer).


Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u0000001VK8CAM&lang=en_US%E2%80%A9&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments