How to Open a Case on IPSec (VPN) Issues

Learn how to collect preliminary information required by the Technical Assistance Center (TAC) to start working on IPSec (VPN) issues. There are multiple daemons responsible for negotiating and installing an IPSec tunnel on the management plane as well as on the data plane.

Management Plane

• ikemgr: Responsible for negotiating phase 1 and phase 2
• keymgr: Responsible for updating the SPI table for all the configured tunnels after ikemgr negotiations.

Dataplane

• mprelay: Responsible for associating the correct SPIs obtained from keymgr to the corresponding tunnel.

IPSec tunnel status window, which can be accessed from the WebGUI > Network > IPSec Tunnels.

Inside that window, you see the status of all of the IPSec VPN tunnels that you have configured on this firewall.

IPSec Tunnel status window showing both P1 and P2 status of every tunnel on this device.

Detail of the second part of the same window showing the IPSec Tunnel Status.

Below are examples of IPSec issues and information needed in each scenario:

Phase 1 down

Please perform the below steps while the issue is happening, and if possible, while the PA firewall is the responder (IKE session being initiated from the peer device). Please note that enabling debugs on ikemgr and mprelay might increase the CPU utilization.

1. Can you ping remote peer IP?
2. Is there any deny all rules blocking ike?
3. > test routing fib-lookup virtual-router <vr-name> ip < remote peer IP >
4. Take IKE pcap on the management plane. This capture will have the control plane packets transferred between the peers to establish the tunnel.
   > debug ike pcap delete
   > debug ike pcap on

5.  Enable debugs on ikemgr

   > debug ike global on debug
6. Enable debugs on mprelay
   > debug mprelay on debug
7. Take a packet capture on the data plane for IKE traffic. Setup the packet filters to capture traffic destined to the peer device IP address over port 500
   Getting Started: Packet Capture
8. Initiate the tunnel from the peer device.
9. > show clock
10. > show counter global filter delta yes packet-filter yes (run this command multiple times)
11. > show session all filter source <value> destination <value> application ike
    Or
    > show session all filter source <value> destination <value> destination-port 500
    > show session id <value>
12. > show vpn ike-sa detail gateway <name> (If you are running PAN-OS 7.x and above)
    > show vpn ike-sa gateway <name>
13. Turn off debugs and pcaps on management plane
    > debug dataplane packet-diag set capture off 
    > debug ike pcap off
    > debug ike global on normal
    > debug mprelay on info
14. Export the ike pcap and attach it to the case
    > scp export debug-pcap from ikemgr.pcap to
15. Attach the packet capture from the management plane as well as from the data plane to the case.
16. Generate a Tech Support file and attach it to the case
    How to Generate and Upload a Tech Support File Using the WebGUI and CLI
17. Please specify the exact time of the issue.
18. Provide the Peer device IPSec configuration so that we can compare it (if the peer device is a PA, please repeat the steps above).

Phase 2 down

Please perform the below steps while the issue is happening, and if possible, while the PA firewall is the responder. Please note that enabling debugs on ikemgr and mprelay might increase the CPU utilization.

1. Is there any deny all rules blocking ipsec-esp or ipsec-esp-udp?
2. Take IKE pcap on the management plane. This capture will have the control plane packets transferred between the peers to establish the tunnel.
   > debug ike pcap delete
   > debug ike pcap on
3. Enable debug on ikemgr
   > debug ike global on debug
4. Enable debug on mprelay
   > debug mprelay on debug
5. Initiate traffic from the peer device.
6. > show clock 
7. > show session all filter source <value> destination <value> application ipsec-esp
   > show session all filter source <value> destination <value> application ipsec-esp-udp > show session id <value>
8. > show vpn ipsec-sa tunnel <tunnel-name>
   > show vpn flow tunnel-id <tunnel-id from previous command>
9. Turn off debugs and pcaps on management plane.
   > debug ike pcap off
   > debug ike global on normal
   > debug mprelay on info
10. Export the ike pcap and attach it to the case
    > scp export debug-pcap from ikemgr.pcap to
11. Attach the packet capture from the management plane to the case.
12. Generate a Tech Support file and attach it to the case
    How to Generate and Upload a Tech Support File Using the WebGUI and CLI
13. Please specify the exact time of the issue.
14. Provide the Peer device IPSec configuration so that we can compare it (if the peer device is a PA, please repeat the steps above).

Why do we need this information?

In order for phase 1 and phase 2 to come up, IKE and ESP sessions should be established. The above information will let us verify that IKE and ESP packets are being received and transmitted. It is important to perform the above steps while the PA firewall is the responder, as this will provide us with useful inforamtion for troubleshooting.

• Tunnel is up but the traffic is not passing
  1. Take a packet capture of the interesting traffic.
     Getting Started: Packet Capture
  2. Initiate traffic.
  3. > show clock
  4. > show counter global filter delta yes packet-filter yes (run this command multiple times)
  5. > show session all filter source <value> destination <value>
     > show session id <value>
  6. > show vpn ipsec-sa tunnel <tunnel-name>
     > show vpn flow tunnel-id <tunnel-id from previous command> (run this command multiple times)

     admin@PA-5060> show vpn ipsec-sa tunnel SAP-SAP 
     
     GwID/client IP  TnID Peer-Address           Tunnel(Gateway)                                Algorithm     SPI(in)  SPI(out) life(Sec/KB)
     --------------- ---- ------------           ---------------                                ---------     -------  -------- ------------
                  90  207 10.1.1.1               SAP-SAP(SAP)                                   ESP/3DES/SHA1 8AE5524F B198C409  25978/0
     
     Show IPSec SA: Total 1 tunnels found. 1 ipsec sa found.
     
     admin@PA-5060> show vpn flow tunnel-id 207
     
     tunnel  SAP-SAP
             id:                     207
             type:                   IPSec
             gateway id:             90
             local ip:               10.1.1.2
             peer ip:                10.1.1.1
             inner interface:        tunnel.10 
             outer interface:        ethernet1/21
             state:                  active
             session:                145372
             tunnel mtu:             9128
             lifetime remain:        25935 sec
             latest rekey:           2865 seconds ago
             monitor:                off
             monitor packets seen:   0
             monitor packets reply:  0
             en/decap context:       23       
             local spi:              8AE5524F
             remote spi:             B198C409
             key type:               auto key
             protocol:               ESP
             auth algorithm:         SHA1
             enc  algorithm:         3DES
             proxy-id local ip:      0.0.0.0/0
             proxy-id remote ip:     0.0.0.0/0
             proxy-id protocol:      0  
             proxy-id local port:    0   
             proxy-id remote port:   0
             anti replay check:      no
             copy tos:               no
             authentication errors:  0
             decryption errors:      0
             inner packet warnings:  0
             replay packets:         0
             packets received 
               when lifetime expired:0
               when lifesize expired:0
             sending sequence:       0
             receive sequence:       0
             encap packets:          387833
             decap packets:          1483008
             encap bytes:            71262032
             decap bytes:            2107755408
             key acquire requests:   12
             owner state:            0
             owner cpuid:            s1dp0
             ownership:              1
      
     

  7. Confirm that there is a route to send the traffic into the tunnel.
     > test routing fib-lookup virtual-router <vr-name> ip < destination IP behind remote peer>
  8. Generate a Tech Support file and attach it to the case
     https://live.paloaltonetworks.com/t5/Featured-Articles/How-to-Generate-and-Upload-a-Tech-Support-File-Using-the-WebGUI/ta-p/60757
  9. Specify the exact time of the issue.

   

  • Why do we need this information?

   

  If the traffic is not passing over the IPSec tunnel, we will need to find out if the issue is in transmitting (encrypting) the traffic or receiving (decrypting) the traffic. The above commands will provide the encap and decap packets counters that will help us narrow the issue down.

   

  • Performance issues:

  Please perform the below steps while the issue is happening.

  1. Take a packet capture of the interesting traffic.
     Getting Started: Packet Capture
  2. > debug dataplane pool statistics
  3. > show counter global filter delta yes packet-filter yes (run multiple times)
  4. > show running resource-monitor second last 5  (run multiple times)
  5. > debug dataplane pow performance (run multiple times)
  6. > show vpn ipsec-sa tunnel <name>
     > show vpn flow tunnel-id  (run multiple times)
  7. Generate a Tech Support file and attach it to the case
     How to Generate and Upload a Tech Support File Using the WebGUI and CLI
  8. Specify the exact time of the issue.

   

  • Why do we need this information?

  The above steps will help us understand the load on the PA during the time of the issue, and if there is any errors on the IPSec tunnel.

  • Helpful KB articles
    1. CLI Commands to Status, Clear, Restore, and Monitor an IPSEC VPN Tunnel
    2. IPSec VPN Error: IKE Phase-2 Negotiation is Failed as Initiator, Quick Mode
    3. How to Troubleshoot IPSec VPN connectivity issues
    4. VPN Tunnel Traffic Encapsulation Incrementing but no Decaps

   

-- Alaauddin Shieha (ashieha)