How to troubleshoot an IKEv1 IPsec VPN tunnel brought down by DPD

• As a best practice, and to avoid a scenario where a DPD configuration mismatch between two IPsec VPN endpoints (for example, in timeout values) could cause one side to tear down the tunnel while the other side still considers it active—resulting in connectivity failure—verify the Dead Peer Detection (DPD) configuration on both endpoints of the IPsec VPN tunnel.
• Use the timestamp of the DPD down event to correlate with other events that could affect the connectivity between the VPN peers.
• Troubleshoot the IPsec VPN connectivity.
• Collect debug packet captures to check if the firewall is sending and receiving DPD packets.

• IPsec VPN tunnel between two Palo Alto Networks firewalls.
• IKEv1
• Dead Peer Detection (DPD)

1. As a best practice, and to avoid a scenario where a DPD configuration mismatch between two IPsec VPN endpoints (for example, in timeout values) could cause one side to tear down the tunnel while the other side still considers it active—resulting in connectivity failure—verify the Dead Peer Detection (DPD) configuration on both endpoints of the IPsec VPN tunnel. For Palo Alto Networks NGFW, navigate to Network > Network Profiles > IKE Gateways > Advanced Options. Ensure that the Dead Peer Detection is enabled and the DPD interval and retry matches the settings of the other end of the tunnel:
   
   
   1. 
   2. Note that while “Dead Peer Detection” (DPD) is a standard term defined in RFC 3706, vendors may use different names for their keepalive or liveness check mechanisms in IPsec VPNs. For Cisco ASA firewalls, as an example, the equivalent feature is called “keepalive” or “ISAKMP keepalive.” Like DPD, this feature ensures that the peer is still available. As a best practice, and to maintain IPsec VPN tunnel stability, verify that the keepalive settings on other vendors’ devices match the IKEv1 DPD settings on Palo Alto Networks devices. Refer to Site-to-Site IPSec VPN between Palo Alto Networks Firewall and Cisco Router is Unstable or Intermittent.
2. When the IPsec tunnel goes down due to DPD the log messages below will show up in the system log:

   > show log system direction equal backward
   2021/01/14 21:05:02 IKE phase-1 SA is down determined by DPD.
   2021/01/13 21:17:24 IKE phase-1 SA is down determined by DPD.
   2021/01/12 20:41:32 IKE phase-1 SA is down determined by DPD. 

   and the messages below will show up in the ikemgr logs.

   > less mp-log ikemgr.log
   2021-01-14 11:25:12 2021-01-14 21:05:02.001 +0200 [INFO]: { 14: 219}: DPD down, rekey vpn tunnel <NIDA_VPN_MACH:ProxyID15> 

   The default interval of sending the DPD packet is every 5 seconds when SA is idle. Upon losing connection, the firewall will 5 DPD retries and after maximum retries are reached, the firewall will tear down phase 1 and phase 2 SAs. Look for the messages above and note the timestamp so you can use it in order to correlate other events which will help in finding the root cause of DPD failure.
3. When the IPsec tunnel goes down because of DPD that is an indication that there is a connectivity issues between the IPsec VPN peers. For more details on how to remediate, refer to How to Troubleshoot IPSec VPN connectivity issue. 
4. For an ongoing issue with DPD, if the issue is resolved by disabling the DPD check and if in doubt that the firewall is receiving or sending the DPD packet which is the empty informational R-U-There packet. Set a maintenance window to collect a debug ikemgr packet capture. Since the DPD is "not persistent" and is only triggered by a Phase 2 rekey, you need to manually trigger a Phase 2 rekey using the test command below to be able to test and check DPD functionality. :

   debug ike pcap on
   test vpn ipsec-sa tunnel <name of the tunnel>
   debug ike pcap off
   view-pcap no-dns-lookup yes no-port-lookup yes debug-pcap ikemgr.pcap
   scp export debug-pcap from ikemgr.pcap to username@host:path

5. Note: Be cautious with above commands: the one highlighted in orange might become a resource intensive command that is why it is recommended to issue it during a maintenance window otherwise make sure that the pcap is on for a short period of time before turning it off using the third command above.
   1. If the firewall is sending and receiving the empty informational DPD packet then the packet capture will look like below:
   2. Enable the debugs for the affected gateway using the command:

      debug ike gateway <name of the gateway> on debug

      The ikemg logs in this case will show the below messages:The ikemg logs in this case will show the below messages:
      

      2024-07-15 16:11:24.016 -0700  [DEBG]: {    2:     }: DPD monitoring.... ip 0 0
      2024-07-15 16:11:24.016 -0700  [DEBG]: {    2:     }: compute IV for phase2
      2024-07-15 16:11:24.016 -0700  [DEBG]: {    2:     }: phase1 last IV:
      2024-07-15 16:11:24.016 -0700  [DEBG]: {    2:     }: encryption(aes)
      2024-07-15 16:11:24.016 -0700  [DEBG]: {    2:     }: phase2 IV computed:
      2024-07-15 16:11:24.016 -0700  [DEBG]: {    2:     }: HASH with:
      2024-07-15 16:11:24.016 -0700  [DEBG]: {    2:     }: hmac(hmac_sha1)
      2024-07-15 16:11:24.016 -0700  [DEBG]: {    2:     }: HASH computed:
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: begin encryption.
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: encryption(aes)
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: pad length = 8
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: encryption(aes)
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: with key:
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: encrypted payload by IV:
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: save IV for next:
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: encrypted.
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: 92 bytes from 10.46.36.241[500] to 10.46.36.240[500]
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: sendto Information notify.
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: DPD R-U-There sent (0)
      2024-07-15 16:11:24.017 -0700  [DEBG]: {    2:     }: rescheduling send_r_u (5).
      2024-07-15 16:11:24.019 -0700  [DEBG]: {    2:     }: receive Information.
      2024-07-15 16:11:24.019 -0700  [DEBG]: {    2:     }: compute IV for phase2
      2024-07-15 16:11:24.019 -0700  [DEBG]: {    2:     }: phase1 last IV:
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: encryption(aes)
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: phase2 IV computed:
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: begin decryption.
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: encryption(aes)
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: IV was saved for next processing:
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: encryption(aes)
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: with key:
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: decrypted payload by IV:
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: decrypted payload, but not trimed.
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: padding len=8
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: decrypted.
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: HASH with:
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: hmac(hmac_sha1)
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: HASH computed:
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: hash validated.
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: begin.
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: seen nptype=8(hash)
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: seen nptype=11(notify)
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: succeed.
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: DPD R-U-There-Ack received
      2024-07-15 16:11:24.020 -0700  [DEBG]: {    2:     }: received an R-U-THERE-ACK
      2024-07-15 16:11:24.020 -0700  [PNTF]: {    2:     }: notification message 36137:R-U-THERE-ACK, doi=1 proto_id=1 spi=30712ff29eddaded 8256275ddba5a1c2 (size=16).
      2024-07-15 16:11:46.016 -0700  [DEBG]: {    2:     }: del packet d0f81e2a:20 size    44, rcp 0
      

   3. Based on the packet capture collected between the two peers of the tunnel, the resolution of the problem may include adjusting DPD timeout values, by increasing the DPD interval and retry values on both endpoints to make the DPD check less strict. Other resolutions could involve correcting any misconfigured policies, configuring proper route entries, or addressing any potential network issues affecting the reachability of the remote peer.

Other references to check: What is Dead Peer Detection and Tunnel Monitoring across IPSec Tunnel?