How to troubleshoot reduced Tunnel Throughput - Knowledge Base - Palo Alto Networks

How to troubleshoot reduced Tunnel Throughput

25546

Created On 08/10/22 07:51 AM - Last Modified 09/15/23 04:58 AM

VPNs

PAN-OS

Next-Generation Firewall

Strata Cloud Manager

Objective

To troubleshoot and identify possible reasons for Reduced Tunnel Throughput

Environment

PAN-OS
IPSec Tunnel
GRE Tunnel

Procedure

Identify any changes on the Network. Some common things to check are:
1. Are there any changes in the network, e.g., routing changes, traffic patterns changes, DNS changes, etc.?
2. Any software upgrades in network devices
3. Was there any new network device added to the path, that can introduce the issue?
Identify any changes on the Palo Alto Device. Some common things to check are:
1. Any configuration change on the firewall, e.g., any new network changes, like new interface/sub-interface, new subnets, new features enabled?
2. Any new policy added on the firewall, e.g., new security policy, new NAT configuration, new Security Profile, etc.?
3. Any upgrades in the firewall, e.g., PAN-OS upgrade, content version upgrade, etc.?
Identify possible resource depletion in the Palo Alto firewall.
1. If the firewall is monitored by Strata Cloud Manager, use How to identify high CPU, Packet Buffer, and Packet Descriptor in the firewall with Strata Cloud Manager
2. For non-Strata Cloud Manager monitored firewalls, use the following steps
  1. Use HOW TO TROUBLESHOOT HIGH PACKET BUFFER OR PACKET DESCRIPTORS USAGE to check if your firewall is having high dataplane resources usage.
  2. Determine if the data plane CPU utilization is high
    1. Under the firewall's GUI, go to DASHBOARD > Widgets > System > click on System Resources
    2. To resolve this issue, use HOW TO TROUBLESHOOT HIGH DATAPLANE CPU.
  3. Determine if the management plane CPU utilization is high
    1. Under the firewall's GUI, go to DASHBOARD > Widgets > System > click on System Resources
    2. To resolve this issue, use TIPS & TRICKS: REDUCING MANAGEMENT PLANE LOAD
Collect packet captures and global counters that can be used in the next step to isolate possible network issues.

NOTE: It is highly recommended to do this section in a maintenance window!

Use GETTING STARTED: PACKET CAPTURE as a reference.

The first two filters will have to be the tunnel endpoints IP address, e.g.

ID 1: Source=10.10.10.1 Destination=10.20.20.1
ID 2: Source=10.20.20.1 Destination=10.10.10.1

The remaining filters will have to be based on the traffic flowing thru the tunnel, e.g.

ID 3: Source=172.16.13.1 Destination=192.168.14.1
ID 4: Source=192.168.14.1 Destination=172.16.13.1

Enable Wireshark and/or tcpdump in the hosts on both ends of the tunnel.
Preparation for data collection
1. Use HOW TO COLLECT THE OUTPUT OF CLI COMMANDS PERIODICALLY USING TERA TERM SCRIPT
2. Use the following commands for the script to collect:

Global counters - show counter global filter packet-filter yes delta yes
Session details ("show session all filter source <> destination <>" to get the session ID and then using this ID run command "show session <id>" to get session details)

i=0
 sendln 'set cli pager off'
 sendln 'show system info'
 :continue

i=i+1
 sendln 'show counter global filter packet-filter yes delta yes'
 sendln 'show session all filter source 10.10.10.1 destination 10.20.20.1'
 sendln 'show session all filter source 172.16.13.1 destination 192.168.14.1'
 sendln 'show session <id-x>'
 sendln 'show session <id-y>'

pause 5
goto continue
end

Note: The above script sample is for reference purposes only; the source and destination IPs may be reversed; it depends on which IP initiated the connection.

Use the collected packet captures and global counters from the previous step to rule out possible factors causing low throughput.
1. Packet drop
  1. use IPSEC PERFORMANCE IMPACTED WHEN REPLAY PROTECTION IS ENABLED
  2. use IPSEC TUNNEL IS UP BUT PACKET IS GETTING DROPPED WITH WRONG SPI COUNTER INCREASE
  3. Are there significant packets missing when comparing the encrypted packets collected on the two firewalls?
2. Fragmentation
  1. do you see the counters "flow_fwd_mtu_exceeded" and/or "flow_ipfrag_frag"?
  2. To resolve the issue, adjust the tunnel MTU on both ends, go to Network > Interfaces > Tunnel > click on the appropriate interface > Advanced > Other info > MTU, try a lower value from the default of 1500 until the counters go away.
3. Out of order packets
  1. Compare the pcaps captured from step 4.a. using the clear text packets, and step 4.b.

How to troubleshoot reduced Tunnel Throughput

Objective

Environment

Procedure

Other users also viewed: