How to mitigate an abnormal increase in "tcp_drop_out_of_wnd" global counter
Objective
To mitigate an abnormal increase in tcp_drop_out_of_wnd global counter.
Counter's description:
This counter tcp_drop_out_of_wnd increments when TCP packets received outside the TCP sliding window are dropped.
The Palo Alto Networks Firewall creates a sliding sequence window starting with the original ACK (the window size is based on the type of traffic within the session). It is expected that the packet sequence numbers within the current session reside within this sliding window. This window adjusts with the type of traffic and whenever new ACK messages are received. The default behavior on the device is to drop packets when sequence numbers are outside this window.
Environment
- Next Generation Firewall
- tcp_drop_out_of_wnd
Procedure
- Check the firewall current TCP settings:
> show running tcp state session with asymmetric path : drop packet Bypass if OO queue limit is reached : no Favor new seg data : no Urgent data : clear Drop if zero after clear urgent flag : yes Check Timestamp option : yes Allow Challenge Ack : no Remove MPTCP option : yes
- It is crucial to identify the traffic affected by these TCP drops. If possible, use traffic profiling tools or refer to recent traffic issues reported by network users that coincide with the timestamp of the increase in this global counter. In the case of successful identification, conducting a packet capture on the firewall for a sample of that traffic will significantly assist in implementing the appropriate permanent fix for this issue.
- Temporary workarounds can be applied to mitigate the issue:
- If the source zone of the affected traffic flows is unknown or covers all networks, the global setting below will disable the TCP sanity checks for the dataplane traffic of the firewall:
> configure # set device config setting tcp asymmetric-path bypass # commit
- If the source zone of the affected traffic flow is known, a zone protection profile can be configured to allow this type of traffic from one or more specific zones while still performing TCP sanity checks for all other zones connected to the firewall:
- Create a zone protection profile in Networks > Network Profiles > Zone Protection
- Set the Zone Protection > TCP Drop > Reject Non-SYN TCP to no
- Set the Zone Protection > TCP Drop > Asymmetric Path to bypass
- Assign this profile to the zone to disable the TCP checks for it.
- If the source zone of the affected traffic flows is unknown or covers all networks, the global setting below will disable the TCP sanity checks for the dataplane traffic of the firewall:
Very Important: Please be aware that disabling these TCP sanity checks can have severe security implications as an attacker could inject packets that would normally be discarded. It is recommended to only set these commands in order to troubleshoot the cause of these packet drops, or to temporarily bypass the checks while addressing the issue, but not as a permanent solution.
- If, upon reviewing the packet capture in step 2, it is determined that the dropped packet counted towards the global counter tcp_drop_out_of_wnd is due to:
- TCP window-related issues: then address this issue by tuning the window sizes appropriately, ensuring compatibility between devices on the network, and optimizing other TCP parameters to achieve better performance and reliability.
- Asymmetric routing then refer to DotW: Issues with Asymmetric Routing.
- An unexpected behavior from the firewall, because the TCP packet dropped has a sequence number within the TCP sliding window size, then investigate whether the issues and corresponding fixes below are applicable to the specific traffic use case:
- Firewall dropping RST from Client after Server's Challenge-ACK Refer to KB2 for more information about the issue and the fix.
- PAN-216314: (PA-3200 Series firewalls only) Fixed an issue where, after upgrading to or from PAN-OS 10.1.9 or PAN-OS 10.1.9-h1, offloaded application traffic sessions disconnected even when a session was active. This occurred due to the application default session timeout value being exceeded. Refer to KB1 for more information about the issue and the releases with the fix.
- Firewall dropping TCP FIN or RST packets for long TCP sessions and marking it as out of window TCP packet even though the ACK numbers are within the window. This issue is fixed in 11.1.0, 11.0.4, 10.1.12, 11.2.0, 10.2.8. For more information about this issue and other releases with the fix reach out to Palo Alto Networks support and mention PAN-223080.