How to mitigate an abnormal increase in "tcp_drop_packet" global counter
12928
Created On 11/29/23 20:08 PM - Last Modified 11/30/23 19:23 PM
Objective
The counter tcp_drop_packet increments when a packet is received by the firewall which has invalid/incorrect data in a packet field or the flow of packets is not adhering to conventional TCP operation and standards, resulting in TCP Reassembly failing. The counter tcp_drop_packet is typically a catch-all counter for all drops in TCP reassembly and is most commonly seen with another 'tcp_' counter which indicates the specific root cause.
Below is an example of the global counter tcp_drop_packet incrementing in the firewall:
> show counter global name value rate severity category aspect description -------------------------------------------------------------------------------- tcp_drop_packet 2 0 warn tcp pktproc packets dropped because of failure in tcp reassemblyNote: This global counter is only significant if seen during troubleshooting a specific traffic flow using Global Counters with Filters for a specific Source and Destination IP Address and this global counter is seen to drop that traffic flow specifically. All other times, this global counter may be safely ignored, and do not proceed with the steps below in this document. If the global counter 'tcp_drop_packet' is seen to be dropping a specific traffic flow being troubleshot using global counters, then proceed below.
Environment
- Next Generation Firewall
Procedure
- Identify which traffic flow by Source IP, Source Port, Destination IP, Destination Port in the network is being dropped by the firewall due to this counter (if any) using Global Counters with Filters
Note: If the tcp_drop_packet global counter is seen in general but no traffic issue is known to be occurring, this counter may be safely ignored. If troubleshooting a specific traffic flow and that traffic flow is being dropped due to the 'tcp_drop_packet' counter specifically, proceed with Step 2 below.
- Run the below CLI command:
> show counter globalLook for any other global counters containing "tcp_" which are incrementing at the same time as tcp_drop_packet and troubleshoot the reason for that more-specific global counter instead.
Example:
If "tcp_drop_packet" and "tcp_drop_out_of_wnd" are both incrementing, proceed with troubleshooting using How to mitigate an abnormal increase in "tcp_drop_out_of_wnd" global counter instead
If "tcp_drop_packet" and "tcp_invalid_ts_option" are both incrementing, proceed with troubleshooting using TCP packets dropped by Firewall due to Invalid Timestamp option instead
If "tcp_drop_packet" and "tcp_exceed_flow_seg_limit" are both incrementing, proceed with troubleshooting using PAN-OS Web Interface Help - TCP Settings instead
- Contact the vendor of the software application sending those TCP packets to have the invalid or incorrect TCP behavior corrected
- If a firewall setting exists (depends on the reason for the packet drop), adjust the specific firewall configuration setting which corresponds to that TCP exception global counter found above in Step 2
Caution: Per PAN-OS Best Practices, it is advised to maintain the default values for the majority of PAN-OS settings. Any modification to default PAN-OS settings should be undertaken with careful consideration, ensuring thoughtful evaluation prior to any change being made.
Example: PAN-OS provides the ability to disable 'Check Timestamp Option', thus stopping packets being dropped with the 'tcp_invalid_ts_option' global counter, but it is not recommended to disable that setting to maintain a secure deployment as well as for connectivity reasons.
Example: PAN-OS provides the ability to disable 'Check Timestamp Option', thus stopping packets being dropped with the 'tcp_invalid_ts_option' global counter, but it is not recommended to disable that setting to maintain a secure deployment as well as for connectivity reasons.
Additional Information
Other global counters seen alongside tcp_drop_packet which further indicate the specific root cause of the packet drop include:
tcp_out_of_sync warn can't continue tcp reassembly because it is out of sync tcp_drop_out_of_wnd warn out-of-window packets dropped tcp_invalid_ts_option info packets with invalid timestamp option tcp_syn_missing info miss SYN packet for tcp session tcp_exceed_flow_seg_limit warn packets dropped due to the limitation on tcp out-of-order queue sizeMost of the above invalid/incorrect TCP packet exceptions can be viewed and verified using a Packet Capture. If needed, take a Packet Capture to view the fields of any TCP packet or flow to verify the information in each field.