How to mitigate an abnormal increase in "pkt_recv_err" global counter
10130
Created On 11/28/23 19:03 PM - Last Modified 07/24/25 07:42 AM
Objective
The counter pkt_recv_err increments when a packet is received by the firewall with incorrect or invalid L2-L4 information. The firewall fails to parse or process the packet, drops the packet, and increments the pkt_recv_err global counter.
Below is an example of the global counter pkt_recv_err incrementing in the firewall:
> show counter global name value rate severity category aspect description -------------------------------------------------------------------------------- pkt_recv_err 100 100 drop packet pktproc Packet receive errorAlthough rare, some applications may send packets that do not adhere to current valid RFC standard for TCP/IP networking, causing this issue to occur. In those cases, the vendor of the software application must resolve that issue with their software.
Note: This global counter is only significant if seen during troubleshooting a specific traffic flow using Global Counters with Filters for a specific Source and Destination IP Address and this global counter is seen to drop that traffic flow specifically. All other times, this global counter may be safely ignored, and do not proceed with the steps below in this document. If the global counter 'pkt_recv_err' is seen to be dropping a specific traffic flow being troubleshot using global counters, then proceed below.
Environment
- Next Generation Firewall
Procedure
- Identify which traffic flow by Source IP, Source Port, Destination IP, Destination Port in the network is being dropped by the firewall due to this counter (if any) using Global Counters with Filters
Note: If the pkt_recv_err global counter is seen in general but no traffic issue is known to be occurring, this counter may be safely ignored. If troubleshooting a specific traffic flow and that traffic flow is being dropped due to the 'pkt_recv_err' counter specifically, proceed with Step 2 below.
- Run the below CLI command:
> show counter globalLook for any other global counters beginning with "pkt_recv" which are incrementing at the same time as pkt_recv_err and troubleshoot the reason for that global counter instead.
Example:
If "pkt_recv_err" and "pkt_recv_multiple_bufs" are both incrementing, proceed with troubleshooting using How to mitigate an abnormal increase in "pkt_recv_multiple_bufs" global counter instead
If "pkt_recv_err" and "pkt_recv_short_pkt" are both incrementing, proceed with troubleshooting using How to mitigate an abnormal increase in "pkt_recv_short_pkt" global counter instead
- Take a packet capture (with filters) at the following capture points:
a. The egress port of the switch/device connected to the firewall (where this packet is coming from)
b. The ingress port of the firewall (where this packet is going to)
b. The ingress port of the firewall (where this packet is going to)
- Open the packet capture(s) in Wireshark and inspect the L2-L4 information in the dropped packets for any of the below issues:
L2: Incorrect length of VLAN tag, incorrect encapsulation
L3: Incorrect checksum, IP packet is truncated, packet too short, packet too large, packet checksum error, IP and UDP/TCP length mismatch, packet larger than firewall interface MTU (may need to enable Jumbo Frames)
L4: TCP/UDP Port 0, Invalid TCP Flag, etc.
L3: Incorrect checksum, IP packet is truncated, packet too short, packet too large, packet checksum error, IP and UDP/TCP length mismatch, packet larger than firewall interface MTU (may need to enable Jumbo Frames)
L4: TCP/UDP Port 0, Invalid TCP Flag, etc.
- Take the appropriate action to resolve the issue according to the findings above. Examples include:
- If the source device or application of these packets is sending them incorrectly / with invalid L2-L4 information in them, contact the vendor of that software application about the reported behavior to have them fix it
- If the VLAN is configured incorrectly on the firewall compared to the VLAN tag in the packets, adjust the VLAN tag to be correct on the firewall or in the packets
- If Jumbo Frames are disabled but your network uses Jumbo Frames, consider enabling Jumbo Frames on the firewall (for packets larger than default interface MTU of 1500)
- If the packets are coming in with a Source Port or Destination Port of 0, go to the source application generating these invalid packets and correct that issue
Additional Information
Other global counters which are commonly seen incrementing at the same time as pkt_recv_err include:
>show counter global | match pkt_recv pkt_recv_err_large drop Packet receive error due to large packet pkt_recv_short_pkt drop Packet receive short packets pkt_recv_empty_buf drop packet pktproc Packet WQE without buffer pkt_recv_multiple_bufs drop packet pktproc Packets received with multiple buffers pkt_recv_throttle_cos drop packet pktproc Packets throttled by QoS control pkt_recv_flush_link drop Packets dropped due to link down in dpdk mode pkt_recv_flush_passive drop Packets drop due to passive VM in dpdk mode pkt_recv_skip_inflight info Packet receive skipped due to high inflight num pkt_recv_multiple_bufs drop Packets received with multiple buffers not from content engine pkt_recv_multiple_bufs_ce info Packets received with multiple buffers from content engine pkt_recv_retry info Full Burst Packets received on retry pkt_recv_sso_wa_err warn Packets received with atomic type but without SSO workaround pkt_recv_zero info Packets received from QoS 0 flow_ipfrag_large_pkt_recv info IP fragment large packet(>16k) received pkt_recv_hop_err drop Packet received with IP HOP error pkt_recv_lro info LRO Packets received pkt_recv_lro_cksum_err drop LRO Packets received but dropped due incorrect cksum pkt_recv_lro_dvflt_err drop LRO Packets received but dropped due to DV Filter pkt_recv_lro_err drop LRO Packets receive error pkt_recv_lro_ipfrag_err drop LRO Packets received but dropped due to IP fragment pkt_recv_lro_mem_err drop LRO Packets received but no memory pkt_recv_lro_notip_err drop LRO Packets drop due to non-IP pkt_recv_lro_prepend_err drop LRO Packets received but prepend errorIf a packet larger than 1500 bytes comes into a firewall interface whose interface is the default interface MTU of 1500, the packet may drop and pkt_recv_err increments. Enabling Jumbo Frames may resolve this issue.