How to Troubleshoot High Packet Buffer or Packet Descriptors Usage

61715

Created On 02/14/22 21:53 PM - Last Modified 11/08/23 21:46 PM

Zone and DoS Protection

PAN-OS

Objective

The article provides general guidance on troubleshooting High Packet Buffer or Packet Descriptors.
A high level of packet buffer usage can result in slowness and latency in user traffic. Many different factors can cause High Packet Buffer Usage. See possible Scenario's below

Burst of unexpected traffic, DoS attack.
For PA-3200, PA-5200, PA-7000’s, high rate of denied traffic that belongs to a single session that is being denied, high rate of fragmented traffic that belongs to a single session.
Associated with High Dataplane CPU. If the dataplane CPU is busy then the packet buffers can fill up. See here for tips on how to reduce dataplane CPU.
Software defect where packet buffers are not being released. In this scenario, packet buffer usage is high even when the traffic going through the firewall is very low.

In such scenarios, consider the following steps to bring back the device to a healthy state:

Environment

Palo Alto Firewalls
Supported PAN-OS
Packet Buffers and Packet Descriptors

Procedure

Scenario A:

Check for threat logs.
Threat logs will be logged only if Packet Buffer Protection (PBP) is enabled.
To check if PBP has been enabled and activated, go here. (create a different KB with all the PBP outputs listed below)

Mitigation for Scenario A and B

Check threat logs to see if any traffic has been identified as the source of the high traffic.
If no threat logs are seen, ensure that Packet Buffer Protection (PBP) is enabled and the configured parameters are sufficient to bring down packet buffer usage. (The Activate threshold for PBP defaults to 80%. If the firewall is sized correctly, buffer utilization should be well below 50%)
1. To check if PBP has been enabled and/or activated, use CLI show session packet-buffer-protection

Output below shows that PBP has not been enabled

> show session packet-buffer-protection
  Packet buffer protection is disabled.

Output below shows that PBP has been enabled, but not activated. An activated PBP means that the packet buffer usage has crossed the threshold and the traffic is now being inspected to find top users of the packet buffer resource.

> show session packet-buffer-protection
 ------------------------------------------------------------------------------------------
dp0
------------------------------------------------------------------------------------------
Packet buffer count based
Congestion: 1/86016 (0%)
System resource usage is low, packet buffer protection not activated.
------------------------------------------------------------------------------------------
dp1
------------------------------------------------------------------------------------------
Packet buffer count based
Congestion: 1/86016 (0%)
System resource usage is low, packet buffer protection not activated.

Output below shows that PBP has been activate that the top abusive session is session id 38492

> show session packet-buffer-protection
-------------------------------------------------------------------------------------------
dp0
-------------------------------------------------------------------------------------------
Packet buffer count based
Congestion: 12431/17203 (72%)
Drop probability: 74%. Percentage drop threshold: 48.
-------------------------------------------------------------------------------------------
|                   |      |            | Drop  |     Packets      |
Session/IP Address |       Zone        | PCS  | Percentage | State | Total  | Dropped | Time till discard
-------------------------------------------------------------------------------------------

38492              | replay_vw_trust   | 4088 | 49         | Yes   | 381171 | 121232  | 59
38492              | replay_vw_untrust | 4024 | 49         | Yes   | 392557 | 125172  | 57
172.16.1.1         | vw-trust          | 31   | 0          | No    | 721    | 0       | 60
51718              | vw-trust          | 9    | 0          | No    | 1      | 0       | 60
172.16.1.2         | vw-untrust        | 4    | 0          | No    | 71     | 0       | 60
51704              | vw-untrust        | 3    | 0          | No    | 0      | 0       | 60
51712              | vw-trust          | 3    | 0          | No    | 0      | 0       | 60
51714              | vw-untrust        | 3    | 0          | No    | 0      | 0       | 60
-------------------------------------------------------------------------------------------

If packet buffer usage is high, but PBP hasn't been activated then determine if the PBP thresholds should be lowered. By default, PBP kicks in when the buffer usage goes to 80%. The active threshold can be lowered so that if there is an abusive session, it can be discarded earlier. To change configuration for PBP, go here: Configure Packet Buffer Protection

Ensure that Zone protection Profiles are in place to protect against packet floods.
Below CLI prints the configured zone protection thresholds for a specific zone. Adjust the thresholds to match the traffic pattern seen by the device.

> show zone-protection zone <zone name>
------------------------------------------------------------------------------------------
Number of zones with protection profile: 1
------------------------------------------------------------------------------------------
Zone test, vsys vsys1, profile tap
------------------------------------------------------------------------------------------
tcp-syn              RED enabled: yes
DP alarm rate:       80000 cps, activate rate:   80000 cps, maximal rate:   80000 cps
current:            193413 packets
dropped:           262219664 packets
------------------------------------------------------------------------------------------
udp                  RED enabled: yes
DP alarm rate:       80000 cps, activate rate:   80000 cps, maximal rate:   80000 cps
current:                 0 packets
dropped:                 0 packets
------------------------------------------------------------------------------------------
icmp                 RED enabled: yes
DP alarm rate:       80000 cps, activate rate:   80000 cps, maximal rate:   80000 cps
current:                 0 packets
dropped:                 0 packets
------------------------------------------------------------------------------------------
other-ip             RED enabled: yes
DP alarm rate:       80000 cps, activate rate:   80000 cps, maximal rate:   80000 cps
current:                 0 packets
dropped:                 0 packets
------------------------------------------------------------------------------------------
icmpv6               RED enabled: yes
DP alarm rate:       80000 cps, activate rate:   80000 cps, maximal rate:   80000 cps
current:                 0 packets
dropped:                 0 packets

If a session is identified through the threat logs or the CLI output of show session packet-buffer-protection, specific action can be taken against that traffic, by creating a DoS policy against known offenders and follow the instructions that are documented in (HIGH ON-CHIP DESCRIPTOR AND PACKET BUFFER USAGE DUE TO POLICY DENY RESULTING IN TRAFFIC LATENCY AND DROPS). Or, the traffic can be stopped upstream of the firewall.

Scenario B:

Run “show running resource-monitor” and look at the output for on-chip packet descriptors. If the value is consistently greater than 90% then the device is hitting scenario B.

Resource utilization (%) during last 5 seconds:
session:
  0   0   0   0   0 
packet buffer:
 75  75  75  75  75 
packet descriptor:
  0   0   0   0   0 
packet descriptor (on-chip):
100 100 100 100 100

Mitigation for Scenario B (same as A)

Scenario C:

Run “show running resource-monitor” to check CPU usage. If CPU usage is above 80% then the device could be hitting the scenario where the build up is being caused by the CPU.

Mitigation for Scenario C

To reduce the CPU usage, please try to reduce the traffic inspection. Following steps could be considered
- Remove Security Profile that associated with the Security Policy. See Identify Sessions That Use Too Much of the On-Chip Packet Descriptor
- Disable Server Response Inspection as per "IMPROVING PERFORMANCE OF HTTP WITH DSRI"
- Configure App Override for the traffic that could potentially cause high CPU/packet buffer usage. As per "HOW TO CREATE AN APPLICATION OVERRIDE "

Scenario D:

If the device doesn’t match any of the conditions above, then this could be a software defect. See below on the data needed when opening a Support Case.

Data to collect for Scenario D

Run the following commands

show running resource-monitor
debug dataplane pow performance
debug dataplane pool statistics
show counter global filter delta yes (multiple times)
show counter global

Note: For the next 2 commands, if the CLI output contains any sessions, print “show session id <session id>” for those sessions, so that the sessions that are occupying the resource can be identified.

show session packet-buffer-protection
show session packet-buffer-protection buffer-latency
Collect a Tech Support File
Traffic logs that cover the time period of when the packet buffers are high