Traffic Latency - Packet Descriptors (on-chip)

27555

Created On 04/20/22 18:04 PM - Last Modified 09/03/24 20:22 PM

Performance

9.1

10.1

11.0

PAN-OS

Strata

Strata Cloud Manager

Symptom

Alert regarding Packet Descriptors (on-chip) being at high levels
Recommendation to identify and mitigate the offending traffic sessions which are utilizing a high amount of Packet Descriptors (on-chip)

Environment

PAN-OS

Cause

High percentage of Packet Descriptors (on-chip) in >show running resource-monitor,
High percentage of ATOMIC usage in >show running resource-monitor ingress-backlogs,

Resolution

To address this alert:

Check to see if Packet Descriptors (on-chip) are above normal values using the below CLI command

>show running resource-monitor

You will see the output below, for example

Resource utilization (%) during last 60 minutes:
packet descriptor (on-chip) (average):
  2   2   2   2   2   2   2   2   2   2   2   2   2   2   2
  2   2   2   2   2   2   2   2   2   2   2   2   2   2   2
  2   2   2   2   2   2   2   2   2   2   2   2   2   2   2
  2   2   2   2   2   2   2   2   2   2   2   2   2   2   2

If these values are higher than normal (Ex: usually 1-50% during the day, but showing 80%+ currently), a certain traffic flow might be abnormally utilizing a high amount of Packet Descriptors (on-chip), which could contribute to latency / traffic processing slowdowns in the firewall, and that traffic flow should be mitigated as soon as possible. You can identify that traffic flow and mitigate it by performing the steps below

Run the below CLI command repeatedly during the time when Packet Descriptors (on-chip) are high:

>show running resource-monitor ingress-backlogs

If you find any session with an abnormally high 'PCT' value - this traffic could be an 'offending' session that is causing the firewall's Packet Descriptors (on-chip) to go high

Example:
Identifying a session using 80%+ in >show running resource-monitor ingress-backlogs output

Identifying a session using 80%+ in >show running resource-monitor ingress-backlogs output

Command Reference: Identify Sessions That Use Too Much of the On-Chip Packet Descriptor

After the offending traffic flow's Source IP Address and Destination IP Address has been identified from >show running resource-monitor ingress-backlogs output, proceed below with the steps below to mitigate the offending session

1. Once the Source IP and Destination IP of the traffic flow contributing to the issue has been identified, shutdown/stop that traffic flow on the upstream/downstream device from which it is sourcing (i.e. stop the traffic at its source or before it comes to the firewall)

Example:
If there is a device flooding syslog packets over UDP port 514 to a particular destination IP, you can remove that syslog server destination IP from that device to stop the flood - see if Packet Descriptors (on-chip) are still high after shutting down that traffic - this will help you identify if that traffic is the cause of the high Packet Descriptors (on-chip)

2. Review the offending traffic flow, and determine if it should be Allowed or Denied in your network (i.e. determine if it is known/legitimate traffic in your network, or if it is unknown/malicious/noise traffic in your network), and based on that decision

If it is supposed to be Allowed, make sure it is getting Allowed by a corresponding Allow rule in your Security Policy
If it is supposed to be Denied, make sure it is getting Denied by a corresponding Deny rule in your Security Policy

3. Enable DoS and Zone Protection features to protect from the flood behavior

Example:
How to Configure DoS Protection
How to Configure Zone Protection
How to Identify and Mitigate traffic flows causing Packet Descriptors on-chip to go high

4. Starting PAN-OS 10.2 If you are having trouble catching the “offending session” in the ingress-backlogs CLI command output above, you can enable the below CLI option:
>set session inflight_monitoring yes

This CLI command will trigger the firewall to automatically capture >show running resource-monitor ingress-backlogs output whenever Packet Descriptors (on-chip) are above 80% (configurable) and write it to a log file located here:
>less mp-log pan_ingress_backlogs.log


Tip: To change the threshold or duration at which the firewall will automatically capture the 'ingress-backlogs' output, you can use the below commands:
>set session ingress_backlogs_threshold <2-100>
(Default: 80%)
>set session ingress_backlogs_duration <0-10>
(Default: 3sec)

WARNING: Remember to disable this option using the below CLI command as soon as you have captured the offending session details you need:
>set session inflight_monitoring no

Additional Information