Traffic Latency - Packet Descriptors (on-chip)
27731
Created On 04/20/22 18:04 PM - Last Modified 09/03/24 20:22 PM
Symptom
- Alert regarding Packet Descriptors (on-chip) being at high levels
- Recommendation to identify and mitigate the offending traffic sessions which are utilizing a high amount of Packet Descriptors (on-chip)
Environment
- PAN-OS
Cause
- High percentage of Packet Descriptors (on-chip) in >show running resource-monitor,
- High percentage of ATOMIC usage in >show running resource-monitor ingress-backlogs,
Resolution
To address this alert:
Check to see if Packet Descriptors (on-chip) are above normal values using the below CLI command
>show running resource-monitor
You will see the output below, for example
Resource utilization (%) during last 60 minutes: packet descriptor (on-chip) (average): 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2If these values are higher than normal (Ex: usually 1-50% during the day, but showing 80%+ currently), a certain traffic flow might be abnormally utilizing a high amount of Packet Descriptors (on-chip), which could contribute to latency / traffic processing slowdowns in the firewall, and that traffic flow should be mitigated as soon as possible. You can identify that traffic flow and mitigate it by performing the steps below
Run the below CLI command repeatedly during the time when Packet Descriptors (on-chip) are high:
>show running resource-monitor ingress-backlogsIf you find any session with an abnormally high 'PCT' value - this traffic could be an 'offending' session that is causing the firewall's Packet Descriptors (on-chip) to go high
Example:
Command Reference: Identify Sessions That Use Too Much of the On-Chip Packet Descriptor
After the offending traffic flow's Source IP Address and Destination IP Address has been identified from >show running resource-monitor ingress-backlogs output, proceed below with the steps below to mitigate the offending session
1. Once the Source IP and Destination IP of the traffic flow contributing to the issue has been identified, shutdown/stop that traffic flow on the upstream/downstream device from which it is sourcing (i.e. stop the traffic at its source or before it comes to the firewall)
Example:
If there is a device flooding syslog packets over UDP port 514 to a particular destination IP, you can remove that syslog server destination IP from that device to stop the flood - see if Packet Descriptors (on-chip) are still high after shutting down that traffic - this will help you identify if that traffic is the cause of the high Packet Descriptors (on-chip)
2. Review the offending traffic flow, and determine if it should be Allowed or Denied in your network (i.e. determine if it is known/legitimate traffic in your network, or if it is unknown/malicious/noise traffic in your network), and based on that decision
If it is supposed to be Allowed, make sure it is getting Allowed by a corresponding Allow rule in your Security Policy
If it is supposed to be Denied, make sure it is getting Denied by a corresponding Deny rule in your Security Policy
3. Enable DoS and Zone Protection features to protect from the flood behavior
Example:
How to Configure DoS Protection
How to Configure Zone Protection
How to Identify and Mitigate traffic flows causing Packet Descriptors on-chip to go high
4. Starting PAN-OS 10.2 If you are having trouble catching the “offending session” in the ingress-backlogs CLI command output above, you can enable the below CLI option:
>set session inflight_monitoring yes
This CLI command will trigger the firewall to automatically capture >show running resource-monitor ingress-backlogs output whenever Packet Descriptors (on-chip) are above 80% (configurable) and write it to a log file located here:
>less mp-log pan_ingress_backlogs.log
Tip: To change the threshold or duration at which the firewall will automatically capture the 'ingress-backlogs' output, you can use the below commands:
>set session ingress_backlogs_threshold <2-100>
(Default: 80%)
>set session ingress_backlogs_duration <0-10>
(Default: 3sec)
WARNING: Remember to disable this option using the below CLI command as soon as you have captured the offending session details you need:
>set session inflight_monitoring no
Additional Information
Refer to our official documentation about DoS Protection and Zone Protection