How to mitigate High DP CPU issue due to an increase in flow of traffic denied at slowpath stage by a security policy
12176
Created On 05/31/23 17:58 PM - Last Modified 08/23/23 22:04 PM
Objective
To mitigate High DP CPU issue due to an increase in flow of traffic denied at slowpath stage by a security policy. Note: This root cause of high DP CPU would have been determined by noticing that the increase in DP CPU coincides with an increase in “flow_policy_deny” global counter and an increase in both "flow_slowpath count" and "flow_slowpath total-us" values in the output of "debug data plane pow performance", indicating that the traffic's sessions are denied at the slowpath stage.
Environment
- Palo Alto Firewall
- DP CPU
- Security Policy with action deny
- Flow slowpath
Procedure
- Identify the source of the traffic experiencing an increase in its session being denied:
- Check the traffic logs: Monitor > Logs > Traffic and use the search filter ( action eq deny).
- In case your search comes up empty that means that you most likely need to enable the logging on the security policy that is denying the traffic. This security policy can possibly be the interzone-default that you need to override to enable the Log at Session End and/or Start.
- Check the ACC tab: ACC and use the action deny filter.
- Check the traffic logs: Monitor > Logs > Traffic and use the search filter ( action eq deny).
- Determine the eligibility of the denied traffic:
- If the traffic is eligible and shouldn't have been denied by the FW:
- Create a security policy for this traffic that has action allow or alert and apply the proper security profile to it.
- If the traffic is not eligible:
- Attempt to stop the traffic at its source or closer to its source.
- Otherwise, use DOS protection policy to protect the firewall against this offending traffic.
- If the offender IP(s) is/are known, then create a DOS protection policy: POLICIES > DoS Protection you need to be able to define the source and destination zones and IPs as well as service port so you can set the action to Deny without affecting other eligible traffic.
- If the offender IP is unknown and port service is unknown, but its determined that it has one source of traffic coming from defined zone and going to defined destinations, then create a DOS protection policy with Classified option enabled, action set to Protect and attach a DOS protection profile to it.
- If the traffic is eligible and shouldn't have been denied by the FW:
In the example shown above the Address is set to source-ip-only the other valid option would be to set it to src-dest-ip-both for more details on why those two options are the only one valid in this case refer to High on-chip descriptor and packet buffer usage due to policy deny resulting in traffic latency and drops (Mitigation step 4). As for which Flood Protection to enable and configure under the DOS Protection Profile that would be based on your findings in step 1 if for example the traffic denied is a UDP traffic then enable the UDP Flood.
- If the offenders are unknown and it is found that multiple sources are triggering the denied traffic, then create a DOS protection policy with Aggregate option enabled, action set to Protect and attach a DOS protection profile to it. Your findings in step 1 will determine which Flood Protection to enable and configure under the DOS Protection Profile for example the traffic denied is a UDP traffic then enable the UDP Flood.
- As best practice and in order to protect the firewall's resources it is always recommended to enable packet buffer protection globally under Device > Setup > Session Settings and per zone under Network > Zones.
Additional Information
For more information on the behavior of the firewall and its impact when high DP processing activity is caused by an increase in flow of traffic denied by a security policy refer to: High on-chip descriptor and packet buffer usage due to policy deny resulting in traffic latency and drops.
For more details on Packet Flow Sequence and slowpath stage refer to Packet Flow Sequence in PAN-OS .