NOTE
- Once you have determined that the policy deny is significantly high or is higher than usual, the next step is to Identify the source of this Traffic.
- If "Log at session-end" is enabled on the policy denying the traffic, the "offending" traffic can be found by filtering the Traffic logs for policy-deny.
Note: In most cases, it is same 6 tuples UDP syslog traffic, that causes the issue.
UDP syslog traffic is usually high volume and uses the same source-port (socket).
Mitigation
1. Once the source of the denied traffic is identified, check if it is feasible to stop this traffic at source or closer to source.
Example: If there is a device that is flooding syslog messages to a particular destination, you can remove the syslog server destination from that device to stop the flood.
2. Allowing traffic with security policies
- Before moving into mitigation techniques, we should make sure the traffic that is supposed to be allowed is indeed allowed by security policies.
- If the traffic needs to be allowed, create the required security policy. Once the traffic is allowed, a session would be installed and the traffic is not subjected to slowpath.
- If traffic pattern in the network is not known, a security policy can be created to allow high volume traffic like syslog from internal/trusted zones(apply security profiles as required).
3. Protection against known offenders (DoS policy)
- If exact IPs of the hosts causing the issue are known, creating a DoS policy with the action “Deny” will help.
- DoS policy rules are specific (source/destination zone, IPs, service port), and they can replace similar security policy with action deny.
- Since DoS policies are evaluated before security policy lookup and do not have large number of entries, packets are blocked earlier thereby saving firewall resources.
4. Protection against unknown offenders (DoS policy)
- Creating a DoS classified policy with the action “Protect”
- A classified DoS policy can be applied with action “Protect” and address matched to “source-ip only” or “src-dest-ip-both”.
***Note: Configured threshold values are only examples, these need to be tweaked based on customer environment
- Once the configured threshold is reached, the DoS policy would create a DoS ip-block-table, which would start dropping packets without being subjected to slowpath.
- In devices that have an offload processor, the block table would be installed in the offload hardware to further reduce the load on the DP CPU.
- For further reading refer : Monitor Block List and DoS Protection Profiles and Policy Rules
Caveat:
- Thresholds for the classified DoS object would change based on customer traffic pattern and network, the default values may not be applicable to all environments.
- For slowpath deny attacks only “source-ip only” or “src-dest-ip-both” would work, using "destination-ip only" does not help.
- For internet facing zones, since source ips could be potentially huge, the firewall doesn’t have the capacity to store counters for every possible IP address on the internet.
- Refer: Classified vs Aggregate DoS profiles
5. Packet Buffer Protection (PBP)
- Packet Buffer Protection (PBP) is a feature available starting with PAN-OS 8.0.
- PBP is preferred, as it is automatic and is triggered based on actual resource utilization, when compared to DoS policy which is triggered on pre-configured connections per second threshold
- PBP protects the firewall both from slowpath and fastpath (existing session) buffer depletion.
- Firewall monitors buffer abusers automatically.
- After reaching the configured activate threshold (default 50%), the firewall starts dropping offending traffic (RED).
- If buffer utilization is above 80% ( this threshold is internally hardcoded and not configurable) for a duration of block hold time a dos block table entry is created.
- Refer: Packet Buffer Protection
In this specific case of slowpath deny usually a combination of PBP + DoS classified policy with the action “Protect” provides better results.Monitoring:SNMP can be leveraged to monitor buffer utilization among other things. DP resources are part of HOST-RESOURCES-MIB. More information can be found here:
SNMP for Monitoring Palo Alto Networks Devicessnmp-mibsList of useful OIDs:1. Description - .1.3.6.1.2.1.25.2.3.1.3.xxxx
Example:.1.3.6.1.2.1.25.2.3.1.3.1011 = STRING: "Slot-1 Data Processor-0 Hardware Packet Buffers"
.1.3.6.1.2.1.25.2.3.1.3.1111 = STRING: "Slot-1 Data Processor-1 Hardware Packet Buffers"
2. Hardware Packet Buffer pool size - .1.3.6.1.2.1.25.2.3.1.5.xxxx
Example:.1.3.6.1.2.1.25.2.3.1.5.1011 = INTEGER: 17203
.1.3.6.1.2.1.25.2.3.1.5.1111 = INTEGER: 17203
3. Current Buffer Utilization - .1.3.6.1.2.1.25.2.3.1.6.xxxx
Example:.1.3.6.1.2.1.25.2.3.1.6.1011 = INTEGER: 122
.1.3.6.1.2.1.25.2.3.1.6.1111 = INTEGER: 128
DoS related counters via SNMP (part of PAN-COMMON-MIB):
MIB Identity | Counter | Description | OID |
panFlowPolicyDeny | flow_policy_deny | Session setup: denied by policy | .1.3.6.1.4.1.25461.2.1.2.1.19.8.10 |
panFlowDosBlkNumEntries | flow_dos_blk_num_entries | Number of entries in DOS block table | .1.3.6.1.4.1.25461.2.1.2.1.19.8.2 |
panFlowDosBlkSwEntries | flow_dos_blk_sw_entries | Number of entries in DOS Software block table | .1.3.6.1.4.1.25461.2.1.2.1.19.8.33 |
panFlowDosBlkHwEntries | flow_dos_blk_hw_entries | Number of entries in DOS Hardware block table | .1.3.6.1.4.1.25461.2.1.2.1.19.8.34 |
panFlowDosDropIpBlocked | flow_dos_drop_ip_blocked | Packets dropped: Flagged for blocking and under block duration by DoS or other modules | .1.3.6.1.4.1.25461.2.1.2.1.19.8.13 |
panFlowDosRuleDrop | flow_dos_rule_drop | Packets dropped: Rate limited or IP blocked | .1.3.6.1.4.1.25461.2.1.2.1.19.8.23 |