How to troubleshoot high dataplane processing latency

12198

Created On 05/14/24 16:51 PM - Last Modified 05/15/24 21:33 PM

Performance

Analysis

10.2

10.1

11.1

11.0

AIOps for NGFW

Next-Generation Firewall

Strata Cloud Manager

Objective

Investigate the root cause of latency.
Mitigate latency in dataplane traffic processing.

Environment

Packet Buffer protection.
Dataplane resources.

Procedure

Root Cause Investigation:

Ensure that Packet Buffer Protection (PBP) is enabled globally and on the suspected offending source zones.
Check the system and threat logs. Look for the following log messages:
1. Threat ID: 8507 / Threat type: Flood / Threat name: PBP Packet Drop.
2. Threat ID: 8508 / Threat type: Flood / Threat name: PBP Session Discarded.
3. Threat ID: 8509 / Threat type: Flood / Threat name: PBP IP.

The threat logs above will be logged only if Packet Buffer Protection (PBP) is enabled.

Check the status of the PBP module, latency measurements, and block list via CLI commands:

PBP and DP:

> show session packet-buffer-protection
> show session packet-buffer-protection buffer-latency
> show running resource-monitor ingress-backlogs
> debug dataplane pow performance
> debug dataplane pow performance | match pbp

Block Lists:

> show dos-block-table all
> show dos-block-table software
> show dos-block-table hardware
> debug dataplane show dos block-table

Check the global counters:
```
> show counter global 
```
Look for :
1. flow_dos_pbp_drop //Increments per packet for RED drop action.
2. flow_dos_pbp_block_session //Increments once when session is discarded (10.0 only).
3. flow_dos_pbp_block_host //Increments once when host is placed in the block list (10.0).
4. flow_dos_drop_ip_blocked //Generic DoS Block per packet counter
Use traffic logs and ACC if offending traffic can't be isolated otherwise.

Mitigation Steps:

Run the CLI command:
```
> show running resource monitor
```
Based on the output, check whether the dataplane exhibits high CPU usage, high packet buffer usage, high packet descriptor usage, or high packet descriptor (on-chip) usage during periods of high dataplane processing latency.
Then refer to the mitigation steps outlined in the following articles when traffic latency coincides with:
1. High dataplane CPU: How to Troubleshoot High Dataplane CPU.
2. High packet buffer or packet descriptors usage: How to Troubleshoot High Packet Buffer or Packet Descriptors Usage.
3. High Packet descriptors (on-chip): Traffic Latency - Packet Descriptors (on-chip).
Ensure that the appropriate thresholds are set for the Packet Buffer protection and confirm that PBP is enabled globally and on the zone level for Block/Discard action to work:
1. Packet Buffer Protection.
2. Packet Buffer Protection Based on Latency.

How to troubleshoot high dataplane processing latency

Objective

Environment

Procedure

Additional Information

Other users also viewed: