High Dataplane CPU after software upgrade
Symptom
-
High dataplane CPU utilization accompanied by elevated packet buffer and packet descriptor usage.
-
Increase in overall packet rate due to "ack storm" introduced by Client accumulation proxy.
-
High C2S with zero S2C packet count
>show session id 1449889335 Session 1449889335 c2s flow: source: 10.10.20.25 [Inside] dst: 11.11.11.11 proto: 6 sport: 49162 dport: 443 state: ACTIVE type: FLOW src user: unknown dst user: unknown s2c flow: source: 11.11.11.11 [Untrust] dst: 2.2.2.2 proto: 6 sport: 443 dport: 49162 state: ACTIVE type: FLOW src user: unknown dst user: unknown Slot : 6 DP : 3 index(local): : 7048759 start time : Wed Jun 26 08:43:46 2025 timeout : 1800 sec time to live : 1800 sec total byte count(c2s) : 536074470 total byte count(s2c) : 0 layer7 packet count(c2s) : 8934544 layer7 packet count(s2c) : 0 vsys : vsys1 application : ssl rule : Outbound_internet service timeout override(index) : False session to be logged at end : True session in session ager : True session updated by HA peer : False session proxied : True layer7 processing : enabled ctd version : 17 URL filtering enabled : True URL category : All_URLs, search-engines, low-risk session via syn-cookies : False session terminated on host : False session traverses tunnel : False session terminate tunnel : False captive portal session : False ingress interface : ae2.100 egress interface : ae1.7 session QoS rule : N/A (class 4) end-reason : unknown Proxy Info: Proxy Flow Index: 1198240, Type: accumulation, Tag: 7048759, Dir: cts Peer Flow Index: 1421833, Type: accumulation, Tag: 7048759, Dir: stc
Environment
- Any firewall platform
- PAN-OS
Cause
In an asymmetric routing environment, when the firewall is only able to see the client-to-server (C2S) flow and the ClientHello does not arrive in a single TCP segment, the ClientHello accumulation proxy is triggered. The session then enters an “ACK storm” state, where the firewall’s accumulation proxy (fptcp) continues to ACK the ClientHello. However, due to the asymmetry, the client has already received the ACK for the ClientHello directly from the server and proceeds to complete the TLS handshake (ChangeCipherSpec). As a result, the firewall proxy and the client become stuck in a loop of repeated ACKs. This ACK storm significantly increases the packet rate, placing additional load on the CPU.
Resolution
A fix for this issue has been provided under bug ID PAN-279500, and there are currently two workarounds available:
Workaround 1: Disable the accumulation proxy using the following command:
>debug dataplane set ssl-decrypt accumulate-client-hello disable yes
Workaround 2: Disable all decryption policies, regardless of whether they are set to no-decrypt or forward proxy.
Additional Information
This issue was observed after upgrading the PAN OS to 10.2.9-h10,10.2.14, 11.1.7, 11.1.8, 12.1.0, 11.2.5, 11.2.8, 10.2.7-h20, 10.2.11-h8, 11.1.2-h17, 11.1.4-h12, 11.2.4-h4, 11.1.6-h1, 10.2.7-h22, 10.2.9-h20, 10.2.10-h13, 10.2.8-h20, 10.2.11-h11, 10.2.12-h5, 10.2.13-h3