High Dataplane CPU after software upgrade

High Dataplane CPU after software upgrade

1453
Created On 07/11/25 00:56 AM - Last Modified 07/17/25 15:44 PM


Symptom


  • High dataplane CPU utilization accompanied by elevated packet buffer and packet descriptor usage.

  • Increase in overall packet rate due to "ack storm" introduced by Client accumulation proxy. 

  • High C2S with zero S2C packet count

>show session id 1449889335

Session 1449889335

c2s flow:
source: 10.10.20.25 [Inside]
dst: 11.11.11.11
proto: 6
sport: 49162 dport: 443
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown

s2c flow:
source: 11.11.11.11 [Untrust]
dst: 2.2.2.2
proto: 6
sport: 443 dport: 49162
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown

Slot : 6
DP : 3
index(local): : 7048759
start time : Wed Jun 26 08:43:46 2025
timeout : 1800 sec
time to live : 1800 sec
total byte count(c2s) : 536074470
total byte count(s2c) : 0
layer7 packet count(c2s) : 8934544
layer7 packet count(s2c) : 0 
vsys : vsys1
application : ssl
rule : Outbound_internet
service timeout override(index) : False
session to be logged at end : True
session in session ager : True
session updated by HA peer : False
session proxied : True 
layer7 processing : enabled 
ctd version : 17
URL filtering enabled : True
URL category : All_URLs, search-engines, low-risk
session via syn-cookies : False
session terminated on host : False
session traverses tunnel : False
session terminate tunnel : False
captive portal session : False
ingress interface : ae2.100
egress interface : ae1.7
session QoS rule : N/A (class 4)
end-reason : unknown

Proxy Info:
Proxy Flow
Index: 1198240, Type: accumulation, Tag: 7048759, Dir: cts 

Peer Flow
Index: 1421833, Type: accumulation, Tag: 7048759, Dir: stc


Environment


  • Any firewall platform
  • PAN-OS


Cause


In an asymmetric routing environment, when the firewall is only able to see the client-to-server (C2S) flow and the ClientHello does not arrive in a single TCP segment, the ClientHello accumulation proxy is triggered. The session then enters an “ACK storm” state, where the firewall’s accumulation proxy (fptcp) continues to ACK the ClientHello. However, due to the asymmetry, the client has already received the ACK for the ClientHello directly from the server and proceeds to complete the TLS handshake (ChangeCipherSpec). As a result, the firewall proxy and the client become stuck in a loop of repeated ACKs. This ACK storm significantly increases the packet rate, placing additional load on the CPU.



Resolution


A fix for this issue has been provided under bug ID PAN-279500, and there are currently two workarounds available:

 

Workaround 1: Disable the accumulation proxy using the following command:

>debug dataplane set ssl-decrypt accumulate-client-hello disable yes

 

Workaround 2: Disable all decryption policies, regardless of whether they are set to no-decrypt or forward proxy.



Additional Information


This issue was observed after upgrading the PAN OS to 10.2.9-h10,10.2.14, 11.1.7, 11.1.8, 12.1.0, 11.2.5, 11.2.8, 10.2.7-h20, 10.2.11-h8, 11.1.2-h17, 11.1.4-h12, 11.2.4-h4, 11.1.6-h1, 10.2.7-h22, 10.2.9-h20, 10.2.10-h13, 10.2.8-h20, 10.2.11-h11, 10.2.12-h5, 10.2.13-h3 



Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA1Ki000000TNY9KAO&lang=en_US&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail