After enabling Network Packet Broker, packet loop within broker cause high packet rates resulting high dataplane CPU
3005
Created On 01/10/23 01:28 AM - Last Modified 06/03/25 03:00 AM
Symptom
- Enabling network packet broker, DP CPU goes high due to very high packet rate. Issue happens intermittently
- The increase in packet rate is due to a packet looping in the broker session.
- Once the broker session is cleared, packet rate as well as DP CPU comes down.
- Discard due to proxy decrypt failure.
- From client, "Fatal, Certificate unknown" received*
CMA-FW-PA01(active)> show session id 1349898 ...... session terminated on host : False session traverses tunnel : False session terminate tunnel : False captive portal session : False ingress interface : ae4.221 egress interface : ae5.533 session QoS rule : N/A (class 4) tracker stage firewall : proxy decrypt failure tracker stage l7proc : ctd proc changed end-reason : decrypt-error .....
- Due to the packet looping, the broker session can get as big as 1G*
CMA-FW-PA01(active)> show session all filter ingress-interface ethernet1/6 min-kb 1048576 -------------------------------------------------------------------------------- ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port]) Vsys Dst[Dport]/Zone (translated IP[Port]) -------------------------------------------------------------------------------- 1673649 none ACTIVE FLOW 172.25.171.62[51410]/Packet-Broker/6 (172.25.171.62[51410]) vsys1 172.25.95.250[8080]/Packet-Broker (172.25.95.250[8080]) {noformat}
- Only when the session is killed, packet rate comes down. With multiple similar sessions, packet rate increases (seen 800000/s in customer environment) which is when DP CPU increases.
FW-PA01(active)> show session info | match "Packet rate:" Packet rate: 53844/s FW-PA01(active)> show session info | match "Packet rate:" Packet rate: 68708/s FW-PA01(active)> show session info | match "Packet rate:" Packet rate: 83558/s FW-PA01(active)> show session info | match "Packet rate:" Packet rate: 101342/s
- Packet Buffer is used up, which result in high DP cpu.
admin@pa-850> show running resource-monitor second last 60 Resource monitoring sampling data (per second): CPU load sampling by group: flow_lookup : 100% flow_fastpath : 100% flow_slowpath : 100% flow_forwarding : 100% flow_mgmt : 100% flow_ctrl : 100% nac_result : 100% flow_np : 100% dfa_result : 100% module_internal : 100% aho_result : 100% zip_result : 100% pktlog_forwarding : 100% lwm : 0% flow_host : 100% fpga_result : 0% CPU load (%) during last 60 seconds: core 0 1 2 3 4 5 6 7 * 100 100 100 100 100 * * * 100 100 100 100 100 * * * 100 100 100 100 100 * * * 100 100 100 100 100 * * * 100 100 100 100 100 * * * 100 100 100 100 100 * * * 100 100 100 100 100 * * * 100 100 100 100 100 * * * 100 100 100 100 100 * * .....output omitted.... packet buffer: 56 67 79 98 98 99 98 94 95 95 99 94 99 99 90 99 100 100 99 99 99 93 52 28 8 7 2 1 1 1 0 4 12 18 24 22 26 18 2 0 2 0 0 11 22 32 51 57 74 74 52 57 56 61 61 67 83 77 72 75 ....
Environment
- Affect any firewalls connect to network packet broker (NPB)
- Any PANOS 10.1.7 or below
Cause
Software Issue.
Resolution
- The issue is fixed PANOS 10.1.8 and 10.2.4 under PAN-194441. Upgrade will resolve the issue.
- PAN-194441- Fixed an issue where the dataplane CPU usage was higher than expected due to packet looping in the broker session when the network packet broker was enabled.