Buffer depletion on PA-VM deployed in Nutanix virtualization platform
7583
Created On 03/18/20 16:30 PM - Last Modified 07/30/20 00:49 AM
Symptom
Packet Buffer utilization is high and all user traffic passing through firewall is impacted
- Review output of below CLI command
> debug dataplane pool statistics
Pow Atomic Memory Pools
[ 0] Work Queue Entries : 98276/98304 0xe02965bf80
[ 1] Packet Buffers : 44/24576 0xe02ae9bf80
- Review dp-monitor Logs
dp-monitor Logs:
dp-monitor.log.4 2019-09-23 18:36:51 :[ 1] Packet Buffers : 18358/24576 0xe02ae9bf80
dp-monitor.log.4 2019-09-23 18:46:50 :[ 1] Packet Buffers : 18304/24576 0xe02ae9bf80
dp-monitor.log.4 2019-09-23 18:56:51 :[ 1] Packet Buffers : 18306/24576 0xe02ae9bf80
dp-monitor.log 2019-09-26 14:17:16 :[ 1] Packet Buffers : 44/24576 0xe02ae9bf80
dp-monitor.log 2019-09-26 14:27:18 :[ 1] Packet Buffers : 42/24576 0xe02ae9bf80
dp-monitor.log 2019-09-26 14:37:20 :[ 1] Packet Buffers : 28/24576 0xe02ae9bf80
dp-monitor.log 2019-09-26 14:47:18 :[ 1] Packet Buffers : 24/24576 0xe02ae9bf80
dp-monitor.log 2019-09-26 14:57:17 :[ 1] Packet Buffers : 17/24576 0xe02ae9bf80
dp-monitor.log 2019-09-26 15:07:15 :[ 1] Packet Buffers : 5/24576 0xe02ae9bf80
-
Run below CLI command to verify if the below counters are incrementing> show counter global filter delta yes | match clone pkt_swbuf_clone 251453 0 info packet pktproc Packets replicated using software buffer flow_tunnel_ipsec_esp_encap_swbuf 142403 0 info flow tunnel Packet encapped: IPSec ESP encrypt clear text pkts with cloned swbuf
Environment
- Platform: PAN-OS
- Deployment: VM-Series
Cause
- Some clear text pkts might be software cloned pkts. Those will cause the encrypted the ESP pkts to leak as the no-free flag of the clear text pkt is copied to the allocated hw pkt (to encrypt).
Resolution
- Workaround: Disable software packet buffer cloning using the below CLI commands:
> configure
# set deviceconfig setting pow wqe-swbuf-ref no
# commit
This issue is addressed in PAN-OS 9.0.8 or later and 9.1 or later