Buffer depletion on PA-VM deployed in Nutanix virtualization platform

Buffer depletion on PA-VM deployed in Nutanix virtualization platform

8433
Created On 03/18/20 16:30 PM - Last Modified 07/30/20 00:49 AM


Symptom


Packet Buffer utilization is high and all user traffic passing through firewall is impacted
 
  • Review output of below CLI command
> debug dataplane pool statistics
 Pow Atomic Memory Pools
[ 0] Work Queue Entries        :    98276/98304    0xe02965bf80
[ 1] Packet Buffers            :    44/24576    0xe02ae9bf80
 
  • Review dp-monitor Logs 
dp-monitor Logs:
dp-monitor.log.4 2019-09-23 18:36:51   :[ 1] Packet Buffers :    18358/24576    0xe02ae9bf80
dp-monitor.log.4 2019-09-23 18:46:50   :[ 1] Packet Buffers :    18304/24576    0xe02ae9bf80
dp-monitor.log.4 2019-09-23 18:56:51   :[ 1] Packet Buffers :    18306/24576    0xe02ae9bf80

dp-monitor.log 2019-09-26 14:17:16   :[ 1] Packet Buffers :       44/24576    0xe02ae9bf80
dp-monitor.log 2019-09-26 14:27:18   :[ 1] Packet Buffers :       42/24576    0xe02ae9bf80
dp-monitor.log 2019-09-26 14:37:20   :[ 1] Packet Buffers :       28/24576    0xe02ae9bf80
dp-monitor.log 2019-09-26 14:47:18   :[ 1] Packet Buffers :       24/24576    0xe02ae9bf80
dp-monitor.log 2019-09-26 14:57:17   :[ 1] Packet Buffers :       17/24576    0xe02ae9bf80
dp-monitor.log 2019-09-26 15:07:15   :[ 1] Packet Buffers :        5/24576    0xe02ae9bf80
 
  • Run below CLI command to verify if the below counters are incrementing
    
    
    > show counter global filter delta yes | match clone
    pkt_swbuf_clone                       251453        0 info      packet    pktproc   Packets replicated using software buffer
    flow_tunnel_ipsec_esp_encap_swbuf     142403        0 info      flow      tunnel    Packet encapped: IPSec ESP encrypt clear 
    text pkts with cloned swbuf


Environment


  • Platform: PAN-OS
  • Deployment: VM-Series


Cause


  • Some clear text pkts might be software cloned pkts. Those will cause the encrypted the ESP pkts to leak as the no-free flag of the clear text pkt is copied to the allocated hw pkt (to encrypt).


Resolution


  1. Workaround: Disable software packet buffer cloning using the below CLI commands:
> configure
# set deviceconfig setting pow wqe-swbuf-ref no
# commit

This issue is addressed in PAN-OS 9.0.8 or later and 9.1 or later


Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/kcSArticleDetail?id=kA10g000000PP8s&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FkcSArticleDetail

Choose Language