Packet Buffer Utilization stuck at High Rates

Packet Buffer Utilization stuck at High Rates

18773
Created On 06/22/19 14:25 PM - Last Modified 06/17/20 01:17 AM


Symptom


  • Constant high rates Packet Buffer utilization showing in Packet Buffer Graph:

User-added image

DP-Monitor.log:
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet buffer (average):
2019-04-22 18:45:20   : 91  91  91  91  91  91  91  91  91  91  91  91  91  91  91 
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet buffer (maximum):
2019-04-22 18:45:20   : 91  91  91  91  91  91  91  91  92  91  91  91  91  91  91 
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet descriptor (average):
2019-04-22 18:45:20   : 28  28  28  28  28  28  28  28  28  28  28  28  27  27  28 
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet descriptor (maximum):
2019-04-22 18:45:20   : 28  28  28  28  28  28  28  28  28  28  28  28  28  28  28 
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet descriptor (on-chip) (average):
2019-04-22 18:45:20   :  1   1   1   1   1   1   1   1   1   1   1   1   1   1   1 
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet descriptor (on-chip) (maximum):
2019-04-22 18:45:20   :  1   2   1   1   1   1   2   1   1   1   1   2   1   1   2 

2019-04-22 18:55:20   :packet buffer (average):
2019-04-22 18:55:20   : 91  91  91  91  91  91  91  91  91  91  91  91  91  91  91 
2019-04-22 18:55:20   :
2019-04-22 18:55:20   :packet buffer (maximum):
2019-04-22 18:55:20   : 91  91  91  91  91  91  91  91  91  91  91  91  91  91  91 
2019-04-22 18:55:20   :
2019-04-22 18:55:20   :packet descriptor (average):
2019-04-22 18:55:20   : 28  28  28  28  28  28  28  28  28  28  28  28  28  28  28 
2019-04-22 18:55:20   :
2019-04-22 18:55:20   :packet descriptor (maximum):
2019-04-22 18:55:20   : 29  28  28  29  28  28  28  28  28  28  28  28  28  28  28 
2019-04-22 18:55:20   :
2019-04-22 18:55:20   :packet descriptor (on-chip) (average):
2019-04-22 18:55:20   :  1   1   1   1   1   1   1   1   1   1   1   1   1   1   1 
2019-04-22 18:55:20   :
2019-04-22 18:55:20   :packet descriptor (on-chip) (maximum):
2019-04-22 18:55:20   :  1   1   1   2   1   1   2   2   2   2   1   2   1   1   1 
  • Constant BGP and LACP flapping showing in System Log:
BGP:
====
2019/04/22 18:59:59 info     routing Produc routed- 34 BGP peer session enters established state. peer IP: 10.203.5.118.
2019/04/22 18:59:57 info     routing Produc routed- 18 BGP peer session enters established state. peer IP: 10.240.26.57.
2019/04/22 18:59:54 high     routing Produc routed- 18 BGP peer session left established state. peer IP: 10.240.24.66.
2019/04/22 18:59:51 high     routing Produc routed- 18 BGP peer session left established state. peer IP: 10.240.28.62.
2019/04/22 18:59:47 high     routing Produc routed- 18 BGP peer session left established state. peer IP: 10.240.25.154.
2019/04/22 18:59:45 info     routing Produc routed- 18 BGP peer session enters established state. peer IP: 10.240.9.234.
LACP:
====
2019/04/22 18:59:46 critical lacp    ethern lacp-up 0  LACP interface ethernet1/23 moved into AE-group ae2.
2019/04/22 18:59:45 critical lacp    ethern nego-fa 0  LACP interface ethernet1/23 moved out of AE-group ae2. Selection state Selected
2019/04/22 18:59:29 critical lacp    ethern lacp-up 0  LACP interface ethernet1/24 moved into AE-group ae2.
2019/04/22 18:59:28 critical lacp    ethern nego-fa 0  LACP interface ethernet1/24 moved out of AE-group ae2. Selection state Selected
2019/04/22 18:59:20 critical lacp    ethern lacp-up 0  LACP interface ethernet1/24 moved into AE-group ae2.
2019/04/22 18:59:19 critical lacp    ethern nego-fa 0  LACP interface ethernet1/24 moved out of AE-group ae2. Selection state Selected
2019/04/22 18:59:10 critical lacp    ethern lacp-up 0  LACP interface ethernet1/23 moved into AE-group ae2.
2019/04/22 18:59:09 critical lacp    ethern nego-fa 0  LACP interface ethernet1/23 moved out of AE-group ae2. Selection state Selected
2019/04/22 18:59:05 critical lacp    ethern lacp-up 0  LACP interface ethernet1/23 moved into AE-group ae2.
  • Packet Buffer Depletion happen showing in DP-Monitor log
2019-04-22 18:45:20   :Hardware Pools
2019-04-22 18:45:20   :[ 0] Packet Buffers            :     5177/57344    0x8000000416ab0500
2019-04-22 18:45:20   :[ 1] Work Queue Entries        :   164451/229376   0x8000000414eb0500
2019-04-22 18:45:20   :[ 2] Output Buffers            :    21750/21760    0x800000041dabed80
2019-04-22 18:45:20   :[ 3] DFA Result                :     4096/4096     0x800000041f200000
2019-04-22 18:45:20   :[ 4] Timer Buffers             :     4093/4096     0x800000041f600000
2019-04-22 18:45:20   :[ 5] LWM Pool                  :     1024/1024     0x80000000fff00000
2019-04-22 18:45:20   :[ 6] ZIP Commands              :     1023/1024     0x800000041f000000
2019-04-22 18:45:20   :[ 7] Dma Cmd Buffers           :     1019/1024     0x8000000414dae880

2019-04-22 18:55:20   :Hardware Pools
2019-04-22 18:55:20   :[ 0] Packet Buffers            :     5161/57344    0x8000000416ab0500
2019-04-22 18:55:20   :[ 1] Work Queue Entries        :   164170/229376   0x8000000414eb0500
2019-04-22 18:55:20   :[ 2] Output Buffers            :    21750/21760    0x800000041dabed80
2019-04-22 18:55:20   :[ 3] DFA Result                :     4096/4096     0x800000041f200000
2019-04-22 18:55:20   :[ 4] Timer Buffers             :     4093/4096     0x800000041f600000
2019-04-22 18:55:20   :[ 5] LWM Pool                  :     1024/1024     0x80000000fff00000
2019-04-22 18:55:20   :[ 6] ZIP Commands              :     1023/1024     0x800000041f000000
2019-04-22 18:55:20   :[ 7] Dma Cmd Buffers           :     1019/1024     0x8000000414dae880
  • Dataplane CPU utilization at low rates
User-added image


Environment


  • Platform: PA-5250
  • PAN-OS: 8.0.12
  • Customer encountered BGP and LACP flapping


Cause


A known issue which was reported in PAN-104116

Fixed in 7.1.21, 8.0.13, & 8.1.4* -- "Fixed an issue where a hardware packet buffer leak caused firewall performance to degrade."


Resolution


Upgrade to either 7.1.21, 8.0.13, 8.1.4 or later versions to resolve known issue.
 


Additional Information


NOTE:
PAN-OS 8.0.X is End of Life as of October 31, 2019


Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PMCb&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Choose Language