Packet Buffer Utilization stuck at High Rates

20039

Created On 06/22/19 14:25 PM - Last Modified 06/17/20 01:17 AM

Hardware

8.1

8.0

7.1

PAN-OS

Symptom

Constant high rates Packet Buffer utilization showing in Packet Buffer Graph:




DP-Monitor.log:
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet buffer (average):
2019-04-22 18:45:20   : 91  91  91  91  91  91  91  91  91  91  91  91  91  91  91 
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet buffer (maximum):
2019-04-22 18:45:20   : 91  91  91  91  91  91  91  91  92  91  91  91  91  91  91 
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet descriptor (average):
2019-04-22 18:45:20   : 28  28  28  28  28  28  28  28  28  28  28  28  27  27  28 
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet descriptor (maximum):
2019-04-22 18:45:20   : 28  28  28  28  28  28  28  28  28  28  28  28  28  28  28 
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet descriptor (on-chip) (average):
2019-04-22 18:45:20   :  1   1   1   1   1   1   1   1   1   1   1   1   1   1   1 
2019-04-22 18:45:20   :
2019-04-22 18:45:20   :packet descriptor (on-chip) (maximum):
2019-04-22 18:45:20   :  1   2   1   1   1   1   2   1   1   1   1   2   1   1   2 

2019-04-22 18:55:20   :packet buffer (average):
2019-04-22 18:55:20   : 91  91  91  91  91  91  91  91  91  91  91  91  91  91  91 
2019-04-22 18:55:20   :
2019-04-22 18:55:20   :packet buffer (maximum):
2019-04-22 18:55:20   : 91  91  91  91  91  91  91  91  91  91  91  91  91  91  91 
2019-04-22 18:55:20   :
2019-04-22 18:55:20   :packet descriptor (average):
2019-04-22 18:55:20   : 28  28  28  28  28  28  28  28  28  28  28  28  28  28  28 
2019-04-22 18:55:20   :
2019-04-22 18:55:20   :packet descriptor (maximum):
2019-04-22 18:55:20   : 29  28  28  29  28  28  28  28  28  28  28  28  28  28  28 
2019-04-22 18:55:20   :
2019-04-22 18:55:20   :packet descriptor (on-chip) (average):
2019-04-22 18:55:20   :  1   1   1   1   1   1   1   1   1   1   1   1   1   1   1 
2019-04-22 18:55:20   :
2019-04-22 18:55:20   :packet descriptor (on-chip) (maximum):
2019-04-22 18:55:20   :  1   1   1   2   1   1   2   2   2   2   1   2   1   1   1

Constant BGP and LACP flapping showing in System Log:

BGP:
====
2019/04/22 18:59:59 info routing Produc routed- 34 BGP peer session enters established state. peer IP: 10.203.5.118.
2019/04/22 18:59:57 info routing Produc routed- 18 BGP peer session enters established state. peer IP: 10.240.26.57.
2019/04/22 18:59:54 high routing Produc routed- 18 BGP peer session left established state. peer IP: 10.240.24.66.
2019/04/22 18:59:51 high routing Produc routed- 18 BGP peer session left established state. peer IP: 10.240.28.62.
2019/04/22 18:59:47 high routing Produc routed- 18 BGP peer session left established state. peer IP: 10.240.25.154.
2019/04/22 18:59:45 info routing Produc routed- 18 BGP peer session enters established state. peer IP: 10.240.9.234.
LACP:
====
2019/04/22 18:59:46 critical lacp ethern lacp-up 0 LACP interface ethernet1/23 moved into AE-group ae2.
2019/04/22 18:59:45 critical lacp ethern nego-fa 0 LACP interface ethernet1/23 moved out of AE-group ae2. Selection state Selected
2019/04/22 18:59:29 critical lacp ethern lacp-up 0 LACP interface ethernet1/24 moved into AE-group ae2.
2019/04/22 18:59:28 critical lacp ethern nego-fa 0 LACP interface ethernet1/24 moved out of AE-group ae2. Selection state Selected
2019/04/22 18:59:20 critical lacp ethern lacp-up 0 LACP interface ethernet1/24 moved into AE-group ae2.
2019/04/22 18:59:19 critical lacp ethern nego-fa 0 LACP interface ethernet1/24 moved out of AE-group ae2. Selection state Selected
2019/04/22 18:59:10 critical lacp ethern lacp-up 0 LACP interface ethernet1/23 moved into AE-group ae2.
2019/04/22 18:59:09 critical lacp ethern nego-fa 0 LACP interface ethernet1/23 moved out of AE-group ae2. Selection state Selected
2019/04/22 18:59:05 critical lacp ethern lacp-up 0 LACP interface ethernet1/23 moved into AE-group ae2.

Packet Buffer Depletion happen showing in DP-Monitor log

2019-04-22 18:45:20   :Hardware Pools
2019-04-22 18:45:20   :[ 0] Packet Buffers            :     5177/57344    0x8000000416ab0500
2019-04-22 18:45:20   :[ 1] Work Queue Entries        :   164451/229376   0x8000000414eb0500
2019-04-22 18:45:20   :[ 2] Output Buffers            :    21750/21760    0x800000041dabed80
2019-04-22 18:45:20   :[ 3] DFA Result                :     4096/4096     0x800000041f200000
2019-04-22 18:45:20   :[ 4] Timer Buffers             :     4093/4096     0x800000041f600000
2019-04-22 18:45:20   :[ 5] LWM Pool                  :     1024/1024     0x80000000fff00000
2019-04-22 18:45:20   :[ 6] ZIP Commands              :     1023/1024     0x800000041f000000
2019-04-22 18:45:20   :[ 7] Dma Cmd Buffers           :     1019/1024     0x8000000414dae880

2019-04-22 18:55:20   :Hardware Pools
2019-04-22 18:55:20   :[ 0] Packet Buffers            :     5161/57344    0x8000000416ab0500
2019-04-22 18:55:20   :[ 1] Work Queue Entries        :   164170/229376   0x8000000414eb0500
2019-04-22 18:55:20   :[ 2] Output Buffers            :    21750/21760    0x800000041dabed80
2019-04-22 18:55:20   :[ 3] DFA Result                :     4096/4096     0x800000041f200000
2019-04-22 18:55:20   :[ 4] Timer Buffers             :     4093/4096     0x800000041f600000
2019-04-22 18:55:20   :[ 5] LWM Pool                  :     1024/1024     0x80000000fff00000
2019-04-22 18:55:20   :[ 6] ZIP Commands              :     1023/1024     0x800000041f000000
2019-04-22 18:55:20   :[ 7] Dma Cmd Buffers           :     1019/1024     0x8000000414dae880

Dataplane CPU utilization at low rates

Environment

Platform: PA-5250
PAN-OS: 8.0.12
Customer encountered BGP and LACP flapping

Cause

A known issue which was reported in PAN-104116

Fixed in 7.1.21, ~~8.0.13~~, & 8.1.4* -- "Fixed an issue where a hardware packet buffer leak caused firewall performance to degrade."

Resolution

Upgrade to either 7.1.21, ~~8.0.13~~, 8.1.4 or later versions to resolve known issue.

Additional Information