How to identify suspect traffic causing latency by monitoring interface/sub-interface global counters

Below are a few helpful procedures to identify suspect traffic or application causing traffic latency on the firewall by isolating which ingress interface the traffic is being received on. 

• Palo Alto Firewall
• Any PAN-OS
• Filtering feature on Packet Capture
• Global Counter with packet-filter option (packet-filter yes)

1. Collect and review global counter name/description reporting high or concerning 'rate' (2nd column).
2. Repeat a few times to see the consistency of the suspect counter reporting a high rate.
3. For example, below 'pkt_outstanding' counter rate is extremely high and concerning in this case. 

> show counter global filter delta yes 

Global counters:
Elapsed time since last sampling: 17.145 seconds

name                                   value     rate severity  category  aspect    description
--------------------------------------------------------------------------------
pkt_outstanding                       589466    34387 info      packet    pktproc   Outstanding packet to be transmitted
pkt_alloc                              74188     4327 info      packet    resource  Packets allocated
session_allocated                        729       42 info      session   resource  Sessions allocated
. . .

4. Isolate which interface/sub-interface is processing the suspect traffic with high 'pkt_outstanding' rate by configuring a packet capture filter on a single interface one at a time
5. This is done by GUI: Monitor > Packet Capture > Configure Filtering > Manage FIlters > Add > etc
6. Monitoring the global counter with the filtered interface. (Note: Using a 'Pre-Parse Match' option is recommended to allow the firewall to monitor packets that do not reach the firewall filtering stage)
7. If the high 'pkt_outstanding' rate is not reported on ethernet1/1, then try other suspect interface and continue to monitor for the high 'pkt_outstanding' rate on global counters by using the 'packet-filter yes' option, etc.

> debug dataplane packet-diag set filter index 1 match ingress-interface ethernet1/1 non-ip include

> show counter global filter delta yes packet-filter yes    (repeat this command a few times to obtain a more reliable/consistent counter)

8. Identify which suspect application (active session) is processing high throughput traffic (larger than 1 GB) by filtering the isolated ingress interface/zone 

> show session all filter ingress-interface ethernet1/1 min-kb 1000000

> show session all filter from L3-Trust min-kb 1000000

9. If  assistance is needed, contact Support.