Routing Loop Causing High Dataplane CPU

• Routing loop in the network where the same packet is bounced between two devices
• Global counter "flow_fwd_l3_ttl_zero" is high

admin@PA> show counter global filter delta yes severity warn

Global counters:
Elapsed time since last sampling: 8.613 seconds

name                  value     rate 	severity   category   aspect    description
-----------------------------------------------------------------------------------
...
:flow_fwd_l3_ttl_zero 11865807 	52 		drop 	   flow 	  forward 	Packets dropped: IP TTL reaches zero

admin@PA> show counter global filter delta yes severity warn
...
:flow_fwd_l3_ttl_zero 25 		62 	 	drop 	   flow 	  forward 	Packets dropped: IP TTL reaches zero

admin@PA> show counter global filter delta yes severity warn
:flow_fwd_l3_ttl_zero 11897747 	53   	drop 	   flow 	  forward 	Packets dropped: IP TTL reaches zero

• Dataplane (DP) resources are depleted and DP CPU constantly throttling with high CPU % average 

2019-04-28 17:35:11.058 -0700  --- cpu
Last 180 seconds
Avg (%)    Max (%)
95          100

•  Dataplane packet descriptor and packet buffer high average value

packet descriptor (on-chip) (average):
 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 --

• In some cases, packet rate can also be very high for a particular interface that is looping. Refer to the article How to check throughput of interfaces.

• Any firewall platform
• PAN-OS
• Routing

1. Enable global counter logging to identify where the source of the loop is coming from by running the following commands from the CLI:

• Set the packet-diag filter for any source and any destination IP address,
  debug dataplane packet-diag set filter match source 0.0.0.0 destination 0.0.0.0
  Turn on the packet-diag filter,
  debug dataplane packet-diag set filter on
  

• Turn on the counter
  debug dataplane packet-diag set log counter flow_fwd_l3_ttl_zero

2. View the system logs and identify the IP addresses and/or subnets of the traffic getting dropped by the firewall
3. Create security policy and block the IP addresses and/or subnets
4. Verify the loop has been resolved by running the following commands from the CLI:

• Check if the DP CPU is dropping
  show running resource-monitor second last 60 

• Check if the counter "flow_fwd_l3_ttl_zero" is no longer showing 
  show counter global filter delta yes severity warn 

5. After verifying, clear the packet-diag settings

• debug dataplane packet-diag clear all
• debug dataplane packet-diag clear log log