Routing Loop Causing High Dataplane CPU

Routing Loop Causing High Dataplane CPU

18037
Created On 04/30/19 00:59 AM - Last Modified 05/02/19 19:25 PM


Symptom


  • Routing loop in the network where the same packet is bounced between two devices
  • Global counter "flow_fwd_l3_ttl_zero" is high
admin@PA> show counter global filter delta yes severity warn

Global counters:
Elapsed time since last sampling: 8.613 seconds

name                  value     rate 	severity   category   aspect    description
-----------------------------------------------------------------------------------
...
:flow_fwd_l3_ttl_zero 11865807 	52 		drop 	   flow 	  forward 	Packets dropped: IP TTL reaches zero
admin@PA> show counter global filter delta yes severity warn
...
:flow_fwd_l3_ttl_zero 25 		62 	 	drop 	   flow 	  forward 	Packets dropped: IP TTL reaches zero
admin@PA> show counter global filter delta yes severity warn
:flow_fwd_l3_ttl_zero 11897747 	53   	drop 	   flow 	  forward 	Packets dropped: IP TTL reaches zero
 
  • Dataplane (DP) resources are depleted and DP CPU constantly throttling with high CPU % average 
From dp-monitor.log:
2019-04-28 17:35:11.058 -0700  --- cpu
Last 180 seconds
Avg (%)    Max (%)
95          100
 
  •  Dataplane packet descriptor and packet buffer high average value
From dp-monitor.log:
packet descriptor (on-chip) (average):
 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 --
 


Environment


  • Any firewall platform
  • PAN-OS
  • Routing


Cause


Due to routing loop in the network

Resolution


  1. Enable global counter logging to identify where the source of the loop is coming from by running the following commands from the CLI:
  • Set the packet-diag filter for any source and any destination IP address,
    debug dataplane packet-diag set filter match source 0.0.0.0 destination 0.0.0.0
    Turn on the packet-diag filter,
    debug dataplane packet-diag set filter on
    
  • Turn on the counter
    debug dataplane packet-diag set log counter flow_fwd_l3_ttl_zero
  1. View the system logs and identify the IP addresses and/or subnets of the traffic getting dropped by the firewall
  2. Create security policy and block the IP addresses and/or subnets
  3. Verify the loop has been resolved by running the following commands from the CLI:
  • Check if the DP CPU is dropping
    show running resource-monitor second last 60 
  • Check if the counter "flow_fwd_l3_ttl_zero" is no longer showing 
    show counter global filter delta yes severity warn 
  1. After verifying, clear the packet-diag settings
  • debug dataplane packet-diag clear all
  • debug dataplane packet-diag clear log log


Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/kcSArticleDetail?id=kA10g000000PLrn&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FkcSArticleDetail