How to mitigate High DP CPU issue due to an increase in an interface counter

How to mitigate High DP CPU issue due to an increase in an interface counter

12970
Created On 02/16/23 17:55 PM - Last Modified 08/23/23 22:06 PM


Objective


To mitigate High DP CPU issue due to to an increase in an interface counter

Environment


  • Palo Alto Firewall
  • DP CPU
  • Interface Bytes/Packets Received
  • Interface Bytes/Packets Transmitted
  • Interface Receive Errors
  • Interface Packets Dropped


Procedure


  1. Identify which interface counter increased around the time of the high DP CPU detection:
    1. Rely on the information provided by the tool that helped determine the problem.
    2. Issue the following CLI multiple times during the high DP CPU issue and track which counter increases at a high rate
      show counter interface all
    3. Check the throughput of the interfaces during the high DP CPU issue using the CLI:
      show system state browser
      and press Shift+L then Enter.
    4. Use the ACC tab and set a global filter based on the suspected interface to be the source of the traffic increase. If needing to look at historical data then set the time to be the one of the high DP CPU's detection.

      2. Apply the remediation to the issue:

  1. If the increase in the number of Bytes or Packets Received and you have determined the source of the traffic using the ACC tab or traffic logs you need to:
    1. If the increase is expected in your network then consider:
      1. Changing the network path of this traffic to bypass the FW.
      2. Changing the config or settings of the source of traffic to reduce the amount of this traffic.
      3. Changing the scheduled time of this traffic flow if possible to be outside peak hours.
    2. If the increase is unexpected in your network then consider:
      1. Eliminating the traffic from your network.
      2. Blocking the traffic from reaching your FW.
      3. Using the FW Zone and DOS protection against the offending traffic.
    3. Protect the Firewall buffer resources by enabling the packet buffer protection.
    4. Check if your FW supports HW offload; If so, then check if offload is enabled if not then enable it.
      set session offload yes
    5. Check if total sessions during the high DP CPU exceeded the supported sessions of your FW platform. If this large amount of sessions is expected consider upgrading your FW to higher capacity platforms. 
      show session info
  2. If the increase in the number of Bytes or Packets Transmitted then based on your investigation using the ACC tab or traffic logs check if if those packets are being sent via a tunnel meaning encapsulated, belong to an encrypted traffic that is being decrypted by the FW or QOS has been applied to that traffic and understand that this is adding a load on the FW's Dataplane CPU so consider applying the appropriate remediation:
    1. If the traffic transmitted is being sent through a tunnel then consider reducing the amount of this traffic.
    2. If the traffic transmitted belongs to a decrypted session then exclude that traffic from decryption.
    3. If QOS has been applied to that traffic then consider disabling QOS on that traffic.
  3. If the increase is in the number of Packets Dropped on the interface then check if the drop is HW_PACKETS_DROPPED or CPU_PACKETS_DROPPED then refer to the information listed in the Logical interface packet drop counter explanation and apply the proper remediation.
  4. If the increase is in the number of Receive Errors on the interface then check if the drop is HW_RECEIVE_ERRORS or CPU_RECEIVE_ERRORS then refer to the information listed in the Difference Between Receive Errors for Hardware and Logical Interface Counters and apply the proper remediation.


Additional Information


HW in HW_PACKETS_DROPPED indicates that the "packets dropped" counter is under the "Hardware interface counters read from CPU" in the output of 
show counter interface all
CPU in CPU_PACKETS_DROPPED indicates that the "packets dropped" counter is under the "Logical interface counters read from CPU" in the output of 
show counter interface all

 


Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000kGzmCAE&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Choose Language