How to mitigate High DP CPU issue due to High Application Usage

• Palo Alto Firewall
• DP CPU
• Application Usage

1. Identify which ports, source IP and destination IP this application uses. 
   1. Go to your FW UI Monitor > Logs > Traffic.
   2. Filter according to: (app eq <name of application>)    
2. Check applipedia to learn more about the high usage application and about its standard ports. 
3. Check if the traffic increase or spike of this application is expected or not in your network:
   1. If expected then consider:
      1. Changing the network path of this traffic to bypass the FW.
      2. Changing the config or settings of the source of traffic to reduce the amount of this traffic.
      3. Changing the scheduled time of this traffic flow if possible to be outside peak hours.
   2. If unexpected then:
      1. Eliminate the traffic from your network.
      2. Block the traffic from reaching your FW.
      3. Use the FW Zone and DOS protection against the offending application traffic.
4. Check if your FW supports HW offload; If so, then check if offload is enabled.    
   1. Hardware offload is supported on the following firewalls: PA-3200 Series, PA-5200 Series, PA-7000 Series and PA-5450 firewall.
   2. To check if HW offload is enabled. 

      > show session info | match offload 

   3. To enable HW offload

      > set session offload yes

   4. The VM-series FWs running version 10.1.0 onwards can support intelligent traffic offload.
   5. A session that no longer sends packets to the CPU for L7 processing, is considered offloaded and the layer7 process shows as completed in the output of :

      > show session id <id#> 

5. Check if total sessions during the high DP CPU exceeded the supported sessions of your FW platform. If this large amount of sessions is expected consider upgrading your FW to higher capacity platforms. 
6. If traffic is legitimate, expected and trusted then:
   1. Add a security policy rule above the current rule it is hitting, but with less security profiles enabled in the Actions tab of the rule; or,
   2. Check the DSRI (disable server response inspection) checkbox in the security policy rule that allows only this application traffic this will disable the layer 7 inspection on the server-to-client direction of the traffic flow and should only be configured on rules where the traffic of the server is trusted; or,
   3. Consider using an application override for this traffic to reduce the load on the DP resources. Keep in mind that this means that this traffic would bypass the content inspection. Application override should be a last resort solution and only used as a temporary approach to reduce the CPU usage. 
7. In addition to the steps above and based on the application which is suspected to be causing the high DP CPU check the steps or notes listed below relative to each application: syslog, paloalto-logging-service, unknown-udp, meraki-cloud-controller, ssl, dns-base, crowdstrike, paloalto-updates, google-base, open-vpn, ping, vmware, ms-ds-smbv2, ms-ds-smbv3, ipsec-esp-udp, ms-update, netflow, rtsp, rtp-base, rtcp, web-browsing, panorama.

1. syslog
   1. If the syslog traffic is sourced from the FW then consider reducing this traffic by changing FW configuration. 
   2. If your FW logs are already being sent to Panorama, then consider sending logs to the syslog server from Panorama instead of directly from Firewall.

Note1: By looking at the start of the session in the traffic logs, check if the syslog session was a very long lived one (maybe days or even weeks) and then recently closed, this might then show a spike in the ACC for the syslog application usage then would be expected and shouldn't be alarming.
Note2: Denying syslog UDP traffic can cause a High DP resource usage . It is best to either allow traffic or route that traffic to bypass FW.

2. paloalto-logging-service
   1. Check if the FW or Panorama source of this traffic are properly configured and consider reducing the amount of logs sent to Cortex Data Lake (CDL) based on the severity of the log, the type of log and which security policy rule(s) logs.
3. unknown-udp
   1. Create custom application(s) for the application(s) unrecognized by the FW and identified as unknown-udp to allow the legitimate apps, deny the unwanted apps, and depending on the nature and usage of the app(s) check if you should consider applying an application override to them .
   2. If an application was previously recognized by the FW and stopped getting recognized after a content update, then revert back the content update and open a support ticket to report and address this problem.
4. meraki-cloud-controller
   1. Check if the meraki-cloud-controller application traffic is trusted in your network and if then you can consider applying an application override to it.
5. ssl
   1. The SSL traffic, if not decrypted, is offloaded and thus shouldn’t be the main contributor of the high usage of dataplane resources if the total number of sessions received on the FW are within the limit of what the platform supports. In that case, it is recommended to investigate the other high usage applications.
   2. If SSL traffic is getting decrypted on the firewall then during the time of high-dp CPU check if the value in the output of :

      > show session all filter ssl-decrypt yes count yes 

      is at any point exceeding Max concurrent decryption sessions of the platform and if so consider either changing your firewall config to reduce the amount of decrypted traffic by creating a decryption exclusion rule for the traffic exempt from decryption or plan to upgrade your FW to a higher capacity platform.
6. dns-base
   1. If the increase or spike of the dns-base traffic is found to be malicious or unexpected in your network then understand the top dns-based attacks your organization should know about and the ways to mitigate them.
7. crowdstrike
   1. Check if the crowdstrike application traffic is trusted in your network and if then you can consider applying an application override to it.
8. paloalto-updates
   1. Schedule the updates during non-peak hours.
   2. If considering the use of application override then base its rules on official Palo Alto Networks update servers. 
9. google-base
   1. Check solution 5 above. Otherwise if the total sessions is within the supported limit then consider to investigate other high usage applications.
10. open-vpn
    1. The open-VPN, if not decrypted is offloaded and is so should not be the main contributor to high FW resource usage. In a scenario where the open-vpn server is sitting behind the firewall, disable SSL inbound decryption for these connections at the firewall, and let the open-vpn server do the decryption process alone; after open-vpn server has decrypted the traffic, forward clear text traffic to the firewall for inspection.
11. ping
    1. There are at least four areas in the Zone Protection that can specifically address the ICMP issue:
       1. Flood Protection from ICMP / ICMPv6.
       2. Packet Based attack protection from ICMP / ICMPv6.
       3. Configure DoS Protection.
       4. Of all the solutions presented here, Packet Buffer Protection is the easiest to implement among the three.
12. vmware
    1. If traffic is expected then consider the solution presented in 3.a above.
13. ms-ds-smbv2
    1. This application if received with high amount by the FW can potentially highly use its resources consider configuring an application override for this traffic.
    2. Check if SMB is using file compression by filtering the traffic according to that application under Monitor > Packet Capture then issuing below command from CLI:

       > show counter global filter packet-filter yes delta yes | match zip

       Based on the result of the counters, you should be able to conclude if SMB traffic is being compressed and, if yes, disable it on the SMB side. This should reduce the CPU cycles for SMB.
14. ms-ds-smbv3
    1. Palo Alto Networks recommends disabling SMB multichannel splitting of files through the Windows PowerShell for maximum protection and inspection of files.
    2. If still seeing High DP CPU after step n. A then use the same approach as the one listed for ms-ds-smbv2 above.
15. ipsec-esp-udp
    1. Troubleshoot High DP CPU Because of IPsec Traffic to check if it's the cause behind the high DP CPU (too much encapsulation/decapsulation, tunnel operations, etc.):
       Be aware that configuring lower-strength cipher and/or encryption algorithm on the VPN Tunnel initiated or terminated on the FW can reduce FW's processing load in some cases.
       For high amount of IPsec traffic passing through the firewall causing High DP CPU use How to Troubleshoot High Dataplane CPU.
    2. Traffic sessions marked as the application 'ipsec-esp-udp' can also be  your users' GlobalProtect VPN sessions (port 4501) in that case make sure that your firewall configuration doesn't exceed the maximum Number of GlobalProtect VPN Tunnels supported on Firewall and follow the steps that can be taken to increase GlobalProtect VPN performance due to increased number of connections (leverage split tunnel feature, don't tunnel video traffic, remove idle users, etc.) which will reduce the amount of data plane resources usage of these GlobalProtect VPN users.
       If needed to troubleshoot GlobalProtect further refer to the resource list.
16. ms-update
    1. Check which possible reason is behind the increase of the ms-update traffic:
       1. A Microsoft Windows update was scheduled to be at the same time for multiple Windows devices in your network.
       2. A large Group Policy got pushed simultaneously to many Windows devices at the same time by your company's IT department. 
    2. Apply the solution based on the reason found:
       1. Turn off Automatic Updates for domain machines, and manage Windows updates via Group Policy instead, choosing when to push them out selectively.
       2. Schedule the Windows updates to happen at a less-impactful time during non-peak hours.
       3. Set a staggered Windows update schedule so the Windows devices in your network do not perform their updates at the same time.
17. netflow
    1. Whether the FW is the receiver or the sender of the netflow traffic, verify the FW configuration, its security policy allowing this traffic and how the netflow traffic counters are incrementing when received or sent on the FW dataplane; use CLI commands:

       > debug log-receiver netflow statistics
       > debug dataplane netflow statistics
       > show counter global | match netflow

    2. Ensure that this FW is not receiving or sending Netflow records at an abnormal or excessive rate as that could indicate a routing loop or other configuration problems in the FW, the sender or the network. Netflow records are processed on the FW DP which means that a high rate of this traffic if either received or sent can impact FW DP CPU.
18. rtsp
    1. The Real Time Streaming Protocol is an application-level protocol for control over the delivery of data with real-time properties. Consider the use of application override for this traffic only if the Application Layer Gateway ALG capability is not needed to be performed by the firewall. Otherwise you might risk breaking the VOIP traffic. Other applications that use ALG are listed here.
19. rtp-base
    1. This traffic contains audio and video packets, it is possible to apply the application override to it if an improvement in performance is needed.
20. rtcp
    1. This traffic contains audio and video packets, it is possible to apply the application override to it if an improvement in performance is needed.
21. web-browsing
    1. Consider disabling SSL decryption for the web traffic that can be exempt from decryption as that will decrease the amount of traffic classified as ‘web-browsing’ (instead, it will remain as ‘ssl’) and would be offloaded by the FW. Note that generally the FW spends more resources (CPU) on inspecting ‘web-browsing’ traffic than on ‘ssl’ traffic.
22. panorama
    1. If a large amount of panorama traffic is unexpected then check if there was any update or configuration change on Panorama, log collector or firewall managed by Panorama that has caused it and if that config change needs to be reverted or adjusted to reduce this amount of traffic.