Troubleshooting DoS Attacks

A Denial-of-Service (DoS) attack is an attack meant to shut down a machine or network resource, making it inaccessible to its intended users. Please read more about DoS in What is a denial of service attack?

 This article gives some insight into how to troubleshoot a few situations due to firewall DoS attacks. This article assumes you are aware of DoS/ Zone Protection Profiles and checking resource usage:

Understanding DoS Protection

How to Troubleshoot High Dataplane CPU

1. Session Table Full

Detected by checking 

show session info | match table

This can be seen in cases where the DoS traffic is allowed by the firewall policies. If the attacker can find open IPs and ports using IP Sweep or Port Scanning techniques, it can launch a flood attack. In the case of TCP, a response from the victim leads to a session getting installed on the firewall for 3600 seconds. Further on, more such SYN packets will lead to session table full, thereby causing legitimate traffic to be affected.

Please note that in these cases ,the rate of flood attack may or may not be high.

Allowing asymmetric bypass or disabling tcp-reject-non-syn can worsen the situation, as it will lead to any TCP packet to install a session on the firewall if the policy and routing allows it.

How to troubleshoot session table full

• In these kind of cases, the best way would be take a dump of the session table and skim through it. In most cases a particular source IP or a targetted IP will be standing out. Use command

  show session all

• Check if the identified target IP and port , or the source of the traffic is valid and should be allowed through the firewall. If yes, can we restrict the firewall rules.
• Mostly in these cases mitigation is the best way. You should implement DoS Protection/ Zone Protection Profiles to mitigate these cases. SYN Cookie mode in SYN Flood Protection can be very useful in dealing with these situations, as it will prevent firewall from installing a session and will send a SYN/ACK packet to the attacker. Unless the attacker responds with a valid ACK, the session will not be established.
• The threshold of the SYN Flood Protection Mechanism should be tuned in to specific requirements after analyzing the normal connection rate to the target IP.

2. Packet Buffer/ Packet Descriptors Full

Detected by checking 

show running resource-monitor
show running resource-monitor ingress-backlogs 

This can be seen in DoS attacks with high packet/ data rate. Can be because of high rate of new connections per second or high packet rate on existing sessions. Even if the connections are getting denied, it would not prevent the buffer to be full as the packets have to be buffered and before sending to the dataplane for processing.

How to troubleshoot packet buffer/packet descriptors full:

• Check ACC tabs and try to use custom filters to see if there is a particular traffic that is exceptionally higher than the other traffic. However ACC tab does not provide good intel against ongoing attack. Reason being, that it take its feed from the traffic summary, traffic log databases and appstats database.

  • If the attack is going on via an ongoing session, it would not have generated a traffic log yet.

  • If the attack comprises of new connections,  ACC tab might not be able to show sessions which are anyway getting failed.

• Check for any live sessions with more data than usual. Use min-kb option in the session filter as under:

  admin@PA-200> show session all filter min-kb 5000
  
  --------------------------------------------------------------------------------
  
  ID          Application    State   Type Flag  Src[Sport]/Zone/Proto (translated IP[Port])
  
  Vsys                                          Dst[Dport]/Zone (translated IP[Port])
  
  --------------------------------------------------------------------------------
  
  10216        pcoip          ACTIVE  FLOW  NS   10.1.1.11[50002]/Trust/17  (10.129.15.24[61158])
  
  vsys1                                          10.101.41.211[4172]/Untrust  (10.101.41.211[4172])

   

• The above is just an example, you can use the value of min-kb as anywhere between 1-1048576, i.e. almost up to 1G. This can provide an insight into sessions with top rate of traffic. Look for anything suspicious.

• You can use the following to find sessions which are overusing the On-Chip Packet Descriptor: 

  Identify Sessions That Use an Excessive Percentage of the On-Chip Packet Descriptor

• Check the live traffic rate on interfaces and find out which interface is receiving excessive traffic. Use CLI

  show system state browser

  followed by shift+L as mentioned in How to Check Throughput of Interfaces. 

  • Check for all relevant ports as per your configuration.

  • You should be focusing on the rx-bytes/s  or rx-unicast/s or rx-multicast/s. This is the rate being shown in the second column when you enable tracking using ‘Y’ and ‘U’.

• After you identify the port on which the attack traffic is being received, you will need to take a short packet dump of the traffic being received on that port.

  • It is recommended to do this on a connected switch by port mirroring the traffic as packet captures might be more resource intensive.

  • However if no other option is available, enable the captures on the Palo Alto Networks firewall with filter as ingress-interface as identified above and run the captures for 10-15 seconds.

• Stop the captures and open with Wireshark. Under Wireshark look under Statistics -> Protocol Hierarchy or Conversations. Check the conversation on IP layer, UDP layer, TCP layer and check for any traffic which is having a high packet count.

  • This can help identify and isolate the attacker (trusted or untrusted). Mostly for trusted source, you can track the affected host and take actions.

  • For an untrusted source, you can ask for help from your ISP to help prevent the source of attack.