Firewall dropping RST from Client after Server's Challenge-ACK

• Firewall dropping RST from Client after Server's "Challenge ACK" preventing client from establishing TCP connections to server.

Details:

• Any client-server architecture where the Server is configured to mitigate "Blind Reset Attack Using the SYN Bit" and sends "Challenge-ACK"
• As a response to client's SYN, the Server challenges by sending an ACK to confirm the loss of the previous connection and the request to start a new connection.
• This challenge ACK has acknowledgement number from previous connection and upon seeing the unexpected ACK, client sends a RST; thus tearing down TCP connection on the server also.
• RFC5961: Blind Reset Attack Using the SYN Bit 
• Here the Firewall drops the RST from Client after Server's "Challenge ACK" preventing client from establishing TCP connections to server.

• Palo Alto Firewalls
• Supported PAN-OS
• TCP connections

• When Palo Alto firewall is placed between such client and server, it doesn't understand such a flow by default.
• Firewall forwards the ACK received from server to the client however the RST from client is dropped by firewall.
• The flow debug logs display the following error message.
• Caution: Flow debug is CPU intensive and should not be run during peak traffic hours.

== xxxx-02-20 13:34:24.975 +0530 ==
Packet received at fastpath stage, tag 97481, type ATOMIC
Packet info: len 60 port 19 interface 19 vsys 1
wqe index 228938 packet 0x0x800000031627d8ce, HA: 0
  Packet decoded dump:
     L2:     00:26:6c:23:81:cd->08:30:6b:a1:e0:13, type 0x0800
     IP:     192.168.0.17->X.X.X.X, protocol 6
     version 4, ihl 5, tos 0x00, len 40,
     id 939, frag_off 0x4000, ttl 128, checksum 22508(0x57ec)
     TCP:    sport 51227, dport 443, seq 972108513, ack 972108513,
     reserved 0, offset 5, window 0, checksum 27948,
     flags 0x04 ( RST), urgent data 0, l4 data len 0
  TCP option:
       Flow fastpath, session 97481 (set work 0x800000030ef51900 exclude_video 0 from sp 0x80000002c393f080 exclude_video 0)
       NAT session, run address/port translation
       handle out of window packet, l7 not done, cutthrough  fin
       Drop invalid RST or FIN packet   

•  Following counters seen for such a flow:

   tcp_drop_packet                        1        0 warn      tcp
pktproc   packets dropped because of failure in tcp reassembly
   tcp_drop_out_of_wnd                    2        0 warn      tcp
resource  out-of-window packets dropped

• As such the TCP connection between both client and server enters into a hung state.
• In other words, the client keeps on trying to establish a new connection while the server continues to respond with a challenge ACK.

1. While dropping the out of window RST is actually an intended behavior, it breaks the Challenge-ACK mechanism.
2. Starting from PanOS 8.0.7 and onward, the following global configuration is provisioned to make the firewall aware of "Challenge-ACK" mechanism.
3. With this configuration, the client's RST will not be dropped, thereby letting the mechanism work unhindered.

>configure
#set deviceconfig setting tcp allow-challenge-ack yes
#commit
#exit
>

4. The setting can also be configured using WebUI under Device > Setup > Session > TCP Settings ( If configuring from the Panorama, use the correct template).

Reference: https://tools.ietf.org/html/rfc5961#section-4 (Blind Reset Attack Using the SYN Bit)
 

• Sample packet capture explaining such a flow:

• • First ACK from the server has acknowledgement number different than SYN's sequence number.
  • Client responding with RST with sequence number derived from challenge ACK.
  • Upon receiving the RST, Server tears down old TCP connection and relies on the SYN retransmission from the client end to re-establish the connection.