Firewall dropping RST from Client after Server's Challenge-ACK

280842

Created On 02/25/19 01:54 AM - Last Modified 03/06/24 16:02 PM

TCP

Deployment

PAN-OS

Symptom

Firewall dropping RST from Client after Server's "Challenge ACK" preventing client from establishing TCP connections to server.

Environment

Any client-server architecture where the Server is configured to mitigate "Blind Reset Attack Using the SYN Bit" and sends "Challenge-ACK"
As a response to client's SYN, the Server challenges by sending an ACK to confirm the loss of the previous connection and the request to start a new connection.
This challenge ACK has acknowledgement number from previous connection and upon seeing the unexpected ACK, client sends a RST; thus tearing down TCP connection on the server also.
RFC5961: https://tools.ietf.org/html/rfc5961#section-4
Sample packet capture explaining such a flow:

> First ACK from the server has acknowledgement number different than SYN's sequence number.
> Client responding with RST with sequence number derived from challenge ACK.
> Upon receiving the RST, Server tears down old TCP connection and relies on the SYN retransmission from the client end to re-establish the connection.

Cause

When Palo Alto firewall is placed between such client and server, it doesn't understand such a flow by default.
Firewall forwards the ACK received from server to the client however the RST from client is dropped by firewall with following reason in flow debug logs (How to enable flow debug ):

           == 2019-02-20 13:34:24.975 +0530 ==
           Packet received at fastpath stage, tag 97481, type ATOMIC
           Packet info: len 60 port 19 interface 19 vsys 1
           wqe index 228938 packet 0x0x800000031627d8ce, HA: 0
           Packet decoded dump:
              L2:     00:26:6c:23:81:cd->08:30:6b:a1:e0:13, type 0x0800
              IP:     192.168.0.17->X.X.X.X, protocol 6
               version 4, ihl 5, tos 0x00, len 40,
               id 939, frag_off 0x4000, ttl 128, checksum 22508(0x57ec)
              TCP:    sport 51227, dport 443, seq 972108513, ack 972108513,
               reserved 0, offset 5, window 0, checksum 27948,
               flags 0x04 ( RST), urgent data 0, l4 data len 0
          TCP option:
              Flow fastpath, session 97481 (set work 0x800000030ef51900 exclude_video 0 from sp 0x80000002c393f080 exclude_video 0)
              NAT session, run address/port translation
              handle out of window packet, l7 not done, cutthrough  fin
              Drop invalid RST or FIN packet

Note: Flow debug is CPU intensive and should not be run during peak traffic hours.

Following counters seen for such a flow:

tcp_drop_packet 1 0 warn tcp pktproc packets dropped because of failure in tcp reassembly
tcp_drop_out_of_wnd 2 0 warn tcp resource out-of-window packets dropped

As such the TCP connection between both client and server enters into a hung state. In other words, the client keeps on trying to establish a new connection while the server continues to respond with a challenge ACK.

Resolution

While dropping the out of window RST is actually an intended behavior, it breaks the Challenge-ACK mechanism.
Starting from PanOS 8.0.7 and onward, the following global configuration is provisioned to make the firewall aware of "Challenge-ACK" mechanism. The client's RST will not be dropped, thereby letting the mechanism work unhindered

>configure
#set deviceconfig setting tcp allow-challenge-ack yes
#commit
#exit
>

Additional Information