SSL connection errors/timeouts through Explicit proxy

• Intermittent connection errors experienced by users accessing internal resources through the explicit proxy.

ERROR_LOGS (See below for the details in the packet diag below)

• Accumulating too many packets 13 max 12
• Upstream reset: reset reason: connection termination, transport failure reason
• Resetting stream due to upstream_reset_after_response_started{connection termination}. Prior headers have already been sent
• Stream reset.

LOG_SIGNATURES

Pan_packet_diag.log: 
===============
debug: pan_proxy_process_app_data_accumulation(pan_proxy_a.c:236): Accumulating too many packets 13 max 12

eproxy.log
==============
[debug][router] [source/common/router/router.cc:1073] [C294395][S9135653462384844879] upstream reset: reset reason: connection termination, transport failure reason
[2025-02-26 15:39:18.274][18205][debug][http] [source/common/http/filter_manager.cc:897] [C294395][S9135653462384844879] Resetting stream due to upstream_reset_after_response_started{conn
ection termination}. Prior headers have already been sent
[2025-02-26 15:39:18.274][18205][debug][http] [source/common/http/conn_manager_impl.cc:204] [C294395][S9135653462384844879] doEndStream() resetting stream

• Palo Alto Networks Firewalls.
• Supported PAN-OS.
• Explicit proxy
• Accumulation proxy.
• A Server with an MSS field missed in the SYN-ACK TCP options.

• The issue is caused by a large client hello, exceeding the default packet limit of the accumulation proxy on the firewall's data plane.
• This large client hello, containing an extended session ticket, is sent by the client during SSL session resumption attempts, and the segmented packets ( from explicit proxy logic due to missing MSS ) fail to reassemble correctly on the accumulation proxy due to the MTU size of 590.
• This results in session closure as the accumulation proxy reaches its packet limit.
• The issue manifested due to the server not sending the MSS option in the SYN-ACK packet.

REMEDIATION_PLAN

1. Fix the MSS value issue from the server. 
2. if not possible and you need to fix it from the firewall, you can Increase the accumulation proxy packet limit: 'debug dataplane set ssl-decrypt accumulate-client-hello packetlimit 16'.
3. The default is 12, therefore if one faces the issue: 
   1. calculate the size of the problematic client hello
   2. <size of the client hello>/590 = X.
   3. X = the number to add in the 2nd point command
4. Use the command with caution with the number, keep it a little bit above the needed. 
5. Although the solution does not not need commit/reboot, it is suggested to use a maintenance window to add and test the command.
6. The command also needs to be re-added in case of an upgrade. 

• The issue can be resolved by increasing the accumulation proxy packet limit using the command 'debug dataplane set ssl-decrypt accumulate-client-hello packetlimit 16'.
• This command modifies the configuration for accumulation proxy to gather data from up to 16 packets which could lead to higher resource usage on the firewall and limit the total number of accumulation proxy sessions supported.
• Our engineering team will engage in internal discussions to potentially enhance the behavior of the accumulation proxy, aiming to prevent session closure due to a large client hello.
• Instances of encountering such small TCP segments are rare in real-world scenarios.
• Packet Captures are given below if an example is needed.