TCP MSS adjustment for IPSec traffic

TCP MSS adjustment for IPSec traffic

Created On 09/25/18 19:21 PM - Last Modified 04/21/20 00:20 AM


For TCP traffic over IPSec Tunnel, the Palo Alto Networks firewall will automatically adjust the TCP MSS in the three-way handshake. This will happen irrespective of the Adjust TCP MSS option enabled on the VPN external interface.

The calculated MSS is the lower of the two values as under:

  1. Tunnel Interface MTU - 40 bytes
  2. MSS Calculated based on Interface MTU, Encryption, Authentication Algorithms

Relation Between Original Packet Size, Encryption Algorithm, Authentication Algorithm and Interface MTU

Lets consider the following situation:

Client  ——— Palto Alto ——— Internet  ——— Remote Firewall ——— Server

                              \____________(IPSec) ____________/


Client MTU : 1500

Server MTU: 1500

MTU on VPN terminating interface : 1500

Tunnel Interface MTU : 1500

Encryption Algo : AES-256-CBC

Authentication Algorithm : SHA1


ESP Overhead: (All sizes in bytes)


Outer IP Header20
Sequence Number4
Initialisation Vector16
ESP Padding [0-15]
Padding Length1
Next Header1
Authentication Data12


So AES-256 with SHA1 produces a maximum overhead of 73 bytes. 


Original Packet Size + Max Overhead  <= 1500

TCP Segment + TCP Header + IP Header + Max Overhead  <= 1500

TCP Segment + 20 bytes + 20 bytes +  73 bytes <= 1500

TCP Segment <= 1387 bytes


If MSS is taken as 1388, then the resulting ESP header in this case will only be 1496 bytes. (Padding will be 10 bytes only)


From above,


  1. MSS Based on Tunnel Interface MTU = 1500 - 20 Bytes (IP Header) - 20 bytes (TCP Header) = 1460 Bytes
  2. MSS Calculated based on Interface MTU, Encryption, Authentication Algorithms = 1388 Bytes


Final MSS Calculated : MIN (1460, 1388) = 1388.


The same calculation can be used for various combination of encryption/authentication algorithms. Some of the known values are:


Size of Initialization Vector for

  • AES : 16 bytes
  • DES : 8 bytes


Size of Authentication Data for

  • MD5/ SHA-1 :  12 bytes
  • SHA-256 : 16 bytes
  • SHA-384 : 24 bytes
  • SHA-512 : 32 bytes


Maximum Size of padding for

  • AES : 15 bytes
  • DES : 7 bytes


  • The above behavior has been tested in PAN-OS 6.0 and later.
  • In the same case above, if you set the MTU of tunnel interface as 1400, then the resulting MSS will be 1360 and not 1388.

The above calculation can also be used to calculate the optimum MSS value for an IPSec tunnel. If the firewall is not auto adjusting the MSS considering the ESP overhead, the proper value of MTU can be set on the tunnel.X interface for TCP adjustment.


For example, if, in the above case, the firewall was not adjusting MSS as per ESP overhead, you can set the tunnel interface MTU to 1387 + 40 = 1427 bytes. This will result in MSS value to be adjusted to same 1387 bytes.


This helps in improving performance of TCP applications over IPSec tunnels.

Additional Information


  • Print
  • Copy Link

Choose Language