TCP MSS adjustment for IPSec traffic

TCP MSS adjustment for IPSec traffic

78416
Created On 09/25/18 19:21 PM - Last Updated 04/21/20 00:20 AM


Resolution

For TCP traffic over IPSec Tunnel, the Palo Alto Networks firewall will automatically adjust the TCP MSS in the three-way handshake. This will happen irrespective of the Adjust TCP MSS option enabled on the VPN external interface.

The calculated MSS is the lower of the two values as under:

  1. Tunnel Interface MTU - 40 bytes
  2. MSS Calculated based on Interface MTU, Encryption, Authentication Algorithms


Relation Between Original Packet Size, Encryption Algorithm, Authentication Algorithm and Interface MTU

Lets consider the following situation:

Client  ——— Palto Alto ——— Internet  ——— Remote Firewall ——— Server

                              \____________(IPSec) ____________/

 

Client MTU : 1500

Server MTU: 1500

MTU on VPN terminating interface : 1500

Tunnel Interface MTU : 1500

Encryption Algo : AES-256-CBC

Authentication Algorithm : SHA1

 

ESP Overhead: (All sizes in bytes)

 

Outer IP Header20
Sequence Number4
SPI4
Initialisation Vector16
ESP Padding [0-15]
Padding Length1
Next Header1
Authentication Data12
  
Total[58-73]

 

So AES-256 with SHA1 produces a maximum overhead of 73 bytes. 

 

Original Packet Size + Max Overhead  <= 1500

TCP Segment + TCP Header + IP Header + Max Overhead  <= 1500

TCP Segment + 20 bytes + 20 bytes +  73 bytes <= 1500

TCP Segment <= 1387 bytes

 

If MSS is taken as 1388, then the resulting ESP header in this case will only be 1496 bytes. (Padding will be 10 bytes only)

 

From above,

 

  1. MSS Based on Tunnel Interface MTU = 1500 - 20 Bytes (IP Header) - 20 bytes (TCP Header) = 1460 Bytes
  2. MSS Calculated based on Interface MTU, Encryption, Authentication Algorithms = 1388 Bytes

 

Final MSS Calculated : MIN (1460, 1388) = 1388.

 

The same calculation can be used for various combination of encryption/authentication algorithms. Some of the known values are:

 

Size of Initialization Vector for

  • AES : 16 bytes
  • DES : 8 bytes

 

Size of Authentication Data for

  • MD5/ SHA-1 :  12 bytes
  • SHA-256 : 16 bytes
  • SHA-384 : 24 bytes
  • SHA-512 : 32 bytes

 

Maximum Size of padding for

  • AES : 15 bytes
  • DES : 7 bytes

Note:

  • The above behavior has been tested in PAN-OS 6.0 and later.
  • In the same case above, if you set the MTU of tunnel interface as 1400, then the resulting MSS will be 1360 and not 1388.

The above calculation can also be used to calculate the optimum MSS value for an IPSec tunnel. If the firewall is not auto adjusting the MSS considering the ESP overhead, the proper value of MTU can be set on the tunnel.X interface for TCP adjustment.

 

For example, if, in the above case, the firewall was not adjusting MSS as per ESP overhead, you can set the tunnel interface MTU to 1387 + 40 = 1427 bytes. This will result in MSS value to be adjusted to same 1387 bytes.

 

This helps in improving performance of TCP applications over IPSec tunnels.



Additional Information

WHEN TO USE ADJUST MSS



Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClW3CAK&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments
Choose Language