For TCP traffic over IPSec Tunnel, the Palo Alto Networks firewall will automatically adjust the TCP MSS in the three-way handshake. This will happen irrespective of the Adjust TCP MSS option enabled on the VPN external interface.
The calculated MSS is the lower of the two values as under:
- Tunnel Interface MTU - 40 bytes
- MSS Calculated based on Interface MTU, Encryption, Authentication Algorithms
Relation Between Original Packet Size, Encryption Algorithm, Authentication Algorithm and Interface MTU
Lets consider the following situation:
Client ——— Palto Alto ——— Internet ——— Remote Firewall ——— Server
\____________(IPSec) ____________/
Client MTU : 1500
Server MTU: 1500
MTU on VPN terminating interface : 1500
Tunnel Interface MTU : 1500
Encryption Algo : AES-256-CBC
Authentication Algorithm : SHA1
ESP Overhead: (All sizes in bytes)
| Outer IP Header | 20 |
| Sequence Number | 4 |
| SPI | 4 |
| Initialisation Vector | 16 |
| ESP Padding | [0-15] |
| Padding Length | 1 |
| Next Header | 1 |
| Authentication Data | 12 |
| | |
| Total | [58-73] |
So AES-256 with SHA1 produces a maximum overhead of 73 bytes.
Original Packet Size + Max Overhead <= 1500
TCP Segment + TCP Header + IP Header + Max Overhead <= 1500
TCP Segment + 20 bytes + 20 bytes + 73 bytes <= 1500
TCP Segment <= 1387 bytes
If MSS is taken as 1388, then the resulting ESP header in this case will only be 1496 bytes. (Padding will be 10 bytes only)
From above,
- MSS Based on Tunnel Interface MTU = 1500 - 20 Bytes (IP Header) - 20 bytes (TCP Header) = 1460 Bytes
- MSS Calculated based on Interface MTU, Encryption, Authentication Algorithms = 1388 Bytes
Final MSS Calculated : MIN (1460, 1388) = 1388.
The same calculation can be used for various combination of encryption/authentication algorithms. Some of the known values are:
Size of Initialization Vector for
- AES : 16 bytes
- DES : 8 bytes
Size of Authentication Data for
- MD5/ SHA-1 : 12 bytes
- SHA-256 : 16 bytes
- SHA-384 : 24 bytes
- SHA-512 : 32 bytes
Maximum Size of padding for
- AES : 15 bytes
- DES : 7 bytes
Note:
- The above behavior has been tested in PAN-OS 6.0 and later.
- In the same case above, if you set the MTU of tunnel interface as 1400, then the resulting MSS will be 1360 and not 1388.
The above calculation can also be used to calculate the optimum MSS value for an IPSec tunnel. If the firewall is not auto adjusting the MSS considering the ESP overhead, the proper value of MTU can be set on the tunnel.X interface for TCP adjustment.
For example, if, in the above case, the firewall was not adjusting MSS as per ESP overhead, you can set the tunnel interface MTU to 1387 + 40 = 1427 bytes. This will result in MSS value to be adjusted to same 1387 bytes.
This helps in improving performance of TCP applications over IPSec tunnels.