Firewall unable to connect to Panorama due to fragmentation

30451

Created On 10/24/19 03:44 AM - Last Modified 04/09/23 14:02 PM

M-100 Appliance

M-200 Appliance

M-500 Appliance

M-600 Appliance

PA-200

PA-3000 Series

PA-3200 Series

PA-5000 Series

PA-5200 Series

PA-7000 Series

PA-820 Firewall

PA-850 Firewall

Device Management

PAN-OS

Symptom

The firewall is unable to connect to Panorama due to fragmentation and MTU issue

Environment

Any Palo Alto Firewalls.
Panorama 7.1 and above.
PAN-OS 7.1 and above.

Cause

Fragmentation on the network devices between Firewall and Panorama causes the issue. The FW to panorama communications are based on SSL (TCP-3978) which is flagged as do-not-fragment.
When a standard ethernet packet (1500bytes long) is sent it might get dropped if the infrastructure has a lower MTU value than 1500.
In the example below the MTU is set as 1500 Bytes (default) and packets of size more than 1400 Bytes will have to be fragmented before they can be sent out of the interface. If the Don't Fragment (DF) flag is set on the IP header of the packet, then the devices along the route will have to drop the packet and send "Fragmentation needed but DF bit set" (type 3, code 4) ICMP error code to the source. Since an SSL (HTTPS) connection is being used, then the packets with the Server Certificate are usually big packets and can get dropped causing the SSL/TLS handshake to fail.

This can be verified using the following three steps.

Confirm on the firewall that Panorama status is seen as disconnected using show panorama-status.

fw01(active)> show panorama-status
Panorama Server 1 : 10.66.57.131
    Connected     : no
    HA state      : disconnected

Panorama Server 2 : 10.32.6.131
    Connected     : no
    HA state      : disconnected

Confirm on the firewall that the connection to Panorama is shown as "established".

fw01(active)> show netstat all yes numeric yes | match 3978
tcp        0      0 10.4.110.26:37305       10.66.57.131:3978         ESTABLISHED
tcp        0      0 10.4.110.26:48055       10.32.6.131:3978         ESTABLISHED

Using TCPdump, capture the packets on firewall management. The PCAP shows "bad hdr length 32 - too long, > 24" message

20:11:44.017947 IP 10.66.57.131.pan-panorama > 10.4.110.26.34412: P 2897:3250(353) ack 296 win 61 <nop,nop,timestamp 1667104066 660833754>
20:11:44.017980 IP 10.4.110.26.34412 > 10.66.57.131.pan-panorama: . ack 1 win 115 <nop,nop,timestamp 660833767 1667104066,nop,nop,sack 1 {2897:3250}>
20:11:44.017986 IP 10.66.57.131.pan-panorama > 10.4.110.26.34412: . [bad hdr length 32 - too long, > 24]
20:11:44.364196 IP 10.66.57.131.pan-panorama > 10.4.110.26.34412: . [bad hdr length 32 - too long, > 24]
20:11:44.367325 IP 10.66.57.131 > 10.4.110.26: tcp

Ping from the firewall to Panorama with no fragment and MTU value set on the firewall. The ping fails with Frag needed messages. The output below will be received if the network devices along the route can reply with an ICMP unreachable.

fw01(active)> ping do-not-fragment yes size 1500 host 10.66.57.131
From 10.4.110.26 icmp_seq=1 Frag needed and DF set (mtu = 1500)
From 10.4.110.26 icmp_seq=1 Frag needed and DF set (mtu = 1500)
From 10.4.110.26 icmp_seq=1 Frag needed and DF set (mtu = 1500)

NOTE: 10.66.57.131 is the Panorama IP

Repeat the above step in the opposite direction, from the panorama to the firewall to determine where our problem is.

Resolution

Solution-1

Determine the exact MTU (Maximum Transmission Unit) that can be used between the Firewall and Panorama. To do this, Ping from the firewall to Panorama (or vice versa) with no fragment option with different MTU values starting with the highest till ping is successful. In this example packet size of 1450 works fine with no fragment option being used.

fw01(active)> ping do-not-fragment yes size 1450 host 10.66.57.131
PING 10.66.57.131 (10.66.57.131) 1450(1478) bytes of data.
1458 bytes from 10.66.57.131: icmp_seq=1 ttl=48 time=238 ms
1458 bytes from 10.66.57.131: icmp_seq=2 ttl=48 time=178 ms

NOTE: 10.66.57.131 is the Panorama IP

Once the MTU size is determined, go to the Management Interface of both the firewall and Panorama and change the MTU to the determined value. In this example, it is 1450.

GUI: Device > Setup > Interfaces > Management >

Once the MTU is changed, Commit the changes on the Firewall and Panorama. Connectivity to Panorama should be established within a few seconds.

Solution-2

Most Firewalls and routers have the capability of adjusting the MSS value on a TCP connection through them. They can rewrite the MSS value on the SYN and SYN-ACK packets exchanged between the Client and Server. This can be used to set the MSS value to the calculated optimal MSS value so that both client and server build their segments to that size.
The TCP MSS, the maximum segment size, is a parameter of the options field of the TCP header that specifies the largest amount of data, specified in bytes, that a computer or communications device can receive in a single TCP segment. It does not include the TCP header (20bytes) or the IP header (20bytes).
This MSS value is announced (often mistakenly called a negotiation) from both sides during the TCP 3WHS.

Each side says: I can accept TCP segments up to the size of x.

Once each side announces its MSS, the remote side is expected to send packets containing a TCP segment no larger than the announced MSS value. Below shows an example of the MSS being announced within the 3WHS:

10.1.1.100 192.1.1.100 [SYN] Seq=0 Len=0 MSS=1460 TSV=556758839
192.1.1.100 10.1.1.100 [SYN, ACK] Seq=0 Ack=1 Win=16484 Len=0 MSS=1280 WS=0  
10.1.1.100 192.1.1.100 [ACK] Seq=1 Ack=1 Win=5888 Len=0

Additional Information