Palo Alto Networks Knowledgebase: Session setup fails due to session hash collision error

Session setup fails due to session hash collision error

8567
Created On 02/07/19 23:59 PM - Last Updated 02/08/19 00:00 AM
Policy
Symptom

Symptoms

  • Inbound connections occasionally fail
  • The firewall is configured with a Dynamic IP and Port source NAT policy for outbound traffic
  • The firewall is configured with a static destination NAT policy for inbound traffic

Diagnosis

On a Palo Alto Networks firewall, a session is defined by two uni-directional flows each uniquely identified by a 6-tuple key: source-address, destination-address, source-port, destination-port, protocol, and security-zone.


A hash collision occurs when the firewall attempts to create a new session with either flow matching an existing session's 6-tuple key. When this occurs, the firewall will drop the incoming packet and increment the global counter called session_hash_insert_duplicate.

 

In the next two diagrams you can see this session collision behavior with two SIP calls. Each diagram includes a local phone, firewall, and a remote phone. Additionally we've enumerated the network legs of the client-server flow and the IP address and UDP port information for each packet.

 

70539-1.1.png

 

  1. The local phone (192.168.1.2) initiates a SIP call to the remote phone (10.10.10.10) with a source and destination UDP port of 5060.
  2. The outbound traffic matches a source NAT policy on the firewall, translating the source IP to 10.10.10.1 and source port to 56611. The firewall then creates a session with two uni-directional flows. Here we can see the session details of the first SIP session created by the firewall.

    > show session id 27954

    Session 27954

    c2s flow:
    source: 192.168.1.2 [l3-trust]
    dst: 10.10.10.10
    proto: 17
    sport: 5060 dport: 5060
    state: ACTIVE type: FLOW
    src user: unknown
    dst user: unknown

    s2c flow:
    source: 10.10.10.10 [l3-untrust]
    dst: 10.10.10.1
    proto: 17
    sport: 5060 dport: 56611
    state: ACTIVE type: FLOW
    src user: unknown
    dst user: unknown


  3. A second connection is attempted by the remote phone shortly afterward. This connection is destined for the firewall's IP address of 10.10.10.1. The firewall has an inbound destination NAT policy configured for the internal phone's IP address of 192.168.1.2. This call is never established, but why?

 

70539-2.1.png

 

After running a filtered packet capture we see that the firewall is dropping the SIP INVITE packets from 10.10.10.10.

The global counters indicate that the drops occur during session creation and are caused by a session hash collision.

 

> show counter global filter delta yes packet-filter yes

 

Global counters:
Elapsed time since last sampling: 9.170 seconds

name value rate severity category aspect description
--------------------------------------------------------------------------------
session_allocated 1 0 info session resource Sessions allocated
session_freed 1 0 info session resource Sessions freed
session_install_error 1 0 warn session pktproc Sessions installation error
session_install_error_s2c 1 0 warn session pktproc Sessions installation error s2c
session_hash_insert_duplicate 1 0 warn session pktproc Session setup: hash insert failure due to duplicate entry

 

The firewall matches the call from 10.10.10.10 with an inbound destination NAT policy and translates the destination IP address from 10.10.10.1 to 192.168.1.2. When the firewall attempts to create a session it checks to see if the session hash matches an existing c2s or s2c flow.

 

If we compare the c2s flow of the first session and the expected s2c flow of the failed session we have the following:

 

c2s flow:
source: 192.168.1.2 [l3-trust]
dst: 10.10.10.10
proto: 17
sport: 5060 dport: 5060
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown
s2c flow:
source: 192.168.1.2 [l3-trust]
dst: 10.10.10.10
proto: 17
sport: 5060 dport: 5060
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown

 

Notice that these two flows are identical. If the firewall had created the second session, it would be unable to identify which flow to use for a matching packet. The firewall does not create the second session due to this conflict.



Resolution

This behavior can manifest itself in unique ways depending on the implementation. For this scenario, a user may reconfigure their source NAT from Dynamic IP and Port to Static NAT. This will allow the second call to match the initial s2c flow instead of creating a new session by the firewall.

 

When session_hash_insert_duplicate is seen in the global counters, keep in mind how NAT policies could affect new sessions that match exisiting flows.



Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClWrCAK&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments
Choose Language