Error:
An unexpected error occurred. Please click Reload to try again.
Error:
An unexpected error occurred. Please click Reload to try again.
UDP sessions stuck after failover - Knowledge Base - Palo Alto Networks

UDP sessions stuck after failover

50676
Created On 11/30/20 02:01 AM - Last Modified 10/23/24 20:17 PM


Symptom


  • UDP traffic not being forwarded to the backup link after a failure on the primary link.
  • The backup egress interface is ignored.
  • This can be a PBF rule or another route.


Environment


  • Any current PAN-OS
  • Dual Homed configuration.
  • Path monitoring for egress interfaces.


Cause


  • The firewall is following sessions.
  • The sessions are easy to track for TCP traffic as we can identify a session start and a session end.
  • In case of UDP, the firewall creates session at the first UDP packet, then the session remain up as the session TTL reaches 0.
  • The session TTL is reset to its default value (by default 30sec) as long there is UDP traffic matching this session.
  • The session does not expire due to continuous incoming packet, the session cannot be purge and the UDP traffic is being stuck to the wrong egress interface.


Resolution


  1. The issue has been addressed under an internal enhancement PAN-244348.
  2. The solution is to upgrade to the one of the following  versions or higher
  3. After upgrade use the CLI command "set session teardown-upon-fwd-zonechange yes"
  4. The default value is false, disabled.  Once set the command is persistent across reboots. 
  5. The setting can be observed under "show session info" command
  6. Another workaround is to clear the UDP sessions using the CLI command " > clear session all filter protocol 17"
Session setup
  TCP - reject non-SYN first packet:             True
  Hardware session offloading:                   True
  Software Cut Through:                          False
  Hardware UDP session offloading:               True
  Run-to-completion mode:                        True
  Tunnel acceleration:                           True
  IPv6 firewalling:                              True
  Strict TCP/IP checksum:                        True
  Strict TCP RST sequence:                       True
  Reject TCP small initial window:               False
  Reject TCP SYN with different seq/options:     True
  Teardown session if forward zone changes:      True        
  ICMP Unreachable Packet Rate:                  200 pps
 


Additional Information


The method shown below is taking advantage of the XMLAPI to clear the sessions.
So the steps below will allow you to automatically clear those sessions when there is a PBF failure. But a similar configuration can be done for path monitoring failure on static route.

Step1: Create an user account for the XML API

For our use case, the user account need to have at least the "device admin" role. 

  1. Go to GUI: Device > Administrators
  2. Click on Add.
  3. Enter the user account information
    create an account for XML API usage
  4. Once the account created, commit the configuration change.

Step2: Generate the XML API key.

Using your web browser, open the URL

https://firewall/api/?type=keygen&user=username&password=password


get the API key

  1. Keep the the key in the notepad as we need it later.

 

Step3: The PBF need to generate a log

  1. Make sure the PBF rule has path monitoring as we will use this log to trigger the action.
  2. Enable the path monitoring on the PBF rule
    PBF path monitoring


Step4: Look for the log to use

  1. Note in the system logs, the logs for the PBF are visible with the filter "(subtype eq pbf)"
the system logs show the logs so we can use those logs to trigger the event
  1. To avoid multiple sessions cleared in short time, use the filter
(subtype eq pbf) and ( description contains ' nexthop is ')

Step5:  Configuration of the HTTP profile.

  1. Go to GUI: Device > HTTP
  2. Click on Add
    Add a HTTP profile
  3. Click on Add .
    Server parameters

 

  1. Set the following parameters for the server :
    Name : any name
    Address : localhost
    Protocol : HTTP
    Port : 80
    HTTP Method : GET
  2. Click on Payload Format.
    payload customization
  3. Click on System (as we are working on system logs).
    Payload format
  4. In the payload format window, set the following values:
    Name : any name
    In Parameters:
    key : the API key retrieved earlier
    type : op
    cmd : <clear><session><all><filter><protocol>17</protocol></filter></all></session></clear>
  5. Once done, click OK twice.

Step6:  Apply the HTTP profile to the log.

  1. Go to GUI: Device > Log Settings
  2. Click on Add 
    set the System Log settings
  3. Give a name to the rule.
  4. Click on the arrow to expand the filter options
    Open the filter builder
  5. Click on Filter Builder.
  6. In the filter builder, you can paste the filter "(subtype eq pbf) and ( description contains ' nexthop is ')".
    Filter Builder
  7. Click OK.
  8. Click on Add in the HTTP section.
    Add the HTTP profile
  9. Select the HTTP profile we created earlier
  10. Click OK.
  11. Click on commit.

Step7:  Verification

  1. To check some sessions are being closed, check the traffic logs and look for session ended for unknown reason.
  2. Correlate with the system logs.

Correlate the traffic logs sessions with the system logs
 

 
  • Try Using username plus password with 26 or fewer characters or less the API key length generated will be 132.
  • If you have 27 or more characters combined for username and password then the API key will be 164 characters.
  • Because the HTTP server profile Payload Format Parameters The maximum length of this field is 134.
  • It is Recommended using a username/password combination of 26 characters or lower, then generate the API key which will be 132 character length and will fit in the HTTP server profile.


Get Your API Key
Configure Log Forwarding



Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000HBmqCAG&lang=en_US&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Choose Language