UDP sessions stuck after failover
50676
Created On 11/30/20 02:01 AM - Last Modified 10/23/24 20:17 PM
Symptom
- UDP traffic not being forwarded to the backup link after a failure on the primary link.
- The backup egress interface is ignored.
- This can be a PBF rule or another route.
Environment
- Any current PAN-OS
- Dual Homed configuration.
- Path monitoring for egress interfaces.
Cause
- The firewall is following sessions.
- The sessions are easy to track for TCP traffic as we can identify a session start and a session end.
- In case of UDP, the firewall creates session at the first UDP packet, then the session remain up as the session TTL reaches 0.
- The session TTL is reset to its default value (by default 30sec) as long there is UDP traffic matching this session.
- The session does not expire due to continuous incoming packet, the session cannot be purge and the UDP traffic is being stuck to the wrong egress interface.
Resolution
- The issue has been addressed under an internal enhancement PAN-244348.
- The solution is to upgrade to the one of the following versions or higher
- After upgrade use the CLI command "set session teardown-upon-fwd-zonechange yes"
- The default value is false, disabled. Once set the command is persistent across reboots.
- The setting can be observed under "show session info" command
- Another workaround is to clear the UDP sessions using the CLI command " > clear session all filter protocol 17"
Session setup
TCP - reject non-SYN first packet: True
Hardware session offloading: True
Software Cut Through: False
Hardware UDP session offloading: True
Run-to-completion mode: True
Tunnel acceleration: True
IPv6 firewalling: True
Strict TCP/IP checksum: True
Strict TCP RST sequence: True
Reject TCP small initial window: False
Reject TCP SYN with different seq/options: True
Teardown session if forward zone changes: True
ICMP Unreachable Packet Rate: 200 pps
Additional Information
The method shown below is taking advantage of the XMLAPI to clear the sessions.
So the steps below will allow you to automatically clear those sessions when there is a PBF failure. But a similar configuration can be done for path monitoring failure on static route.
Step1: Create an user account for the XML API
For our use case, the user account need to have at least the "device admin" role.
- Go to GUI: Device > Administrators
- Click on Add.
- Enter the user account information
- Once the account created, commit the configuration change.
Step2: Generate the XML API key.
Using your web browser, open the URL
https://firewall/api/?type=keygen&user=username&password=password
- Keep the the key in the notepad as we need it later.
Step3: The PBF need to generate a log
- Make sure the PBF rule has path monitoring as we will use this log to trigger the action.
- Enable the path monitoring on the PBF rule
Step4: Look for the log to use
- Note in the system logs, the logs for the PBF are visible with the filter "(subtype eq pbf)"
- To avoid multiple sessions cleared in short time, use the filter
(subtype eq pbf) and ( description contains ' nexthop is ')
Step5: Configuration of the HTTP profile.
- Go to GUI: Device > HTTP
- Click on Add
- Click on Add .
- Set the following parameters for the server :
Name : any name Address : localhost Protocol : HTTP Port : 80 HTTP Method : GET
- Click on Payload Format.
- Click on System (as we are working on system logs).
- In the payload format window, set the following values:
Name : any name In Parameters: key : the API key retrieved earlier type : op cmd : <clear><session><all><filter><protocol>17</protocol></filter></all></session></clear>
- Once done, click OK twice.
Step6: Apply the HTTP profile to the log.
- Go to GUI: Device > Log Settings
- Click on Add
- Give a name to the rule.
- Click on the arrow to expand the filter options
- Click on Filter Builder.
- In the filter builder, you can paste the filter "(subtype eq pbf) and ( description contains ' nexthop is ')".
- Click OK.
- Click on Add in the HTTP section.
- Select the HTTP profile we created earlier
- Click OK.
- Click on commit.
Step7: Verification
- To check some sessions are being closed, check the traffic logs and look for session ended for unknown reason.
- Correlate with the system logs.
- Try Using username plus password with 26 or fewer characters or less the API key length generated will be 132.
- If you have 27 or more characters combined for username and password then the API key will be 164 characters.
- Because the HTTP server profile Payload Format Parameters The maximum length of this field is 134.
- It is Recommended using a username/password combination of 26 characters or lower, then generate the API key which will be 132 character length and will fit in the HTTP server profile.