UDP sessions stuck after failover
35589
Created On 11/30/20 02:01 AM - Last Modified 01/27/22 03:39 AM
Symptom
- UDP traffic not being forwarded to the backup link after a failure on the primary link.
- The backup egress interface is ignored.
- This can be a PBF rule or another route.
Environment
- Any current PAN-OS
- Dual Homed configuration.
- Path monitoring for egress interfaces.
Cause
The firewall is following sessions.
The sessions are easy to track for TCP traffic as we can identify a session start and a session end.
However, this is not possible to do the same on UDP.
For UDP, the firewall creates session at the first UDP packet, then the session remain up as the session TTL reaches 0. The session TTL is reset to its default value (by default 30sec) as long there is UDP traffic matching this session.
Because the session does not expired because of the continuous incoming packet, the session cannot be purge and the UDP traffic is being stuck to the wrong egress interface.
Resolution
The simple solution is to clear the UDP sessions stuck.
> clear session all filter protocol 17
Generally this is not possible to clear sessions manually every time there is a failure.The method shown below is taking advantage of the XMLAPI to clear the sessions.
So the steps below will allow you to automatically clear those sessions when there is a PBF failure. But a similar configuration can be done for path monitoring failure on static route.
Step1: Create an user account for the XML API
For our use case, the user account need to have at least the "device admin" role.- Go to GUI: Device > Administrators
- Click on Add.
- Enter the user account information
- Once the account created, commit the configuration change.
Step2: Generate the XML API key.
Using your web browser, open the URL
https://firewall/api/?type=keygen&user=username&password=password
- Keep the the key in the notepad as we need it later.
Step3: The PBF need to generate a log
- Make sure the PBF rule has path monitoring as we will use this log to trigger the action.
- Enable the path monitoring on the PBF rule
Step4: Look for the log to use
- Note in the system logs, the logs for the PBF are visible with the filter "(subtype eq pbf)"
- To avoid multiple sessions cleared in short time, use the filter
(subtype eq pbf) and ( description contains ' nexthop is ')
Step5: Configuration of the HTTP profile.
- Go to GUI: Device > HTTP
- Click on Add
- Click on Add .
- Set the following parameters for the server :
Name : any name Address : localhost Protocol : HTTP Port : 80 HTTP Method : GET
- Click on Payload Format.
- Click on System (as we are working on system logs).
- In the payload format window, set the following values:
Name : any name In Parameters: key : the API key retrieved earlier type : op cmd : <clear><session><all><filter><protocol>17</protocol></filter></all></session></clear>
- Once done, click OK twice.
Step6: Apply the HTTP profile to the log.
- Go to GUI: Device > Log Settings
- Click on Add
- Give a name to the rule.
- Click on the arrow to expand the filter options
- Click on Filter Builder.
- In the filter builder, you can paste the filter "(subtype eq pbf) and ( description contains ' nexthop is ')".
- Click OK.
- Click on Add in the HTTP section.
- Select the HTTP profile we created earlier
- Click OK.
- Click on commit.
Step7: Verification
- To check some sessions are being closed, check the traffic logs and look for session ended for unknown reason.
- Correlate with the system logs.
Additional Information
- Try Using username plus password with 26 or fewer characters or less the API key length generated will be 132.
- If you have 27 or more characters combined for username and password then the API key will be 164 characters.
- Because the HTTP server profile Payload Format Parameters The maximum length of this field is 134.
- It is Recommended using a username/password combination of 26 characters or lower, then generate the API key which will be 132 character length and will fit in the HTTP server profile.
Get Your API Key
Configure Log Forwarding