UDP sessions stuck after failover

34838

Created On 11/30/20 02:01 AM - Last Modified 01/27/22 03:39 AM

HTTP

Policy Based Forwarding

XML API

High Availability

PAN-OS

Symptom

UDP traffic not being forwarded to the backup link after a failure on the primary link.
The backup egress interface is ignored.
This can be a PBF rule or another route.

Environment

Any current PAN-OS
Dual Homed configuration.
Path monitoring for egress interfaces.

Cause

The firewall is following sessions.
The sessions are easy to track for TCP traffic as we can identify a session start and a session end.

However, this is not possible to do the same on UDP.
For UDP, the firewall creates session at the first UDP packet, then the session remain up as the session TTL reaches 0. The session TTL is reset to its default value (by default 30sec) as long there is UDP traffic matching this session.

Because the session does not expired because of the continuous incoming packet, the session cannot be purge and the UDP traffic is being stuck to the wrong egress interface.

Resolution

The simple solution is to clear the UDP sessions stuck.

> clear session all filter protocol 17

Generally this is not possible to clear sessions manually every time there is a failure.

The method shown below is taking advantage of the XMLAPI to clear the sessions.
So the steps below will allow you to automatically clear those sessions when there is a PBF failure. But a similar configuration can be done for path monitoring failure on static route.

Step1: Create an user account for the XML API

For our use case, the user account need to have at least the "device admin" role.

Go to GUI: Device > Administrators
Click on Add.
Enter the user account information
Once the account created, commit the configuration change.

Step2: Generate the XML API key.

Using your web browser, open the URL

https://firewall/api/?type=keygen&user=username&password=password

Keep the the key in the notepad as we need it later.

Step3: The PBF need to generate a log

Make sure the PBF rule has path monitoring as we will use this log to trigger the action.
Enable the path monitoring on the PBF rule

Step4: Look for the log to use

Note in the system logs, the logs for the PBF are visible with the filter "(subtype eq pbf)"

the system logs show the logs so we can use those logs to trigger the event

To avoid multiple sessions cleared in short time, use the filter

(subtype eq pbf) and ( description contains ' nexthop is ')

Step5: Configuration of the HTTP profile.

Go to GUI: Device > HTTP
Click on Add
Click on Add .

Set the following parameters for the server :

Name : any name
Address : localhost
Protocol : HTTP
Port : 80
HTTP Method : GET

Click on Payload Format.
Click on System (as we are working on system logs).

In the payload format window, set the following values:

Name : any name
In Parameters:
key : the API key retrieved earlier
type : op
cmd : <clear><session><all><filter><protocol>17</protocol></filter></all></session></clear>

Once done, click OK twice.

Step6: Apply the HTTP profile to the log.

Go to GUI: Device > Log Settings
Click on Add
Give a name to the rule.
Click on the arrow to expand the filter options
Click on Filter Builder.
In the filter builder, you can paste the filter "(subtype eq pbf) and ( description contains ' nexthop is ')".
Click OK.
Click on Add in the HTTP section.
Select the HTTP profile we created earlier
Click OK.
Click on commit.

Step7: Verification

To check some sessions are being closed, check the traffic logs and look for session ended for unknown reason.
Correlate with the system logs.

Correlate the traffic logs sessions with the system logs

Additional Information

Try Using username plus password with 26 or fewer characters or less the API key length generated will be 132.
If you have 27 or more characters combined for username and password then the API key will be 164 characters.
Because the HTTP server profile Payload Format Parameters The maximum length of this field is 134.
It is Recommended using a username/password combination of 26 characters or lower, then generate the API key which will be 132 character length and will fit in the HTTP server profile.

Get Your API Key
Configure Log Forwarding