URL Lookup Behavior For PAN-OS Releases

34007

Created On 04/21/20 08:03 AM - Last Modified 11/16/24 21:46 PM

URL Category

URL Filtering

8.1

9.0

9.1

PAN-OS

Panorama

Symptom

This article describes the conditions for which URL Lookup is done. When packets might get dropped and when a session might be let through even if the PAN-DB server is not reachable.

Environment

Cause

Traffic for which a URL Lookup is done

A valid PAN-DB URL Filtering License is installed on the device
Traffic is HTTP/HTTPS (cleartext, SSL, or decrypted SSL)
URL doesn’t exist in the DP or MP cache
At least one security policy for the associated vsys has a URL category or URL security profile configured

To check if URL lookup was applied to a session, run show session id <id> and look for:

URL filtering enabled                : True

In certain scenarios, if the URL Lookup processing is slow, packets might get dropped and sessions denied.

URL Lookup behavior in PAN-OS 8.1

URL category for the URL is present in the DP cache
1. The URL category is used to do a security policy lookup. The resulting action is applied to all packets for that session
URL category for the URL is not present in the DP cache
1. If traffic is allowed based on parameters other than the URL category, the first packet (which is often a GET request) is transmitted to the server while waiting for URL categorization from MP cache or cloud
2. Subsequent packets (mostly server response packet) are dropped till the URL categorization is received. Packets are dropped with the counter 'url_request_pkt_drop'. Once the URL category is known, we go to Step 3.
3. With the newly learned URL categor(ies), a security policy re-evaluation is done. Traffic is allowed based on security policy evaluation.
Slow or no connectivity to the PAN-DB cloud:
- If the PAN-DB cloud is completely inaccessible and the URL entry is not present in the MP and DP URL caches, then the URL request operation will time out after 5 seconds and the URL category will be set to 'PAN_URL_CATEGORY_NOT_RESOLVED'. Another security policy re-evaluation is done with the URL category set to 'PAN_URL_CATEGORY_NOT_RESOLVED'. Traffic is allowed based on security policy evaluation.

URL Lookup behavior in PAN-OS 9.0
The behavior is identical to PAN-OS 8.1 except for the following enhancement:
A new option "Hold Client Request for category lookup" is available.
Configurable in 9.0 via CLI -

set deviceconfig setting ctd hold-client-request <yes/no>

If hold-client-request is set to "yes",

If URL category is not available, the first packet is *not* transmitted to the server. (This is different from 8.1 where the first packet is always transmitted to the server, even if the URL category is not available).
All subsequent packets (client or server) are held in DP ie not transmitted from the firewall until URL categorization for first packet is received and the processing of the first packet is completed and transmitted out.

URL Lookup behavior in PAN-OS 9.1

Starting from 9.1, customer can configure 'hold-client-request' in UI as well.
The behavior is identical to PAN-OS 9.0 except for the following changes:

The time that DP will wait for the URL request operation to complete defaults to 2 seconds instead of 5 seconds
The "Hold Client Request for category lookup" is configurable via GUI under Device > Setup > Content-ID