URL Categorization in SSL and Policy Lookup Work-Flow - Knowledge Base - Palo Alto Networks

URL Categorization in SSL and Policy Lookup Work-Flow

40615

Created On 09/26/18 13:54 PM - Last Modified 06/06/24 18:52 PM

URL Category

Policy

8.1

9.0

9.1

PAN-OS

Symptom

PAN-OS performs URL Category lookup for URLs encountered in HTTP headers and SSL Handshake payloads, as long as a valid URL Filtering License is installed. Using URL Category in policy evaluation and applying URL-Filtering profile action is subject to a couple of conditions as highlighted below.

HTTP Connection - Clear-Text Traffic:

As soon as the URL is encountered in an HTTP header, PAN-OS will derive the URL category and use the URL category to look-up the relevant security policy (along with all the information already available including the app-id). If the selected security-policy has a URL Filtering Profile attached to it, then the URL category action defined in the URL filtering profile will be taken.

In the following sections, the focus will be on SSL connections. The work-flow depends on whether the SSL connection is subjected to decryption using SSL Forward-Proxy feature.

Note-1: As of PAN-OS 8.1, the URL Category can be added as a match condition in Security Policies, QoS Policies, Decryption Policies, and Authentication Policies. The focus will be on Decryption Policies and Security Policies in this article.
Note-2: Although the article refers to SSL, the correct term is indeed TLS.

Environment

PAN-OS 8.1 and above.
Palo Alto Firewall.

Resolution

SSL Decryption is either not enabled or not applicable:

As soon as SSL Client Hello is processed by PAN-OS, the app-id is set to SSL.
Note that SSL Client Hello should be right after the 3-way handshake as per normal protocol packet flow. Sometimes there is custom SSL communication triggered from agents(not web browsers) which send some data prior to Client Hello. In this case the SSL decoder will not even come into play, app-id will be potentially identified as unknown-tcp hence URL Category match will not work
Client Hello (as of TLS 1.2) usually contains an extension named Server Name Indication (SNI) - This contains the Fully Qualified Domain Name (FQDN) of the Server.
Example: If the user accesses https://www.example.com using a web-browser, it initiates an SSL connection with the Client Hello containing SNI = www.example.com
In case the SNI is present, then PAN-OS will derive the URL Category, which in case of www.example.com is "computer-and-internet-info"
At this point, PAN-OS will perform a security-policy re-evaluation with this additional information of app-id = SSL and URL Category = computer-and-internet-info
In case the SNI is not present, PAN-OS will perform a security-policy re-evaluation with the additional information of app-id = SSL
As soon as SSL Server Hello with the Server Certificate Payload is received, PAN-OS will try to derive the URL Category using the Common Name (CN) of the Server Certificate Subject DN.
Note: PAN-OS will also try to derive the app-id using the Server Certificate payload, which will not be covered in this article. So app-id remains SSL.
If the URL Category derived using the Server Certificate CN is a new information, then PAN-OS will perform a security-policy re-evaluation with this additional information of app-id = SSL and URL Category = URL Category of Server Certificate CN.

SSL Decryption is enabled and is applicable:

As soon as SSL Client Hello is processed by PAN-OS, the app-id is set to SSL.
PAN-OS will perform a security-policy re-evaluation with the additional information of app-id = SSL
In case the Client Hello SNI is present, then PAN-OS will derive the URL Category, which in case of www.example.com is "computer-and-internet-info"
At this point, the URL category is used by PAN-OS to lookup decryption-policies alone. With this PAN-OS SSL Forward Proxy functionality is engaged.
It is important to note that PAN-OS will not use the URL Category to re-evaluate security policies. Neither does it apply the URL category action defined in the relevant URL filtering profile. and the justification is:
a) The URL category is derived using the FQDN as presented by the Client (web-browser); PAN-OS makes an informed decision that the application using the SSL connection will likely present the full URL (URL including the File location - www.example.com/images/amsterdam.png), which will be visible post-decryption and it may match a custom URL category.
b) URL categorization and filtering action will strictly apply to HTTP based traffic and during SSL handshake, it is not guaranteed that HTTP traffic will flow through the SSL channel.
c) If the URL Category action requires PAN-OS to present a response-page over HTTP to the end-user, it will have to wait until the SSL handshake is complete.
If a decrypted SSL connection terminates gracefully without sending any application data or if the SSL connection does not send any HTTP data, then the traffic logs will show incorrect security-policies aka rules if URL-Category match conditions are configured, because they are ignored by design.

Note: The justification provided above may not be fool-proof, however as described, the change in work-flow is the current design.