How to Troubleshoot HIP Match Issues
Objective
Security Policies with HIP Profiles are not being matched as expected and as a result the Global Protect user traffic is affected. So this article would help isolate the issue and either fix or engage support with data captured from the time of issue.
Environment
Any supported PAN OS (verified up to 9.0.0) and GP agent version with valid GP gateway license for using HIP based configurations.
Procedure
When a GP user reports that he is currently unable to access some resources after connecting to GP successfully, here are a few things we can check to quickly narrow down the issue with less user intervention.
- First, note down the time of issue.
- Check Monitor > Traffic Logs on firewall for GP client's IP address as source and see if the security rule is matching correctly. Let's say, the policy with HIP profile attached is not seen to be hit and traffic is matching other rule somewhere below the order.
- You can check if the user hit the security policy by running this command :
>test security-policy-match source <ip> source-user <> from <zone> destination <ip> protocol <> check-hip-mask yes4.Check Monitor > HIP Match Logs on the firewall to see if there is a match for the HIP Profile (and HIP objects used inside that HIP profile). Let's assume we are not seeing any match logs for that particular object/profile.
5.To extract the GP user IP address, Computer name and User name which is required further, first verify a few command outputs:
> show global-protect-gateway current-user
Domain-User Name : domain\username
Computer : USERNAME-PC
Primary Username : domain\username
Client : Microsoft Windows 7 Enterprise Edition Service Pack 1, 64-bit
VPN Type : Device Level VPN
Mobile ID :
Client OS : Windows
Private IP : 192.168.31.4
Private IPv6 : ::
Public IP (connected) : 10.193.89.141
Public IPv6 : ::
ESP : exist
SSL : none
Login Time : Mar.22 15:12:06
Logout/Expiration : Apr.21 16:12:06
TTL : 2585579
Inactivity TTL : 7996
Run the following with the correct username format:
Note : In Muti-Vsys firewalls make sure the target vsys is set before running the debug command
> set system settings target-vsys <vsys name>
In case of single vsys firewall you can skip the above command and proceed.
> debug user-id dump hip-report ip 192.168.31.4 user domain\username computer USERNAME-PC
<?xml version="1.0" encoding="UTF-8"?>
<hip-report>
<md5-sum>f04e9bc8ad5f75fcbcd5b48cf6247ce4</md5-sum>
<user-name>username</user-name>
<domain>domain</domain>
<host-name>USERNAME-PC</host-name>
<host-id>99584afe-4060-4eba-8640-0dbfd3e0d2f5</host-id>
<ip-address>192.168.31.4</ip-address>
<ipv6-address></ipv6-address>
<generate-time>03/22/2019 08:12:22</generate-time>
<hip-report-version>4</hip-report-version>
<categories>
<entry name="disk-encryption">
<list>
<entry>
<ProductInfo>
<Prod vendor="Microsoft Corporation" name="BitLocker Drive Encryption" version="6.1.7600.16385"/>
<drives>
<entry>
<drive-name>C:\</drive-name>
<enc-state>unencrypted</enc-state>
</entry>
</drives>
</ProductInfo>
</entry>
</list>
</entry>
The firewall will perform HIP Matching with the help of this HIP report against ALL of the HIP Objects and Profiles and will create HIP Match Logs for us to view under Monitor section. This is done immediately after the firewall receives the report regardless of the HIP objects/profiles used in policies.
This command output would be long which contains the XML of the entire HIP report the GP agent sent to the firewall and this output needs to be checked in real-time.
If you do not see any output for this command, then collect the GP Client Logs as the issue could be any listed (but not limited) below and further steps do not apply.
- GP agent didn't detect host information properly (due to some compatibility issues with the OPSWAT 3rd party libraries say) OR
- GP agent didn't pass on the report to GP Gateway firewall after connecting successfully.
- Look for the specific host information that didn't match, in this instance there was a HIP Object named UC-Drive used (so far only this one for illustration) in a HIP Profile named U-Drives which in turn is referenced in a security policy named U-Block. For example:
- Next look at the HIP Profile or Object configured (preferably from CLI to match the content with HIP Report as is).
# show profiles hip-objects UC-Drive
UC-Drive {
disk-encryption {
criteria {
encrypted-locations {
c$ {
encryption-state {
is unencrypted;
}
}
}
is-installed yes;
}
exclude-vendor no;
}
}
Snapshot from 8.1 Web UI:
# show profiles hip-profiles U-Drives
U-Drives {
match '"UC-Drive" ';
}
Snapshot from 8.1 Web UI:
Until now, we know that the HIP Report sent by the GP agent was received by the PA firewall, stored in the local DB but it failed to match against the existing HIP Objects thus not generating any HIP Match Logs under Monitor tab and also not matching the Deny rule U-Block which contains the HIP profile in it.
- Compare the outputs from Step 5 and 6:
It would have failed to match if the drive name was set to c:\ instead of C:\ because the configuration (that we checked using show config command earlier) should exactly match the data present in the HIP report (that we dumped using debug command earlier)
- To highlight the below output:
IP address: 192.168.31.4 (vsys1)
User: domain\username
From: GP
Idle Timeout: 10766s
Max. TTL: 10766s
Group(s): domain\username(2)
cn=domain users,cn=users,dc=domain,dc=local(2147483648)
cn=domain admins,cn=users,dc=domain,dc=local(2147483650)
cn=vpn,cn=users,dc=domain,dc=local(2147483652)
NOTE:
It does not contain any information about the HIP profiles used in policy (yet).
Also the best way to tell if the HIP profile is attached to a security policy would be by checking the following output, this is a very significant command that prints the security rules that are currently active in the data plane - which means it won't show disabled rules:
> show running security-policy
"U-Block; index: 1" {
from any;
source any;
source-region none;
to L3-Outside;
destination any;
destination-region none;
user any;
hip-profile U-Drives;
category any;
application/service 0:any/any/any/any;
action deny;
icmp-unreachable: no
terminal no;
}
- Change the HIP object UC-Drive configuration as follows > click OK to commit:
HIP Match Log is also generated for the HIP Object UC-Drive as follows:
> show user ip-user-mapping ip 192.168.31.4
IP address: 192.168.31.4 (vsys1)
User: domain\username
From: GP
Idle Timeout: 2591993s
Max. TTL: 2591993s
HIP profiles that user belong to (used in policy)
HIP profile(s): U-Drives
Group(s): domain\username(2)
cn=domain users,cn=users,dc=domain,dc=local(2147483648)
cn=domain admins,cn=users,dc=domain,dc=local(2147483650)
cn=vpn,cn=users,dc=domain,dc=local(2147483652)
NOTE
Now we can see the HIP Profiles matching for this IP address AND that are used in policy listed as expected.
Also the traffic started matching the U-Block security policy that contains the HIP Profile U-Drives
The example taken was just a manual configuration mistake, however the same troubleshooting steps apply for any changes that can occur due to PAN OS upgrade, GP Agent version update, etc.,
Anytime you see the HIP Match broken or being inconsistent, capturing this information would either reveal the root cause of help with the investigation of the issue.