Troubleshooting GlobalProtect

Troubleshooting GlobalProtect

1553679
Created On 09/25/18 20:40 PM - Last Modified 03/15/24 20:36 PM


Symptom


Issues related to GlobalProtect can fall broadly into the following categories:

 

– GlobalProtect unable to connect to portal or gateway
– GlobalProtect agent connected but unable to access resources
– Miscellaneous

This article lists some of the common issues and methods for troubleshooting GlobalProtect. The article assumes you are aware of the basics of GlobalProtect and its configuration. Refer to the GlobalProtect resource guide.

Tools used for troubleshooting

Tools and utilities for troubleshooting on the client machine
 

Ping/Traceroute

To verify reachability to the portal/gateway

Nslookup

To make sure that the FQDNs for the portal/gateway are getting resolved

Ipconfig/ Ifconfig/ Netstat -nr / Route print

To verify the GlobalProtect adapter settings and routes installed by the GlobalProtect client

MMC (Windows)/Keychain Access (OSX)

To install and verify the installed client/root CA certificates

Wireshark

To capture transaction between the GlobalProtect client and the portal/gateway

Web Browser

To download the GlobalProtect client and to confirm successful SSL connection between the client and the portal/gateway

GlobalProtect Client Status/Detail tab

To check the status of the connection
GlobalProtect client logsTo check detailed debug logs from the GlobalProtect client

 



Environment


  • Any Pan-OS
  • Any GP client
  •  Existing GlobalProtect infrastructure configured 


Resolution


Tools used for troubleshooting on the firewall

 

1) Packet Captures

Dataplane Captures: How to Run a Packet Capture
(
For transactions between the client and the portal/gateway. Useful to see if the firewall is dropping any packets on the dataplane. it could also be useful to confirm if the ISP handle the traffic (specially UDP) correctly and not misroute or drop it, But not very helpful with SSL offload enabled since packets might be missing.)

Management Port Captures : How To Packet Capture (tcpdump) On Management Interface
(For transactions between the firewall and the Authentication server (LDAP, Radius and Kerberos)) 

2) Firewall's Debug Logs: Might need to enable debug for more detailed information:

 

appweb3-sslvpn.log
(PAN OS 9.1 and 10.1)/
gpsvc.log
(PAN OS 10.2 and higher)

 

Main log file for all SSL VPN related activities (Portal responses, gateway responses, certificate authentication, Cookie authentication override) also can be used to track communication with other daemons.

 

pan_packet_diag.log

 

To verify the handling of initial SSL request from Client on the dataplane, after which the communication is sent to the sslvpn / GPSVC daemon on the management plane (MP).

 

authd.log

 

For authentication issues related to GlobalProtect login.

rasmgr.log 

For client login/logout events and Portal and gateway configuration matching issues. 

useridd.log For User-IP mappings, group mapping and HIP checks.

3) CLI commands: Useful GlobalProtect CLI Commands.

4) Traffic logs: To verify connections coming from the client for the portal/gateway and for checking details of sessions from a connected GlobalProtect client to resources. 
 

General Troubleshooting approach

First make sure of the Compatibility matrix:
Where Can I Download and Install the GlobalProtect App?
1) Verify that the configuration has been done correctly as per documents suiting your scenario.

2) On the client, make sure the GlobalProtect client is installed, if this is not the first time you are connecting to GlobalProtect.

3) Use nslookup on the client to make sure the client can resolve the FQDNs for the portal/gateway.

4) Open a web browser and enter the URL : https://portal-fqdn/global-protect/prelogin.esp and/or https://gateway-fqdn/ssl-vpn/prelogin.esp. This will make sure that the SSL communication between the client and the portal/gateway is working fine. The web browser easily helps us check the certificate coming from the portal/gateway. If there are certificate issues, browser errors can help isolate those. Below are some examples: 
– Signing Authority is not trusted
– Common Name in the certificate is different from SNI requested by client, or SAN does not contain proper DNS name
– Certificate validity expired
– Any issues in certificate chain

5) If the browser page above is not loading properly, check with Wireshark to see if the TCP handshake is complete or not. Use filter ip.addr==<Portal IP> or ip.addr==<gatewayIP> as appropriate.

6) If the SYN packet is going out and no ACK is received, move to the firewall and see if the sessions are getting formed, and if packets are getting dropped. Use dataplane debugs or captures combined with global counters to check the same. Check security policies, NAT, etc. to make sure traffic is not getting dropped.

7) In the above case, sometimes it is also helpful to check if dataplane resources are healthy. Check the following commands to find any resource over-utilization:

     > show running resource-monitor
     > debug dataplane pool statistics

8) Check appweb3-sslvpn.log / gpsvc.log for  more information, if packets are not getting dropped on the dataplane.

9) From the browser, if the GlobalProtect login page is loading properly, it might ask for the client certificate if client certificate-based authentication is enabled on the portal.

10) Check whether the proper client certificate is loaded into the user's certificate store for the browser and GP app and the machine's certificate store for GP app.


11) If you are getting the error 'valid Client Certificate is required,' import the client certificate into the client's user certificate store and/or the client's machine certificate store.
'Valid client certificate is required' error accessing portal address on Firefox
Internet Explorer Browser Error: "Valid client certificate required"


12) Try logging in to the GlobalProtect Portal Web page. This will confirm that the authentication is working fine.

13) If unable to log in, check the firewall authd logs to see what is the error. The following document can be helpful if using LDAP authentication: How to Troubleshoot LDAP Authentication


14) If you are able to login in to the Portal Web page, download and install the GlobalProtect client, if it is not already installed.

15) Open the GlobalProtect client, and enter the required settings (Username/ Password / Portal) and click connect.

16) Notice the message displayed on the Status tab.

17) Collect the logs on the GlobalProtect client, as mentioned in the tools used section, and open the PanGPS.log file in the zipped folder.

18) Go through the logs, and based on error messages, take corrective action or troubleshoot.

19) Simultaneously, you might be required to check the mp-log/appweb3-sslvpn.log/ gpsvc.log on the firewall for more information.


 

Common Issues

GlobalProtect unable to connect to portal or gateway

After following the above troubleshooting approach, if you are receiving the following errors:

1) Could not connect to Portal (or similar symptoms)
– GlobalProtect Client Error: did not find portal address
– GlobalProtect Client not Connecting
– GlobalProtect Client Stuck at Connecting when Workstation is on the Local Network
– GlobalProtect Client Unable to Connect on Newly Installed Machine

2) Required client certificate is not found
– GlobalProtect failed to connect - required client certificate is not found

3) 'Server certificate verification failed' or 'Protocol error. Check server certificate
– GP Client Error: Gateway Protocol Error, Check Server Certificate
– Unable to Access GlobalProtect Due to Error (3659)

4) Failed to SetDoc. Message: errors getting GlobalProtect config
– GlobalProtect Client Error: "Failed to SetDoc. Message: errors getting GlobalProtect config"

5) [OCSP] The result of Certificate status query is unavailable
– OCSP Validation of Client Certificate Not Working

6) Discovering Network
– How To Troubleshoot Driver Issues in GlobalProtect that cause "Discovering Network" to be stuck.

7) IpReleaseAddress failed: The RPC server is unavailable 
– Try uninstalling any other virtual adapters
– Try reinstalling the GlobalProtect client after removing all the components
– Try stopping and starting the RPC Services:
     – – Click on start and go to Run window.
     – – On Run, type services.msc
     – – Locate the Remote procedure Call service.
     – – Start Remote procedure Call service, by right clicking the service. If it is started, stop it and start it again.
     – – Restart the PC and see if the problem persists.
     – – You can also try to reinstall Windows OS on the machine.
     – – Contact Technical Support if issue persists.

8) Element not found 
– Try updating the Microsoft patches on the client machine.
– Try installing a different GlobalProtect client version.
– Check Palo Alto release notes for any reported issues.
– Contact Technical Support if issue persists.

9) Failed to find PANGP virtual adapter interface
– Disable WMI services. Run - services.msc - WMI - stop the services.
– Delete the files under: C:\Windows\System32\wbem\Repository
– Delete GlobalProtect related files, uninstalled GlobalProtect, make sure that the virtual adapter disappeared.
– Reboot the machine, reinstall, and check the status. 
– Reinstalling the client OS might help if the situation permits.
– Contact Technical Support if issue persists.

10) Failed to get default route entry
– Uninstall Reinstall the GlobalProtect client
– If a newer version of the GlobalProtect client is available and if the situation permits, try installing the newer version.
– Try to restart the Windows DHCP : Run - services..msc - DHCP Client - Stop the service, Start the service.
– Update the Microsoft patches or hot fix more detail on this link
– Contact Technical Support if issue persists.

11) GP App reports an error of The virtual adapter was not set up correctly due to a delay
– Need to rebuild WMI repository. follow the steps on this troubleshooting link
– If any issues, contact Technical Support.

12) Assign private IP address failed
– Check if the IP address pool has enough IPs
– Check if the IP pool does not overlaps with the IP of the Client PC.
– Check if the User Group used in Global Protect > gateway > Client Configuration > Network Setting is properly included in the Group Mappings on the firewall and firewall is able to fetch the group from the AD server.
– Check if the user belongs to the correct group as mentioned in the Network Settings of Client Configuration under GP gateway.

GlobalProtect agent connected but unable to access resources

1) Check whether the GlobalProtect Client Virtual Adapter is getting an IP address, DNS Suffix and Access Routes for the remote resources. You can use the GlobalProtect Client Panel Detail tab or the command line tools like ipconfig/all, ifconfig, nslookup, netstat -nr, route print etc. for the same.

2) Check to see that port 4501 is not blocked on the Palo Alto Networks firewall or the client side (firewall on PC) or somewhere in between, as this is used by IPsec for the data communication between the GlobalProtect client and the firewall. Pcaps on the client physical interface or pcaps and debugs on the firewall can help to make sure packets are not getting dropped anywhere.

3) Check whether the Firewall is configured with proper security policies to allow the traffic from the IP pool allotted to the GlobalProtect Client Virtual Adapter. The policy should be configured from the zone of the tunnel interface to the zone of the protected resource. Tools like traffic logs, packet captures, dataplane debugs with global counters can be used to troubleshoot this. Packet captures on the Client on the GlobalProtect Virtual Adapter can help to compare the packets as sent by the client with what is received on the firewall and vice versa.

4) Check for SSL decryption being enabled for GP traffic, which could break any browser-based or non-browser application's traffic.

5) Check whether there is proper route for the IP pool used by GlobalProtect on the network for reply traffic. If you are using dynamic routing, then you need to redistribute these routes to the routing protocol from Palo Alto Networks. Captures on the Palo Alto Networks firewall for unencrypted traffic can help find out if firewall is sending the packets out towards the resources and if it is getting any response.

6) Check whether the Firewall is getting the IP-User Mapping from the GlobalProtect client. Verify using > show user ip-user-mapping ip <ip> to make sure the firewall is able to find the group the user is a part of. If the group mapping is not populated properly, then troubleshoot the User-ID issue, detail configuration and troubleshooting is on the below link:

User-ID Configuring and Troubleshooting

7) Check whether the firewall is getting the HIP data from the GlobalProtect Client, and if the HIP object is configured properly and allowed in the security rule. How to Troubleshoot HIP Data


Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClkBCAS&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Choose Language