GlobalProtect on Windows : Pre-Logon Tunnel fails to establish intermittently
41429
Created On 03/03/21 18:09 PM - Last Modified 03/04/21 14:36 PM
Symptom
GlobalProtect Pre-Logon Tunnel, as the name suggests, is a GlobalProtect Tunnel created between the end-point and the GlobalProtect gateway "before" the user logs in to the end-point. This article describes an issue one might encounter while deploying pre-logon configuration in Windows PCs. The pre-logon tunnel establishment workflow in Windows is as follows:
- Once Windows finishes booting, GlobalProtect Service (PanGPS) starts. PanGPS identifies that Pre-Logon is enabled based on the registry setting and starts a Pre-Logon thread.
- Similarly, when all the user sessions are terminated i.e. when the Windows user logs out, Windows notifies PanGPS and this kicks off a Pre-Logon thread.
- The entire Pre-Logon Tunnel establishment is handled by PanGPS and must be finished before Windows notifies PanGPS about a user session.
- As soon as PanGPS learns about a User Logon event, it proceeds to terminate Pre-Logon thread. Naturally, there are two outcomes:
- If the Pre-Logon Tunnel is established by the time Windows notifies PanGPS about a user session, then the pre-logon tunnel is retained and the pre-logon thread is terminated.
- If the Pre-Logon Tunnel is being negotiated when Windows notifies PanGPS about a user session, then the pre-logon tunnel establishment is either aborted or disconnected abruptly and the pre-logon thread is terminated.
- Once the user login event is complete, depending on the Connect Method and the "Pre-Logon Tunnel Rename Timeout" value, either the pre-logon tunnel is retained while the user-tunnel is established and gracefully renamed to the user tunnel or the pre-logon tunnel is terminated and the user tunnel is established.
When the sequence of events follow a normal pattern and PanGPS is given enough time prior to the user login event, which is usually less than a minute, then then Pre-Logon Tunnel will establish consistently. However, we are observing an issue where the Pre-Logon Tunnel fails to establish intermittently even though the User has not logged in to the system yet. In the following sections, we will review the different stages of operation to help us identify the problem.
In a working scenario, the following sequence of events are observed [as seen in PanGPS.log file]:
Once the PC boots up:
(P5452-T5876)Info ( 156): 03/03/21 14:03:15:292 ####################### Start PanGPS service (ver: 5.2.5-66) ####################### (P5452-T7296)Info (10158): 03/03/21 14:03:17:173 PrelogonThread starts (P5452-T7296)Debug(2377): 03/03/21 14:03:17:173 ----portal processing starts---- (P5452-T7296)Debug(6897): 03/03/21 14:03:17:252 ----Portal Pre-login starts---- (P5452-T7296)Debug(8260): 03/03/21 14:03:17:454 portal status is Connected (P5452-T7324)Debug(5473): 03/03/21 14:03:17:600 ----Network Discover starts---- (P5452-T7324)Info (1385): 03/03/21 14:03:26:844 CPanGatewayList::PickGatewayBaseOnWeight, PAN_NEW_GATEWAY_SELECTOR, chose prefered gateway index =7, gateway is Amsterdam (P5452-T7324)Debug(3502): 03/03/21 14:03:26:848 ----Gateway Pre-login starts---- (P5452-T7324)Debug(3909): 03/03/21 14:03:27:123 ----Gateway Login starts---- (P5452-T7324)Debug(2645): 03/03/21 14:03:27:500 ----Tunnel creation starts---- (P5452-T7324)Debug(6849): 03/03/21 14:03:31:384 --Set state to Connected
Once the user logs in:
(P5452-T5456)Debug(1534): 03/03/21 14:25:41:344 Session 1, username roadrunner. (P5452-T5456)Debug(1543): 03/03/21 14:25:41:344 Session 1, domain name ACME. (P5452-T5456)Info (1578): 03/03/21 14:25:42:017 User acme\roadrunner logs in on session 1 (P5452-T7160)Info (10231): 03/03/21 14:26:13:294 PrelogonThread exits (P5452-T7284)Debug(2444): 03/03/21 14:26:13:294 User just logs in (P5452-T7284)Debug(2377): 03/03/21 14:26:13:851 ----portal processing starts---- ... and so on
In the non-working scenario, the following sequence may be observed [as seen in PanGPS.log file]
Here, as soon as the PC boots up followed by PanGPS process start-up, Windows notifies that the user session has been created and the session has been locked off. With this, Pre-Logon Tunnel establishment cannot happen because PanGPS learns about an active user session. What we do know however is that the user has not officially logged in i.e. the user has not yet made an attempt to login by typing the credentials:
(T4456)Info ( 144): 02/24/21 18:27:37:334 ####################### Start PanGPS service (ver: 5.2.5-66) ####################### (T5192)Info (9835): 02/24/21 18:27:38:537 PrelogonThread starts (T3244)Debug(1416): 02/24/21 18:27:47:287 First logon user. (T3244)Debug(1467): 02/24/21 18:27:47:303 Session 1, username roadrunner. (T3244)Debug(1476): 02/24/21 18:27:47:303 Session 1, domain name ACME. (T3244)Debug( 302): 02/24/21 18:27:55:678 Received session change, event type 7, session 1 (T3244)Info (1556): 02/24/21 18:27:55:678 lock off session 1 (T5192)Debug(4492): 02/24/21 18:28:05:459 Prelogon flag is 1 (T5192)Debug(4470): 02/24/21 18:28:05:459 Start ProcessServerPortal tiggered by prelogon flag (T5192)Info (4107): 02/24/21 18:28:05:459 UpdatePrelogonStateForSSO() - Pre-logon tunnel state = Connecting (T5192)Debug(2233): 02/24/21 18:28:05:740 ----portal processing starts---- (T5192)Debug(6682): 02/24/21 18:28:07:006 ----Portal Pre-login starts---- (T5192)Debug(1244): 02/24/21 18:28:10:240 set certname to none-selected (T5192)Debug(6827): 02/24/21 18:28:10:240 prelogin to portal result is (null) (T5192)Debug(7123): 02/24/21 18:28:10:240 Failed to pre-login to the portal globalprotect.acme.corp with return value 12044(A certificate is required to complete client authentication).
In this scenario, the pre-logon tunnel establishment failed because PanGPS did not make an attempt to query the machine certificate store causing portal pre-login failure. However, the trigger is Windows notifying PanGPS about a user session before the pre-logon tunnel negotiation is over.
Windows may notify PanGPS about the User Session at different stages of pre-logon tunnel negotiation, so the error messages in the logs may vary. The most important thing here is Windows notifying PanGPS about a User session before the pre-logon tunnel establishment is over and much before the user has actually entered the credentials to login to the PC.
Environment
Windows 10 Endpoints using GlobalProtect Clients with connect method set to Pre-Logon.
Cause
This issue is caused by a feature in Windows, which can either be called "Automatic sign-in" or "Fast Logon". In order to speed-up the login process and re-open the applications that were open prior to a Restart, Windows can be configured to use the previous sign-in information to finish setting up the User Session right after the restart process has finished. This is enabled by default and can be disabled using the option located at:
Windows Settings > Accounts > Sign-In Options
"Use my sign-in info to automatically finish setting up my device after an update or restart"
If Windows Automatic Sign-In creates the user session prior to PanGPS starting up, then we will see the following entry in the PanGPS.log file:
(T2564)Info ( 144): 02/24/21 18:27:37:334 ####################### Start PanGPS service (ver: 5.2.5-66) ####################### (T2564)Info (1517): 02/24/21 18:27:37:334 Enumerate session: user acme\roadrunner logs in on session 1
However, recently we have observed scenarios where Windows Automatic Sign-In creates a User Session after PanGPS starts up, which leads to erratic pre-logon tunnel behavior. As highlighted above, When Windows Automatic Sign-In happens, PanGPS.log makes it evident, however the pre-logon tunnel failure log is not so evident or consistent. In other words:
- If Windows Automatic Sign-in notifies PanGPS about the User Session prior to Pre-Logon Tunnel negotiation is over, then the pre-logon tunnel will either fail or be disconnected abruptly.
- If Windows Automatic Sign-in notifies PanGPS about the User Session after the Pre-Logon Tunnel negotiation is over, then the pre-logon tunnel will remain connected naturally.
Resolution
Windows Automatic Sign-In or Fast-Logon should be disabled for GlobalProtect Pre-Logon functionality to be effective and consistent.