What information and logs required to collect when troubleshooting split-tunnel Domain and application related issues
20133
Created On 07/05/21 14:44 PM - Last Modified 08/04/21 20:12 PM
Question
What information and logs required to collect when troubleshooting split-tunnel Domain and application related issues?
Environment
- GlobalProtect App 4.1+ and with Pan-OS 8.1 and later releases
- Window 7 Service Pack 2 and later release and MacOS 10.10 and later releases.
- GlobalProtect gateway configured with split tunneling: eg, Domain and Application
Answer
On Window device:
- The first step is to verify whether the configuration on the gateway for 'Split Tunnel Domain' or 'Split Application' has been pushed correctly on the GlobalProtect app or not.
- Verify the connection table on the client machine and confirm that specific applications are going via a physical interface and not from the tunnel interface, netstat -anob command can be used. We can also check the PCAP or firewall to see if there is traffic leaking.
- Change the logging level to "Dump" to make sure that PanGPS.log contains the details logs related to split-tunnel functionality( Under GlobalProtect app>Setting>Troubleshooting>Logging Level >Dump). Make sure to mark the time of the test (when the issue has been reproduced), along with the domain being accessed.
- Enable Wireshark packet captures on the client machine, on the physical and tunnel interface to track the traffic for a specific domain. It is required to take the packet captures for both physical and tunnel interfaces when troubleshooting the split-tunnel issue.
- For detailed Window Kernel side logs, which allow us to see the interaction between the GlobalProtect filter driver and the kernel, use DebugView.
- Run dbgview.exe as Administrator
- "Enable Verbose Kernel Output" and Start "Capture Kernel" (Ctrl +K)
- Note: This can generate a large number of logs and may also impact endpoint performance. Please enable this only when it is requested by TAC or engineering teams.
On macOS device:
- The first step is to verify whether the configuration on the gateway for 'Split Tunnel Domain' or 'Split Application' has been pushed correctly on the GlobalProtect app or not.
- Verify the connection table on the client machine and confirm that specific application are going via a physical interface and not from the tunnel interface, ‘netstat -arn’ or 'lsof -n -i | grep <application>' command can be used. We can also check the PCAP or firewall to see if there is traffic leaking.
- Check whether GlobalProtect system extension is active using
$ systemextensionsctl list
--- com.apple.system_extension.network_extension
enabled active teamID bundleID (version) name [state]
* * PXPZ95SK77 com.paloaltonetworks.GlobalProtect.client.extension (5.2.5-66/1) GlobalProtectExtension [activated enabled]
Run sudo launchctl list | grep palo command to confirm the presence of
NetworkExtension.com.paloaltonetworks.GlobalProtect.client.extension
--- com.apple.system_extension.network_extension
enabled active teamID bundleID (version) name [state]
* * PXPZ95SK77 com.paloaltonetworks.GlobalProtect.client.extension (5.2.5-66/1) GlobalProtectExtension [activated enabled]
Run sudo launchctl list | grep palo command to confirm the presence of
NetworkExtension.com.paloaltonetworks.GlobalProtect.client.extension
- Steps to collect information:
- In Terminal, running "sudo tcpdump -i all -k INP -w gptest.pcapng" to capture packets.
- Change the logging level to "Dump" to make sure that PanGPS.log contains the details logs related to split-tunnel functionality( Under GlobalProtect app>Setting>Troubleshooting>Logging Level >Dump).
- Start to reproduce the issue.
- Once the issue reproduced, stop packet capture and collect GP client logs. (Under GlobalProtect app>Setting>Troubleshooting>Collect logs)
- Collect gptest.pcapng and GP logs
- Make sure to mark the time of the test (when the issue has been reproduced), along with the domain name and the process accessing the domain.
- Make sure to change the GlobalProtect client logging level back to debug
Additional Information
- For include/exclude applications, please be aware that you MUST have all those applications installed before GP login. Otherwise, you have to reconnect GlobalProtect after you installed those applications. Double-check that all application paths configured include/exclude application list exist on client device before GlobalProtect login.
- Split-tunneling rules only apply to TCP/UDP traffic, so ICMP/ping is not subject to split-tunneling rules. Do not use ping to test whether split-tunnel rules are applied.
- Check if there is a 3rd party product that can prevent GlobalProtect from properly using filters/extensions to perform split-tunnel operations. Most of the time, conflicts are found with DLP(Data Loss Prevention), AV/AM (Anti-Virus/Anti-Malware), and other VPN types of software. In this case, we need to investigate whether the issue is on the GlobalProtect side or 3rd party vendor.
- For more troubleshooting tips and tricks related to Split Tunnel Domain & Application, please refer to the document.
- For the configuration guide of this feature refer to Optimized Split Tunneling for GlobalProtect and GlobalProtect: Implement Split Tunnel Domain and Applications