How to troubleshoot connection failure to the monitored server for PAN-OS integrated User-ID Agent

show user server-monitor statistics

show user server-monitor state <server-name>

• Under: Device > User Identification > User Mapping then click on the gear icon of the Palo Alto Networks User-ID Agent Setup
• Go to the Server Monitor Account tab.
• For more details, Refer Step 5 in Configure User Mapping Using the PAN-OS Integrated User-ID Agent.

1. Connection Timeout
2. Kerberos Error
3. Connection Refused
4. Access Denied
5. Authentication failed
6. Not Connected

1. Check if the firewall is able to reach the server this can be checked by pinging the server or using traceroute. If the service route to the monitored server is the mgmt interface, use CLI command:

   ping host <IP address of the monitored server>
   traceroute host <IP address of the monitored server>

   If the service rote to the monitored server is the dataplane interface, use CLI command:

   ping source <IP address of the dataplane interface> host <IP address of the monitored server>
   
   traceroute source <IP address of the dataplane interface> host <IP address of the monitored server>

less mp-log useridd.log

Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2062): failed to get krb5 tgt ticket with error -1765328228.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2076): failed to get krb5 tgt ticket with error -1765328366.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2062): failed to get krb5 tgt ticket with error 11.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2063): failed to get krb5 tgt ticket with error 145.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2073): failed to get krb5 tgt ticket with error 16.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2074): failed to get krb5 tgt ticket with error -1765328237.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2063): failed to get krb5 tgt ticket with error -1765328378.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2063): failed to get krb5 tgt ticket with error -1765328370.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2063): failed to get krb5 tgt ticket with error -1765328360.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:1982): failed to get krb5 tgt ticket with error -1765328316.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2063): failed to get krb5 tgt ticket with error -1765328353. 

1. If the message seen is similar to:

   Error: pan_user_id_winrm_query(pan_user_id_win.c:2794): Connection failed. response code = 0, error: Couldn't connect to server in vsys 1
   Error: pan_user_id_winrm_query(pan_user_id_win.c:2805): Connection failed. response code = 0, error: Couldn't resolve host name in vsys 1 

   refer to Server Monitoring Status stuck in "Connection Refused (0)" when using WinRM-HTTP/WinRM-HTTPS.
2. If the message seen is similar to:

   Error:  pan_user_id_winrm_query(pan_user_id_win.c:2805): Connection failed. response code = 0, error: Peer certificate cannot be authenticated with given CA certificates in vsys 1

   refer to  Error message: Peer certificate cannot be authenticated with given CA certificates and Server monitoring using WinRM-HTTPS status shows "Connection Refused (0)" after the renewal of the server certificate.
3. If the message seen is similar to:

   Error: pan_user_id_winrm_query(pan_user_id_win.c:2805): Connection failed. response code = 0, error: SSL peer certificate or SSH remote key was not OK in vsys

   refer to Failed to connect to WINRM over https , Unable to get basic constraints.

1. If the message seen is similar to:

   Error:  pan_user_id_winrm_query(pan_user_id_win.c:2805): Connection failed. response code = 500, error: (null) in vsys 1

   follow the instructions in the KB: Dedicated Service Account required Active Directory Security Groups for WinRM Agentless User-ID.
2. If the message seen is similar to:

   Error:  pan_user_id_win_wmic_log_query(pan_user_id_win.c:1670): log query for SPFPL-SRV-AD.shapoorjipallonjifinance.com failed: NTSTATUS: NT_STATUS_ACCESS_DENIED - Access denied

   Ensure that you have properly configured and set a Dedicated Service Account for the User-ID Agent. For Microsoft Window server check if the affected by Windows patch KB5014692 breaks WMI for User-ID.

1. If the message seen is similar to:

   Error:  pan_user_id_winrm_query(pan_user_id_win.c:2794): Connection failed. response code = 401, error: (null) in vsys 1

   If you're using WinRM-HTTP, consider whether it's necessary to switch to HTTPS, which is more secure. If using HTTPS, ensure that a valid certificate is installed and configured on the domain controller for WinRM. Refer to WinRM-HTTP fails with the error 401 and check that the proper configuration steps have been followed as listed in Configure Server Monitoring Using WinRM.

1. Check if the firewall is passive in an A/P HA setup.

   > show high-availability all
   Group 1:
     Mode: Active-Passive
     Local Information:
       Version: 1
       Mode: Active-Passive
       State: passive (last 1 days)  <<<
       Device Information:

   if so then this is expected behavior. After fail-over when passive firewall becomes active then monitored server status should show as Connected.
2. Otherwise check the user-id logs: if the message seen is similar to:

   Error: pan_user_id_win_log_query(pan_user_id_win.c:1349): log query for AD-Server failed: NTSTATUS: NT code 0x80041003 - NT code 0x80041003

   follow the instructions in the KB: Agentless User-ID Connection to Active Directory Server Not Connected.

Error:  pan_user_id_winrm_query(pan_user_id_win.c:2805): Connection failed. response code = 100, error: Timeout was reached in vsys 1

or

Error:  pan_user_id_winrm_query(pan_user_id_win.c:2809): Connection failed. response code = 100, error: Failure when receiving data from the peer in vsys 1 

This may indicate an unstable connection between the firewall and the Windows Server. 

1. Check for network connectivity problems. Use the information mentioned in step 15 of this document about the TCP ports and how to perform a packet capture.
2. Ensure that the username configured in the Server Monitor Account has the correct format. In the UI, navigate to Device > User Identification > User mapping > Palo Alto Networks User-ID Agent Setup > Server Monitor Account
3. Review the Windows server configuration. Consider increasing the maximum timeout value to 300s to allow WinRM sufficient time for the WMI service to respond. Refer to Installation and configuration for Windows Remote Management. 

show user server-monitor state all
UDP Syslog Listener Service is enabled
SSL Syslog Listener Service is enabled
Proxy: Server1(vsys: vsys1) Host: Server1(10.10.10.16)
number of log messages : 0   <<<<
number of auth. success messages : 0
number of active connections : 0
total connections made : 0
Proxy: Server2 UDP(vsys: vsys1) Host: Server2 UDP(10.10.10.15)
number of log messages : 0    <<<<
number of auth. success messages : 0

1. For monitored server connection via firewall management use CLI

   tcpdump filter host <IP address of the monitored server> snaplen 0
   view-pcap mgmt-pcap mgmt.pcap
   

2. For monitored server connection via firewall dataplane use CLI, set a packet capture on the firewall Getting Started: Packet Capture.
3. For information about which TCP service port is used in the connection between the Firewall and the monitored server (found under the Directory Servers) connection refer to below:
   1. For WMI, the initial connection uses TCP/135, but utilizes RPC which gets assigned random ports in the 49152-65535 range
   2. For WinRM-HTTP the port is TCP/5985
   3. For WinRM-HTTPS, the port is TCP/5986
4. For information about which service port is used in the connection between the Firewall and the monitored server (found under the Syslog Servers) connection refer to below:
   1. For UDP Syslog Listener connection the port is UDP/514.
   2. For SSL Syslog Listener connection the port is TCP/6514.