How to troubleshoot connection failure to the monitored server for PAN-OS integrated User-ID Agent

2056

Created On 02/08/24 18:10 PM - Last Modified 03/11/24 16:11 PM

Alerts

User-ID agent

User Mapping

User-ID

10.0

10.2

10.1

11.1

11.0

PAN-OS

AIOps for NGFW

Next-Generation Firewall

Strata Cloud Manager

Objective

Troubleshooting connection failure between firewall and the monitored server.

Environment

Palo Alto Firewall
PAN-OS integrated User-ID Agent / Agentless User-ID
Server-monitor host / monitored server

Procedure

Determine which monitored server is disconnected:
```
show user server-monitor statistics
```
Use UI Device > User Identification > User Mapping and check the Status column in the Server Monitoring window.
Check further details with regard to the disconnected monitored server:
```
show user server-monitor state <server-name>
```
Check the service route from the firewall to the monitored server: Device > Setup > Services > Service Route Configuration > UID Agent.
If the management interface (aka Default ) is configured as UID Agent service route, and if permitted IP addresses are configured under Device > Setup > Interface > Management, ensure that the monitored server IP address is included in that list.
If the dataplane interface is configured as UID Agent service route and if an interface management profile is configured for that interface with permitted IP addresses, ensure that the monitored server IP address is included in that list.
Ensure that the username and password for the service account, which the User-ID agent will use to access the monitored servers, are up to date. ( This step is applicable when monitoring Exchange servers and domain controllers).
- Under: Device > User Identification > User Mapping then click on the gear icon of the Palo Alto Networks User-ID Agent Setup
- Go to the Server Monitor Account tab.
- For more details, Refer Step 5 in Configure User Mapping Using the PAN-OS Integrated User-ID Agent.
The status of the monitored server found in the output of the command under the Directory Servers in a failure status can be:
If the monitored server status is showing as Connection timeout then check the network connection to the server.
1. Check if the firewall is able to reach the server this can be checked by pinging the server or using traceroute. If the service route to the monitored server is the mgmt interface, use CLI command:
```
ping host <IP address of the monitored server>
traceroute host <IP address of the monitored server>
```
  If the service rote to the monitored server is the dataplane interface, use CLI command:
```
ping source <IP address of the dataplane interface> host <IP address of the monitored server>

traceroute source <IP address of the dataplane interface> host <IP address of the monitored server>
```

If the monitored server status is showing as Kerberos Error then read the user-id log

less mp-log user-id.log

and search for the Kerberos Error Codes. Some errors seen in the field are listed in the additional Information section with their meaning and proper remediation steps:

Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2062): failed to get krb5 tgt ticket with error -1765328228.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2076): failed to get krb5 tgt ticket with error -1765328366.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2062): failed to get krb5 tgt ticket with error 11.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2063): failed to get krb5 tgt ticket with error 145.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2073): failed to get krb5 tgt ticket with error 16.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2074): failed to get krb5 tgt ticket with error -1765328237.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2063): failed to get krb5 tgt ticket with error -1765328378.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2063): failed to get krb5 tgt ticket with error -1765328370.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2063): failed to get krb5 tgt ticket with error -1765328360.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:1982): failed to get krb5 tgt ticket with error -1765328316.
Error:  pan_user_id_krb5_init_ticket(pan_user_id_win.c:2063): failed to get krb5 tgt ticket with error -1765328353.

If the monitored server status is showing as Connection refused then read the user-id logs:
1. If the message seen is similar to:
```
Error: pan_user_id_winrm_query(pan_user_id_win.c:2794): Connection failed. response code = 0, error: Couldn't connect to server in vsys 1
Error: pan_user_id_winrm_query(pan_user_id_win.c:2805): Connection failed. response code = 0, error: Couldn't resolve host name in vsys 1 
```
  refer to Server Monitoring Status stuck in "Connection Refused (0)" when using WinRM-HTTP/WinRM-HTTPS.
2. If the message seen is similar to:
```
Error:  pan_user_id_winrm_query(pan_user_id_win.c:2805): Connection failed. response code = 0, error: Peer certificate cannot be authenticated with given CA certificates in vsys 1
```
  refer to Error message: Peer certificate cannot be authenticated with given CA certificates and Server monitoring using WinRM-HTTPS status shows "Connection Refused (0)" after the renewal of the server certificate.
3. If the message seen is similar to:
```
Error: pan_user_id_winrm_query(pan_user_id_win.c:2805): Connection failed. response code = 0, error: SSL peer certificate or SSH remote key was not OK in vsys
```
  refer to Failed to connect to WINRM over https , Unable to get basic constraints.
4. If the message seen is similar to:
```
Error:  pan_user_id_winrm_query(pan_user_id_win.c:2794): Connection failed. response code = 401, error: (null) in vsys 1
```
  If you're using WinRM-HTTP, consider whether it's necessary to switch to HTTPS, which is more secure. If using HTTPS, ensure that a valid certificate is installed and configured on the domain controller for WinRM. Refer to WinRM-HTTP fails with the error 401 and check that the proper configuration steps have been followed as listed in Configure Server Monitoring Using WinRM.
If the monitored server status is showing as Access Denied then read the user-id logs:
1. If the message seen is similar to:
```
Error:  pan_user_id_winrm_query(pan_user_id_win.c:2805): Connection failed. response code = 500, error: (null) in vsys 1
```
  follow the instructions in the KB: Dedicated Service Account required Active Directory Security Groups for WinRM Agentless User-ID.
2. If the message seen is similar to:
```
Error:  pan_user_id_win_wmic_log_query(pan_user_id_win.c:1670): log query for SPFPL-SRV-AD.shapoorjipallonjifinance.com failed: NTSTATUS: NT_STATUS_ACCESS_DENIED - Access denied
```
  Ensure that you have properly configured and set a Dedicated Service Account for the User-ID Agent. For Microsoft Window server check if the affected by Windows patch KB5014692 breaks WMI for User-ID.
If the monitored server status is showing as Not Connected then:
1. Check if the firewall is passive in an A/P HA setup.
```
> show high-availability all
Group 1:
  Mode: Active-Passive
  Local Information:
    Version: 1
    Mode: Active-Passive
    State: passive (last 1 days)  <<<
    Device Information:
```
  if so then this is expected behavior. After fail-over when passive firewall becomes active then monitored server status should show as Connected.
2. Otherwise check the user-id logs: if the message seen is similar to:
```
Error: pan_user_id_win_log_query(pan_user_id_win.c:1349): log query for AD-Server failed: NTSTATUS: NT code 0x80041003 - NT code 0x80041003
```
  follow the instructions in the KB: Agentless User-ID Connection to Active Directory Server Not Connected.

The status of the monitored server found in the output of the command under the Syslog Servers will show as Connected or Not connected for Syslog Listener configured to use SSL and will show as N/A for Syslog Listener configured to use UDP port. To check whether the firewall is receiving log messages from the Syslog server use the CLI command:

show user server-monitor state all
UDP Syslog Listener Service is enabled
SSL Syslog Listener Service is enabled
Proxy: Server1(vsys: vsys1) Host: Server1(10.10.10.16)
number of log messages : 0   <<<<
number of auth. success messages : 0
number of active connections : 0
total connections made : 0
Proxy: Server2 UDP(vsys: vsys1) Host: Server2 UDP(10.10.10.15)
number of log messages : 0    <<<<
number of auth. success messages : 0

In the output above the Server1 uses SSL connection and Server2 uses UDP connection
The number of log messages is 0 which indicates that the firewall is not receiving log messages from the Syslog Listener.
The output of the command above will also help to verify whether "User-ID Syslog Listener-SSL" or "User-ID Syslog Listener-UDP" has been enabled on the interface that the firewall uses to collect user mappings.
Syslog listener can be configured by using the guide on How to Configures a Custom Syslog Sender and Test User Mappings and the guide Configure the PAN-OS Integrated User-ID Agent as a Syslog Listener .

If none of the above helps in isolating the issue then collect a packet capture between the firewall and the monitored server then open a support case.
1. For monitored server connection via firewall management use CLI
```
tcpdump filter host <IP address of the monitored server> snaplen 0
view-pcap mgmt-pcap mgmt.pcap
```
2. For monitored server connection via firewall dataplane use CLI, set a packet capture on the firewall Getting Started: Packet Capture.
3. For information about which TCP service port is used in the connection between the Firewall and the monitored server (found under the Directory Servers) connection refer to below:
  1. For WMI, the initial connection uses TCP/135, but utilizes RPC which gets assigned random ports in the 49152-65535 range
  2. For WinRM-HTTP the port is TCP/5985
  3. For WinRM-HTTPS, the port is TCP/5986
4. For information about which service port is used in the connection between the Firewall and the monitored server (found under the Syslog Servers) connection refer to below:
  1. For UDP Syslog Listener connection the port is UDP/514.
  2. For SSL Syslog Listener connection the port is TCP/6514.

Additional Information

error -1765328228: Cannot contact any KDC for realm '<>'. Refer to Kerberos error "-1765328228" observed after configuring with WinRM-HTTP or WinRM-HTTPS for Agentless User-ID .

error -1765328366: Client's credentials have been revoked. Refer to KDC_ERR_CLIENT_REVOKED.

error 11: Resource temporarily unavailable. Follow similar recommendation as for error code 0 Server Monitoring Status stuck in "Connection Refused (0)" when using WinRM-HTTP/WinRM-HTTPS.

error 145: Connection timed out. Follow similar recommendation as for error code 0 Server Monitoring Status stuck in "Connection Refused (0)" when using WinRM-HTTP/WinRM-HTTPS.

error 16: Device or resource busy. Refer to configuration Configure Server Monitoring Using WinR. If needed use the following command to test the authentication with the service account: test authentication authentication-profile <authentication-profile-name> username <username> password

error -1765328237: KDC reply did not match expectations. Refer to Server Monitoring status shows "Kerberos error" when using WinRM-HTTP Transport Protocol.

error -1765328378: Client '<>' not found in Kerberos database. Refer to doc.

error -1765328370: KDC has no support for encryption type. Follow the below article on "Cannot join with service account after enabling a group policy to disable RC4 and enable AES128 and 256. KRB5KDC_ERR_ETYPE_NOSUPP (-1765328370): KDC has no support for encryption type/ (4295213)" refer to doc.

error -1765328360: Preauthentication failed. Authentication server (AS) on KDC states that the authentication has failed and closes the connection so the authentication process is not completed problem is on the Authentication server side. Check Mark the “do not require kerberos preauthentication” in the AD solved the issue. Validate your configuration on the AD side using: Configure Server Monitoring Using WinRM .

error -1765328316: Realm not local to KDC. Refer to Server monitoring connection status "Kerberos error"

error -1765328353: Decrypt integrity check failed. If you're using WinRM-HTTP, consider whether it's necessary to switch to HTTPS, which is more secure. If using HTTPS, ensure that a valid certificate is installed and configured on the domain controller for WinRM. Refer to Configure Server Monitoring Using WinRM.

How to troubleshoot connection failure to the monitored server for PAN-OS integrated User-ID Agent

Objective

Environment

Procedure

Additional Information

Other users also viewed: