How to Resolve Server Monitoring Connection Failures Caused by Kerberos Errors

Resolve server monitoring connection failures caused by Kerberos Errors.

• NGFW
• User-ID
• Server monitoring

1. Review the WinRM configuration over HTTP or HTTPS with Kerberos. Refer to Configure Server Monitoring Using WinRM.

2. If needed, use the following command to test the authentication with the service account: 

   > test authentication authentication-profile <authentication-profile-name> username <username> password

3. Look for the krb5 error code in the useridd.log, use the CLI command:

   less mp-log useridd.log

    and search for its meaning as listed below:
   1. Error -1765328228: Cannot contact any KDC for realm '<>' indicates that the firewall or Panorama (the client) cannot reach the Key Distribution Center (KDC), which is typically the Active Directory Domain controller. This is usually a network connectivity or DNS resolution issue.
      1. Recommendation: Verify that the firewall or Panorama can resolve the domain controller's hostname and communicate over the appropriate ports, typically port 88 for Kerberos. Refer to Kerberos error "-1765328228" observed after configuring with WinRM-HTTP or WinRM-HTTPS for Agentless User-ID.
   2. Error -1765328366: Client's credentials have been revoked means the service account credentials used by the firewall or Panorama for User-ID have been revoked on the Active Directory server. This typically occurs if the account is disabled, locked out, or its Ticket-Granting Ticket (TGT) has been explicitly revoked.
      1. Recommendation: Refer to KDC_ERR_CLIENT_REVOKED in Error: KRB5KDC_ERR_CLIENT_REVOKED (-1765328366): Client's credentials have been revoked. How to identify from the client that a user account has been locked out.
   3. Error 11: Resource temporarily unavailable suggests a temporary issue on the firewall, Panorama, or the network that prevents the Kerberos client from getting the necessary resources to contact the KDC. It has been observed in situations where there's an abrupt interruption of the connection between the firewall or Panorama and the Domain Controller.
      1. Recommendation: Troubleshoot the network connection between the firewall or Panorama and the KDC. Restarting the userid process may resolve the issue. For that, you need to schedule a maintenance window, as this is disruptive. Additionally, refer to the recommendation for a similar problem where the error code is 0, Server Monitoring Status stuck in "Connection Refused (0)" when using WinRM-HTTP/WinRM-HTTPS. 
   4. Error 145: Connection timed out indicates that the firewall's or Panorama's attempt to connect to the KDC timed out. This is typically a network connectivity issue where the firewall or Panorama sends a request to the Domain Controller but does not receive a response within the expected time frame. This could be due to network latency, packet loss, or a firewall or router between the PAN-OS device and the DC blocking the communication.
      1. Recommendation: Troubleshoot the network connection between the firewall or Panorama and the KDC. Refer to the recommendation for a similar issue where the error code is 0, Server Monitoring Status stuck in "Connection Refused (0)" when using WinRM-HTTP/WinRM-HTTPS. 
   5. Error 16: Device or resource busy suggests that a required resource for the Kerberos authentication process was temporarily unavailable. 
      1. Recommendation: This can be caused by the Kerberos client on the firewall or Panorama being unable to acquire a necessary lock to perform ticket-related operations. Since this could be related to configuration, review the configuration as mentioned in step 1 of this article.
   6. Error -1765328237: KDC reply did not match expectations means that the firewall received a reply from the KDC, but it was not in the expected format or was otherwise malformed. Because the firewall or Panorama cannot process the unexpected reply, it fails to acquire the TGT. 
      1. Recommendation: Refer to Server Monitoring status shows "Kerberos error" when using WinRM-HTTP Transport Protocol.
   7. Error -1765328378: Client '<>' not found in Kerberos database indicates that the username of the service account configured on the firewall for User-ID does not exist in the Active Directory (Kerberos database).
      1. Recommendation: Ensure that the correct username is configured in the Server Monitoring settings on the firewall or Panorama. Refer to KDC_ERR_C_PRINCIPAL_UNKNOWN Returned in S4U2Self Request.
   8. Error -1765328370: KDC has no support for encryption type.
      1. Recommendation: Refer to Cannot join with service account after enabling a group policy to disable RC4 and enable AES128 and 256. KRB5KDC_ERR_ETYPE_NOSUPP (-1765328370): KDC has no support for encryption type/ (4295213).
   9. Error -1765328360: Preauthentication failed.
      1. The authentication server (AS) in KDC states that the authentication has failed and closes the connection, so the authentication process is not completed. The problem is on the Authentication server side. Checking the box option of “do not require Kerberos preauthentication” in the AD may solve the issue.
      2. Recommendation: Validate your configuration on the AD side using: Configure Server Monitoring Using WinRM.
   10. Error -1765328316: Realm not local to KDC. 
       1. Recommendation: Refer to Server monitoring connection status "Kerberos error" -1765328316: Realm not local to KDC.
   11. Error -1765328353: Decrypt integrity check failed.
       1. Recommendation:
          1. Set up User-ID with WinRM over HTTPS. Import the Domain Controller certificate, then check if the connection is established. Validate the configuration using Configure Server Monitoring Using WinRM.
          2. This may mean that there is a mismatch in the encryption types supported by the firewall or Panorama and the KDC. Ensure that both the service account and the KDC are configured to use matching encryption types.

In Kerberos, KDC stands for Key Distribution Center. It’s a centralized authentication service that issues and manages the cryptographic keys used in the Kerberos authentication process. The KDC has two main components:

1. Authentication Service (AS):

   1. Verifies a user’s identity (for example, using a username and password).

   2. Issues a Ticket-Granting Ticket (TGT) if the credentials are valid.

2. Ticket Granting Service (TGS):

   1. Uses the TGT to issue service tickets that allow access to specific network services without re-entering credentials.

The KDC is like a trusted authority that gives out “passes” (tickets) proving that a user or system is who they claim to be, so they can securely access other services within the network.