How To Troubleshoot Connection Failure To LDAP Servers

• Firewall
• LDAP server

1. Check the firewall system logs for the following event-id "auth-server-down" this can be done from UI under Monitor > Logs > System  with following filter ( eventid eq auth-server-down ) or from CLI using the following command:

> show log system direction equal backward eventid equal "auth-server-down"

2. From the firewall CLI, check if the number of received authentication requests to authenticate against the LDAP server is equal or less then the received responses:

> debug authentication connection-show protocol-type LDAP

3. Test the authentication against the LDAP server using the command:

    > test authentication authentication-profile ldap-auth-profile username user-id password
   Enter password :
   

   In case that the firewall is not able to contact the LDAP server because of connection failure between the firewall and the LDAP server then the following message would show as a result of test command issued in step 3: 

   Do allow list check before sending out authentication request...
   name "user-id" is in group "all"
   
   Authentication to LDAP server at 10.16.0.14 for user "user-id"
   Egress: 10.10.168.130
   Type of authentication: plaintext
   Starting LDAP connection...
   Failed to create a session with LDAP server 
   Authentication failed against LDAP server at 10.16.0.14:389 for user "user-id"
   
   
   Authentication failed for user "user-id"
   

4. After completing step 3 check the authd logs and look for the following message "Can't contact LDAP server" since that message would show up if the firewall lost connection to the LDAP server used in an authentication profile and if a user attempts to authenticate against the LDAP server as seen in below example:

   tail mp-log authd.log
   
   2022-07-14 11:55:59.619 -0700 Error: pan_authd_ldap_bind(pan_authd_shared_ldap.c:646): Failed to bind ldap (Can't contact LDAP server)
   2022-07-14 11:55:59.620 -0700 Error: pan_auth_create_a_ldap_session(pan_auth_svr_cctxt.c:2038): Failed to bind, get out
   2022-07-14 11:55:59.620 -0700 Error: _recreate_a_ldap_session(pan_auth_service_handle.c:538): failed to re-create 0th LDAP session for server: 10.16.0.14:389
   2022-07-14 11:55:59.620 -0700 LDAP auth server 10.16.0.14 is down !!!
   
   2022-07-14 11:55:59.620 -0700 debug: auth_svr_set_flag_retry_interval(pan_auth_svr.c:746): set retry-interval flag to "true" at Thu Jul 14 11:55:59 2022
   2022-07-14 11:55:59.620 -0700 debug: _start_sync_auth(pan_auth_service_handle.c:617): _report_server_not_avail() succeeded
   2022-07-14 11:55:59.620 -0700 debug: pan_auth_response_process(pan_auth_state_engine.c:4322): auth status: auth server not available
   2022-07-14 11:55:59.620 -0700 debug: pan_auth_response_process(pan_auth_state_engine.c:4562): Auth FAILED for user "user-id" thru <"ldap-auth-profile", "shared">: remote server 10.16.0.14 of server profile "LDAP-NEW" is down, or in retry interval, or request timed out (elapsed time 30 secs, max allowed 60 secs)
   2022-07-14 11:55:59.620 -0700 debug: pan_auth_response_process(pan_auth_state_engine.c:4601): Authentication failed: <profile: "ldap-auth-profile", vsys: "shared", username "user-id">

   Now that you know that the problem is that the firewall is not able to establish a connection to the LDAP server use the below steps to further troubleshoot the problem.
5. On the Firewall, check the Service Route to the LDAP server Device > Setup > Services > Service Route Configuration > click Customize > LDAP:

1. Check IP connection between firewall and the LDAP server.

   > ping host <IP address of LDAP server>

   If ping is successful then proceed to (b) otherwise check physical layer1 and data link layer2 on your network.
2. Perform a traceroute check to the LDAP server:

   > traceroute host <IP address of the LDAP server>

   Similarly perform a traceroute check from the LDAP to the management IP address of the firewall.
3. Check Permitted IP Address (Device > Setup> Interfaces > click Management > Permitted IP Addresses)
4. Perform a tcpdump on the firewall management interface using this command if TCP port is 389 otherwise replace 389 with corresponding port number (Note for TLS the port used is 636):

   > tcpdump filter "port 389" snaplen 0

5. Export the tcpdump packet capture to a scp or tftp server and analyze it to root cause the connection issue between firewall and the LDAP server.

   > scp export mgmt-pcap from mgmt.pcap to username@host:path

   Take packet capture on the LDAP server
   What to look for in tcpdump captures from previous steps
   Look for the completion of the TCP handshake. If the 3way handshake does not complete, then check if an intermediate device could be dropping this traffic.
   If using TLS then check if SSL handshake completed. If the SSL handshake doesn’t complete, then check that the SSL certificate on the LDAP server has not expired.
   If the handshake completes, compare the PCAPs on the two devices to determine which device might be resetting/closing the connection. 

4. If service route is dataplane interface then from the firewall CLI:
   1. Check IP connection between firewall dataplane interface and the LDAP server.

      > ping source <IP address of the dataplane interface> host <IP address of LDAP server>

      If ping is successful then proceed to b otherwise check physical layer1 and data link layer2 on your network.
   2. Perform a traceroute check to the LDAP server:

      > traceroute source <IP address of the dataplane interface> host <IP address of the LDAP server>

      Similarly perform a traceroute check from the LDAP server command line to the IP address of the dataplane of the firewall.
   3. Check TCP connection between firewall and the LDAP server by performing a packet capture on the dataplane using GUI. Check knowledge base Getting Started: Packet Capture
   4. Check the session details on the firewall CLI.

      > show session all filter source <IP address of the dataplane interface> destination <IP address of the LDAP server>

      session should show active if discarded then check if firewall security policy, nat and routing.
   5. If above checks are done then check if any firewall or device in your network is blocking this connection.

2022-07-15 14:10:54.852 -0700 ldap uri: ldap://10.16.0.14:389
2022-07-15 14:10:54.852 -0700 debug: pan_authd_ldap_init_set(pan_authd_shared_ldap.c:982): set source route into ldap option: 10.16.0.1
2022-07-15 14:10:54.852 -0700 Succeed to init LDAPp=0x564adee38270 for entry 0
2022-07-15 14:10:54.852 -0700 b_ssl: No
2022-07-15 14:10:54.852 -0700 debug: pan_authd_ldap_bind(pan_authd_shared_ldap.c:637): binding with binddn user-id@domain.com
2022-07-15 14:10:54.857 -0700 Error:  _parse_ldap_bind_result(pan_authd_shared_ldap.c:282): bind failed (extracted from parsed bind result) (code: 49) (string: Invalid credentials) (additional info: 80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580)
2022-07-15 14:10:54.857 -0700 Error:  _parse_ldap_bind_result(pan_authd_shared_ldap.c:286): wrong password was provided
2022-07-15 14:10:54.857 -0700 Error:  pan_authd_ldap_bind(pan_authd_shared_ldap.c:659): _parse_ldap_bind_result() for DN user-id@domain.com
2022-07-15 14:10:54.857 -0700 Error:  pan_auth_create_a_ldap_session(pan_auth_svr_cctxt.c:2038): Failed to bind, get out
2022-07-15 14:10:54.857 -0700 Error:  _recreate_a_ldap_session(pan_auth_service_handle.c:538): failed to re-create 0th LDAP session for server: 172.16.0.14:389
2022-07-15 14:10:54.857 -0700 LDAP auth server 10.16.0.14 is down !!!