How To Troubleshoot Connection Failures To Syslog Servers
74224
Created On 02/10/22 23:16 PM - Last Modified 08/23/23 22:37 PM
Objective
Troubleshoot connection failures to log-forwarding servers (syslog server).
Environment
- Firewall
- Log-forwarding server (syslog server)
Procedure
- From the firewall, check if syslog-ng is running with the CLI
-
admin@PA> debug syslog-ng status syslog-ng (pid 3578 3577) is running...
-
- From the firewall, check if syslog-ng sends out data or drops data using CLI. Check if syslog-ng has connection stats to the server. It is expected to see the network socket information towards the syslog server.
-
> debug syslog-ng stats
-
- On the Firewall, check the Service Route to the Log Collector
Device > Setup > Services > Service Route Configuration > click Customize > Syslog
- If Service Route is set to "Use Management Interface for all" or "Use Default" then from the firewall CLI:
- Check IP connection between firewall and the syslog server.
-
ping host <IP address of syslog server>
If ping is successful then proceed to (b) otherwise check physical layer1 and data link layer2 on your network. - Perform a traceroute check to the syslog server:
traceroute host <IP address of the syslog server>
Similarly perform a traceroute check from the syslog server to the management IP address of the firewall. - Check TCP connection between firewall and the syslog server using this command if TCP port is 514 otherwise replace 514 with corresponding port number.
show netstat numeric-host yes numeric-port yes all yes | match 514
Connection should show established if not then. - Check Permitted IP Address (Device > Setup> Interfaces > click Management > Permitted IP Addresses)
- Perform a tcpdump on the firewall management interface using this command if TCP port is 514 otherwise replace 514 with corresponding port number.
tcpdump filter "port 514" snaplen 0
- Export the tcpdump packet capture to a scp or tftp server and analyze it to root cause the connection issue between firewall and the syslog server.
scp export mgmt-pcap from mgmt.pcap to username@host:path
- Take packet capture on syslog server
- What to look for in tcpdump captures from previous steps
- Look for the completion of the TCP handshake. If the 3way handshake does not complete, then check if an intermediate device could be dropping this traffic.
- If using TLS then check if SSL handshake completed. If the SSL handshake doesn’t complete, then check that the SSL certificate on the Syslog server has not expired.
- If the handshake completes, compare the PCAPs on the two devices to determine which device might not be closing the connection.
- If service route is dataplane interface then from the firewall CLI:
- Check IP connection between firewall dataplane interface and the syslog server.
ping source <IP address of the dataplane interface> host <IP address of syslog server>
If ping is successful then proceed to b otherwise check physical layer1 and data link layer2 on your network. - Perform a traceroute check to the syslog server:
traceroute source <IP address of the dataplane interface> host <IP address of the syslog server>
Similarly perform a traceroute check from the syslog server command line to the IP address of the dataplane of the firewall. - Check TCP connection between firewall and the syslog server by performing a packet capture on the dataplane using GUI.
- Check IP connection between firewall dataplane interface and the syslog server.
Check knowledge base Getting Started: Packet Capture
- Check the session details on the firewall CLI.
show session all filter source <IP address of the dataplane interface> destination <IP address of the syslog server>
session should show active if discarded then check if firewall security policy, nat and routing.
- If above checks are done then check if any firewall or device in your network is blocking this connection.