How To Troubleshoot Connection Failures To Syslog Servers

147670

Created On 02/10/22 23:16 PM - Last Modified 10/21/25 18:45 PM

Log Forwarding

Reporting and Logging

10.0

10.1

Strata Cloud Manager

Objective

Troubleshoot connection failures to log-forwarding servers (syslog server) in PAN-OS versions

Note: Starting from PAN-OS 11.1.x the syslog-ng is deprecated and its role transitioned to logrcvr for firewall and logd for Panorama.

Environment

Palo Alto Firewall
Log-forwarding server (syslog server)

Procedure

Check whether the process responsible for forwarding logs to the syslog server is running on the firewall:
1. for PAN-OS pre-11.1, check the syslog-ng. Use the CLI command:
```
> debug syslog-ng status
syslog-ng (pid 3578 3577) is running...
```
2. for PAN-OS 11.1 and post-11.1, check the logrcvr. Use the CLI command:
```
> show system software status | match logrcvr
Process  logrcvr                        running  (pid: 7537)
```

Verify that the process is sending out the data and if there are any drops:

for PAN-OS pre-11.1, check if syslog-ng has connection stats to the server. It is expected to see the network socket information towards the syslog server.

> debug syslog-ng stats

SourceName;   SourceId;    SourceInstance;   State;   Type;   Number

destination;dst30;;a;processed;112
source;src_sctp;;a;processed;0
dst.network;    dst30#0;    tcp,10.3.232.112:514;    a;    dropped;    0
dst.network;dst30#0;tcp,10.3.232.112:514;a;processed;113

for PAN-OS 11.1 and post-11.1, use the CLI command:

> show syslog-ssl-conn-validation
syslogng ssl connection validation settings:
all-conns: enforce
crl: skip
ocsp: skip
eku: skip

> show system state filter sw.logrcvr.syslog*

sw.logrcvr.syslog_avg_send_rate: 448
sw.logrcvr.syslog_drop_count: 0
sw.logrcvr.syslog_enqueue_count: 8187
sw.logrcvr.syslog_queue_depth: 0
sw.logrcvr.syslog_sent_count: 8187

For PAN-OS pre-11.1, 11.1 and post-11.1, use the CLI command to check for syslog forwarding log drops :

> debug log-receiver statistics
External Forwarding stats:
Type Enqueue Count Send Count Drop Count Queue Depth Send Rate(last 1min)
syslog 4152556162 4151059521 1850057 0 110737

On the Firewall, check the Service Route to the Log Collector

Device > Setup > Services > Service Route Configuration > click Customize > Syslog

If Service Route is set to "Use Management Interface for all" or "Use Default" then from the firewall CLI:
1. Check IP connection between firewall and the syslog server.
2. ```
ping host <IP address of syslog server>
```
  If ping is successful then proceed to (b) otherwise check physical layer1 and data link layer2 on your network.
3. Perform a traceroute check to the syslog server:
```
traceroute host <IP address of the syslog server>
```
```
Similarly perform a traceroute check from the syslog server to the management IP address of the firewall.
```
4. Check TCP connection between firewall and the syslog server using this command if TCP port is 514 otherwise replace 514 with corresponding port number.
```
show netstat numeric-host yes numeric-port yes all yes | match 514
```
  Connection should show established if not then.
5. Check Permitted IP Address (Device > Setup> Interfaces > click Management > Permitted IP Addresses)
6. Perform a tcpdump on the firewall management interface using this command if TCP port is 514 otherwise replace 514 with corresponding port number.
```
tcpdump filter "port 514" snaplen 0
```
7. Export the tcpdump packet capture to a scp or tftp server and analyze it to root cause the connection issue between firewall and the syslog server.
```
scp export mgmt-pcap from mgmt.pcap to username@host:path
```
8. Take packet capture on syslog server

What to look for in tcpdump captures from previous steps:
1. Look for the completion of the TCP handshake. If the 3way handshake does not complete, then check if an intermediate device could be dropping this traffic.
2. If using TLS then check if SSL handshake completed. If the SSL handshake doesn’t complete, then check that the SSL certificate on the Syslog server has not expired.
3. If the handshake completes, compare the PCAPs on the two devices to determine which device might not be closing the connection.

If service route is dataplane interface then from the firewall CLI:

Check IP connection between firewall dataplane interface and the syslog server.

ping source <IP address of the dataplane interface> host <IP address of syslog server>

If ping is successful then proceed to b otherwise check physical layer1 and data link layer2 on your network.

Perform a traceroute check to the syslog server:

traceroute source <IP address of the dataplane interface> host <IP address of the syslog server>

Similarly perform a traceroute check from the syslog server command line to the IP address of the dataplane of the firewall.

Check TCP connection between firewall and the syslog server by performing a packet capture on the dataplane using GUI.

Check knowledge base Getting Started: Packet Capture

Check the session details on the firewall CLI.

show session all filter source <IP address of the dataplane interface> destination <IP address of the syslog server>

session should show active if discarded then check if firewall security policy, nat and routing.

If above checks are done then check if any firewall or device in your network is blocking this connection.

Additional Information

For Panorama, starting with version 11.1, logd is the process responsible for maintaining the connection to the syslog server and forwarding logs to it. The CLI command to check if logd is running on Panorama is:

> show system software status | match logd
Process  logd               running  (pid: 3970)

Use the CLI command to check the syslog forwarding log drops:

> show system state filter sw.logd.syslog*

sw.logd.syslog_drop_count: 0
sw.logd.syslog_enqueue_count: 160240
sw.logd.syslog_queue_depth: 0
sw.logd.syslog_sent_count: 160240

and the CLI command:

> debug log-collector log-collection-stats show log-forwarding-stats

syslog enqueued count: 902112620
syslog sent count: 902112620
syslog dropped count: 0
syslog Queue depth:

In PAN-OS 12.2.x and later, a new command, debug log-receiver syslog-connections, is available to troubleshoot syslog connections.
The command will also be backported in the PAN-OS 12.1.5 when released.
This command adds a Status column, which lists one of the four possible states for each syslog connection.

created / in handshake: Initial states for UDP/TCP and SSL sockets, respectively.
ready: The socket is prepared to transmit log data.
error: An issue occurred while attempting to write to the socket.
closed: The connection is in a state other than the above.

Example output of the command below

admin@PA-VM(active)> debug log-receiver syslog-connections

Syslog Server     Port       Status   Protocol   Socket        Logs Sent     Logs Dropped
  0.0.0.0      514       closed        TCP     1024                0               0
10.4.0.30      514        ready        UDP     1026                2               0

How To Troubleshoot Connection Failures To Syslog Servers

Objective

Environment

Procedure

Additional Information

Other users also viewed: