Log Forwarding to Syslog Delayed Troubleshooting
84505
Created On 09/26/18 13:47 PM - Last Modified 11/12/19 22:06 PM
Symptom
The received log times of the syslog have been delayed for an hour or up to 7 days and the customer network environment is stable. Any information in the Palo Alto Networks device can tell the log forward status with the syslog server.
This document describes how to troubleshoot a delayed log received at the syslog server.
Resolution
There needs to be a determination where the delay occurs:
- If the traffic sent from Palo Alto Networks firewall is received immediately by the syslog server, check if the log entries were delayed.
- If the log entries are delayed and found in PCAP, perform the following steps:
- Determine PA state (DP/MP) whether it has resource issues.
- Check log forwarding statistics for syslog. `> debug log-receiver statistics`. Check for syslog enqueue count for unusually high value.
- Check related processes are working properly. Restart them if necessary. `> debug software restart process log-receiver` "Note: missing process" - Sastera
- Reduce logging activities and observe any difference.
> debug log-receiver statistics
Logging statistics
------------------------------ -----------
Log incoming rate: 0/sec
Log written rate: 0/sec
Corrupted packets: 0
Corrupted URL packets: 0
Corrupted HTTP HDR packets: 0
Logs discarded (queue full): 0
Traffic logs written: 64229
URL logs written: 0
Wildfire logs written: 0
Anti-virus logs written: 0
Anti-virus logs written: 0
Spyware logs written: 0
Attack logs written: 0
Vulnerability logs written: 0
Fileext logs written: 2
URL cache age out count: 0
URL cache full count: 0
URL cache key exist count: 0
URL cache wrt incomplete http hdrs count: 0
URL cache rcv http hdr before url count: 0
URL cache full drop count(url log not received): 0
URL cache age out drop count(url log not received): 0
Traffic alarms dropped due to sysd write failures: 0
Traffic alarms dropped due to global rate limiting: 0
Traffic alarms dropped due to each source rate limiting: 0
Traffic alarms generated count: 0
Log Forward count: 0
Log Forward discarded (queue full) count: 0
Log Forward discarded (send error) count: 0
Summary Statistics:
Num current drop entries in trsum:0
Num cumulative drop entries in trsum:0
Num current drop entries in thsum:0
Num cumulative drop entries in thsum:0
External Forwarding stats:
Type Enqueue Count Send Count Drop Count Queue Depth Send Rate(last 1min)
syslog 22158 22158 0 0 0
snmp 0 0 0 0 0
email 0 0 0 0 0
raw 0 0 0 0 0
- If the log entries are not delayed and received immediately from the syslog server PCAP, then check the syslog server.
Note: Before proceeding with packet capture at the log server, set a filter to just focus on Palo Alto Networks mgmt IP.
Steps for PCAP Comparison
- PCAP at Palo Alto Networks firewall, use the following CLI command:
> tcpdump filter "port 514" snaplen 0
Press Ctrl-C to stop capturing:
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C12 packets captured
24 packets received by filter
0 packets dropped by kernel
> view-pcap mgmt-pcap mgmt.pcap
20:35:56.914210 IP 192.168.1.1.53393 > 192.168.1.120.syslog: SYSLOG user.info, length: 411
20:35:58.914761 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 405
20:35:58.914910 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 406
20:35:58.915046 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 404
20:36:44.918449 IP 192.168.1.1.59424 > 192.168.1.120.syslog: SYSLOG user.info, length: 406
...
> scp export mgmt-pcap from mgmt.pcap to telee@192.168.1.21:.
- PCAP at syslog server (Linux in the example below):
# tcpdump port 514 -s 0 -w pa.pcap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C13 packets captured
13 packets received by filter
0 packets dropped by kernel
# tcpdump -r pa.pcap
reading from file pa.pcap, link-type EN10MB (Ethernet)
20:36:49.550890 IP 192.168.1.1.33231 > 192.168.1.120.syslog: SYSLOG user.info, length: 406
20:37:23.554831 IP 192.168.1.1.53393 > 192.168.1.120.syslog: SYSLOG user.info, length: 411
20:37:25.555158 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 405
20:37:25.555231 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 406
20:37:25.555653 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 404
20:38:11.559826 IP 192.168.1.1.59424 > 192.168.1.120.syslog: SYSLOG user.info, length: 406
...
#
- Compare the two PCAP files with Wireshark. Match the packet (log times) and observe how much delay in capture times, as shown below:
Additional Information
See also How To Packet Capture (tcpdump) On Management Interface