Palo Alto Networks Knowledgebase: Log Forwarding to Syslog Delayed Troubleshooting

Log Forwarding to Syslog Delayed Troubleshooting

16434
Created On 02/07/19 23:43 PM - Last Updated 02/07/19 23:44 PM
Resolution

Overview

This document describes how to troubleshoot a delayed log received at the syslog server.

 

Issue

The received log times of the syslog have been delayed for an hour or up to 7 days and the customer network environment is stable. Any information in the Palo Alto Networks device can tell the log forward status with the syslog server.

 

Details

There needs to be a determination where the delay occurs:

    1. If the traffic sent from Palo Alto Networks firewall is received immediately by the syslog server, check if the log entries were delayed.
    2. If the log entries are delayed and found in PCAP, perform the following steps:
      1. Determine PA state (DP/MP) whether it has resource issues.
      2. Check log forwarding statistics for syslog. `> debug log-receiver statistics`. Check for syslog enqueue count for unusually high value.
      3. Check related processes are working properly. Restart them if necessary. `> debug software restart process log-receiver`  "Note: missing process" - Sastera
      4. Reduce logging activities and observe any difference.

        > debug log-receiver statistics
        Logging statistics
        ------------------------------ -----------
        Log incoming rate:             0/sec
        Log written rate:              0/sec
        Corrupted packets:             0
        Corrupted URL packets:         0
        Corrupted HTTP HDR packets:    0
        Logs discarded (queue full):   0
        Traffic logs written:          64229
        URL logs written:              0
        Wildfire logs written:         0
        Anti-virus logs written:       0

        Anti-virus logs written: 0
        Spyware logs written:          0
        Attack logs written:           0
        Vulnerability logs written:    0
        Fileext logs written:          2
        URL cache age out count:       0
        URL cache full count:          0
        URL cache key exist count:     0
        URL cache wrt incomplete http hdrs count: 0
        URL cache rcv http hdr before url count: 0
        URL cache full drop count(url log not received): 0
        URL cache age out drop count(url log not received): 0
        Traffic alarms dropped due to sysd write failures: 0
        Traffic alarms dropped due to global rate limiting: 0
        Traffic alarms dropped due to each source rate limiting: 0
        Traffic alarms generated count:  0
        Log Forward count:             0
        Log Forward discarded (queue full) count: 0
        Log Forward discarded (send error) count: 0

        Summary Statistics:
        Num current drop entries in trsum:0
        Num cumulative drop entries in trsum:0
        Num current drop entries in thsum:0
        Num cumulative drop entries in thsum:0
        External Forwarding stats:
        Type         Enqueue Count  Send Count     Drop Count      Queue Depth         Send Rate(last 1min)
        syslog          22158          22158            0              0                        0
        snmp              0              0              0              0                        0
        email             0              0              0              0                        0
        raw               0              0              0              0                        0

    3. If the log entries are not delayed and received immediately from the syslog server PCAP, then check the syslog server.

 

Note: Before proceeding with packet capture at the log server, set a filter to just focus on Palo Alto Networks mgmt IP.

 

Steps for PCAP Comparison

  1. PCAP at Palo Alto Networks firewall, use the following CLI command:
    > tcpdump filter "port 514" snaplen 0

    Press Ctrl-C to stop capturing:
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    ^C12 packets captured
    24 packets received by filter
    0 packets dropped by kernel

    > view-pcap mgmt-pcap mgmt.pcap
    20:35:56.914210 IP 192.168.1.1.53393 > 192.168.1.120.syslog: SYSLOG user.info, length: 411
    20:35:58.914761 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 405
    20:35:58.914910 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 406
    20:35:58.915046 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 404
    20:36:44.918449 IP 192.168.1.1.59424 > 192.168.1.120.syslog: SYSLOG user.info, length: 406
    ...

    > scp export mgmt-pcap from mgmt.pcap to telee@192.168.1.21:.

  2. PCAP at syslog server (Linux in the example below):
    # tcpdump port 514 -s 0 -w pa.pcap
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    ^C13 packets captured
    13 packets received by filter
    0 packets dropped by kernel
    # tcpdump -r pa.pcap
    reading from file pa.pcap, link-type EN10MB (Ethernet)
    20:36:49.550890 IP 192.168.1.1.33231 > 192.168.1.120.syslog: SYSLOG user.info, length: 406
    20:37:23.554831 IP 192.168.1.1.53393 > 192.168.1.120.syslog: SYSLOG user.info, length: 411
    20:37:25.555158 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 405
    20:37:25.555231 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 406
    20:37:25.555653 IP 192.168.1.1.60783 > 192.168.1.120.syslog: SYSLOG user.info, length: 404
    20:38:11.559826 IP 192.168.1.1.59424 > 192.168.1.120.syslog: SYSLOG user.info, length: 406
    ...
    #

  3. Compare the two PCAP files with Wireshark. Match the packet (log times) and observe how much delay in capture times, as shown below:
    Screen Shot 2015-01-11 at 20.45.49.pngScreen Shot 2015-01-11 at 20.43.05.png

 

See also

How To Packet Capture (tcpdump) On Management Interface

 

owner: telee



Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CloWCAS&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments
Choose Language