Delay in logs sent from CDL to syslog server
12056
Created On 03/20/23 18:48 PM - Last Modified 06/13/23 17:49 PM
Question
Why my Syslog Server is not receiving the latest logs?
Environment
Sender: Cortex Data Lake
Receiver: Any Syslog Receivers as syslog-ng, Splunk, QRadar, Sentinel among others.
Answer
Cortex Data Lake (CDL) will start sending available logs to the Syslog Receiver configured in the Log Forwarding Profile once you complete to configure the filters and save the profile.
Sometimes you could see that the logs received in the Syslog Server might not be the latest ones, getting logs from previous hours or even days.
This is usually related to some sort of limitation on the Syslog Server that prevents it to handle the volume of logs CDL is sending.
Some common causes are:
- Max Log Rate: Depending on the vendor / application you're using on your server, there are some specs related to the maximum log rate the server can handle, so you need to check and configure the parameters on your server to accept a batch size of 500 logs, or 2.25 MB (500 logs x 4500B/log = 2.25MB). This does not apply to Google Chronicle, which accepts only 1 MB of data at a time, or a batch size of 250 logs.
- Server performance: You need to check the resources available on the server (CPU, RAM, disk, etc.) and confirm if any element could be causing performance issues.
- Server unable to handle the traffic: Often times, the server doesn't show any outstanding value from resource consumption, even though it's not able to handle the traffic. You can verify if this is your case by checking the status of the Recv-Q on your server; this can be done by running netstat command as in the example below:
> netstat -an | grep 6415
The output from the command given above is similar to the one shown below:
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 3223 0 11.10.32.24.8002 11.10.32.12.64672 ESTABLISHED
The value for the Recv-Q should be 0, otherwise, your server is buffering traffic in the Recv-Q which means that your server is not able to handle all the traffic received. If the buffer for the Recv-Q gets exhausted, the syslog server will cause the Send-Q on Cortex Data Lake side to increase and eventually to stop sending traffic until the Recv-Q on syslog receiver server is cleared.