Syslog traffic are dropped despite of low log rate
Symptom
- Some syslog traffic from firewall or log collector is not received on syslog server
- Following error messages are seen in log file (logrcvr.log on firewall and logd.log on log collector) which means the syslog forwarding task queue in logrcvr (or logd) is full and needs to be increased or adjusted.
Error: pan_logforward_enqueue_new(pan_logforward.c:2794): LOGFWD: enqueue task to syslog taskq(q_depth 0), log dropped
Other potential indicators
> debug log-collector log-collection-stats show incoming-logs | match Incoming is low
> debug log-collector log-collection-stats show incoming-logs | match "syslog dropped count:" is high
> potentially reports run on the syslog server log lower log rate than customer expects for their environment
Environment
- PAN-OS 11.1 or greater
- Firewall
- Log Collector (Panorama)
Cause
With PAN-OS 11.1 and greater, PAN-OS uses logrcvr (firewall) or logd (Panorama/LC) to send syslog traffic, where previously syslog-ng was used, and the default buffer starts at 16384 (16k) where fine-tuning may be necessary for any given implementation or platform.
Resolution
The syslog forwarding task queue can be updated to 32k, 64k, 128k, 256k, 512k up to 999999
> debug log-receiver param-tuning task-queue size ?
<value> <2048-999999> set task queue size (2048 - 999999)
Example: To update syslog forwarding task queue to 32k (32768 bytes), please use the following procedure (log-receiver process restart is required to update the task queue parameter):
> debug log-receiver param-tuning task-queue size 32768
> debug software restart process log-receiver
To verify the updated task queue size, the following command can be used
> debug log-receiver param-tuning task-queue show
32768