Syslog forwarding reporting increased drop counters and High send-q values when forwarding to third party syslog servers

• Palo Alto configured with Syslog forwarding towards third party syslog servers
• The resources on Palo Alto are normal with no high MP CPU.
• High logging rate environment
• No drops are seen in the output of CLI command "debug log-receiver statistics"
• High Send-q values in netstats towards the syslog server socket, visible in mp-monitor logs or the output of "show netstat all yes numeric-ports yes numeric yes programs yes | match syslog-ng":

  Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
  
  tcp        0      2205234  IPADDRESS:44159         IPADDRESS:8514    ESTABLISHED 10160/syslog-ng
  

• Drops seen seen in the output of ">debug syslog-ng stats"

  dst.network;dst10#0;tcp,IP-ADDRESS:8514;a;dropped;VALUE HIGHER THAN ZERO

• Palo Alto Firewalls
• PAN-OS versions below 11.1.x
• Syslog Forwarding
• Third party syslog server

• High Send-Q means the data is put on TCP/IP send buffer, but it is not sent or it is sent but not ACKed.
• High value in Send-Q can be related to performance issues on the TCP peer.
• Syslog forwarding is sending logs in real time, if there is a congestion in the network or the remote server, once the buffers are full the logs are dropped and will not reach the syslog server. It will not be resent.
• There is a limited buffer in syslog-ng (around 10k size) also task-q in logrcvr, but once these buffers are full messages will be dropped.

Note: Hint files are not supported for syslog forwarding, it is confirmed that only available for CDL and Panorama forwarding.

1. Increase the third party Syslog server receive buffer double the size or more of the log rate.
2. Create multiple rules in the same log forwarding profile.
   OR
3. Configure multiple log forwarding profiles pointing to the same syslog-server, each log forwarding profiles attached to a different security policy.
4. Ensure correct configuration to avoid duplicate logs on the syslog server.

Note: Starting PAN-OS 11.1, syslog-ng process is no longer used. Refer How To Troubleshoot Connection Failures To Syslog Servers.