Firewall TCP SYSLOG behaviour

• Firewall was connected to TCP syslog server and the connection was lost. There are 2 scenarios described here:
• Scenario 1 - tcp syslog server is not listening on a tcp port and it is sending tcp-rst for syn packet. Either syslog was disabled or traffic is rejected by the network/server.
  • a the begging show netstat all yes numeric-host yes numeric-port yes | match 514, is showing established and after some time it went to close_wait state,
  • after some time Firewall start  initiate new tcp connection and received reset from the server,
  • as firewall was unable to connect to the server, it will mark it as unreachable and Critical System log will be generated "Syslog server <Server-ip> is being marked unreachable" and it will set backoff duration for 1 min,
  • after 1 min, firewall will try to initiate new connection, if it fail again backoff timer will be doubled
  • after 16 min server will be marked as permanently unreachable and the Critical System log will be generated "Syslog server (10.194.89.128) permanently marked unreachable after 5 retries" and no further retry is done,

2025/08/12 07:15:19 critical general                          general                   0  Syslog server (<Syslog-ip>) permanently marked unreachable after 5 retries
2025/08/12 06:29:15 critical general                          general                   0  Syslog server <Syslog-ip> is being marked unreachable
2025/08/12 05:59:16 critical general                          general                   0  Syslog server <Syslog-ip> is back online

• • logrcv logs (less mp-log logrcv.log) will show:

2025-08-12 06:35:39.562 -0700 Error:  _pan_init_sock(pan_syslog.c:960): server(<Syslog-ip>) marked unreachable for backoff seconds: 240
2025-08-12 06:35:39.562 -0700 create new cache socket:-1 for <Syslog-ip>
2025-08-12 06:42:25.202 -0700 Error:  _pan_init_sock(pan_syslog.c:934): server(<Syslog-ip>) is set to unreachable: Connection refused
2025-08-12 06:42:25.203 -0700 Error:  _pan_init_sock(pan_syslog.c:960): server(<Syslog-ip>) marked unreachable for backoff seconds: 480
2025-08-12 06:42:25.203 -0700 create new cache socket:-1 for <Syslog-ip>
2025-08-12 06:58:01.697 -0700 Error:  _pan_init_sock(pan_syslog.c:934): server(<Syslog-ip>) is set to unreachable: Connection refused
2025-08-12 06:58:01.697 -0700 Error:  _pan_init_sock(pan_syslog.c:960): server(<Syslog-ip>) marked unreachable for backoff seconds: 960
2025-08-12 06:58:01.698 -0700 create new cache socket:-1 for <Syslog-ip>
2025-08-12 07:15:19.924 -0700 Error:  _pan_init_sock(pan_syslog.c:934): server(<Syslog-ip>) is set to unreachable: Connection refused
2025-08-12 07:15:19.925 -0700 Error:  _pan_init_sock(pan_syslog.c:952): Syslog server (<Syslog-ip>) permanently marked unreachable after 5 retries
2025-08-12 07:15:19.925 -0700 Error:  _pan_init_sock(pan_syslog.c:960): server(<Syslog-ip>) marked unreachable for backoff seconds: 960

• • if the syslog become reachable Critical System log will be generate:

2025/08/12 05:59:16 critical general                          general                   0  Syslog server <Syslog-ip> is back online

• Scenario 2 - connection to tcp syslog server is blocked and there is no response for syn packet
  • a the begging show netstat all yes numeric-host yes numeric-port yes | match 514, is showing established and after some time it went to close_wait state,
  • after some time Firewall start  initiate new tcp connection and no response is received,
  • there is no system logs generated,
  • show netstat command will show SYN_SENT

show netstat all yes numeric-host yes numeric-port yes | match 514 was showing SYN_SENT
tcp        0      1 YYYY:51262      <Syslog-ip>:514       SYN_SENT

• • logrcv logs (less mp-log logrcv.log) will show:

2025-08-12 23:26:45.483 -0700 MS: peer watch. sock=27 curtime=140645 recvtime=140583 errcount=1
2025-08-12 23:26:48.281 -0700 Error:  _pan_syslog(pan_syslog.c:1962): TCP send failure: data size: 288, sent: 0, errno: 145
2025-08-12 23:26:49.324 -0700 Error:  _pan_syslog(pan_syslog.c:1957): TCP send failure, socket is broken errno (32)
2025-08-12 23:26:49.504 -0700 create new cache socket:1024 for <SYSLOG-IP>

• PAN-OS 11.1 
• 1 tcp syslog server

• Syslog server is unreachable,
• tcp connection to the server is blocked by network device,
• tcp server is rejecting tcp connections.

• check the path to the server and confirm that traffic is not blocked and server is listening on particular port,
• For scenario 1 - commit on the firewall or restarting logrcv solves the issue,

debug software restart process log-receiver

• For scenario 2 - tcp syslog server should recover automatically 

• How To Troubleshoot Connection Failures To Syslog Servers