Log forwarding over SSL fails due to CRL check
1603
Created On 06/10/25 06:58 AM - Last Modified 08/16/25 02:56 AM
Symptom
- No logs seen on the SYSLOG server
- packet capture taken on the NGFW and the SYSLOG server show the connection is ended by the NGFW with a TCP RST with an alert message - Fatal (2), Internal Error (80)
- No connection established between the NGFW and the server x.x.x.x when using "show netstat all yes numeric-hosts yes | match x.x.x.x"
- Drop counters for syslog stats is incrementing
admin@NGFW> debug log-receiver statistics | match syslog
Type Enque Count Send Count Drop Count Queue Depth Send Rate (last 1 min)
syslog 0 0 0 0 0 ** wait few seconds ** admin@NGFW> debug log-receiver statistics | match syslog syslog 0 0 10 0 0
Environment
- PAN-OS 11.1+
- Log forwarding over SSL
- CRL (Certificate Revocation List)
Cause
- The CRL check failing.
- This information can be seen in the logrcvr.log (or logd.log in case of Panorama)
admin@NGFW> tail follow yes mp-log logrcvr.log
:23:07.712 +1000 connecting to remote address @ fd 138
:23:07.712 +1000 Pre. send buffer limit=46080. s=138
:23:07.712 +1000 Post. send buffer limit=2097152. s=138
:23:07.752 +1000 Error: pan_logforward_enqueue_new(pan_logforward.c:2996): LOGFWD: enqueue task to syslog taskq(q_depth 0), log dropped
:23:07.802 +1000 Error: delete_old_peer_certs(pan_syslog.c:1082): Too many old certs, deleted old peer cert: /tmp/srvr.crt.218799
:23:07.802 +1000 NO CRL
:23:07.803 +1000 Error: delete_old_peer_certs(pan_syslog.c:1082): Too many old certs, deleted old peer cert: /tmp/srvr.crt.218801
:23:08.129 +1000 CRL URL http://crl3.digicert.com/DigiCertGlobalRootG2.crl cert /tmp/srvr.crt.218812 certcnt 2
:23:08.129 +1000 CRL status [Errno 104] Connection reset by peer rv 0 end
2025-06-10 14:23:08.129 +1000 Error: _pan_syslog(pan_syslog.c:1616): error in SSL_connect
2025-06-10 14:23:08.132 +1000 syslog fwd: update syslog cache. dst ip: x.x.x.x, dst port: 443. cache ip: x.x.x.x, cache port: 443. vsys: vsys1.
2025-06-10 14:23:08.505 +1000 Error: pan_logforward_enqueue_new(pan_logforward.c:2996): LOGFWD: enqueue task to syslog taskq(q_depth 0), log dropped
Resolution
Workaround:
- Check the current current ssl verification check status using the command show syslog-ssl-conn-validation.
admin@NGFW> show syslog-ssl-conn-validation
syslogng ssl connection validation settings:
all-conns: enforce
crl: enforce
ocsp: enforce
eku: enforce
- Based on the above settings, disable the CRL check using the command set syslog ssl-conn-validation explicit CRL skip OCSP enforce EDU enforce.
- Note: The value of OCSP and EDU can be skipped as well. Based on the above input, the other checks are kept as it is.
admin@fw-a> set syslog ssl-conn-validation explicit CRL skip EKU enforce OCSP enforce
Setting syslogng ssl connection validation: crl = skip ocsp = enforce eku = enforce
Permanent solution
- Investigate the cause of failure between the firewall and the CRL endpoint.
- In the output shared above, check the reachability to the CRL URL http://crl3.digicert.com/DigiCertGlobalRootG2.crl.
Additional Information