Firewall stops sending logs to Strata Logging Service

• Logrcvr process growing slowly until restart due to OOM
• Missing logs on SLS or logs are not sent by the firewall at all.
• When using "debug log-receiver queue-stats" the "discarded counters" are incrementing and "comp_taskq" task is stuck at a certain value.

> debug log-receiver queue-stats 
Logging statistics 
------------------------------ ----------- 
Log incoming rate: 0/sec 
Log written rate: 25168/sec 
Logs discarded (queue full): 6531286340 
Ring buffer entries: 32768/32768 
Traffic taskqs: 
   comp_taskq: 99/128 
   disk_flush_taskq: 0/256 
   summary_calc_taskq: 0/64 
   summary_flush_taskq: 0/16 
   logpurger_taskq: 0/1 
   bdx_taskqs[ 0]: 0/128 
   bdx_taskqs[ 1]: 0/128 
   bdx_taskqs[ 2]: 0/128

• Palo Alto Firewalls
• PAN-OS lower than 10.2.14 or 11.1.8
• Strata Logging Service (SLS)

Software issue PAN-268800: large number of logs caused the logrcvr process to stop responding.

1. For a permanent fix, upgrade to PAN-OS version 10.2.14 or 11.1.8 when available.
2. As a workaround, restart the logrcvr process using CLI from time to time to clear the logs backlogs and avoid the OOM.

debug software restart process log-receiver

In CLI, review the following logs to confirm that certain logs are not being sent to SLS or Log collector:

> less mp-log lorcvr.log:
18:57:52.217 +0200 Error:  pan_logrcvr_enqueue_dp_block(pan_log_receiver.c:10208): Error submitting task for rb_taskq
18:57:53.474 +0200 Error:  pan_logrcvr_enqueue_dp_block(pan_log_receiver.c:10208): Error submitting task for rb_taskq
18:57:55.174 +0200 Error:  pan_logrcvr_enqueue_dp_block(pan_log_receiver.c:10208): Error submitting task for rb_taskq
18:57:56.633 +0200 Error:  pan_logrcvr_enqueue_dp_block(pan_log_receiver.c:10208): Error submitting task for rb_taskq

> debug log-receiver queue-stats
Logging statistics
------------------------------ -----------
Log incoming rate:             0/sec
Log written rate:              7504/sec
Logs discarded (queue full):   2439074 741
Ring buffer entries:           32768/32768 <<<<<< Ring buffer exhausted

> less mp-log mp-monitor.log - Look for the following log and review if it is growing:
Logs discarded (queue full):   2232890188
Logs discarded (queue full):   2303086554

In the CLI review mp-monitor logs to confirm the slowly memory consumption grow:

less mp-log mp-monitor.log.4 - The oldest one:
logrcvr              20957               0 kB      6967752 kB      9595032 kB      7239260 kB
less mp-log mp-monitor.log - The newest one:
logrcvr              20957               0 kB      7517344 kB     10063024 kB      7808804 kB