High CPU use by logrcvr process and system log message "Number of hints on disk has exceeded 5000 due to log forward failures"
2995
Created On 06/17/25 09:18 AM - Last Modified 01/26/26 16:30 PM
Symptom
- High CPU use by process logrcvr
> show system resources
top - 08:24:17 up 4 days, 4:20, 0 users, load average: 35.31, 35.79, 35.97
Tasks: 608 total, 38 running, 570 sleeping, 0 stopped, 0 zombie
%Cpu(s): 70.4 us, 1.3 sy, 0.0 ni, 27.6 id, 0.0 wa, 0.7 hi, 0.1 si, 0.0 st
MiB Mem : 128744.8 total, 860.8 free, 20934.3 used, 106949.7 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 74400.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13088 root 20 0 3202224 1.4g 109328 S 107.7 1.1 8674:32 logrcvr
- Message "Number of hints on disk has exceeded 5000 due to log forward failures" observed in the system log
> show log system direction equal backward | match "hints on disk"
2025/04/29 05:05:37 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
2025/04/29 04:05:35 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
2025/04/29 03:05:33 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
2025/04/29 02:05:31 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
2025/04/29 01:05:30 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
Environment
- Palo Alto firewall not managed by Panorama
- PAN-OS 10.1 and above
Cause
- A log forwarding profile is configured for Panorama or Cloud Logging, but The firewall cannot send logs because is not managed by Panorama, and Cloud Logging isn't enabled.
- Logs generated by matching policies using the misconfigured log forwarding profile cause hints for these logs to build up and logrcvr process high CPU use.
Resolution
- Verify the number of hints on disk is greater than 5000 and increasing
> debug log-receiver rawlog_fwd show hints-stats Hints statistics Number of hints on disk : 13526 Number of hints acked : 0 Number of hints corrupted : 0 Number of hints purged : 0 Number of hints discarded : 0 - Correct the log forwarding profile and log settings to remove Panorama or Cloud Logging as a destination under the following locations:
- Objects > Log Forwarding
- Device > Log Setting
- Commit the configuration
- Verify the number of hints on disk has stopped increasing by running the following command at 1 min intervals:
> debug log-receiver rawlog_fwd show hints-stats - Clear the accumulated hints. This must be done 64 hints at a time by running the following command repeatedly until all hints have been cleared:
> debug log-receiver rawlog_fwd clear hints-all - Verify logrcvr CPU use has returned to normal levels
> show system resources
Additional Information
- How To Clear the "hints" counters after checking number of hints on disk
- Another reason for hint exceeded message: System Log Contains "Number of hints on disk has exceeded 5000 due to log forward failures."