High CPU use by logrcvr process and system log message "Number of hints on disk has exceeded 5000 due to log forward failures"

High CPU use by logrcvr process and system log message "Number of hints on disk has exceeded 5000 due to log forward failures"

2381
Created On 06/17/25 09:18 AM - Last Modified 01/26/26 16:30 PM


Symptom


  • High CPU use by process logrcvr
> show system resources

top - 08:24:17 up 4 days,  4:20,  0 users,  load average: 35.31, 35.79, 35.97
Tasks: 608 total,  38 running, 570 sleeping,   0 stopped,   0 zombie
%Cpu(s): 70.4 us,  1.3 sy,  0.0 ni, 27.6 id,  0.0 wa,  0.7 hi,  0.1 si,  0.0 st
MiB Mem : 128744.8 total,    860.8 free,  20934.3 used, 106949.7 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  74400.1 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
13088 root      20   0 3202224   1.4g 109328 S 107.7   1.1   8674:32 logrcvr
  • Message "Number of hints on disk has exceeded 5000 due to log forward failures" observed in the system log
> show log system direction equal backward | match "hints on disk"
2025/04/29 05:05:37 high     general        general 0  Number of hints on disk has exceeded 5000 due to log forward failures.
2025/04/29 04:05:35 high     general        general 0  Number of hints on disk has exceeded 5000 due to log forward failures.
2025/04/29 03:05:33 high     general        general 0  Number of hints on disk has exceeded 5000 due to log forward failures.
2025/04/29 02:05:31 high     general        general 0  Number of hints on disk has exceeded 5000 due to log forward failures.
2025/04/29 01:05:30 high     general        general 0  Number of hints on disk has exceeded 5000 due to log forward failures.


Environment


  • Palo Alto firewall not managed by Panorama
  • PAN-OS 10.1 and above


Cause


  • A log forwarding profile is configured for Panorama or Cloud Logging, but The firewall cannot send logs because  is not managed by Panorama, and Cloud Logging isn't enabled.
  • Logs generated by matching policies using the misconfigured log forwarding profile cause hints for these logs to build up and logrcvr process high CPU use.


    Resolution


    1. Verify the number of hints on disk is greater than 5000 and increasing 
      > debug log-receiver rawlog_fwd show hints-stats
      
      Hints statistics
      Number of hints on disk     :       13526
      Number of hints acked       :           0
      Number of hints corrupted   :           0
      Number of hints purged      :           0
      Number of hints discarded   :           0
      
    2. Correct the log forwarding profile and log settings to remove Panorama or Cloud Logging as a destination under the following locations:
      1. Objects  > Log Forwarding
      2. Device   > Log Setting
    3. Commit the configuration
    4. Verify the number of hints on disk has stopped increasing by running the following command at 1 min intervals:
      > debug log-receiver rawlog_fwd show hints-stats
    5. Clear the accumulated hints.  This must be done 64 hints at a time by running the following command repeatedly until all hints have been cleared:
      > debug log-receiver rawlog_fwd clear hints-all
    6. Verify logrcvr CPU use has returned to normal levels
      > show system resources


    Additional Information




    Actions
    • Print
    • Copy Link

      https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA1Ki000000k9rBKAQ&lang=en_US&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail