System Log Contains "Number of hints on disk has exceeded 5000 due to log forward failures."

102108

Created On 04/27/19 11:24 AM - Last Modified 04/06/23 18:54 PM

Logs

Device Management

Investigation and Response

PAN-OS

Symptom

The system logs contains "Number of hints on disk has exceeded 5000 due to log forward failures."

Firewall > show log system
.....
2019/04/14 10:27:43 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
2019/04/14 09:27:38 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
2019/04/14 08:27:33 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
2019/04/14 06:27:01 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
...

Environment

For PAN-OS 9.1, 10.0 this affects PA-5200 and PA-7k Series Firewall
Starting 10.1 the error message can show up on all NGFW platforms

Cause

These alerts are generated every hour if the number of hints exceeds 5000.
The system generates the alert every hour to basically alert about a possible issue with logs not getting forwarded. Hints get created when a block of logs cannot be sent to Panorama, Log Collector or Cortex Data Lake.
The logs cannot be sent for one of the following reasons:

The Next Generation Firewall is disconnected from log collector for any reason.
The logging rate is very high and the logs sent rate does not match with the logs creation rate.
The logging rate is very high and cannot be ingested and ACK by log collector or Cortex Data Lake.

After connection is reestablished the next-generation firewall starts resending logs to Panorama, log collector or Cortex Data Lake. Once the number of hints drops below 5000, the alerts stop.

Resolution

The alert in itself is working as expected. If the hints cross 5000 mark, the system alerts are triggered.

STEP 1: Check the hints on the next-generation firewall, run the below commands on the next-generation firewall:

For PA-5200 and PA-7k running 10.0 and earlier.

debug management-server rawlog_fwd show hints-stats 
debug management-server rawlog_fwd_dpi show hints-stats 
debug management-server rawlog_fwd_trial show hints-stats 
debug log-receiver rawlog_fwd show hints-stats

For all platforms starting 10.1

debug log-receiver rawlog_fwd show hints-stats

STEP 2: Check and verify the IP address of the firewall's log forwarding destination server:

show logging-status

STEP 3: Check if the logging rate is exceeding device capacity.

STEP 4: Check if next-generation firewall is connected to:

a- Log collector.

To check if next-generation firewall is forwarding logs to Panorama and log collector refer: Verify Log Forwarding to Panorama.
Troubleshoot connectivity link between firewall and log collector.
Check the health of the log collector and if all its processes are in "GREEN" state.
```
show log-collector all
```
The command above starting 10.2 will reflect the status of logd, vldmgr, vlds and es when issued on the CLI of Panorama managing the log-collectors.

b- Cortex Data Lake

NOTE: If further help is needed in troubleshooting this problem, then reach out to Palo Alto Networks support.

Additional Information