System Log Contains "Number of hints on disk has exceeded 5000 due to log forward failures."

157406

Created On 04/27/19 11:24 AM - Last Modified 11/25/24 19:03 PM

Logs

Device Management

Investigation and Response

PAN-OS

AIOps for NGFW

Strata Cloud Manager

Symptom

The system logs contains "Number of hints on disk has exceeded 5000 due to log forward failures."

Firewall > show log system
.....
2019/04/14 10:27:43 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
2019/04/14 09:27:38 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
2019/04/14 08:27:33 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
2019/04/14 06:27:01 high general general 0 Number of hints on disk has exceeded 5000 due to log forward failures.
...

For firewalls monitored by Strata Cloud Manager (SCM), an alert titled "Log Loss due to Log Forwarding Failure" is generated by the affected firewalls.

Environment

For PAN-OS 9.1, 10.0 this affects PA-5200 and PA-7k Series Firewall
Starting 10.1 the error message can show up on all NGFW platforms

Cause

These alerts are generated every hour if the number of hints exceeds 5000. The system generates the alert to indicate a potential issue with logs not being forwarded. Hints are created when a block of logs cannot be sent to Panorama, a Log Collector, or Strata Logging Service (SLS).

Logs may fail to be sent due to one or more of the following reasons:

Network connectivity issues between the firewall and the log forwarding destination (Panorama, Log Collector or Strata Logging Service).
High logging rate:
- The logging rate is very high and the logs sent rate does not match with the logs creation rate.
- The log forwarding destination (Log Collector or Cortex Data Lake) cannot ingest or acknowledge logs at the required rate due to the high logging volume.

Once the connection is reestablished, the next-generation firewall resumes sending logs to Panorama, the Log Collector, or Strata Logging Service. When the number of hints drops below 5000, the alerts stop.

Resolution

The alert in itself is working as expected. If the hints cross 5000 mark, the system alerts are triggered.

STEP 1: Check the hints on the next-generation firewall, run the below commands on the next-generation firewall:

For all platforms starting 10.1

debug log-receiver rawlog_fwd show hints-stats

For PA-5200 and PA-7k running 10.0 and earlier.

debug management-server rawlog_fwd show hints-stats 
debug management-server rawlog_fwd_dpi show hints-stats 
debug management-server rawlog_fwd_trial show hints-stats 
debug log-receiver rawlog_fwd show hints-stats

STEP 2: Check and verify the IP address of the firewall's log forwarding destination server:

show logging-status

And compare the Last Log Created date, the Last Log Fwded, and the output of the command:

show clock

STEP 3: Check if the logging rate is exceeding device capacity.

STEP 4: Check if next-generation firewall is connected to:

a- Log collector.

To check if next-generation firewall is forwarding logs to Panorama and log collector refer: Verify Log Forwarding to Panorama. In the UI navigate to OBJECT > Log Forwarding and DEVICE > Log Settings.
Troubleshoot connectivity link between firewall and log collector.
Check the health of the log collector and if all its processes are in "GREEN" state. Starting with PAN-OS 10.2, the following command displays the status of logd, vldmgr, vlds and es when issued on the CLI of a Panorama managing the Log Collectors.
```
show log-collector all
```

b- Strata Logging Service

Check the output of the command:
```
request logging-service-forwarding status
```
and refer to Troubleshoot connectivity between firewall and Cortex Data Lake.
View Strata Logging Service status.
Strata Logging Service Monitoring.

NOTE: If further help is needed in troubleshooting this problem, then reach out to Palo Alto Networks support.

Additional Information

If these hints are being seen post an upgrade or a reboot of the log collector, wait for at least a day. The hints might clear out on their own. When ES is restarted on a LC, it can take some time for it to catch up with the inflow of logs.
If the configured log forwarding destination is no longer in use, or if log loss is not a concern, you can clear the hints by issuing the following command:
```
debug log-receiver rawlog_fwd clear hints-all
```
Refer to How To Clear the "hints" counters after checking number of hints on disk. Also check the relevant scenario in the link provided here.
In the output of the command:

show logging-status