How To Troubleshoot The Log Forwarding Failure From Firewall To Cortex Data Lake (CDL)
17671
Created On 07/28/22 21:47 PM - Last Modified 04/22/24 07:43 AM
Objective
Troubleshooting the log forwarding failure from firewall to Cortex Data Lake (CDL)
Environment
- Palo Alto Networks Firewall
- Cortex Data Lake (CDL)
Procedure
- Check the connectivity between firewall and Cortex Data Lake using
> request logging-service-forwarding status
and in case of issue troubleshoot the problem using the steps listed in How To Troubleshoot The Connection Failure To Cortex Data Lake (CDL). - Check if logs are getting generated on the firewall side:
> show counter global filter delta yes | match log > debug log-receiver statistics
Check if the counters are incrementing in the last CLI above.
- Check if the correct log-forwarding action is configured and enabled on the logs that are being generated:
> show log traffic direction equal backward query equal "actionflags has fwd"
- Check if logs are getting sent from the firewall to CDL:
- For logging-service forwarding option Enable Cortex Data Lake selected use:
> show logging-status
- For logging-service forwarding option Enable Duplicate Logging (Cloud and On-Premise) selected use:
> debug log-receiver rawlog_fwd_trial stats global show
- For logging-service forwarding option Enable Cortex Data Lake selected use:
- Check if there is a huge number of packets on the outgoing queue for the connection to CDL waiting to be forwarded:
> show netstat program yes numeric yes verbose yes | match <CDL IP address>
by checking the third column in that output that represents the Send-Q (send queue). This command is applicable if the firewall is using management interface to forward logs to cortex data lake.
- To force CDL connection when connection to CDL is up and checks listed in step 1 are verified but logs are still not getting forwarded use:
> debug software restart process management-server > debug software restart process log-receiver
Restarting above daemons on the firewall should not affect the dataplane traffic but will cause the connection loss to firewall management for few minutes it is advised to use those commands with caution. - Check if the firewall is hitting an issue where firewalls repeatedly connected and disconnected to Cortex Data Lake due to a probing issue with ID PAN-153440 fixed in releases 8.1.18, 9.0.12, 9.1.6, 10.0.3 or later.
- Check if firewall has log compression enabled:
> show system setting logging log-compression
If enabled then make sure that your firewall is running release 10.0.5 or later to obtain firewall's best behavior in forwarding logs when log compression is enabled. - If none of the above fixes your problem, then contact our technical support team.
Additional Information
Note 1: Logging-service forwarding option is selected under Device > Setup > Management > Logging Service (9.1 or earlier) / Cortex Data Lake (10.0 or later)