Troubleshooting firewall connectivity issues with Logging Service
148091
Created On 12/02/18 16:29 PM - Last Modified 12/04/19 02:49 AM
Objective
How to troubleshoot firewall connectivity issues with Logging Service?
Environment
- Panorama.
- Palo Alto Firewalls.
- PAN-OS 8.0.5 or greater.
Procedure
Currently, we can configure on-premise hardware-based and vm-based firewalls and cloud firewalls part of GlobalProtect Cloud Services to forward logs to the Logging Service.
On-premise(hardware-based and VM-based) firewalls need to be managed by Panorama. Please review/verify the configuration that needs to be in-place for log-forwarding before continuing further, see the following document:
Configure the Firewalls to Forward Logs to the Logging Service
If the log-forwarding agent is not active, it could mean that it might not have valid licenses:
- Check for a valid license. Check Device > Licenses page on FW UI to make sure they have Logging Service license and Premium Support or by running request license info on the command line in operational mode.
- If the required licenses are missing, login to Panorama that is managing the firewall(s) in question and deploy the Logging Service Licenses from Panorama > Device Deployment > Licenses > Refresh, select the firewalls the license needs to deployed to and click Refresh
- Verify Licenses are installed on the firewall.
- The firewall in question may not have the required Logging Service endpoints(Ingest/Query FQDNs) to connect to. Running request logging-service-forwarding customerinfo show might return the error as shown in the example below.
Firewall> request logging-service-forwarding customerinfo show
Server error : Unable to read the LCaaS customer information. Please re-fetch region info
Firewall> less mp-log lcaas_agent.log
ERROR Failed to fetch ingest/query FQDN for cust xxxxxxx
The above example shows the error message in the log when the firewall management interface is not able to access the API and Licensing endpoints for the Logging Service on port 444. Verify all the TCP Ports and FQDNs required for the Logging Service have been allowed appropriately. Ports used for Logging service are listed in the document TCP Ports and FQDNs Required for Logging Service.
You can run the following command to fetch the customer info:
Firewall> request logging-service-forwarding customerinfo fetch
Verify:
Firewall> request logging-service-forwarding customerinfo show
Ingest endpoint: 9286a54d-3915-4497-a888-42f789e09a33.in2-lc-prod-us.gpcloudservice.com
Query endpoint: 9286a54d-3915-4497-a888-42f789e09a33.api2-lc-prod-us.gpcloudservice.com:444
Customer ID: 121053001
Region : americas
- Or the firewall may not have the certificate required to establish an SSL connection with the Logging Service. Check if the firewall received a valid certificate from Panorama by running request logging-service-forwarding certificate info. The example below shows a scenario where the SSL connection fails due to a bad certificate or due to its unavailability on the firewall(s).
Firewall> less mp-log logrcvr.log
Error: pan_comm_get_tcp_conn_gen(comm_utils.c:567): COMM: cannot connect. remote ip=<LCAAS IP> port=3978 err=Connection timed out(110) sock=134
Error: pan_mgmt_secure_conn_ocsp_crl_check(pan_sec_conn_client.c:208): [Secure conn cert verify stage] cert[CN: *.lc.prod.us.cs.paloaltonetworks.com/emailAddress=support@paloaltonetworks.com] revoked, not valid!
Error: pan_sec_conn_client_validation_impl(pan_sec_conn_client.c:400): [Secure conn] Failed to verify OCSP/CRL for the certificate
Error: pan_mgmt_peer_validation(pan_sec_conn_client.c:665): Peer validation for server[LCAAS IP] failed
Error: pan_conn_mgr_do_connect(cs_conn.c:11173): Untrusted LCaaS cert detected, Error connecting to LCaaS
- Verify all the TCP Ports and FQDNs required for the Logging Service have been allowed appropriately, see the following document:
- You can delete and re-fetch the certificate from Panorama to get the correct/updated certificate.
Firewall> request logging-service-forwarding certificate delete
Firewall> request logging-service-forwarding certificate fetch
- Verify:
Firewall> request logging-service-forwarding certificate info
Certificate chain verification: OK
Public and private key pair match: Yes
Certificate expired: No
Validity
Not Before: Oct 24 02:10:44 2018 GMT
Not After : Jan 22 02:10:44 2019 GMT
- Once the firewall gets the required certificate and FQDN end-point(s), it should start connecting to the LCAAS and can be verified in the output of the command show logging-status. The Last Log Fwded column contains recent (up-to-the-minute) date/time stamps, and the numbers in the Total Logs Fwded. column should be incrementing over time. If you notice Last Logs Fwded, Last Seq Num Fwded and Last Seq Num Acked does not, then there could be a problem on the edge/perimeter/intermediate firewall(s) or Logging Service receptor dropping the logs. Verify all the TCP Ports and FQDNs required for the Logging Service have been allowed appropriately.
- If the counters in show logging-status are not increasing after a few minutes, it means either logs are not getting generated or perhaps logs are getting generated but do not have the correct log-forwarding profile applied to them.
- Take few samples of log generation stats to rule out log generation issue by runnning:
Firewall>debug log-receiver statistics
- If the counters from debug log-receiver statistics are incrementing, run the following command to verify if the correct log-forwarding action is configured and enabled on the logs that are being generated.
Firewall> show log traffic direction equal backward query equal "actionflags has fwd"
If you run the command above a few times, it will print logs generated locally, most recent first and the ones that have correct log action forwarding and the command does not print log records(in the time window you are expecting, for example, today's time-stamp), you are likely to have incorrect log-forwarding profile applied to the security policy.