Log Forwarding to CDL not working due to incorrect region configuration
16885
Created On 06/04/21 09:34 AM - Last Modified 08/15/22 21:01 PM
Symptom
- Prisma firewall is not connected to CDL receptor hence logs are not being forwarded.
admin@GPGW_XXXXX_netherlands-central_> show logging-status
-----------------------------------------------------------------------------------------------------------------------------
Type Last Log Created Last Log Fwded Last Seq Num Fwded Last Seq Num Acked Total Logs Fwded
-----------------------------------------------------------------------------------------------------------------------------
> CMS 0
Not Sending to CMS 0
> CMS 1
Not Sending to CMS 1
>Log Collection Service
'Log Collection log forwarding agent' is active but not connected <<<<<
config Not Available Not Available 0 0 0
system Not Available Not Available 0 0 0
threat Not Available Not Available 0 0 0
traffic Not Available Not Available 0 0 0
.....(Output Omitted)....
If the log-forwarding agent is active and not connected, it means that the firewall might not have the required FQDNs/endpoints to connect.
- Customer information is correctly fetched on the cloud firewall:
admin@GPGW_XXXXX_netherlands-central_> request logging-service-forwarding customerinfo show
Ingest endpoint: 0d5cc924-4234-44af-9359-48b87fb061c3.in2-lc-prod-eu.gpcloudservice.com
Query endpoint: 0d5cc924-4234-44af-9359-48b87fb061c3.api2-lc-prod-eu.gpcloudservice.com:444
Customer ID: <removed>
Region : europe
- Certificate is also present on the firewall and is valid:
admin@GPGW_XXXXX_netherlands-central_> request logging-service-forwarding certificate info
Certificate chain verification: OK
Public and private key pair match: Yes
Certificate expired: No
Certificate details:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 6085537258678521253 (0x54742be4eaf00da5)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Palo Alto Networks Inc.-SJC-Client-Issuing-CA2 G2, O=Palo Alto Networks Inc., C=US
Validity
Not Before: May 27 08:45:55 2021 GMT
Not After : Aug 25 08:45:55 2021 GMT
Subject: CN=<removed>/serialNumber=7DC00A752A25D8C, OU=Cloud, L=Palo Alto Networks, ST=CA, C=US
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:b1:55:2c:20:ab:86:c5:96:4e:41:65:45:11:c9:
00:1b:1e:9e:ca:97:39:fe:db:86:fe:85:74:8e:d3:
1f:9a:3c:5b:e5:de:3b:6a:01:41:19:b3:67:74:51:
e5:45:20:be:7f:b1:1e:64:b1:5f:b6:26:36:d6:65:
24:43:8a:e1:04:8d:1e:e2:84:24:f8:0b:4a:2d:53:
e1:e7:68:72:14:6c:66:71:99:a2:61:66:23:df:7e:
d5:e4:ee:d7:fa:24:c7:03:85:57:3d:7a:28:1a:62:
a7:78:02:97:2a:c1:02:78:b7:8c:fe:e1:06:a0:61:
a9:b4:f5:2f:d6:41:ee:81:26:ae:86:01:18:20:13:
9b:c9:38:2d:4e:57:a6:c1:ab:3c:ab:6e:da:ce:18:
2b:37:76:fd:84:1c:b2:66:b5:1a:dc:c9:9b:b0:47:
9b:20:9c:8f:bb:00:cf:69:d9:13:ce:a7:63:27:82:
1b:f3:29:b1:45:4b:83:88:e5:32:36:08:d5:4e:bb:
ca:2e:c9:a3:a4:dd:c8:34:e1:ad:35:1a:ba:01:0f:
f4:c2:28:be:29:e6:be:a4:a9:6c:63:e5:54:6c:5c:
da:f0:6c:70:a4:93:3a:fa:73:33:70:7b:9f:22:8b:
cc:06:44:7d:57:b3:4d:fe:09:e0:d9:b1:1a:21:ae:
15:35
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
keyid:CA:1B:6B:CC:69:86:51:74:11:6D:93:F9:F8:DF:E2:A1:D9:4D:26:9B
Authority Information Access:
CA Issuers - URI:https://crl.paloaltonetworks.com/pan-client-issuing-sjc-ca2-g2.crt
OCSP - URI:http://ocsp.paloaltonetworks.com/ocsp
X509v3 Certificate Policies:
Policy: 1.3.6.1.4.1.25461.3.2.3.4
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 CRL Distribution Points:
Full Name:
URI:http://crl.paloaltonetworks.com/pan-client-issuing-sjc-ca2-g2.crl
X509v3 Subject Key Identifier:
38:EA:B5:68:38:E3:8A:C4:72:6B:67:11:48:C6:19:13:FB:0A:AC:D9
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
Signature Algorithm: sha256WithRSAEncryption
2b:d7:ab:7a:94:3f:08:56:2a:cd:78:44:a8:d4:b7:91:90:a1:
09:5e:48:52:4c:67:a4:3f:3f:09:90:9b:71:e1:b2:53:a2:9c:
b4:74:ab:10:47:d4:6e:2d:e1:8f:20:a5:2b:02:95:8b:89:97:
39:f3:0c:06:55:d8:43:96:e0:6c:d7:22:93:62:1b:f0:00:62:
..........
c8:40:04:c0:aa:59:a5:f2:e6:87:67:2a:87:18:2e:b3:e0:7a:
1c:0c:01:e4:88:6f:6d:de:b7:3e:a1:30:7c:4f:6f:5a:13:2a:
ce:f4:41:ee:21:e1:74:13:c3:db:a7:b4:24:bd:68:c1:ee:b2:
71:77:d7:28:f0:99:41:ba:a0:2e:00:fc:53:ae:41:05:ed:3e:
bd:c4:f2:b8:e4:ad:5d:10:f9:e6:ae:f6:fa:c7:e8:d4:be:f1:
60:7b:8f:ab:31:0f:02:ab:7c:d9:9b:72:b1:60:87:30:8a:4c:
95:13:06:d7:39:9a
Successfully fetch LCaaS certificate info
- In ms.log (less mp-log ms.log) one can see error:
2021-06-04 10:03:34.700 +0200 Error: _setup_tcp_conn_info(src_panos/lcs_agent.c:594): Error getting iplist for address: fqdn: (null) port:3978
- In logrcvr.log (less mp-log logrcvr.log)one can see errors indicating issues with the preference list
2021-05-27 01:51:31.307 -0700 ------------ PHASE2 ------------
2021-05-27 01:51:31.307 -0700 pan_nf_update_config_cache()
2021-05-27 01:51:31.307 -0700 Error: pan_logrcvr_lc_sroute_changed(pan_logrcvr_parser.c:788): Could not find entry for panorama-log-forwarding
2021-05-27 01:51:31.307 -0700 Error: pan_logrcvr_lc_sroute_changed(pan_logrcvr_parser.c:788): Could not find entry for paloalto_updates
2021-05-27 01:51:31.307 -0700 config phase 2: update url_cache_timeout from 0 to 5
2021-05-27 01:51:31.307 -0700 Collector preference list does not exist /opt/pancfg/mgmt/global/lcaas-pref.xml
2021-05-27 01:51:31.307 -0700 Error: _load_pref_list_and_connect(pan_rlog_fwd.c:4057): Error parsing the destination LCs preference list
2021-05-27 01:51:31.307 -0700 Collector preference list does not exist /opt/pancfg/mgmt/global/lcaas-pref.xml
2021-05-27 01:51:31.307 -0700 Error: _load_pref_list_and_connect(pan_rlog_fwd.c:4057): Error parsing the destination LCs preference list
2021-05-27 01:51:31.307 -0700 ------------ PHASE2 ends with result:1 ------------
2021-05-27 01:51:33.358 -0700 Updating logrcvr with the latest "hostname" info: PA-VM
2021-05-27 01:51:35.513 -0700 FQDN::unregistering 0 FQDNs not used anymore ...
2021-05-27 01:51:35.513 -0700 FQDN::registering 0 new FQDNs ...
2021-05-27 01:51:40.242 -0700 Loading PaloAltoNetworks URL categories...Environment
- Prisma Access Firewall
- Cortex Data Lake (CDL)
- Log forwarding
Cause
As a next step one can check the preference list which shows FQDN (null):
admin@GPGW_XXXXX_netherlands-central_> show log-collector preference-list
Logging Service Preference List
Forward to all: Yes
Serial Number: PANW_LOG_RECEPTOR_SRV FQDN: (null)
If you check firewall logging service configuration using "> show config merged" command it shows region americas while customer information shows region europe
setting {
logging {
enhanced-application-logging {
enable yes;
}
logging-service-forwarding {
logging-service-regions americas;
enable yes;
}
}
Panorama running configuration confirms that the region is set to americas under Service_Conn_Template
<template>
<entry name="Service_Conn_Template">
<settings>
<default-vsys>vsys1</default-vsys>
</settings>
<description>Service Connection Template (Use the Cloud Services plugin to edit)</description>
<config>
<devices>
<entry name="localhost.localdomain">
<vsys>
<entry name="vsys1"/>
</vsys>
<deviceconfig>
<setting>
<logging>
<logging-service-forwarding>
<logging-service-regions>americas</logging-service-regions>
<enable>yes</enable>
</logging-service-forwarding>
</logging>
</setting>
Resolution
To resolve this issue the region should be set to match with the region from the customer information. You can change the region from Panorama either through CLI or GUI:
Option 1:
Option 2:
Before following the steps below, please make sure that all the config changes are committed and pushed on the Panorama.
- Connect to Panorama via SSH.
- Go to configuration mode with the command:
> configure
- Run the commands below:
set template Service_Conn_Template config deviceconfig setting logging logging-service-forwarding logging-service-regions europe
- Exit from the configuration mode with the command:
> exit
- Go to Panorama GUI then commit and push the config changes to Prisma Access
Additional Information
to analyze ms.log and logrcvr.log log files use cli commands:
>less mp-log ms.log
>less mp-log logrcvr.log