Verifying Cortex Data Lake connectivity on a Palo Alto firewall
51652
Created On 08/02/19 14:35 PM - Last Modified 09/15/22 20:55 PM
Symptom
Logs are not seen on Explore app or on Panorama (When logs are sent to only Cortex Data Lake)
Environment
- Palo Alto Firewall
- PAN-OS 8.0 and 8.1
- Log forwarding to Cortex Data Lake (CDL)
Resolution
This procedure is valid for PanOS 8.0.X. It is also valid for PanOS 8.1.X when duplicate logging is not enabled.
Verifying Cortex Data Lake functionality:
1. Run the command below and note Customer ID (It is unique for every customer) and Region info (Currently it can be Europe or Americas based on which location was chosen during the initial setup for Data Lake)
> request logging-service-forwarding customerinfo show
Sample output for the command:
Ingest endpoint: xxxxxxxx.in3.lcaas-beta.us.paloaltonetworks.com
Query endpoint: xxxxxxxx.api3.lcaas-beta.us.paloaltonetworks.com:444
Customer ID: Customer_Tenant_Id
Region : Selected Region
2. Run the command below and check the CN value in the certificate. It should be same with Customer ID seen at previous step. Serial number should be same with serial number of the firewall.
> request logging-service-forwarding certificate info
Sample output for the command:
....Omitted output....
Validity
Not Before: May 1 04:48:36 2018 GMT
Not After : Jul 30 04:48:36 2018 GMT
Subject: C=US, L=Palo Alto Networks, OU=Cloud/serialNumber=FW_S/N, CN=Customer_Tenant_Id
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
3. If you don’t see correct S/N or customer id (CN parameter of the certificate), you should run the commands below and then follow the step 1 and 2 to check if serial number and CN values are correct or not.
> request logging-service-forwarding certificate delete
> request logging-service-forwarding certificate fetch
4. You should check “show logging-status” output to be sure about log forwarding is successful.
Sample output for the command “show logging-status”: (except 7k and 5200 series)
• US: 65.154.226.0/24
• EU: 154.59.126.0/24
Look for the ‘Log collection log forwarding agent’ is active and connected to <IP_address> line. Ip address can change due to chosen region. You can see ip address from subnets below based on the region.
If you see log forwarding agent is active but not connected in this output, you should restart mgmtsrvr process to refresh connection to Cortex Data Lake. You will not need to restart process with PanOS 8.1.8. There will be an enhancement to refresh connection without restart.
Run the command to restart management server:
> debug software restart process management-server
Sample output for the command “show logging-status”: (for 7k and 5200 series)
• US: 65.154.226.0/24
• EU: 154.59.126.0/24
Look for the ‘PANW_LOG_RECEPTOR_SRV’ both MS(mgmtserver), LR (Log receiever)are active and connected to <IP_address> line. Ip address can change due to chosen region. You can see ip address from subnets below based on the region.
5200 and 7K Series have different architecture because of high log rate. Log receiver process is responsible for forwarding traffic, threat etc. logs
Mgmtserver is responsible for forwarding System and Config logs.
If you see connection status is inactive for MS or LR in this output, you should restart mgmtsrvr process and log receiver to refresh connection to Cortex Data Lake. You will not need to restart processes with PanOS 8.1.8. There will be an enhancement to refresh connection without restart.
Run the command to restart management server:
debug software restart process management-server
Run the command to restart log receiver:
debug software restart process log-receiver
Verifying Cortex Data Lake functionality (PanOS 8.1.X when duplicate logging is enabled)
1. Run the command below and note Customer ID(It is unique for every customer) and Region info(Currently it can be Europe or Americas based on which location was chosen during the initial setup for Data Lake)
> request logging-service-forwarding customerinfo show
Sample output for the command:
Ingest endpoint: xxxxxxxx.in3.lcaas-beta.us.paloaltonetworks.com
Query endpoint: xxxxxxxx.api3.lcaas-beta.us.paloaltonetworks.com:444
Customer ID: Customer_Tenant_Id
Region : Selected Region
2. Run the command below and check the CN value in certificate it should be same with Customer ID in previous step. Serial number should be same with serial number of the firewall.
> request logging-service-forwarding certificate info
Sample output for the command:
....Omitted output....
Validity
Not Before: May 1 04:48:36 2018 GMT
Not After : Jul 30 04:48:36 2018 GMT
Subject: C=US, L=Palo Alto Networks, OU=Cloud/serialNumber=FW_S/N, CN=Customer_Tenant_Id
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
3. If you don’t see correct S/N or customer id (CN parameter of certificate), you should follow the procedure below:
> request logging-service-forwarding certificate delete
> request logging-service-forwarding certificate fetch
Then follow the step 1 and 2 to check information is matching.
4. When duplicate logging enabled you will see outputs for the commands below:
> request logging-service-forwarding status
Will result with: LCaaS forwarding enabled: No (When duplicate logging is not enabled you see “Yes”)
> show system state | match lcaas
Will result with: cfg.lcaas-enabled: False (When duplicate logging is not enabled, you see “True”)
> show logging-status (When duplicate logging is enabled, you see only ip address of log collectors in customer’s network. It is expected not seeing any connection to subnets US: 65.154.226.0/24 EU: 154.59.126.0/24)
Will result with: Panorama log forwarding agent is active but not connected to Logging Service
5. You can verify logs are forwarded to Data Lake with commands below:
When duplicate logging is enabled show logging-status will show only logs that are being sent to the Panorama.
In order to verify that logs are being sent in the current setup you need to run the following commands:
> debug log-receiver rawlog_fwd_trial stats global show (Will show log statistics that are sent to the cloud, important to follow drop counters. Run the command 2-3 times, check increase for drop counter)
> debug log-receiver rawlog_fwd_dpi stats global show (Will show log statistics for enhanced application logs that are sent to the cloud, important to follow drop counters. Run the command 2-3 times, check increase for drop counter)
> debug log-receiver rawlog_fwd_trial evtmgr (Will show the connection to the Cortex Data Lake instance)
servers=1 timers=1 proxies=0 msg_class=1 int_timer=13 last_id=1000001
debug counters:
56134 56134 67 0
56142 56142 0 0
0 0 56100 56100
0 0 2 1
0 0
error counters:
56098 0 0 0
0 0
currtime=337001 last_check=337000
Server port=0 fd=-100 ssl=no clients=1 max_pending_msg=250
triallr-74.217.90.122-def 1000001 26 2 0 0
msg total=0 in=0 out=0 outdated=0 leak=0 option_bytes=0 data_bytes=0
6. If you see increase for drop counters, follow the steps below:
(If it is not 7k and 5200 series)
Run the command to restart management server:
> debug software restart process management-server
(For 7k and 5200 series)
Run the command to restart management server:
> debug software restart process management-server
Run the command to log receiver:
> debug software restart process log-receiver
Attachments