SCP Log export failing with Error message: Out of memory condition detected, kill process messages in system Log

24483

Created On 11/09/19 00:01 AM - Last Modified 08/21/20 00:21 AM

Logs

Reporting and Logging

8.1

PAN-OS

Symptom

When monitoring the system logs using GUI: Monitor > System, Out of memory messages are seen and various processes or process IDs are being automatically killed.

critical general        general 0  Out of memory condition detected, kill process 5931
critical general        general 0  Out of memory condition detected, kill process 10810
critical general        general 0  "Abnormal system memory usage detected, restarting mgmtsrvr with virtual memory 1346836 KB
critical general        general 0  "Abnormal system memory usage detected, restarting mgmtsrvr with virtual memory 1309988 KB

The following command will display if there are any active / established TCP connections to the scp server.

> show netstat programs yes numeric yes | match "xx.xx.xx.xx" (where xx.xx.xx.xx  is the IP address of the scp server)

The following command will also show if any log export processes are running

> show system resources follow

and look for any instances of "pan_logquery" jobs that are running.

Based on the above commands, the SCP export and pan_logquery jobs can to be running for several hours or even days. The log export schedule is set on the firewall under GUI: Device > Scheduled Log Export > Scheduled Export Start Time setting.

Environment

PAN-OS 8.1
Palo Alto Firewall.

Cause

The issue is because the firewall is not able to make a connection to the SCP server. This in turn could be because the firewall has not accepted the SCP server host keys.

Resolution

Ensure the SCP connection to the Log Server is properly Established.
Test SCP Server Connection" button to verify the server connection and if necessary, accept the server host keys if the firewall prompts for it.

This action is required on each firewall even if the SCP log export configuration is pushed from Panorama.

Refer: Scheduled Log Export Documentation,

"If you set the Protocol to SCP, you must click this button to test connectivity between the firewall and the SCP server and then verify and accept the host key of the SCP server."

Additional Information