How to Identify and Troubleshoot a Process that Exited or Restarted in PAN-OS

19230

Created On 08/11/22 00:33 AM - Last Modified 08/23/23 18:44 PM

Processes

9.0

9.1

10.0

10.2

10.1

PAN-OS

Panorama

Strata Cloud Manager

Objective

Identifying the date and timestamp a process exited or restarted in PAN-OS

Environment

PAN-OS

Procedure

Navigate to Monitor > Logs > System Logs and look for any logs which contain the word 'kill', 'exit, 'restart', or 'down'

Use the below search filter if needed:
( description contains 'exit' ) or ( description contains 'restart' ) or ( description contains 'kill' ) or ( description contains 'down' )

Examples of System Logs you may see include:

2021/12/09 13:08:43 info     vpn            ike-dae  IKE daemon has exited.
2017/09/18 19:35:42 critical ha datapla HA Group 1: Dataplane is down: brdagent exiting
2021/12/09 12:59:53 critical general       general supervisor: Exited 1 times, must be manually recovered.
2021/12/03 02:04:29 medium  general       general report_gen: Exited with code 0.
2022/01/26 13:34:23 high     general        general  tasks: forcing exit, tell parent
2021/12/09 12:59:56 critical general        general  The dataplane is restarting.
2022/01/26 13:34:23 high     general        general  all_pktproc_7: got max gdb failure event, telling tasks to restart
2021/12/09 12:02:05 critical general        general  Out of memory condition detected, kill process 6235
2017/04/26 02:45:47 critical general general "Abnormal system memory usage detected, restarting mgmtsrvr with virtual memory 1478376 KB

Run the CLI Command below to identify the date + timestamp the process restarted

> less mp-log md_info.log
2022-08-11 01:52:43.169 -0700 INFO: data_plane: exited
2022-08-11 01:52:53.477 -0700 CRITICAL: The dataplane is restarting.
2022-07-18 22:32:10.913 -0700 INFO: data_plane: exited, Core: False, Exit signal: SIGKILL

Note: CLI Command for versions below PAN-OS 10.2.0: > less mp-log masterd.log

Determine what software feature(s) that process/daemon is responsible for using this document: Commonly Used Processes/Daemons
You can troubleshoot and resolve the issue with that process using these resources: Resource List: Performance And Stability

Additional Information

Some firewall models contain multiple data planes and/or multiple slots, so the CLI commands will differ slightly:

Single-DP Platforms (PA-400 PA-850 PA-3000 PA-3200 PA-3400)

> less mp-log md_info.log
> less dp-log md_info.log

Note: Some logs will be in mp-log vs. dp-log depending on the process. See this document for reference

Multi-DP Platforms (PA-5000 PA-5200 PA-5400)

less mp-log md_info.log
less dp0-log md_info.log
less dp1-log md_info.log
less dp2-log md_info.log

Example:
If you see this in Monitor > System Logs
2022/01/14 15:38:40 critical general dp2-path_monitor: Exited 1 times, must be manually recovered.

You would run this command:

less dp2-log md_info.log

Multi-Slot Multi-DP Platforms (PA-5450 PA-7000)

less mp-log md_info.log
less fpp-log md_info.log
less s1dp0-log md_info.log (s1dp0 = Slot 1, Data Plane 0 - replace Slot # and DP # accordingly)

Example:
If you see this in Monitor > System Logs
2021/04/07 12:33:33 high general general 0 slot2: exiting because of path monitor failure
2021/04/07 12:33:33 high general general 0 slot2-path_monitor: exiting because service missed too many heartbeats
2021/04/07 12:33:33 critical general general 0 Internal packet path monitoring failure, restarting slot 2
2021/04/07 12:33:20 high general general 0 all_pktproc_5: exiting because missed too many heartbeats

You would use the command:

> less s2dp1-log masterd.log
2021-04-07 12:33:20.506 +0000 ERROR: all_pktproc_5: exiting because missed too many heartbeats

How to Identify and Troubleshoot a Process that Exited or Restarted in PAN-OS

Objective

Environment

Procedure

Additional Information

Other users also viewed: