How to Identify and Troubleshoot a Process that Exited or Restarted in PAN-OS
18315
Created On 08/11/22 00:33 AM - Last Modified 08/23/23 18:44 PM
Objective
Identifying the date and timestamp a process exited or restarted in PAN-OS
Environment
- PAN-OS
Procedure
- Navigate to Monitor > Logs > System Logs and look for any logs which contain the word 'kill', 'exit, 'restart', or 'down'
Use the below search filter if needed:
( description contains 'exit' ) or ( description contains 'restart' ) or ( description contains 'kill' ) or ( description contains 'down' )
Examples of System Logs you may see include:
( description contains 'exit' ) or ( description contains 'restart' ) or ( description contains 'kill' ) or ( description contains 'down' )
Examples of System Logs you may see include:
2021/12/09 13:08:43 info vpn ike-dae IKE daemon has exited. 2017/09/18 19:35:42 critical ha datapla HA Group 1: Dataplane is down: brdagent exiting 2021/12/09 12:59:53 critical general general supervisor: Exited 1 times, must be manually recovered. 2021/12/03 02:04:29 medium general general report_gen: Exited with code 0. 2022/01/26 13:34:23 high general general tasks: forcing exit, tell parent 2021/12/09 12:59:56 critical general general The dataplane is restarting. 2022/01/26 13:34:23 high general general all_pktproc_7: got max gdb failure event, telling tasks to restart 2021/12/09 12:02:05 critical general general Out of memory condition detected, kill process 6235 2017/04/26 02:45:47 critical general general "Abnormal system memory usage detected, restarting mgmtsrvr with virtual memory 1478376 KB
- Run the CLI Command below to identify the date + timestamp the process restarted
> less mp-log md_info.log 2022-08-11 01:52:43.169 -0700 INFO: data_plane: exited 2022-08-11 01:52:53.477 -0700 CRITICAL: The dataplane is restarting. 2022-07-18 22:32:10.913 -0700 INFO: data_plane: exited, Core: False, Exit signal: SIGKILL
Note: CLI Command for versions below PAN-OS 10.2.0: > less mp-log masterd.log
- Determine what software feature(s) that process/daemon is responsible for using this document: Commonly Used Processes/Daemons
- You can troubleshoot and resolve the issue with that process using these resources: Resource List: Performance And Stability
Additional Information
Some firewall models contain multiple data planes and/or multiple slots, so the CLI commands will differ slightly:
- Single-DP Platforms (PA-400 PA-850 PA-3000 PA-3200 PA-3400)
> less mp-log md_info.log > less dp-log md_info.log
Note: Some logs will be in mp-log vs. dp-log depending on the process. See this document for reference
- Multi-DP Platforms (PA-5000 PA-5200 PA-5400)
less mp-log md_info.log less dp0-log md_info.log less dp1-log md_info.log less dp2-log md_info.log
Example:
If you see this in Monitor > System Logs
2022/01/14 15:38:40 critical general dp2-path_monitor: Exited 1 times, must be manually recovered.
You would run this command:
If you see this in Monitor > System Logs
2022/01/14 15:38:40 critical general dp2-path_monitor: Exited 1 times, must be manually recovered.
You would run this command:
less dp2-log md_info.log
- Multi-Slot Multi-DP Platforms (PA-5450 PA-7000)
less mp-log md_info.log less fpp-log md_info.log less s1dp0-log md_info.log (s1dp0 = Slot 1, Data Plane 0 - replace Slot # and DP # accordingly)
Example:
If you see this in Monitor > System Logs
2021/04/07 12:33:33 high general general 0 slot2: exiting because of path monitor failure
2021/04/07 12:33:33 high general general 0 slot2-path_monitor: exiting because service missed too many heartbeats
2021/04/07 12:33:33 critical general general 0 Internal packet path monitoring failure, restarting slot 2
2021/04/07 12:33:20 high general general 0 all_pktproc_5: exiting because missed too many heartbeats
You would use the command:
If you see this in Monitor > System Logs
2021/04/07 12:33:33 high general general 0 slot2: exiting because of path monitor failure
2021/04/07 12:33:33 high general general 0 slot2-path_monitor: exiting because service missed too many heartbeats
2021/04/07 12:33:33 critical general general 0 Internal packet path monitoring failure, restarting slot 2
2021/04/07 12:33:20 high general general 0 all_pktproc_5: exiting because missed too many heartbeats
You would use the command:
> less s2dp1-log masterd.log 2021-04-07 12:33:20.506 +0000 ERROR: all_pktproc_5: exiting because missed too many heartbeats