VM-Series Firewall or Panorama Crashed from Multiple Sources of System Clock
35214
Created On 04/18/19 19:21 PM - Last Modified 07/31/20 00:11 AM
Symptom
VM-Series firewall or Panorama deployed on VMware ESXi host is restarted and crashed because of a time synchronization between the virtual machine and ESXi server.
– Unable to reach Panorama web interface or SSH.
System is restarted by the init script on firewall and generates logs below under mp\masterd_details.log:
mp masterd_detail.log.1 2019-02-09 10:44:16 2019-02-09 10:44:16.319 -0500 DEBUG: all: Got event stop_event mp masterd_detail.log.1 2019-02-09 10:44:16 2019-02-09 10:44:16.320 -0500 DEBUG: all: Calling stop_event mp masterd_detail.log.1 2019-02-09 10:44:16 2019-02-09 10:44:16.321 -0500 INFO: all: group stop event reason - triggered by init script stop
Check messages.log:
mgmt ntpd[8128]: 0.0.0.0 0613 03 spike_detect -3462.970058 s mgmt ntpd[8128]: 0.0.0.0 061c 0c clock_step -3462.967588 s mgmt ntpd[8128]: 0.0.0.0 0615 05 clock_sync mgmt ntpd[8128]: 0.0.0.0 c618 08 no_sys_peer mgmt ntpd[8128]: 0.0.0.0 0628 08 no_sys_peer
Environment
- VM Series Panorama.
- PAN-OS 7.1 and above.
Cause
When you turn on periodic time synchronization, VMware Tools sets the time of the guest operating system to be the same as the time of the host. After time synchronization occurs, VMware Tools checks once every minute to determine whether the clocks on the guest and host operating systems still match. If not, the clock on the guest operating system is synchronized to match the clock on the host. This may cause excessive CPU and memory over-commitment.
VM kernel hangs when both clock/time sources (i.e., PA-VM and ESXi host) are enabled. This will interfere with NTP operation and may make the clock behave erratically followed by firewall crashes.
Resolution
ESXi host and NTP clock on Panorama or the VM-Series firewall cannot be set together. Disable VMware Tools periodic time synchronization between the virtual machine and the host operating system on ESXi.
STEP 1: Power off the virtual machine.
STEP 2: Right click on any of the virtual machine and click Edit Settings. This will open up the properties of virtual machine.
STEP 3: Select the Options Tab.
STEP 4: Click VMware Tools on the left pane.
STEP 5: On the right pane unselect the check box, "Synchronize guest time with host."
STEP 6: Power on the virtual machine.
The time synchronization checkbox controls only whether the time is periodically re-synchronized while the virtual machine is running. Even if this box is unselected, by default, VMware Tools synchronizes the virtual machine's time after a few specific events that are likely to leave the time incorrect.
To completely disable time synchronization, you must set some properties in the virtual machine configuration file located under this perimeter: '/vmfs/volumes/datastore_name/vm_name/vm_name.vmx
time.synchronize.continue = "0"
time.synchronize.restore = "0"
time.synchronize.resume.disk = "0"
time.synchronize.shrink = "0"
time.synchronize.tools.startup = "0"
time.synchronize.tools.enable = "0"
time.synchronize.resume.host = "0"
Follow this VMware KB article for steps to make the above changes: https://kb.vmware.com/s/article/1189