Panorama VM inaccessible via GUI or SSH
27219
Created On 07/12/20 19:40 PM - Last Modified 07/20/20 23:40 PM
Symptom
- Panorama VM is running on VMware ESXi Hypervisor.
- The device freezes intermittently and unable to access GUI or CLI until Panorama is rebooted from host.
- No cores or crashes are generated on Panorama.
System Logs:
2019/11/12 03:35:43 2019/11/12 03:36:49 info general general 0 0007NSX00013 Residual commit job snapshots were found. Will be cleared.
2019/11/11 14:53:10 2019/11/11 14:54:19 info general general 0 0007NSX00013 Residual commit job snapshots were found. Will be cleared.
2019/11/10 18:03:22 2019/11/10 18:04:30 info general general 0 0007NSX00013 Residual commit job snapshots were found. Will be cleared.
2019/11/12 03:35:43 2019/11/12 03:36:49 info general general 0 0007NSX00013 Residual commit job snapshots were found. Will be cleared.
2019/11/11 14:53:10 2019/11/11 14:54:19 info general general 0 0007NSX00013 Residual commit job snapshots were found. Will be cleared.
2019/11/10 18:03:22 2019/11/10 18:04:30 info general general 0 0007NSX00013 Residual commit job snapshots were found. Will be cleared.
- Check vmware.log file of the affected Panorama VM in ESXi host techsupport:
<YYYY-MM-DD>T<time>Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: done with snapshot 'smvi_UUID': 0
<YYYY-MM-DD>T<time>Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40).
<YYYY-MM-DD>T<time>Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
<YYYY-MM-DD>T<time>Z| vmx| I120: Vix: [18631 guestCommands.c:1926]: Error VIX_E_TOOLS_NOT_RUNNING in
MAutomationTranslateGuestRpcError(): VMware Tools are not running in the guest</time></time></time></time>
Environment
- Platform: Panorama on VMware
- PAN-OS / Plugin Version: Any
- Deployment: Existing
Cause
- Snapshot with Quiesce is enabled on VMware Hypervisor for this Panorama VM.
- This issue occurs when the call by VMware Tools to freeze the Linux filesystem takes too long due to high I/O on the virtual machine. During the freeze operation, VMware Tools does not respond, causing the host to abort the quiesced snapshost operation.
- When the virtual machine has completed the filesystem freeze, it asks the host to take a snapshot of the virtual machine. However, since the host has previously aborted the snapshot operation due to the VMware Tools times out, this task is aborted and the filesystem remains in a frozen state.
Resolution
- Do not use the VMware snapshots functionality on the VM-Series on ESXi. Snapshots can impact performance and result in intermittent and inconsistent packet loss.
- This operation is not supported by PAN-OS devices.
- Disable VSS quiesced application based snapshots in virtual machine. Follow instructions: https://kb.vmware.com/s/article/2146204
- If you need configuration backups, use Panorama, or from the firewall, use Export named configuration snapshot (Device > Set up > Operations). Using Export named configuration snapshot exports the firewall’s active configuration (running-config.xml) and allows you to save it to any network location.