PA-VM on Microsoft Hyper-V hangs and becomes unresponsive intermittently
20137
Created On 04/02/20 15:33 PM - Last Modified 04/02/20 16:45 PM
Symptom
- VM-series firewall hosted on Microsoft Hyper-V Hypervisor intermittently hangs and becomes unresponsive.
- CLI and SSH access to the firewall is lost and management port is unreachable.
- Firewall needs to be hard reboot from host to restore the connectivity.
- In PA-VM console logs noticed errors, "[sda] Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters."
- In var/log/messages logs:
2020-03-09 20:04:23 Mar 9 20:04:23 mgmt kernel: sd 2:0:0:0: [sda] Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters.
2020-03-09 20:04:50 Mar 9 20:04:50 mgmt kernel: hv_storvsc 00000000-0000-8899-0000-000000000000: cmd 0x2a scsi status 0x2 srb status 0x82
2020-03-09 20:04:50 Mar 9 20:04:50 mgmt kernel: hv_storvsc 00000000-0000-8899-0000-000000000000: stor pkt ffff8801ced331c0 autosense data valid - len 18
2020-03-09 20:04:50 Mar 9 20:04:50 mgmt kernel: storvsc: Sense Key : 0x6 [current]
2020-03-09 20:04:50 Mar 9 20:04:50 mgmt kernel: storvsc: ASC=0x3f ASCQ=0x2
2020-03-09 20:04:50 Mar 9 20:04:50 mgmt kernel: sd 2:0:0:0: [sda] Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters.
2020-03-09 20:04:50 Mar 9 20:04:50 mgmt kernel: hv_storvsc 00000000-0000-8899-0000-000000000000: cmd 0x2a scsi status 0x2 srb status 0x82
2020-03-09 20:04:50 Mar 9 20:04:50 mgmt kernel: hv_storvsc 00000000-0000-8899-0000-000000000000: stor pkt ffff8801ced331c0 autosense data valid - len 18
2020-03-09 20:04:50 Mar 9 20:04:50 mgmt kernel: storvsc: Sense Key : 0x6 [current]
2020-03-09 20:04:50 Mar 9 20:04:50 mgmt kernel: storvsc: ASC=0x3f ASCQ=0x2
2020-03-09 20:04:50 Mar 9 20:04:50 mgmt kernel: sd 2:0:0:0: [sda] Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters.
Environment
- Platform: PA-VM on Microsoft Hyper-V
- PAN-OS / Plugin Version: Any
- Deployment: Existing
Cause
- This error is not related to PAN-OS, but it is due to system/VM backup on MSFT Hyper-V i.e. Volume Shadow Copy Service (VSS) to backup and restore virtual machines (VMs). Refer: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/hypervbackupapproaches
- Improper handling of some error codes in Hyper-V storvsc driver is causing system lockup after hot backup.
- Consequence: It is possible to observe block subsystem getting stuck after performing hot backup of Hyper-V Virtual Machine.
- Bug and patch are updated for this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1502601
Resolution
Disable Volume Shadow Copy Service (VSS) integration for VM-Series firewall on Microsoft Hyper-V.