Azure network interface in Failed status after failover

Upon HA failover, the newly active firewall instance cannot pass traffic. Looking up on the Azure Console, its found that Azure network interface in Failed status
 

• Login to Azure Portal > Resource Group > Click on Network Interface > Check if its in Failed state

• Login to newly active Firewall  CLI & Execute following command to review plugin logs

> less mp-log pan_vm_plugin.log

2019-10-17 12:55:23.537 +1030 vm_ha_state_trans INFO: : Put Request Failed: 429
2019-10-17 12:55:23.537 +1030 vm_ha_state_trans INFO: : URL: https://management.azure.com//subscriptions/1ee0ff47-561b-43e2-a3b1-79f6d1a663ea/resourceGroups/corp-network-fw-ase-rg/providers/Microsoft.Network/networkInterfaces/fwprdc52-fwprdc52-eth3?api-version=2018-12-01

• Log snippet indicates, multiple PUT request failure for VM-Series plugin to attach or detach the secondary IPs on failed Azure Network Interface

• Platform: VM-Series Firewall
• PAN-OS / Plugin Version: 9.0.1 / 1.0.5
• Deployment: Azure

• The Azure Network Interface goes into failed state because VM-Series plugin makes an API request to attach secondary IP to currently active instance while the IP detach process from previously active instance is on-going on Azure 

1. To recover the Azure interface from failed state, either Azure support can be engaged or following steps can be tried:

a. Assign dummy IP in same subnet to the interface in failed state, this would make interface in functional state.

b. Re-assign the dis-appeared secondary IPs to Azure NIC

c. Failover should work after this.

2. This issue has been addressed in VM-Series plugin version 1.0.8
3. If the issue still persists on 1.0.8 than the problem can be with Azure backend (as per below log snippet) and you might need to engage Azure support however you can still apply above workaround.

> pan_vm_plugin.log  logs:

Probe result for detaching NIC: /subscriptions/1ee0ff47-561b-43e2-a3b1-79f6d1a663ea/resourceGroups/corp-network-fw-ase-rg/providers/Microsoft.Network/networkInterfaces/fwprdc51-fwprdc51-eth2 is : {u’status’: u’Failed’, u’error’: {u’message’: u’An error occurred.‘, u’code’: u’InternalServerError’, u’details’: []}}
2020-01-14 13:32:15.402 +1030 vm_ha_state_trans INFO: : Interface /subscriptions/1ee0ff47 56143e2-a3b1-79f6d1a663ea/resourceGroups/corp-network-fw-ase-rg/providers/Microsoft.Network/networkInterfaces/fwprdc51-fwprdc51-eth2 status: Failed