Secondary IP(s) of Azure Network Interface(s) do not move to newly active unit upon HA failover.

Upon HA failover, the newly active firewall instance cannot pass traffic. Looking up on the Azure console, we notice the secondary IP(s) of Network Interface(s) did not transfer to newly active firewall VM despite having correct DNS and Internet connectivity.
 

• Review Plugin logs to understand and verify the failure events on the active firewall:
  > less mp-log pan_vm_plugin.log

2020-01-06 19:42:05.752 +0900 vm_ha_state_trans INFO: : Host instance name: rmbpalovm300e41
2020-01-06 19:42:05.817 +0900 vm_ha_state_trans INFO: : URL for get request: https://management.azure.com/subscriptions/d20e2c80-c92e-487c-be20-aa0901bb1a01/resourceGroups/rmb-rg-palovm300-e-41/providers/Microsoft.Compute/virtualMachines/rmbpalovm300e42?api-version=2015-06-15
2020-01-06 19:42:05.855 +0900 vm_ha_state_trans INFO: : Get Request Failed: 403
2020-01-06 19:42:05.855 +0900 vm_ha_state_trans INFO: : URL: https://management.azure.com/subscriptions/d20e2c80-c92e-487c-be20-aa0901bb1a01/resourceGroups/rmb-rg-palovm300-e-41/providers/Microsoft.Compute/virtualMachines/rmbpalovm300e42?api-version=2015-06-15
2020-01-06 19:42:05.855 +0900 vm_ha_state_trans INFO: : Fail to get peer_vm_info from peer_vm_name rmbpalovm300e4

• Above log snippet shows API calls made by the VM-Series plugin to Azure Fabric failed due to in-sufficient credentials, as a result VM-Series plugin could not get the information for peer VM.

• The Azure AD returned HTTP 403 message, which is actually HTTP forbidden message sent from Azure Fabric.

• Platform: VM-Series Firewall
• PAN-OS / Plugin Version: 9.1.0/ -
• Deployment: Azure

• Lack of access privileges (IAM role) to the VM-Series instance in order to make API call to Azure Fabric for moving the secondary IP(s) to newly active device

• Please refer following two documents to assign contributor IAM Role for the service Principal.

https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal

https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles