HA Failover failed on VM-series deployed on AWS with HTTP Error 401: Unauthorized

• Upon HA failover, the newly Active firewall instance cannot pass traffic as dataplane interfaces are down.
• On AWS portal, the Elastic Network Interfaces (ENIs) do not get transferred to newly Active firewall instance.
• Management interface has internet connectivity.
• IAM Role with sufficient privileges is attached to both firewall instances.
• Errors are seen in VM-series plugin logs on the newly active firewall:

> less mp-log pan_vm_plugin.log

2022-06-22 06:15:10.356 -0700 vm_host_init INFO: : AWS get_meta_data called http://169.254.169.254/latest/ meta-data/iam/security-credentials
2022-06-22 06:15:10.356 -0700 vm_host_init INFO: : AWS get_meta_data failed with exception HTTP Error 401: Unauthorized
2022-06-22 06:15:10.356 -0700 vm_host_init INFO: : iamrole

2022-06-22 06:15:10.403 -0700 vm_host_init INFO: : AWS get_meta_data called http://169.254.169.254/latest/ meta-data/instance-id
2022-06-22 06:15:10.403 -0700 vm_host_init INFO: : AWS get_meta_data failed with exception HTTP Error 401: Unauthorized
2022-06-22 06:15:10.403 -0700 vm_host_init INFO: : inst_id

2022-06-22 06:15:10.403 -0700 vm_host_init INFO: : AWS get_meta_data called http://169.254.169.254/latest/ meta-data/instance-type
2022-06-22 06:15:10.404 -0700 vm_host_init INFO: : AWS get_meta_data failed with exception HTTP Error 401: Unauthorized
2022-06-22 06:15:10.404 -0700 vm_host_init INFO: : inst_type

• System state on the newly active firewall displays "empty" for iamrole and instance.

> show system state filter cfg.platform.*
...
cfg.platform.iamrole: <empty>
cfg.platform.instance-id: <empty>
cfg.platform.instance-type: <empty>
...

• Platform: PA-VM
• Deployment: AWS
• PAN-OS: Any
• VM-series plugin: Any

• VM-series plugin is unable to fetch instance metadata from AWS.
• If Instance Metadata services in AWS has HttpTokens parameter set to Required, every request to access the instance metadata requires a token.
• This caused VM-series plugin fails to get necessary instance metadata values.
• Consult with customer cloud team or AWS support to confirm the current value.

1. Change HttpTokens parameter to Optional through AWS CLI

aws ec2 modify-instance-metadata-options --instance-id <instance_id> --http-tokens optional

2. Firewall needs to be rebooted after changing the parameter above, as metadata values are fetched during system startup.
3. Verify system state on the firewall after reboot (below values are just example):

> show system state filter cfg.platform.*
...
cfg.platform.iamrole: IAM-HA-ROLE
cfg.platform.instance-id: i-080ce004604604d2e
cfg.platform.instance-type: m5.xlarge
...