PA-VM deployed in AWS does not move ENI’s to newly active unit upon HA failover
11238
Created On 03/18/20 15:32 PM - Last Modified 04/06/20 17:26 PM
Symptom
Upon HA failover, the newly active firewall instance cannot pass traffic as dataplane interfaces are down. Looking up on the AWS end, we notice the Elastic Network Interfaces (ENI’s) did not transfer to newly active firewall instance despite having correct IAM Roles.
- Review Plugin logs to understand and verify the failure events on the active firewall:
>less mp-log pan_vm_plugin.log
2020-01-30 06:14:21.214 -0800 vm_ha_state_trans INFO: : Local instance:i-095c5a11b86c2ea5a Remote instance:i-04f41d74e42fa8d32
2020-01-30 06:18:55.919 -0800 vm_ha_state_trans INFO: : EC2 get interface info failed for instance-id:i-04f41d74e42fa8d32
HTTPSConnectionPool(host='ec2.us-east-1.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x287e650>, 'Connection to ec2.us-east-1.amazonaws.com timed out. (connect timeout=60)'))
2020-01-30 06:14:21.214 -0800 vm_ha_state_trans INFO: : Local instance:i-095c5a11b86c2ea5a Remote instance:i-04f41d74e42fa8d32
2020-01-30 06:18:55.919 -0800 vm_ha_state_trans INFO: : EC2 get interface info failed for instance-id:i-04f41d74e42fa8d32
HTTPSConnectionPool(host='ec2.us-east-1.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x287e650>, 'Connection to ec2.us-east-1.amazonaws.com timed out. (connect timeout=60)'))
- Above log snippet shows API calls made by the VM-Series plugin to AWS EC2 services were timed out.
- When HA failover is triggered, VM-Series plugin makes API calls to AWS EC2 services in order to detach dataplane interfaces from primary instance-id and reattach them to the secondary instance. Since this API call is timing out
- Verify elastic IP is associated with private IP assigned to management interface (eth0)
- Verify management interface can reach out to Internet. If you encounter reachability issues, troubleshoot network issues within AWS VPC
Environment
- Platform: PA-VM
- PAN-OS / Plugin Version: 8.1.4 / -
- Deployment: AWS
Cause
- Reachability issues to Internet from management interface either due to an elastic IP not associated with private IP assigned to management interface or AWS routing table for management subnet is routing traffic to incorrect next hop or firewall’s trust interface
Resolution
- Associate elastic IP to private IP assigned to management interface of the firewall instance
- Ensure management traffic is routed out correctly to AWS Internet gateway address and not through any of the dataplane interfaces of the FW
- Alternatively, VPC endpoints can be used to reach AWS service points if elastic IP assignment is not permitted.