PA-VM deployed in AWS does not move ENI’s to newly active unit due to DNS issues
4386
Created On 03/18/20 15:47 PM - Last Modified 04/06/20 17:27 PM
Symptom
Upon HA failover, the newly active firewall instance cannot pass traffic as dataplane interfaces are down. Looking up on the AWS end, we notice the Elastic Network Interfaces (ENI’s) did not transfer to newly active firewall instance despite having correct IAM Roles and Internet connectivity.
- Review Plugin logs to understand and verify the failure events on the active firewall:
>less mp-log pan_vm_plugin.log
2020-01-30 07:19:57.813 -0800 vm_ha_state_trans INFO: : Local instance:i-095c5a11b86c2ea5a Remote instance:i-04f41d74e42fa8d32
2020-01-30 07:21:16.735 -0800 vm_ha_state_trans INFO: : EC2 get interface info failed for instance-id:i-04f41d74e42fa8d32
Could not connect to the endpoint URL: "https://ec2.us-east-1.amazonaws.com/"
2020-01-30 07:19:57.813 -0800 vm_ha_state_trans INFO: : Local instance:i-095c5a11b86c2ea5a Remote instance:i-04f41d74e42fa8d32
2020-01-30 07:21:16.735 -0800 vm_ha_state_trans INFO: : EC2 get interface info failed for instance-id:i-04f41d74e42fa8d32
Could not connect to the endpoint URL: "https://ec2.us-east-1.amazonaws.com/"
- Above log snippet shows API calls made by the VM-Series plugin to AWS EC2 services failed because firewall could not resolve FQDN “ec2.us-east-1.amazonaws.com”
- Verify DNS server is configured under Device > Setup > Services > Primary DNS Server
- Check connectivity from management interface to DNS server using ping. If this fails, troubleshoot network connectivity from Firewall to DNS server
- If ping succeeds that implies, we have layer 3 connectivity, perform traceroute to identify any devices in path.
- Perform tcpdump on management interface of the firewall to verify DNS requests are sent out
> tcpdump filter “host <dns_server_IP>”
- Verify AWS VPC Security Group and subnet NACLis allowing UDP port 53 traffic towards the DNS server
Environment
- Platform: PA-VM
- PAN-OS / Plugin Version: 9.0.x / -
- Deployment: AWS
Cause
- Firewall is unable to resolve DNS for EC2 service endpoint
Resolution
- DNS requests were blocked by AWS Security Group/ subnet NACL