ARP Entries are "incomplete" when PA-VM is restarted from Openstack KVM
17672
Created On 07/12/20 22:20 PM - Last Modified 08/25/20 22:37 PM
Symptom
- VM-Series is deployed on Openstack KVM Red Hat Enterprise Linux Server.
- When VM instance is stopped/started from KVM host running command below, causes ARP entries set to "incomplete"
# nova stop [--all-tenants] <server> [<server> ...] # nova start [--all-tenants] <server> [<server> ...]
- Packets are dropped on the firewall.
- The issue applies to both Packet MMAP and DPDK mode.
- No traffic drops noticed when PA-VM is restarted from PAN-OS CLI or GUI.
Logs:
> show arp all interface ip address hw address port status ttl -------------------------------------------------------------------------------- ethernet1/2 10.64.0.11 (incomplete) ethernet1/2 i 1 Global counters: Elapsed time since last sampling: 656.227 seconds name value rate severity category aspect description -------------------------------------------------------------------------------- pkt_sent_err_drop 8 0 error packet pktproc Packet send error drop flow_fwd_l3_noarp 1 0 drop flow forward Packets dropped: no ARP flow_host_decap_err 14 0 drop flow mgmt Packets dropped: decapsulation error from control plane # lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 7.7 (Maipo) Release: 7.7 Codename: Maipo
Environment
- Platform: VM-Series on Openstack KVM Red Hat Enterprise Linux Server
- PAN-OS / Plugin Version: Any
- Deployment: Existing
Cause
When running Openstack KVM CLI commands, "nova stop" and "nova start":
- Reboot of the PA-VM from the KVM host is leaving the SR-IOV PCI Express (PCIe) Virtual Function (VF) in an abnormal state and unable to pass the broadcast traffic from the PCIe Physical Function (PF) to the firewall.
- This fails ARP resolution on firewall causing packet drops.
- When running NOVA stop, the open-stack resets the configured MAC-VLAN filters on the VF interfaces on the PF. So it needs to set the VLAN on the VF interfaces after a NOVA start in order to set the MAC-VLAN filters.
- This is a PCIe Physical Function (PF) IXGBE driver issue.
- From the Guest OS perspective, PA-VM will not be able to modify the VLAN filter on host.
Resolution
- Reboot VM-Series using PAN-OS CLI command "> request restart system" or GUI under 'Device > Setup > Operations' tab.
- Or re-configure VLAN on the VF interfaces after NOVA stop and start command every time.