Incomplete ARP Entry or Firewall Responds to Every ARP Request on the Network

Incomplete ARP Entry or Firewall Responds to Every ARP Request on the Network

155374
Created On 09/25/18 19:36 PM - Last Modified 06/13/24 07:43 AM


Symptom


There can be two symptoms:

The Palo Alto Networks firewall has an incomplete ARP entry for a host on the network (for example, default gateway):

 

> show arp all
maximum of entries supported :      2500
default timeout:                    1800 seconds
total ARP entries in table :        1
total ARP entries shown :           1
status: s - static, c - complete, e - expiring, i - incomplete

interface         ip address      hw address        port              status   ttl
--------------------------------------------------------------------------------
ethernet1/4       10.108.121.1    (incomplete)      ethernet1/4         i      1
 


The firewall is responding to every ARP request on the network. On the endpoint, select any random IP address, try to ping it and you'll see an ARP entry with the firewall's IP MAC:

$ ping -w 2  10.108.121.251
$ ping -w 2  10.108.121.252
$ arp -an
? (10.108.121.251) at 00:1b:17:00:04:13 [ether] on eth4
? (10.108.121.252) at 00:1b:17:00:04:13 [ether] on eth4
 


    Cause


    It is likely there is an incorrectly configured source NAT policy with a mask length that is not /32. For example, if an interface is configured with IP address 10.108.121.2/24, then the NAT IP should be configured as 10.108.121.2/32 (with /32 mask). The NAT IP in this example should not be configured as 10.108.121.2/24.

    Resolution


    With a large number of NAT rules, it can be difficult to narrow down the policy. Three methods to identify the NAT rule are described below. The first two are safe to perform, the third option involves enabling debugs on the dataplane and should be used cautiously.

    Method 1

    Identify the offending pool:

    > show running global-ippool
    Idx  Type             From                             To              Num   Ref. Cnt
    ---- ---------------- -------------------------------- --------------- ---   ----------
    3    Dynamic IP/Port  0.0.0.0-255.255.255.255          10.108.121.5    1     1
    6    Dynamic IP/Port  0.0.0.0-255.255.255.255          10.108.121.0    256   1
     

    Note: Pool 6 is using 256 addresses in 10.108.121.0 network.

    To determine which policy, run the following command, then press "/" (slash), then type in: "idx: 6" (there is space between double colon and 6 and if needed go back a little bit by pressing the up arrow key).

    > show running nat-policy
    [...]
    dmz_Out {
            from dmz;
            source any;
            to outside;
            to-interface ethernet1/4 ;
            destination any;
            service  any/any/any;
            translate-to "src: 10.108.121.0-10.108.121.255 (dynamic-ip-and-port) (pool idx: 6)";
            terminal no;
    }
     

    See the incorrectly configured rule is dmz_out.


    Method 2

    Run a single command, which basically tells the firewall to output all rule names and src NAT translations, where a range of IPs is used. In this case, the rule name that precedes the translation is the offending rule.

    > show running nat-policy | match {\|src:[^\(]*-
    
    "Rule 1" {
    smtp04-in {
    smtp04-out {
    smtp03-out {
    "Internet outbound" {
    dmz_Out {
            translate-to "src: 10.108.121.0-10.108.121.255 (dynamic-ip-and-port) (pool idx: 6)";
    "Rule 4" {
    "Rule 5" {
    smtp03-in {
     

     


    Method 3


    Important! Use cautiously, because this method enables debugs on the dataplane.

    Enable debug on DP:

    > debug dataplane packet-diag clear all
    > debug dataplane packet-diag set filter match non-ip only
    > debug dataplane packet-diag set filter on
    > debug dataplane packet-diag set log feature flow arp
    > debug dataplane packet-diag set log on
    > debug dataplane packet-diag clear log log
     

    After trying to send the communication through the firewall (or pinging from the firewall default gateway):

    > ping source 10.108.121.253 host 10.108.121.1
     

    Review the DP debug files:

    > less dp-log pan_task_*
     

    It is possible to go to the next file by pressing "n"

    At some point, the following appears, which may be similar to:

    Received ARP packet from port ethernet1/4
    Packet decoded dump:
    L2:     00:50:56:a3:10:5a->00:1b:17:00:04:13, type 0x0806
    ARP:    hardware type 0x0001
            protocol type 0x0800
            hardware size 6
            protocol size 4
            opcode REPLY
            sender mac address 00:50:56:a3:10:5a
            sender ip address 10.108.121.1
            target mac address 00:1b:17:00:04:13
            target ip address 10.108.121.253
    ARP packet sent from translated IP in NAT rule index 5 in vsys 1
    ARP packet sent to interface ethernet1/4 IP
    ARP packet parse complete, learn: no, target myself: yes, gratuitous ARP: no
     

    In the example above, the firewall states that someone is using the IP address, which firewall it is using in NAT rule index 5.

    Note: Index 5 means only active policies (disabled policies do not count) and it starts from 0. The easiest way is to again run command "> show running nat-policy" and count policies.


    Fix Details

    This is how the rule looked in the WebGUI and CLI:

    2014-07-30 15_34_59-HQ_PAN_01.png

    > show config running
    [...]
    dmz_Out {
         to outside;
         from dmz;
         source any;
         destination any;
         service any;
         nat-type ipv4;
         source-translation {
              dynamic-ip-and-port {
                   translated-address 10.108.121.211;
              }
         }
         to-interface ethernet1/4;
    }
     


    At a glance, there is nothing wrong with this rule, however after some investigation see that the 10.108.121.211 is really an object:

           

    10.108.121.211 {
                  ip-netmask 10.108.121.211/24;
                }
     

    This would be difficult to find just by browsing through the WebUI.

    Change the netmask to the appropriate one (most likely /32) and verify. See how the rule looks after the change:

     

    > show running nat-policy
    dmz_Out {
            from dmz;
            source any;
            to outside;
            to-interface ethernet1/4 ;
            destination any;
            service  any/any/any;
            translate-to "src: 10.108.121.211 (dynamic-ip-and-port) (pool idx: 7)";
            terminal no;
    }
    
    > show running global-ippool
    Idx  Type             From                             To              Num   Ref. Cnt
    ---- ---------------- -------------------------------- --------------- ---   ----------
    3    Dynamic IP/Port  0.0.0.0-255.255.255.255          10.108.121.5    1     1
    7    Dynamic IP/Port  0.0.0.0-255.255.255.255          10.108.121.211  1     1
     


    In the example above, note a different pool ID used by the very same rule, but only a single IP address is used.


     



    Additional Information


    Particular scenario in AWS

    Actions
    • Print
    • Copy Link

      https://knowledgebase.paloaltonetworks.com/kCSArticleDetail?id=kA10g000000Cla2&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FkCSArticleDetail

    Choose Language