DNS Service Route (Management Plane) causing DNS Proxy traffic failures

• DNS resolution failures for DNS Proxy users (including Explicit Proxy users, which may manifest as):
  • Explicit Proxy users are unable to access web resources
  • Browsing to HTTP or HTTPS sites result in errors as seen below:

    HTTPS: Error code: PR_END_OF_FILE_ERROR
    HTTP: no healthy upstream

• Management routing table shows the DNS Proxy server using the Management IP address as the source
  

  > debug dataplane internal vif route 0 
  -- skipping for brevity
  1.1.1.1 via <mgmt-gw-ip> dev eth0 table srcif src <mgmt-IP>
  9.9.9.9 via <mgmt-gw-ip> dev eth0 table srcif src <mgmt-IP>

• Dnsproxyd.log may show DNS Proxy queries with 0 results

  >tail follow yes mp-log dnsproxyd.log
  [source/extensions/common/dynamic_forward_proxy/dns_cache_impl.cc:296] main thread resolve complete for host 'www.bing.com:443'. 0 results

• Any Palo Alto Networks hardware firewall (VM's do not display the option to use "MGT")
• Any PAN-OS version
• DNS Proxy configured to use the same Primary/Secondary DNS server IP addresses as the Management Plane DNS (Device > Setup > Services > Primary/Secondary DNS Server)
• Management Plane DNS Service Route source interface set to use "MGT" instead of "Use Default" (Device > Setup > Services > Service Route Configuration > DNS)
   

When DNS Proxy is configured with the same DNS server IP addresses as the Management Plane DNS server (configured under Device > Setup > Services > Primary/Secondary DNS Server), internal requests from the Management Plane are sent out via the Management interface/IP address, and external client (proxied) DNS requests coming in through data plane interfaces will be forwarded out via data plane interfaces to the same destination IP address.

The Management Plane Service Route configuration is only meant to be used to alter the internally generated DNS requests (e.g. FQDN address objects) and forward them out via a specified dataplane interface instead of the Management Plane interface/IP address.

If a Service Route is configured for DNS with the "MGT" option (see screenshot above, only available on hardware platforms), it creates a high priority forwarding "IP rule" for the destination IP address (specified DNS Server) that has higher precedence than the data plane virtual routes.
When an external client DNS request (proxied) reaches the firewall, it will now be sent out of the Management Plane instead of the data plane due to this rule, with the source IP address of the data plane interface. When the DNS response is sent back to the firewall, it is likely that it will be dropped due to this inconsistency, and DNS resolution for the client will fail.

Set the Management DNS Service Route (Device > Setup > Services > Service Route Configuration > DNS) to “Use Default” when the Primary/Secondary DNS Server IP addresses for Management DNS and DNS Proxy are the same.

With this configuration the internally generated DNS requests will be sent out of the Management interface, and the external client (proxied) DNS requests will be forwarded via the corresponding data plane interfaces/virtual routes.