Troubleshooting outage due to DNS security where firewall does not show logs

• Firewall admin applies strict Anti-Spyware policy with DNS security enabled which has action as sinkhole. 
• Once the changes are applied. The clients can no longer access internet and names resolution starts failing.  The traffic logs on the firewall show DNS traffic aged out and Not blocked.
• The threat logs do not show any relevant logs with sinkhole  action or dns-security category.

• Palo Alto Firewalls or Prisma Access Firewalls
• PAN-OS 9.0.x or above
• DNS security feature enabled.

• Packet capture with on the firewall for DNS traffic and check the global counters. Refer How to check Global Counters for a specific source and destination IP address for detailed procedure. 

ctd_dns_req_lookup_miss                    2        1 info      ctd       pktproc   DNS request signature lookup not found
ctd_dns_res_fwd_unresolved                10        8 info      ctd       pktproc   DNS response forwarded as not resolved
ctd_dns_wait_pkt_drop                     43       36 drop      ctd       pktproc   DNS packet drop when waiting
ctd_dns_modify_ttl                       221      187 info      ctd       pktproc   DNS response packet changed TTL due to timeout

• The counters above indicates the packet is dropped due to firewall not being able to reach backend server.
• Check the output of DNS security info.

> show dns-proxy dns-signature info 

Cloud URL: dns.service.paloaltonetworks.com:443
Last Result: Timeout was reached ( 0 sec ago ).  <<<
Last Server Address: m.n.z.q
Parameter Exchange: Interval 300 sec
Whitelist Refresh: Interval 43200 sec 
Request Waiting Transmission: 0
Request Pending Response: 41
Cache Size: 0

• The above output indicates the firewall is not able to reach dns server (dns.service.paloaltonetworks.com)
• Upon further troubleshooting, user finds the issue to be a policy block on the firewall. 

2021/02/04 20:14:00 paloalto-dns-se trust                           53867             x.x.x.x
Inside Intrazone_default deny            untrust                         443               m.n.z.q

1. Create allow rule with relevant data and FQDN for dns.service.paloaltonetworks.com and app as paloalto-dns-security to fix the issue.
2. Once the allow rule is created, the output of DNS signature info changes.

>show dns-proxy dns-signature info

Cloud URL: dns.service.paloaltonetworks.com:443
Last Result: Good ( 1 sec ago )
Last Server Address: m.n.z.q
Parameter Exchange: Interval 300 sec
Whitelist Refresh: Interval 43200 sec 
Request Waiting Transmission: 0
Request Pending Response: 1
Cache Size: 168

3. Now the threat logs should start showing the relevant logs as per configuration.

show dns-proxy dns-signature info
show dns-proxy dns-signature content
debug dnsproxyd dns-signature info
debug dnsproxyd dns-signature counters
test dns-proxy dns-signature fqdn <value>