How to do basic debugging for DNS Security before you open a support case
50937
Created On 02/24/21 23:01 PM - Last Modified 08/13/24 08:54 AM
Objective
This article covers a few debugging steps for DNS Security. In most cases, it will help you identify and solve the issue, if the issue is still not resolved please open a support case with Palo Alto Networks Support using this information.
Environment
- Any Palo Alto Networks Firewall
- PAN-OS 9.x.x,10.x.x and above
- DNS Security license
Procedure
Following are basic debugging steps for DNS Security feature configuration verification, license, and cloud connectivity. During the process, you may identify the issue by yourself, If not, please open a support case with the following information.
1. Check the license on the Firewall
The result should look like as follows.
> request license info License entry: Feature: DNS Security Description: Palo Alto Networks DNS Security License Serial: xxxxxxxxxxxx Issued: January xx, 2021 Expires: January xx, 2024 Expired?: no Base license: PA-VM
2. Check the DNS Security status by visiting https://status.paloaltonetworks.com/
3. Check the connection information
The result should look like as follows.
> show dns-proxy dns-signature info Cloud URL: dns.service.paloaltonetworks.com:443 Last Result: Good ( 46 sec ago ) Last Server Address: 130.211.8.196 Parameter Exchange: Interval 1800 sec Whitelist Refresh: Interval 86400 sec ( Due 71954 sec ) Request Waiting Transmission: 0 Request Pending Response: 0 Cache Size: 10000
If the Cloud connection is not in Good state, the expected behaviour is that Firewall sends a verdict lookup request to the cloud(which will fail) and also forward the client dns request to the name server. Since the cloud is not reachable the query will be dropped. However after an MP timeout, firewall will trigger a re-transmission of the DNS request and its response should go through.
Following global counters can be observed:
Following global counters can be observed:
ctd_dns_wait_pkt_drop 3 0 drop ctd pktproc DNS packet drop when waiting
ctd_dns_pkt_retransmit 3 0 info ctd pktproc number of DNS pkt retransmitted
4. For the DNS Security feature to be enabled and working, the DNS Security action is recommended to be set to "sinkhole" (see here). When a new spyware-profile is created, the default action is dictated by the Palo Alto Content release, please double-check the action. If the action is "allow", DNS Security will not work.
- For PAN-OS 9.x.x add "Palo Alto Networks DNS Security" as follows.
- For PAN-OS 10.x.x, you should select based on the different categories provided by DNS Security.
5. Check the DNS-proxy counters as they provide tons of useful information in multiple rows of counters.
- From these rows, check the "Signature query API" where you want to check request, and request_error counters.
- These counters have three columns, the first column is cumulative, the second column is the delta since the last issue of op-command, the third column is the delta per second.
- Another counter to notice is latency. The time is in millisecond (ms), including max, min, avg, followed by a bucketed break down of data.
>show dns-proxy dns-signature counters Signature query API: [request ] : 59 +7 +0 /sec [request_error ] : 0 +0 +0 /sec [initial_connection ] : 40 +6 +0 /sec [response ] : 59 +7 +0 /sec [latency ] : max 21 (ms) min 0(ms) avg 17(ms) 50 or less : 19 100 or less : 0 200 or less : 0 400 or less : 0 else : 0
6. The default latency value is 100ms. In case you see DNS request is falling into "200 or less : 0", you can configure "cloud-dns-timeout" as following.
# set deviceconfig setting ctd cloud-dns-timeout <value> <0-60000> set cloud DNS signature query timeout in milliseconds # commit
7. You can check the cache for DNS-proxy by the following command. This command will list all cache and can be a long list. This can be reduced by selecting only one.
> show dns-proxy dns-signature cache ==> will bring all 10000 entries, please select one. Cache size: 10000 > show dns-proxy dns-signature cache fqdn < name for only one domain> > show dns-proxy dns-signature cache fqdn italic.com Domain Verdict GTID TTL Hits ---------------------------------------------------------------------------- *.italic.com White list 31388 0
8. Check the global counter before and after the DNS request.
> show counter global | match ctd_dns
ctd_dns_req_lookup_action 1 0 info ctd pktproc DNS request signature lookup yield actions
ctd_dns_req_lookup_noaction 1151 0 info ctd pktproc DNS request signature lookup yield no actions
ctd_dns_req_lookup_miss 83 0 info ctd pktproc DNS request signature lookup not found
ctd_dns_wait_pkt_drop 87 0 drop ctd pktproc DNS packet drop when waiting
ctd_dns_sess_state_verify_failed 1 0 drop ctd pktproc DNS packet drop due to session state invalid
ctd_dns_malicious_reply 2 0 info ctd pktproc MP malicious response received
ctd_dns_benign_reply 72 0 info ctd pktproc MP benign response received
ctd_dns_failed_reply 10 0 info ctd pktproc MP failed response received
9. This is one of the important debug commands that can test the DNS functionality from the Firewall as this command will trigger the signature lookup from the Firewall Management Plane.
Following command is a command that emulates the dns query from the Management Plane and help to check the connectivity.
Note: In the second command output, you can see the counter increased. (The output was modified to show only the relevant counters.)
> debug dnsproxyd dns-signature query bypass-cache yes fqdn test-malware.testpanw.come
Debug dns-signature command successful.
> show dns-proxy dns-signature counters
[response_send ] : 1 +0 +0 /sec
[request_enqueue ] : 1 +0 +0 /sec
[request_process ] : 1 +0 +0 /sec
[request_batch ] : 1 +0 +0 /sec
[response_batch ] : 1 +0 +0 /sec
[response_complete ] : 1 +0 +0 /sec
[cloud_query ] : 1 +0 +0 /sec
Signature query API:
[request ] : 1 +0 +0 /sec
[response ] : 1 +0 +0 /sec
[latency ] :
max 12 (ms) min 0(ms) avg 12(ms)
50 or less : 1
100 or less : 0
200 or less : 0
400 or less : 0
else : 0
10. Now select a client behind the firewall and make sure that the client traffic is hitting the policy with a spyware profile with DNS security enabled. Send a DNS request to a malicious domain and collect the global counters again. If the counters are changing, that means a request was sent and received.
From the firewall CLI:
Note: Please DO NOT use the command "clear dns-proxy dns-signature cache all" because it may cause an unexpected reboot.
> clear dns-proxy dns-signature cache fqdn test-malware.testpanw.com
From the client:
$ nslookup test-malware.testpanw.com ;; connection timed out; no servers could be reached ===> most of the time it may timeout $ nslookup test-dga.testpanw.com $ nslookup test-malware.testpanw.com
From the firewall CLI:
> show counter global filter delta yes | match ctd_dns
ctd_dns_req_lookup_action 1 0 info ctd pktproc DNS request signature lookup yield actions
ctd_dns_req_lookup_miss 1 0 info ctd pktproc DNS request signature lookup not found
ctd_dns_malicious_reply 1 0 info ctd pktproc MP malicious response received
ctd_dns_wildcard_reply 1 0 info ctd pktproc MP wildcard response received
ctd_dns_req_lookup_em 1 0 info ctd pktproc DNS cache lookup matched
ctd_dns_request_mp 1 0 info ctd pktproc number of requests sent to MP
ctd_dns_telemetry_req 2 0 info ctd pktproc number of telemetry requests sent to MP
ctd_dns_pkt_denied 1 0 info ctd pktproc number of DNS pkt denied
ctd_dns_action_block 2 0 info ctd pktproc DNS signature trigger block action
ctd_dns_action_log 2 0 info ctd pktproc DNS signature trigger log action
ctd_dns_action_pktlog 2 0 info ctd pktproc DNS signature trigger packet capture action
11. Login to AutoFocus and check the DNS Security tab, if the DNS Security is working properly you will be able to see your DNS query request.
12. Please attach all this information to a support case.
Additional Information
How to configure the DNS security
How to add an exception for only one DGA domain while blocking the DGA category