WildFire Upload Cancelled by DP
12623
Created On 01/23/19 20:02 PM - Last Modified 05/02/23 08:07 AM
Symptom
- A client downloads an unknown PE file (created by a developer - hash is unknown to WildFire) which should trigger an upload to WildFire. Instead, wildfire-upload.log shows that the uploads are cancelled by DP:
wildfire-upload.log 2018-10-03 12:50:20 2018-10-03 12:50:20 +0200: stegno.exe pe cancelled - by DP PUB 21980 1478 112110 0x4034 allow wildfire-upload.log 2018-10-08 14:40:08 2018-10-08 14:40:08 +0200: stegno.exe pe cancelled - by DP PUB 22757 1481 991030 0x4034 allow wildfire-upload.log 2018-10-08 14:52:41 2018-10-08 14:52:41 +0200: stegno.exe pe cancelled - by DP PUB 190749 1484 281470 0x4034 allow wildfire-upload.log 2018-10-08 15:08:08 2018-10-08 15:08:08 +0200: stegno.exe pe cancelled - by DP PUB 59443 1488 112110 0x4034 allow wildfire-upload.log 2018-10-08 15:29:44 2018-10-08 15:29:44 +0200: stegno.exe pe cancelled - by DP PUB 234409 1495 110650 0x4034 allow 1 52020 0 109 0 172.20.10.40:64594 172.20.31.30:80 98d1d24a59f340716095e978bd3a5094d56626472f8761644059cc85c4f0f9d7 wildfire-upload.log 2018-10-08 15:32:45 2018-10-08 15:32:45 +0200: stegno.exe pe cancelled - by DP PUB 109010 1496 217230 0x4034 allow 1 52020 0 109 0 172.20.10.40:64614 172.20.31.30:80 2377ee33ea7d65fd851c001a071bb5032243afcc53392e7b5ea381863c2bc5cc wildfire-upload.log 2018-10-08 15:39:34 2018-10-08 15:39:34 +0200: stegno.exe pe cancelled - by DP PUB 23556 1499 326730 0x4034 allow 1 52020 0 109 0 172.20.10.40:64655 172.20.31.30:80 93aecc4564ed6dd7beeefb75132f1612b89c2a0420ee9253f1f5bb5f608b50b7 wildfire-upload.log 2018-10-08 15:40:51 2018-10-08 15:40:51 +0200: stegno.exe pe cancelled - by DP PUB 116244 1502 1357490 0x4034 allow 1 52020 0 109 0 172.20.10.40:64672 172.20.31.30:80 3271fbbfeb472de6f959d9bbe96b54ef0265f5483f1ccd3a9ba497d3bd17f845
- The hashes in the log file do not match the real file hash: 18219154e5b345e8f2096458bfb609702e731ca53ad0b505260e981119207998 and they differ in each attempt.
- Here is an example of the session information:
admin@pan01> show session id 184522
Session 184522
c2s flow:
source: 172.20.10.40 [Office]
dst: 172.20.31.30
proto: 6
sport: 53261 dport: 80
state: INIT type: FLOW
src user: unknown
dst user: unknown
s2c flow:
source: 172.20.31.30 [DMZ]
dst: 172.20.10.40
proto: 6
sport: 80 dport: 53261
state: INIT type: FLOW
src user: unknown
dst user: unknown
start time : Mon Oct 15 10:11:09 2018
timeout : 15 sec
total byte count(c2s) : 18902
total byte count(s2c) : 4899646
layer7 packet count(c2s) : 308
layer7 packet count(s2c) : 3230
vsys : vsys1
application : web-browsing
rule : Webserver
session to be logged at end : True
session in session ager : False
session updated by HA peer : False
layer7 processing : completed
URL filtering enabled : False
session via syn-cookies : False
session terminated on host : False
session traverses tunnel : False
captive portal session : False
ingress interface : ethernet1/1.100
egress interface : ethernet1/1.121
session QoS rule : N/A (class 4)
tracker stage firewall : TCP RST - client
tracker stage l7proc : ctd queue limit
end-reason : tcp-rst-from-client
- From the global counters, we can gather the information regarding the ctd information:
admin@pan01> show counter global filter packet-filter yes delta yes
Global counters:
Elapsed time since last sampling: 8.938 seconds
name value rate severity category aspect description
--------------------------------------------------------------------------------
pkt_outstanding 3540 396 info packet pktproc Outstanding packet to be transmitted
pkt_alloc 1243 139 info packet resource Packets allocated
session_allocated 1 0 info session resource Sessions allocated
session_freed 13 1 info session resource Sessions freed
session_installed 1 0 info session resource Sessions installed
flow_host_pkt_xmt 2 0 info flow mgmt Packets transmitted to control plane
appid_ident_by_simple_sig 1 0 info appid pktproc Application identified by simple signature
appid_proc 1 0 info appid pktproc The number of packets processed by Application identification
dfa_dte_request_total 1239 138 info dfa offload The total number of dfa match using dte
dfa_hte_in_cache_lookup 1239 138 info dfa offload The total number of requests to an in cache HFA graph
dfa_session_change 1 0 info dfa offload when getting dfa result from offload, session was changed
dfa_hfa_lookup_too_many_matches 1 0 info dfa resource too many matches in HFA lookup
ctd_err_sw 1 0 info ctd pktproc ctd sw error
ctd_file_forward 1 0 info ctd pktproc The number of file forward found
ctd_bloom_filter_nohit 4 0 info ctd pktproc The number of no match for virus bloom filter
ctd_fwd_session_init 1 0 info ctd pktproc Content forward: number of successful action init
ctd_fwd_session_send 2474 276 info ctd pktproc Content forward: number of successful action send
ctd_fwd_session_fini 1 0 info ctd pktproc Content forward: number of successful action fini
ctd_fwd_session_cancel_send 1 0 info ctd pktproc Content forward: number of cancel requests sent
ctd_fwd_err_tcp_state 1 0 info ctd pktproc Content forward error: TCP in establishment when session went away
fpga_request 1238 138 info fpga offload The outstanding requests to FPGA
aho_fpga 1238 138 info aho resource The total requests to FPGA for AHO
aho_fpga_data 1855970 207649 info aho resource The total data size to FPGA for AHO
ctd_exceed_queue_limit 1 0 warn ctd resource The number of packets queued in ctd exceeds per session's limit, action bypass
ctd_process 1 0 info ctd pktproc session processed by ctd
ctd_pkt_slowpath 1238 138 info ctd pktproc Packets processed by slowpath
log_traffic_cnt 10 1 info log system Number of traffic logs
ctd_http_range_response 1 0 info ctd system Number of HTTP range responses detected by ctd
--------------------------------------------------------------------------------
Total counters shown: 28
--------------------------------------------------------------------------------
Environment
model: PA-3020 sw-version: 8.0.8 global-protect-client-package-version: 4.0.7 app-version: 8072-5053 app-release-date: 2018/10/02 14:29:35 av-version: 2759-3268 av-release-date: 2018/10/08 04:02:51 threat-version: 8072-5053 threat-release-date: 2018/10/02 14:29:35 wf-private-version: 0 wf-private-release-date: unknown url-db: paloaltonetworks wildfire-version: 286084-288681 wildfire-release-date: 2018/10/08 10:40:08 platform-family: 3000 vpn-disable-mode: off multi-vsys: on operational-mode: normal
Cause
As the session detail indicates, in this particular scenario, the cause of the DP's cancel was "tracker stage l7proc : ctd queue limit".
We can also find that the following global counter is incremented.
ctd_exceed_queue_limit 1 0 warn ctd resource The number of packets queued in ctd exceeds per session's limit, action bypass
- It means the ctd_queue is full and traffic will bypass inspection.
- The firewall skips content inspection when the content inspection queue is full.
- In this case, the DP cancels the file transfer.
Resolution
To prevent malicious content from bypassing the firewall and thus WildFire uploads, make sure the following option is disabled under [Device > Setup > Content-ID] in the WebUI:
- Go to Device> Setup > Content-ID to disable Forward segments exceeding TCP content inspection queue
NOTE:
Palo Alto Networks recommends to disable the option to ensure maximum security practices.
Please refer to the following best practice guide for more detail.
https://docs.paloaltonetworks.com/advanced-threat-prevention/administration/threat-prevention/best-practices-for-securing-your-network-from-layer-4-and-layer-7-evasions