How to troubleshoot CTD processing values spiked or running high in > debug dataplane pow performance

> debug dataplane pow performance
func                                  max-μs   avg-μs        count     total-μs  ac-max-μs  ac-avg-μs         ac-count      ac-total-μs
sml_vm                                     0      276491782.0  0            0         65        1.1           312208           369019
ctd_token                                  0      36.0           27197221  0        180       20.8            18000           374440
detector_run_p1                            0      0.0            0            0         66        4.9             8509            42053
detector_run_p2                            0      0.0            0            0         27        1.5             8786            13669
regex_lookup                               0      0.0            0            0       1093        7.0            11494            81464

• PAN-OS
• CTD (Content & Threat Detection)

• A certain new traffic flow was introduced that is large or heavy on the firewall's inspection engine such as: unknown-udp, unknown-tcp, ms-ds-smb, mssql-db, etc.
• An abnormally high quantity or rate of a heavier type of traffic is going through the firewall such as: unknown-udp, unknown-tcp, ms-ds-smb, mssql-db, etc.
• All or a large amount of traffic goes through one Security Policy allow rule with 'strict' Security Profile(s) attached (especially when the amount of traffic is approaching that model's maximum Performance and Capacity - in this case, sml_vm, ctd_token, detector_run_p1, and detector_run_p2 being high would be expected).
• All or most traffic goes through a single broad Security Profile rule that has many heavy or strict Security Profiles configured on it, especially if there are high amounts of unclassified traffic (unknown-udp, unknown-tcp) or high data rate traffic (ms-ds-smb, mssql-db, etc.) passing through the firewall
• An inefficient Custom URL Filtering, Custom Application, or Custom Threat signature/object was configured which uses too many wildcards, uses nothing but a single wildcard, is too broad, etc. (this misconfiguration will often cause the function regex_lookup to go high specifically). To resolve this, remove any performance-impacting, inefficient wildcards or patterns in URL Filtering Objects, Custom Applications, or Custom Signatures - see Nested Wildcard(*) in URLs May Severely Affect Performance for more details

1. Take a baseline of these values before the issue occurred (or check historical values). If the current values for sml_vm, ctd_token, detector_run_p1, and detector_run_p2 are much higher than previous values seen, then they might be the culprit of the high CPU or traffic issue. If the current values are around the same as old values when the issue was not occurring, then they might not be the culprit or root cause of the issue. If the value(s) are abnormally higher than the baseline values before the issue started occurring, then proceed to Step 2 below.
2. Identify which new traffic flow (by Source IP, Source Port, Destination IP, Destination Port, Application) was introduced which started causing these value(s) to increase
   1. Navigate to ACC > Network Activity tab > view the Source IP Activity, Destination IP Activity, and Application Usage charts
   2. Review and assess any traffic flows in the above charts that stand out as outliers taking up a high amount of: bytes, sessions, packet rate, etc. (especially new traffic flows that have shown up only since the issue began occurring). Look for one specific traffic flow or application which is showing up in the charts only when the issue is occurring and is not showing up when the issue is not occurring
   3. Run the CLI Command: > show running application statistics to check if any applications have an abnormally high value for any column (especially for heavier data rate applications such as: web-browsing, unknown-udp, unknown-tcp, dns, ms-ds-smb, ms-ds-smbv2, ms-ds-smbv3, and mssql-db, or any Custom Applications)
   4. If any traffic flows or applications stand out as possible offenders in the above steps i.e. where their values are higher than at normal, working, non-issue times when there is not a traffic/high CPU issue), then temporarily halt that traffic flow through the firewall and see if the CTD-related values in > debug dataplane pow performance (sml_vm, ctd_token, detector_run_p1, or detector_run_p2) go back down to normal levels and if the traffic issue is now gone
3. Reduce any inspections done on that specific traffic flow by:
   1. Create a Custom Application for that traffic (so that it does not get classified as unknown-udp or unknown-tcp)
   2. Send that traffic instead through a Security Policy rule which does not have any Security Profile(s) attached (or use a less strict profile with less signatures enabled)
   3. Enable DSRI on the Security Policy rule which that traffic takes
   4. Create an Application Override for that traffic (only use if the traffic is trusted by your organization)