Vulnerability scanning is automatically enabled if the custom app is based off a "base app" like HTTP or SMB and also based on the settings of that policy's vulnerability/spyware profile.
Note: The spyware checkbox in the screenshot is a non-operational.
To enable Anti-Virus Scanning
Anti-Virus Scanning for Custom-Application is done by setting the Virus-Identification flag to "yes" as follows:
Multi-Vsys Platforms :
# set vsys vsys1 application myapp virus-ident yes