Evident Auto-Remediation is not working as expected

Created On 09/26/18 13:44 PM - Last Modified 07/19/22 23:08 PM

Symptom

Symptoms

Configured auto-remediation with SNS and Lambda function, but the offending resource is not automatically remediated.

Diagnosis

Configure an AWS resource that would trigger the signature.
Wait for the fail alert to generate. It may take up to 2 scan intervals (by default, 1 scan interval is 15 minutes).
Once the alert is created, check if the offending resource is remediated. If it is remediated, then auto-remediation is configured correctly.
If not, then check the Lambda function's CloudWatch logs.
If no events are recorded, then the SNS Integration was likely not configured properly. Follow the SNS Integration section below to troubleshoot.
If an event is recorded, check the details and look for one of the following messages:
- '=> Nothing to do.'
- '=> No <resource type> to evaluate.'
- '=> Error: <error message>'
If you find any of the above messages, then troubleshoot using the AWS Lambda Function section below.

Resolution

SNS Integration

Check SNS Integration configuration
- Make sure the signature in question is "checked"
- Make sure integration triggers for both fail and warn alerts
- Note the SNS Topic ARN for later.

Ensure the SNS Integration is "Active".

Ensure that the Lambda function is subscribed to the SNS Topic configured for Evident's SNS Integration. Keep in mind that the same topic name can be re-used across regions, so be sure to check if the region is the same.

AWS Lambda Function

'=> Nothing to do.'

Depending on the script, this could either mean that the alert generated was not a fail alert or it wasn't a fail or warn alert. Compare the fail alert's "started at" timestamp and the CloudWatch event's timestamp to make sure it is the same event. If it is, please contact Palo Alto Networks support for further assistance.

'=> No <resource type> to evaluate.'

The Lambda function failed to retrieve the alert's resource ID. If the script contains this reference:

metadata['attributes']['data']['resource_id']

Change it to:

alert['data']['attributes']['resource']

If the issue persists, please contact Palo Alto Networks support for further assistance.

'=> Error: <error message>'
Please contact Palo Alto Networks support and provide the error message.