AWS services throttled (rate-limit exceeded)

AWS services throttled (rate-limit exceeded)

4584
Created On 09/26/18 13:45 PM - Last Updated 07/17/19 22:30 PM


Symptom

Symptoms

  1. Reports contain error alerts with the following message:
    "message" => "The signature is being rate-limited by AWS.  Please stagger API usage so the signature is not running concurrently with other processes on the same service or request an API rate limit increase from AWS for the impacted service."
  2. Other tools reports RequestLimitExceeded exceptions when making AWS API calls.

Diagnosis

Identify which services are hitting the API threshold:

  • View rate-limited error alerts and determine the offending signature and service
  • Analyze CloudTrail events to find which AWS API method failed with "Too many requests" exception (error code: 429)


Resolution

Evident Monitoring can potentially generate a large number of AWS API calls during a scan cycle.  If AWS's rate limit is reached, these API requests may result in rate-limit exceeded exceptions for both Evident Monitoring and other tools.  To reduce API calls made by Evident Monitoring, perform the following or each AWS service that is experiencing rate-limiting issues:

 

  1. Increase the scan interval for specific services to lower the overall number of API calls.  You can configure the scan interval from Evident Monitoring Web UI, go to Control Panel -> External Accounts -> Scan Interval.  Increasing the scan interval will reduce the overall number of API calls, but this will not reduce the number of API calls made per scan cycle.
  2. Disable signatures that you do not need.  You can disable signatures by going to Control Panel -> Disabled Signatures (https://esp.evident.io/control_panel/disabled_signatures). Evident Monitoring will not run any disabled signatures; however, do note that existing alerts may take up to an hour to end.
  3. Reduce the number of AWS assets.  For certain services, the number of assets (e.g. EC2 instances, EBS Volumes) is proportional to the number of API calls that Evident Monitoring makes.  Cleaning up unused assets can lower the number of API calls made per scan cycle.
  4. Stagger your other API-heavy workloads so they run at different times from your Evident Monitoring scans.
  5. Contact AWS to increase the rate limit for you account.


Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClnZCAS&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments
Choose Language