Encountering an "API Plan Limit Exceeded" Error Even When Traffic Is Well Underneath the Set Rate Limit.
Document ID : KB000008735
Last Modified Date : 14/02/2018
 Show Technical Document Details
Issue:

When load testing (or very high traffic to) an API call which uses the Rate Limit assertion set by an API Plan or Account Plan in the CA API Developer Portal ("Portal"), the following error may be encountered even when traffic is well underneath the value set for the Rate Limit assertion:

API Plan Limit Exceeded
Environment:
- Portal version 3.5 and lower, and CA API Gateway ("Gateway") version 9.2 and lower.
Cause:

When a Rate Limit is configured on an API Plan or Account Plan in the Portal, it will cause API calls to be throttled to stay underneath the value set for the Rate Limit.

By default, the Portal will configure the Rate Limit assertion (found in the associated policy viewable in Policy Manager) with the value chosen earlier for the Maximum Requests Per Second function, and have the Spread Limit Over X Sec Window function disabled, and an action set to Throttle. This does not normally cause issues with typical traffic volumes, but depending on the frequency of traffic/API calls (especially if it's being load tested), some requests can be throttled in order to stay under the value set for the Maximum Requests Per Second function, negatively impacting the API client sending the request.

The Rate Limit Assertion does some math for the Maximum Requests Per Second, which is not always well understood. If the value is set to 100 (for example), then the Rate Limit assertion will process all traffic as expected as long as there is no more than 1 request per 1/100th of a second. If two calls come in at under 1/100th of a second apart (which often happens when using load testing tools) for example, then the second request will be throttled. If a request is throttled, it is considered a failure case and will throw the error to the client.
Resolution:

If there is a high traffic load (i.e. a load test running), and there is an expectation that it still be handled as it is viewed as under the Maximum Requests Per Second value, then it is recommended to enable the Spread Limit Over X Sec Window function on the Rate Limit assertion. The Spread Limit Over X Sec Window function allows for bursts of traffic, which includes traffic coming in under the buffer that is set by the value for Maximum Requests Per Second which is often the case with load tests. The Gateway product documentation (more details found in the Additional Information section below) explains the following for the Spread Limit Over X Sec Window function on the Rate Limit assertion:

  • What it does: "Determines whether to allow a burst of requests to be spread across a window of time or whether to enforce a hard cap."
  • When the Spread Limit Over X Sec Window is disabled, then "the Gateway only accepts requests arriving no sooner than 1/limit of a second. For example, if the Max requests per second is 100, at least 1/100 second must have elapsed between requests. Requests that arrive sooner are either throttled or shaped (based on the "When limit exceeded" setting). Disallowing burst traffic is recommended only for advanced users."
  • It also has a note which reads: "It is not recommended to disable burst traffic on a counter that will be servicing multiple concurrent requests, particularly at high rates. Doing so can lead to unintended throttling or delaying of multiple requests that arrive at exactly the same time."

The recommended resolution is to enable the Spread Limit Over X Sec Window function on the Rate Limit assertion, typically with a value of 60 seconds or higher. The gotcha is that the Portal does not automatically enable that function, however it can be configured to after a couple of file changes and a restart of the Portal service. This can be achieved by following the steps below on the Portal:

  1. Connect to the Portal via SSH to get a terminal, or connect on a console level.
  2. Edit the following file using vi: /opt/Deployments/lrs/server/webapps/ROOT/templates/apiplan.xml
    1. Change <L7p:HardLimit booleanValue="true"/> to <L7p:HardLimit booleanValue="false"/>
  3. Edit the following file using vi: /opt/Deployments/lrs/server/webapps/ROOT/templates/accountplan.xml
    1. Change <l7:HardLimit>true</l7:HardLimit> to <l7:HardLimit>false</l7:HardLimit>
  4. Restart the Portal for the changes to take effect.
  5. Sync the API Plans and Account Plans using the buttons on the Layer 7 Gateway plugin page in the /admin section of the Portal website to initiate a sync.
  6. Refresh the Policy Manager to the Gateway and view the changes. The Rate Limit assertion should now have the Spread Limit Over X Sec Window function enabled.

After the changes above, all new API Plans or Account Plans will also have that automatically applied. This is a useful resolution if the Spread Limit Over X Sec Window function is required in the environment, particularly if developers do not have access to Policy Manager to make the changes manually.
Additional Information: