Rate Limiting and Retries: Best Practices When Calling a Moderation API

Calling a moderation API in production is like calling any other third-party service in production: it will fail sometimes. The question is whether your app degrades gracefully, hammers the vendor into a fuller outage, or drops user requests silently. This post is a checklist for writing a client that does the right thing.

1. Respect the rate limit before you hit it

Every moderation API rate-limits. Most expose the current budget in headers on every response:

X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 47
X-RateLimit-Reset: 1716480000

A good client reads these on every response and applies client-side throttling when Remaining drops below a safety threshold. The naive alternative, firing requests until you get a 429, is a great way to DoS yourself.

2. Retry on the right errors, not all of them

A quick classifier for HTTP responses:

Read the Retry-After header if present: it gives you the server's opinion on when to come back.

3. Exponential backoff with jitter

Linear retries cause thundering herds after an outage: everyone retries at the same intervals and brings the service down again the moment it recovers. Use exponential backoff with random jitter:

function backoffMs(attempt: number): number {
    const base = 200;          // 200ms
    const cap = 10_000;        // cap at 10s
    const exp = Math.min(cap, base * 2 ** attempt);
    return Math.random() * exp; // "full jitter"
}

// attempt 0 -> 0 to 200ms
// attempt 1 -> 0 to 400ms
// attempt 2 -> 0 to 800ms
// attempt 3 -> 0 to 1.6s
// attempt 4 -> 0 to 3.2s

"Full jitter" (random between 0 and the capped exponential) spreads the retries better than "equal jitter" or no jitter at all. Cap attempts at three to five unless you have a strong reason to keep going.

4. Idempotency keys for safety

Sending the same moderation check twice because of a retry is fine, since the result is the same. But if you are running moderation as part of a larger mutation (comment write, user ban), you do not want side effects to fire twice. Use idempotency keys:

POST /v1/moderate/text
Idempotency-Key: 7f3c9e2a-1b8d-4e6f-9a1b-2c8d7e3f4a5b

The server returns the same response for the same key for a window (usually 24 hours). Generate the key once per user action, not per HTTP attempt.

5. Circuit breakers for sustained outages

If the API is down, retrying every call adds load without value. A circuit breaker detects sustained failure (e.g. >50% error rate over 30 seconds) and skips the API entirely for a cooldown period. During the breaker-open window, you:

• Fall back to a safer default, usually "let the content through but mark it as unchecked".
• Increase human review rates on that traffic after the fact.
• Alert the oncall.

Libraries like resilience4j (JVM), opossum (Node), or Laravel's HTTP client features do this for you.

6. Timeouts, tighter than you think

A moderation API that normally responds in 50ms should not have a 30-second timeout. Set your client timeout to two or three times your p99, not to "whatever the HTTP library defaults to". A misconfigured timeout is the most common reason a slow dependency cascades into an outage.

const client = new ToxicFilter({
    apiKey: process.env.TOXICFILTER_KEY,
    timeoutMs: 200,          // not 30_000
    retries: 3,
    backoff: 'exponential-jitter',
});

7. Decide your fallback behaviour before the incident

When the moderation API is unreachable and your circuit breaker opens, what does your app do?

• Fail-open: let content through unmoderated. Your app stays up. Risk: attackers learn to attack during outages.
• Fail-closed: reject all content. Your app is safe. Risk: you break your own product during any vendor incident.
• Degraded mode: fall back to a local keyword filter or a self-hosted lightweight classifier. Middle ground; more engineering work.

There is no universally right answer. What matters is that the choice is made in advance, documented in a runbook, and the behaviour is tested with fault injection. The worst time to decide your fallback policy is at 3am during the actual outage.