Beta docs: the shipped API is POST /api/v1/generate for gemini and sarvam. Some pages below describe planned features. See the shipped quick start.

Guides

Retries

What the gateway retries for you, and how to replay safely on your side.

The gateway retries internally before it ever returns a failure. When an error reaches your code, the gateway has already exhausted its own attempts for that request.

What the gateway retries

SituationHandling
Provider 429 / rate limitRetry with exponential backoff, up to 2 attempts
Provider 5xx or gateway timeoutRetry once on a different regional edge if available
Timeout while waiting for the first tokenFail fast with a retryable error
Your request body invalidNever retried — a 422 comes straight back

These defaults are per request. They are not configurable in v1; per-client overrides are on the roadmap.

Provider fallback

When routing is automatic and a provider fails, the gateway may complete your request on another provider instead of retrying the same one. The response reports the actual provider so you always know who answered.

Forced providers skip fallback

If you force provider: "anthropic", requests never fall back to another provider — the failure surfaces instead.

Retrying on your side

retry.ts
async function chatWithRetry(client, messages, attempts = 3) {
  for (let i = 0; i < attempts; i++) {
    try {
      return await client.chat({ messages });
    } catch (err) {
      if (!err.retryable) throw err;
      const backoff = Math.min(1000 * 2 ** i, 8000);
      await sleep(backoff + Math.random() * 250);
    }
  }
  throw new Error("still failing after retries");
}

Don't double-retry

One layer of retries

Because the gateway already retries, keep client retries count low (2–3) or the same slow request will multiply its attempts across both layers.