Beta docs: the shipped API is POST /api/v1/generate for gemini and sarvam. Some pages below describe planned features. See the shipped quick start.

Guides

Streaming

Stream tokens as they are generated, for a chat experience that feels instant.

Set stream: true on a chat request and the gateway responds with server-sent events, one chat.completion.chunk per increment of generated output instead of one big JSON body.

Streaming a chat response

stream.ts
const stream = await client.chat.stream({
  messages: [{ role: "user", content: "Write a haiku about routers" }],
});

let text = "";

for await (const chunk of stream) {
  text += chunk.delta;
  process.stdout.write(chunk.delta);
}

console.log(chunk.usage); // final cumulative usage

The chunk shape

JSON
data: { "id":"chatcmpl_01J8R2aKq...", "object":"chat.completion.chunk",
  "provider":"anthropic", "model":"claude",
  "choices":[{ "index":0, "delta":{ "content":" routers" },
  "finish_reason":null }] }

data: { "id":"chatcmpl_01J8R2aKq...", ...,
  "choices":[{ "index":0, "delta":{}, "finish_reason":"stop" }], "usage": { "prompt_tokens":9, "completion_tokens":12, "total_tokens":21 } }

data: [DONE]

Cancelling a stream

abort.ts
const controller = new AbortController();

const stream = await client.chat.stream(
  { messages: [{ role: "user", content: "Explain LLMs" }] },
  { signal: controller.signal },
);

// later, from a stop button:
controller.abort();

Aborting is free

Cancelling a stream stops provider billing at the last emitted token. Unused generation time is not billed by OpenLink.

Streaming limitations

  • Streams have the same rate limits as non-streaming calls — each stream counts as one request.
  • max_tokens still caps a streamed completion.
  • Temperature and provider selection work exactly as in non-streaming mode.