Guides
Streaming
Stream tokens as they are generated, for a chat experience that feels instant.
Set stream: true on a chat request and the gateway responds with server-sent events, one chat.completion.chunk per increment of generated output instead of one big JSON body.
Streaming a chat response
const stream = await client.chat.stream({
messages: [{ role: "user", content: "Write a haiku about routers" }],
});
let text = "";
for await (const chunk of stream) {
text += chunk.delta;
process.stdout.write(chunk.delta);
}
console.log(chunk.usage); // final cumulative usageThe chunk shape
data: { "id":"chatcmpl_01J8R2aKq...", "object":"chat.completion.chunk",
"provider":"anthropic", "model":"claude",
"choices":[{ "index":0, "delta":{ "content":" routers" },
"finish_reason":null }] }
data: { "id":"chatcmpl_01J8R2aKq...", ...,
"choices":[{ "index":0, "delta":{}, "finish_reason":"stop" }], "usage": { "prompt_tokens":9, "completion_tokens":12, "total_tokens":21 } }
data: [DONE]Cancelling a stream
const controller = new AbortController();
const stream = await client.chat.stream(
{ messages: [{ role: "user", content: "Explain LLMs" }] },
{ signal: controller.signal },
);
// later, from a stop button:
controller.abort();Aborting is free
Cancelling a stream stops provider billing at the last emitted token. Unused generation time is not billed by OpenLink.
Streaming limitations
- Streams have the same rate limits as non-streaming calls — each stream counts as one request.
max_tokensstill caps a streamed completion.- Temperature and provider selection work exactly as in non-streaming mode.