Streaming

Set stream: true on a chat completion to receive tokens as they are generated, over Server-Sent Events (text/event-stream). Every chat model in the catalog streams, including the ones Kontinent reaches through a non-OpenAI upstream (Claude on Bedrock and on Vertex): the gateway decodes their native formats and re-emits OpenAI-shaped chunks.

Embeddings have no incremental form: sending stream: true to /v1/embeddings is a 400, not a silently ignored flag. A vector framed as an event stream is no use to anyone, and pretending to accept the parameter would leave you waiting for deltas that never come.

const stream = await client.chat.completions.create({
  model: "mistral/mistral-small-4",
  messages: [{ role: "user", content: "Write a haiku about Europe." }],
  stream: true,
});
 
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  if (chunk.usage) console.error("\nusage:", chunk.usage);
}

stream = client.chat.completions.create(
    model="mistral/mistral-small-4",
    messages=[{"role": "user", "content": "Write a haiku about Europe."}],
    stream=True,
)
 
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print("\nusage:", chunk.usage)

curl -N https://api.kontinent.ai/v1/chat/completions \
  -H "Authorization: Bearer $KONTINENT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"mistral/mistral-small-4","stream":true,
       "messages":[{"role":"user","content":"Hallo!"}]}'

Wire format

Each event is a data: line carrying a JSON chunk, terminated by a sentinel:

data: {"id":"chatcmpl-01J…","object":"chat.completion.chunk","created":1721822400,"model":"mistral/mistral-small-4","choices":[{"index":0,"delta":{"role":"assistant","content":"Hal"},"finish_reason":null}]}

data: {"id":"chatcmpl-01J…","object":"chat.completion.chunk","created":1721822400,"model":"mistral/mistral-small-4","choices":[{"index":0,"delta":{"content":"lo!"},"finish_reason":null}]}

data: {"id":"chatcmpl-01J…","object":"chat.completion.chunk","created":1721822400,"model":"mistral/mistral-small-4","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-01J…","object":"chat.completion.chunk","created":1721822400,"model":"mistral/mistral-small-4","choices":[],"usage":{"prompt_tokens":3,"completion_tokens":8,"total_tokens":11}}

data: [DONE]

Three things to rely on:

• The first chunk usually announces the role (delta.role: "assistant"), and never repeats it. Guaranteed on the chat paths Kontinent builds itself (every model on Bedrock, Claude on Vertex); on passthrough providers the upstream's own chunks are forwarded untouched, and a few omit it. Treat the role as optional and default to assistant. (/v1/responses has no chat chunks and no choices[].delta: its named events carry their own delta payloads with the actual text, and the role arrives as "role": "assistant" on the message output item.)
• The usage chunk has an empty choices array. Kontinent injects stream_options: {"include_usage": true} upstream so it is always requested; if you set stream_options yourself, your value wins. When a provider omits usage anyway, the gateway estimates tokens. Billing still happens, and the record is flagged internally as estimated.
• [DONE] means the answer is complete. It is the only positive signal that nothing is missing; see Failure mid-stream for the negative one.

Keep-alive comments

While the model is still working, the gateway sends SSE comment lines so nothing in the path mistakes a thinking model for a dead connection:

: KONTINENT PROCESSING

A comment is not an event. Per the SSE specification any line beginning with : carries no data, and every conforming client ignores it. You may use it to drive a loading indicator.

Parsing the stream

The safest option is not to parse it yourself. These clients handle SSE framing, comments, multi-line data: fields and buffering correctly:

• the official OpenAI SDKs (point baseURL at https://api.kontinent.ai/v1)
• the Vercel AI SDK
• eventsource-parser, if you want the framing handled but the chunks yourself

If you do write the loop, it has to survive four things: a chunk split across two TCP reads, a comment line, the [DONE] sentinel, and an error chunk.

const res = await fetch("https://api.kontinent.ai/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.KONTINENT_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "bedrock/claude-sonnet-5",
    messages: [{ role: "user", content: "Hallo!" }],
    stream: true,
  }),
});
 
// Failures BEFORE the first byte are plain JSON with a real status code.
if (!res.ok) {
  const { error } = await res.json();
  throw new Error(`${error.code}: ${error.message}`);
}
console.log("generation:", res.headers.get("X-Generation-Id"));
 
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";
let complete = false;
 
// A labelled loop, so `break read` can leave both loops from the inner one.
read: while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
 
  let nl: number;
  while ((nl = buffer.indexOf("\n")) !== -1) {
    const line = buffer.slice(0, nl).trim();
    buffer = buffer.slice(nl + 1);
 
    if (line === "" || line.startsWith(":")) continue; // blank or keep-alive
    if (!line.startsWith("data:")) continue;
 
    const payload = line.slice(5).trim();
    if (payload === "[DONE]") { complete = true; break; }
 
    const chunk = JSON.parse(payload);
 
    // Failures AFTER the first byte arrive here; the status was already 200.
    if (chunk.error) {
      throw new Error(`${chunk.error.code}: ${chunk.error.message} (partial answer)`);
    }
    if (chunk.usage) console.error("usage:", chunk.usage);
 
    const delta = chunk.choices?.[0]?.delta;
    if (delta?.reasoning) process.stderr.write(delta.reasoning); // thinking
    if (delta?.content) process.stdout.write(delta.content);     // answer
  }
  if (complete) break read;
}
 
// EOF is not success. A connection that dropped before `[DONE]` leaves a
// truncated answer that looks exactly like a short one.
if (!complete) throw new Error("stream ended without [DONE]: answer is incomplete");

import json, requests
 
with requests.post(
    "https://api.kontinent.ai/v1/chat/completions",
    headers={"Authorization": f"Bearer {api_key}"},
    json={
        "model": "bedrock/claude-sonnet-5",
        "messages": [{"role": "user", "content": "Hallo!"}],
        "stream": True,
    },
    stream=True,
) as r:
    # Failures BEFORE the first byte are plain JSON with a real status code.
    if r.status_code != 200:
        err = r.json()["error"]
        raise RuntimeError(f"{err['code']}: {err['message']}")
    print("generation:", r.headers.get("X-Generation-Id"))
 
    complete = False
    for line in r.iter_lines(decode_unicode=True):
        if not line or line.startswith(":"):   # blank or keep-alive
            continue
        if not line.startswith("data:"):
            continue
        payload = line[5:].strip()
        if payload == "[DONE]":
            complete = True
            break
        chunk = json.loads(payload)
 
        # Failures AFTER the first byte arrive here; the status was already 200.
        if "error" in chunk:
            raise RuntimeError(f"{chunk['error']['code']}: {chunk['error']['message']} (partial answer)")
        if chunk.get("usage"):
            print("\nusage:", chunk["usage"])
 
        delta = (chunk.get("choices") or [{}])[0].get("delta", {})
        if delta.get("reasoning"):
            print(delta["reasoning"], end="", flush=True)   # thinking
        if delta.get("content"):
            print(delta["content"], end="", flush=True)     # answer
 
    # EOF is not success: without [DONE] the answer is truncated.
    if not complete:
        raise RuntimeError("stream ended without [DONE]: answer is incomplete")

Reasoning models

Thinking streams in the same delta, and it arrives before the answer:

data: {"choices":[{"index":0,"delta":{"role":"assistant","reasoning":"Comparing the ","reasoning_details":[{"type":"reasoning.text","text":"Comparing the ","index":0}]}}]}

data: {"choices":[{"index":0,"delta":{"reasoning":"tenths…","reasoning_details":[{"type":"reasoning.text","text":"tenths…","index":0}]}}]}

data: {"choices":[{"index":0,"delta":{"content":"9.9 is larger."}}]}

delta.reasoning is a string you can concatenate and print. delta.reasoning_details is the structured form you send back to continue a thinking conversation. See Reasoning for the request parameter, the detail types, and how encrypted thinking is reported.

Tool calls

Tool calls stream as delta.tool_calls, on every chat model. The two Claude paths (Bedrock and Vertex) are translated into exactly this shape, so the wire format does not tell you which upstream served:

data: {"choices":[{"index":0,"delta":{"role":"assistant","tool_calls":[{"index":0,"id":"call_a","type":"function","function":{"name":"get_weather","arguments":""}}]}}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"ci"}}]}}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"ty\":\"Berlin\"}"}}]}}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

Three rules, and they are OpenAI's:

• index is the key, not the order of arrival. The first fragment of a call carries id, type and function.name; every fragment after it carries argument text only. Two parallel calls interleave, so accumulate into a map keyed by index.
• Arguments are a JSON string that arrives in pieces. A single fragment is not parseable JSON. Concatenate first, JSON.parse once finish_reason has arrived.
• finish_reason: "tool_calls" is how the turn ends when the model wants a tool run. Anything else, including "error", means the call you hold may be incomplete: never run a tool from a stream that did not finish.

To continue the loop, send the assistant turn back with its accumulated tool_calls, then one {"role": "tool", "tool_call_id": "...", "content": "..."} message per call. With a reasoning model, replay that assistant turn's reasoning_details too: Claude verifies its own thinking and refuses to continue from thinking it cannot attribute to itself. The official SDKs do the accumulation for you.

Streaming /v1/responses

The Responses API streams named events rather than anonymous chunks, each carrying a monotonic sequence_number:

event: response.created
data: {"type":"response.created","sequence_number":0,"response":{"id":"resp_01J…","status":"in_progress",…}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":1,"output_index":0,"item":{"type":"reasoning","id":"rs_01J…"}}

event: response.reasoning_text.delta
data: {"type":"response.reasoning_text.delta","sequence_number":2,"item_id":"rs_01J…","output_index":0,"content_index":0,"delta":"Comparing the "}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":7,"item_id":"msg_01J…","output_index":1,"content_index":0,"delta":"9.9 is larger."}

event: response.completed
data: {"type":"response.completed","sequence_number":11,"response":{"id":"resp_01J…","status":"completed","output":[…],"usage":{…}}}

The full event set is response.created, response.output_item.added, response.reasoning_text.delta, response.reasoning_text.done, response.content_part.added, response.output_text.delta, response.output_text.done, response.content_part.done, response.function_call_arguments.delta, response.function_call_arguments.done, response.output_item.done, and response.completed. Keep-alive comments and the [DONE]-free failure rule apply here too. A broken generation ends with response.failed, whose response object carries status: "failed", an error, and whatever output was produced before the break.

A tool call is its own output item of type function_call, carrying call_id, name and arguments. Its arguments stream as response.function_call_arguments.delta fragments and are only parseable once response.function_call_arguments.done has repeated them whole. To continue the loop, send the item back in input as {"type": "function_call", "call_id": "...", "name": "...", "arguments": "..."} followed by {"type": "function_call_output", "call_id": "...", "output": "..."}.

Encrypted-only thinking produces no response.reasoning_text.delta, because there is no plaintext to stream. The blob still reaches you on the reasoning item of the final event.

See the API reference for the request shape.

Cancellation

Stop reading and close the connection, with AbortController in JavaScript, or by leaving the with block in Python. Kontinent closes the upstream connection in turn, so providers that honour cancellation stop generating immediately.

const controller = new AbortController();
const stream = await client.chat.completions.create(
  { model: "mistral/mistral-small-4", messages, stream: true },
  { signal: controller.signal },
);
// later:
controller.abort();

Timeouts and keep-alives

The keep-alive deliberately does not reset the 90 s upstream limit. If it did, a provider that died mid-generation would hold the connection open indefinitely while the gateway kept reassuring you. The two answer different questions: is the client still reachable and is the provider still working.

When the 90 s limit does fire, you get the failure chunk described next, not a silent truncation.

Failure mid-stream

If the model defines fallback providers, upstream 401/403/429/5xx/connect errors are retried on the next provider, only before the first byte reaches you. That is the last moment a different provider can still serve the whole answer.

After the first byte the status code is already 200 and cannot be changed, so the failure is announced inside the stream:

data: {"id":"chatcmpl-01J…","object":"chat.completion.chunk","created":1721822400,"model":"bedrock/claude-sonnet-5","provider":"bedrock","error":{"code":"upstream_error","message":"the provider failed mid-generation; the answer is incomplete"},"choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}]}

• The error object sits at the top level, beside the ordinary chunk fields.
• finish_reason is "error", so a client that only tracks finish reasons still notices.
• provider names the upstream that dropped, which is what makes the failure actionable.
• No [DONE] follows. Detect the failure by testing each chunk for error, or for finish_reason: "error".

Treat it as retriable on your side: the text you hold is partial, and the tokens already generated are billed. Errors that happen before streaming starts are ordinary JSON with a real status code (400, 401, 402, 429, 502, 503). See Handling errors.

Response headers

Three headers tell you what happened to a request, streaming or not. X-Generation-Id is on every response, errors included; X-Provider and X-Served-Model are on every response a provider served. All headers are listed in Request & response headers.

X-Generation-Id is the handle for reconciling a single call against your logs and your invoice. On the endpoints whose response body Kontinent builds itself (Bedrock, Vertex Anthropic, /v1/responses) the body's id is that value with an endpoint prefix: chatcmpl-<id> for chat completions, resp_<id> for /v1/responses. Strip the prefix before comparing; the two are not literally equal.

X-Provider and X-Served-Model are the only place you can see a silent failover. For a sovereignty product that is not a debugging nicety: you chose a model partly for where it runs, and after a fallback the answer may have come from your second choice.

Checklist for a robust stream loop

1. Check the HTTP status before reading the body.
2. Record X-Generation-Id and X-Provider.
3. Buffer reads and split on newlines, because a chunk can straddle two TCP reads.
4. Skip blank lines and lines starting with :.
5. Stop on [DONE], and treat any other ending as incomplete.
6. Test each chunk for error / finish_reason: "error".
7. Read usage from the chunk with empty choices.
8. Render delta.reasoning separately from delta.content. Thoughts are not the answer.
9. Accumulate delta.tool_calls by index, concatenate function.arguments, and parse them only after finish_reason: "tool_calls".

Related