> ## Documentation Index
> Fetch the complete documentation index at: https://docs.withhopper.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Chat Completions

> OpenAI-compatible chat completions on gemma-4-31b with ~37 ms TTFT on short prompts

Hopper serves Gemma 4 31B, Google's open model, through an OpenAI-compatible chat completions endpoint. Point any OpenAI client at `https://api.withhopper.com/v1` and request model `gemma-4-31b`; the context window is 25,000 tokens. TTFT is \~37 ms on short prompts, measured from us-west-2 with cold unique prompts on a keyed connection.

## Make a request

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.withhopper.com/v1",
      api_key="sk_hopper_...",
  )

  response = client.chat.completions.create(
      model="gemma-4-31b",
      messages=[{"role": "user", "content": "Confirm the appointment for 3pm."}],
  )
  print(response.choices[0].message.content)
  ```

  ```typescript TypeScript theme={null}
  import OpenAI from "openai";

  const client = new OpenAI({
    baseURL: "https://api.withhopper.com/v1",
    apiKey: "sk_hopper_...",
  });

  const response = await client.chat.completions.create({
    model: "gemma-4-31b",
    messages: [{ role: "user", content: "Confirm the appointment for 3pm." }],
  });
  console.log(response.choices[0].message.content);
  ```

  ```bash cURL theme={null}
  curl https://api.withhopper.com/v1/chat/completions \
    -H "Authorization: Bearer sk_hopper_..." \
    -H "Content-Type: application/json" \
    -d '{
      "model": "gemma-4-31b",
      "messages": [{"role": "user", "content": "Confirm the appointment for 3pm."}]
    }'
  ```
</CodeGroup>

Requests are forwarded to the serving engine unchanged, so any parameter the OpenAI chat API accepts passes through. Other `/v1/*` routes, such as `GET /v1/models` and `POST /v1/completions`, are forwarded the same way.

## Streaming

Pass `stream: true` and the response arrives as SSE chunks in the OpenAI format.

```python theme={null}
stream = client.chat.completions.create(
    model="gemma-4-31b",
    messages=[{"role": "user", "content": "Confirm the appointment for 3pm."}],
    stream=True,
    stream_options={"include_usage": True},  # optional: get the final usage chunk
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

The gateway meters every stream regardless of what you send: if the request omits `stream_options.include_usage`, the gateway injects it upstream, reads the usage chunk for billing, and strips it from your stream. Set `stream_options: {"include_usage": true}` yourself and the usage chunk is delivered to you as the final data event.

Spend headers differ between the two modes because a stream's cost is unknown until it ends:

| Response      | `x-hopper-request-cost-usd` | `x-hopper-credits-remaining-usd` |
| ------------- | --------------------------- | -------------------------------- |
| Non-streaming | Exact cost of this request  | Balance after this request       |
| Streaming     | Absent                      | Pre-request balance              |

## Tool calling and structured outputs

Tool calling and structured outputs are enabled on `gemma-4-31b`. Define tools in the OpenAI format:

```python theme={null}
response = client.chat.completions.create(
    model="gemma-4-31b",
    messages=[{"role": "user", "content": "Book a table for two at 7pm."}],
    tools=[{
        "type": "function",
        "function": {
            "name": "book_table",
            "parameters": {
                "type": "object",
                "properties": {
                    "party_size": {"type": "integer"},
                    "time": {"type": "string"},
                },
                "required": ["party_size", "time"],
            },
        },
    }],
)
print(response.choices[0].message.tool_calls)
```

`response_format` for structured outputs passes through the same way.

## Keep one warm connection

Every turn of a voice agent makes an LLM request, and a fresh TCP + TLS handshake adds multiple round trips to that turn — the default OpenAI client pays it repeatedly, because its 5 s keepalive lapses in normal conversational gaps. `hopper-client` is a drop-in replacement with an httpx transport tuned for voice:

| Setting                     | OpenAI default | Hopper default | Why it changes                                                                                     |
| --------------------------- | -------------- | -------------- | -------------------------------------------------------------------------------------------------- |
| `http2`                     | off            | on             | One connection multiplexes many streams and stays reusable across turns.                           |
| `keepalive_expiry`          | 5 s            | 300 s          | Spans normal conversational gaps so the next turn reuses the connection instead of re-handshaking. |
| `max_keepalive_connections` | 100            | 20             | With HTTP/2 one connection carries many streams, so the warm pool stays small.                     |
| `connect` timeout           | 5 s            | 3 s            | Fail a bad connection fast so the caller can retry or hedge instead of stalling a live turn.       |

Measured against the same endpoint, these defaults roughly halve median TTFT; the numbers and benchmark script are in the [client post](https://withhopper.com/blog/llm-client-voice-agents).

Create the client once, at startup, and reuse it for every turn.

```python theme={null}
from hopper import AsyncHopper

# initialize once — reused for every turn
client = AsyncHopper(base_url="https://api.withhopper.com/v1", api_key="sk_hopper_...")
```

Everything else is the OpenAI SDK you already use. One warm connection also fits the rate limit: 600 requests per minute is per key, not per connection, so a single HTTP/2 connection multiplexing all streams costs you nothing.

## Limits

| Limit            | Value                    | Past it                    |
| ---------------- | ------------------------ | -------------------------- |
| Request body     | 10 MB                    | 413 `body_too_large`       |
| Upstream timeout | 300 s                    | 502 `upstream_unavailable` |
| Context window   | 25,000 tokens            | The request is rejected    |
| Rate limit       | 600 requests/min per key | 429 with `Retry-After`     |