Chat Completions

Hopper serves Gemma 4 31B, Google’s open model, through an OpenAI-compatible chat completions endpoint. Point any OpenAI client at https://api.withhopper.com/v1 and request model gemma-4-31b; the context window is 25,000 tokens. TTFT is ~37 ms on short prompts, measured from us-west-2 with cold unique prompts on a keyed connection.

Make a request

from openai import OpenAI

client = OpenAI(
    base_url="https://api.withhopper.com/v1",
    api_key="sk_hopper_...",
)

response = client.chat.completions.create(
    model="gemma-4-31b",
    messages=[{"role": "user", "content": "Confirm the appointment for 3pm."}],
)
print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.withhopper.com/v1",
  apiKey: "sk_hopper_...",
});

const response = await client.chat.completions.create({
  model: "gemma-4-31b",
  messages: [{ role: "user", content: "Confirm the appointment for 3pm." }],
});
console.log(response.choices[0].message.content);

curl https://api.withhopper.com/v1/chat/completions \
  -H "Authorization: Bearer sk_hopper_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-4-31b",
    "messages": [{"role": "user", "content": "Confirm the appointment for 3pm."}]
  }'

Requests are forwarded to the serving engine unchanged, so any parameter the OpenAI chat API accepts passes through. Other /v1/* routes, such as GET /v1/models and POST /v1/completions, are forwarded the same way.

Streaming

Pass stream: true and the response arrives as SSE chunks in the OpenAI format.

stream = client.chat.completions.create(
    model="gemma-4-31b",
    messages=[{"role": "user", "content": "Confirm the appointment for 3pm."}],
    stream=True,
    stream_options={"include_usage": True},  # optional: get the final usage chunk
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

The gateway meters every stream regardless of what you send: if the request omits stream_options.include_usage, the gateway injects it upstream, reads the usage chunk for billing, and strips it from your stream. Set stream_options: {"include_usage": true} yourself and the usage chunk is delivered to you as the final data event. Spend headers differ between the two modes because a stream’s cost is unknown until it ends:

Response	`x-hopper-request-cost-usd`	`x-hopper-credits-remaining-usd`
Non-streaming	Exact cost of this request	Balance after this request
Streaming	Absent	Pre-request balance

Tool calling and structured outputs

Tool calling and structured outputs are enabled on gemma-4-31b. Define tools in the OpenAI format:

response = client.chat.completions.create(
    model="gemma-4-31b",
    messages=[{"role": "user", "content": "Book a table for two at 7pm."}],
    tools=[{
        "type": "function",
        "function": {
            "name": "book_table",
            "parameters": {
                "type": "object",
                "properties": {
                    "party_size": {"type": "integer"},
                    "time": {"type": "string"},
                },
                "required": ["party_size", "time"],
            },
        },
    }],
)
print(response.choices[0].message.tool_calls)

response_format for structured outputs passes through the same way.

Keep one warm connection

Every turn of a voice agent makes an LLM request, and a fresh TCP + TLS handshake adds multiple round trips to that turn — the default OpenAI client pays it repeatedly, because its 5 s keepalive lapses in normal conversational gaps. hopper-client is a drop-in replacement with an httpx transport tuned for voice:

Setting	OpenAI default	Hopper default	Why it changes
`http2`	off	on	One connection multiplexes many streams and stays reusable across turns.
`keepalive_expiry`	5 s	300 s	Spans normal conversational gaps so the next turn reuses the connection instead of re-handshaking.
`max_keepalive_connections`	100	20	With HTTP/2 one connection carries many streams, so the warm pool stays small.
`connect` timeout	5 s	3 s	Fail a bad connection fast so the caller can retry or hedge instead of stalling a live turn.

Measured against the same endpoint, these defaults roughly halve median TTFT; the numbers and benchmark script are in the client post. Create the client once, at startup, and reuse it for every turn.

from hopper import AsyncHopper

# initialize once — reused for every turn
client = AsyncHopper(base_url="https://api.withhopper.com/v1", api_key="sk_hopper_...")

Everything else is the OpenAI SDK you already use. One warm connection also fits the rate limit: 600 requests per minute is per key, not per connection, so a single HTTP/2 connection multiplexing all streams costs you nothing.

Limits

Limit	Value	Past it
Request body	10 MB	413 `body_too_large`
Upstream timeout	300 s	502 `upstream_unavailable`
Context window	25,000 tokens	The request is rejected
Rate limit	600 requests/min per key	429 with `Retry-After`

Get started

LLM

Text to Speech

Speech to Text

Platform

Chat Completions

Make a request

Streaming

Tool calling and structured outputs

Keep one warm connection

Limits

​Make a request

​Streaming

​Tool calling and structured outputs

​Keep one warm connection

​Limits

Make a request

Streaming

Tool calling and structured outputs

Keep one warm connection

Limits