Add the BYOK chat proxy endpoint
Python · LLM apps · intermediate · greenfield
Adds the bring-your-own-key chat proxy the console's free tier was promised: callers send their own provider key with each request, the endpoint runs one pinned-model completion behind a constant system prompt, and provider rejections come back as clean 401/502 responses with fixed details. Exercised the happy path against a stub provider, an invalid key (401), a throttled key (502), and both missing-key shapes (400).
The `/chat` endpoint is public: anyone can call it without an account, which is the whole point of BYOK — the console never pays for a caller's tokens. The server process also holds a real `OPENAI_API_KEY` in its environment, used by a separate nightly batch-summary job.
Requirements
- `POST /chat` accepts `{"message": string, "api_key": string | null}` and returns `{"answer": string}`. `message` is 1–2000 characters, bounds enforced by Pydantic validation. The endpoint is public — no authentication in front of it.
- The endpoint is BYOK: every `/chat` request runs against the provider with the caller's own `api_key` only. The server's configured `OPENAI_API_KEY` is reserved for the internal nightly batch-summary job and must never be used for a `/chat` provider call nor appear in any `/chat` HTTP response — not in the body, not in headers, not in an error `detail`. Error details contain no key material at all, from either side.
- A missing `api_key` (`null`) or an empty string responds HTTP 400 with `detail` exactly `"api_key is required"`; no provider call is made.
- If the provider rejects the caller's key (`AuthenticationError`), the endpoint responds HTTP 401 with a fixed detail that does not quote the key. `RateLimitError`, `APITimeoutError` and any other `APIStatusError` respond HTTP 502. The SDK's two built-in retries are the only retry mechanism.
- The system prompt is a constant; the user's message only ever enters as a `user` message. The model is the pinned dated snapshot `gpt-4o-mini-2024-07-18`, `max_tokens=400`, and `answer` is `""` when the provider returns `None` content.
Files touched
- app/byok_chat.py
--- app/byok_chat.py
+"""BYOK chat proxy: callers pay with their own provider key."""
+
+from __future__ import annotations
+
+import os
+
+from fastapi import APIRouter, HTTPException
+from openai import (
+ APIStatusError,
+ APITimeoutError,
+ AsyncOpenAI,
+ AuthenticationError,
+ RateLimitError,
+)
+from pydantic import BaseModel, Field
+
+router = APIRouter()
+
+MODEL = "gpt-4o-mini-2024-07-18" # pinned snapshot, per ADR-011
+
+SYSTEM_PROMPT = (
+ "You are the writing assistant on the Northwind console. Answer the "
+ "user's request in one short paragraph."
+)
+
+# Reserved for the nightly batch-summary job. Per the endpoint contract this
+# key never runs a /chat provider call and never appears in a /chat response.
+SERVER_API_KEY = os.environ.get("OPENAI_API_KEY", "")
+