Django and httpx: An LLM Proxy That Keeps Provider API Keys Away From Callers
Smarter‘s Proxy is a Django view, built on httpx, that forwards requests to OpenAI, Anthropic, Google Gemini or Azure OpenAI with the provider’s API key added on the way out, so the people and programs that call the model never hold that key. Callers keep using the provider’s own SDK. They change two arguments, the base URL and the API key, and from then on the credential that can spend money at the provider lives in exactly one place. This post describes how the Proxy does that, what it protects against, and what it costs to run one.
The problem with handing out provider keys
An LLM provider’s API key is a bearer credential with a credit card behind it. Whoever holds it can run up a bill, read the account’s uploaded files, cancel its fine-tuning jobs, and, depending on the provider, create more keys. Most providers offer coarse project scoping at best, and none of them knows which of your developers, CI jobs or customer-facing apps made a given request.
So the key gets copied. It goes into a developer’s .env, a notebook, a GitHub Actions secret, a Docker image someone built in a hurry, a browser extension’s settings. Each copy is a place it can leak from, and when one leaks there is no way to tell which copy it was. Rotating it means finding and updating every copy at once, so in practice it isn’t rotated.
The usual answer is a gateway. Callers authenticate to something you control, and that thing holds the real key. Smarter’s Proxy is that gateway, built from parts a Django developer already knows.
What a caller sees
From the caller’s side, the only change is the client’s constructor. Here is the example from the Proxy app’s README.
from openai import OpenAI
client = OpenAI(
base_url="https://<smarter>/api/v1/proxy/openai/",
api_key="<Smarter API key>",
)
client.chat.completions.create(
model="gpt-6-luna",
messages=[{"role": "user", "content": "Hi"}],
)
Every SDK puts its key in a different header, so the Proxy looks for a Smarter API key in all of them. This is from authentication.py.
AUTHORIZATION_SCHEMES = ("bearer", "token")
API_KEY_HEADERS = ("HTTP_X_API_KEY", "HTTP_X_GOOG_API_KEY", "HTTP_API_KEY")
"""The request.META keys of the x-api-key (Anthropic), x-goog-api-key (Google Gemini) and api-key (Azure OpenAI) headers."""
Whatever header it arrives in, the key is checked by the same Django REST Framework authentication class as the rest of Smarter’s API, so it has to exist and be active. The passthrough views accept nothing else. A logged-in web console session does not authenticate there, which keeps the endpoint out of reach of cross-site requests that ride on a browser’s cookies.
The swap
Behind the endpoint, a Proxy is a SAM resource, declared in YAML like everything else in Smarter. Here is the built-in one for Anthropic, from the example-manifests repository, trimmed.
apiVersion: smarter.sh/v1
kind: Proxy
metadata:
name: anthropic
spec:
provider: anthropic
apiKey: anthropic_api_key
baseUrl: https://api.anthropic.com/
auth:
header: x-api-key
scheme: ""
headers:
anthropic-version: "2023-06-01"
allowedPaths:
- v1/messages
- v1/messages/count_tokens
- v1/models
- v1/models/*
timeout: 300
spec.apiKey names a Smarter Secret, not a key. The Secret is stored encrypted, and the Proxy decrypts it for each request, just before it builds the outgoing headers. Here is the order of operations, from ProxyForwarder.forward() in services.py.
| Step | What happens | Refused with |
|---|---|---|
| 1 | The caller’s Smarter API key is authenticated | 401 |
| 2 | The Proxy name resolves to one the caller may read, which is their own, their account’s, or a built-in one | 404 |
| 3 | The Proxy is active, and the path matches allowedPaths | 403 |
| 4 | The base URL’s host resolves only to public addresses | 503 |
| 5 | No budget’s resource lock forbids another charge | 402 |
| 6 | Headers are filtered, the provider key is added, and the request is sent | 502 or 504 |
| 7 | Token usage is read from the response, streamed or not, and charged |
Step 6 is the one that matters most. The incoming headers are filtered through a deny list before anything is added, from const.py.
DROPPED_REQUEST_HEADERS = (
*HOP_BY_HOP_HEADERS,
*CREDENTIAL_HEADERS,
"host",
"content-length",
"cookie",
"forwarded",
"x-forwarded-for",
"x-forwarded-host",
"x-forwarded-proto",
"x-forwarded-port",
"x-real-ip",
"x-csrftoken",
"accept-encoding",
)
The caller’s Smarter key never reaches the provider, and neither do Smarter’s cookies, its CSRF token, or the caller’s IP address. Then the Proxy’s own fixed headers are added, and the provider’s key goes in last, so nothing the caller sends can overwrite it.
retval = {
name: value
for name, value in headers.items()
if name.lower() not in DROPPED_REQUEST_HEADERS and name.lower() not in replaced
}
retval.update(proxy.headers or {})
retval[proxy.auth_header] = proxy.auth_header_value(api_key)
On the way back, set-cookie, strict-transport-security and the hop-by-hop headers are dropped from the provider’s response, and an X-Smarter-Proxy header names the Proxy that answered.
The guards around the swap
A gateway that adds a valuable credential to outgoing requests is itself a target. The interesting attacks are not against the caller’s key but against the gateway’s willingness to attach the provider’s key to a request someone else shaped. The Proxy has a guard for each of the obvious ones.
Whose key goes where
Anyone who can write a Proxy manifest chooses its base URL. Without a rule, a user could write a Proxy that sends the platform’s OpenAI key to a server they run. Proxy.may_use_secret() closes that hole, because a Proxy may only send a Secret that belongs to its owner’s account. The built-in Proxies are owned by the Smarter admin, so they can use the platform’s Secrets, and every account can call them, but nobody in another account can point a new Proxy at those Secrets.
Which paths
allowedPaths is a list of glob patterns, relative to the base URL. The built-in manifests limit callers to inference endpoints. That matters because the provider key behind them can usually do much more than inference. Before matching, normalize_path() refuses anything that could climb out of the base URL, such as .. and . segments, empty segments, backslashes, embedded ://, and percent-encoded dots and slashes. An empty allowedPaths allows every path, so a Proxy you write yourself should list the paths it needs.
Which hosts
A base URL whose host resolves to a private, loopback or link-local address is refused, unless a superuser owns the Proxy. That stops a Proxy from being used to reach the cluster’s own services or a cloud metadata endpoint. I think of it as a check on configuration rather than a network control. A network policy that limits the pods’ egress is the second layer, and a deployment that cares about this should have both.
Whose money
Before forwarding, the Proxy asks the budget system whether the Proxy, its Provider, the caller or the caller’s account is locked. If any is, the caller gets HTTP 402 in the error shape that the OpenAI and Anthropic SDKs already know how to report. After the response, the token usage is read from the body or, for a stream, from the server-sent events as they pass through, and charged to all four. A long streamed completion is charged when the stream ends, including when the client disconnects early.
And not in the tests
The development containers hold real provider keys, and Smarter’s unit tests run there. The forwarder refuses to make a real call during a test run.
if _transport_factory is not None:
return _transport_factory()
if running_unit_tests():
raise ProxyConfigurationError(
"Refusing to call a real LLM provider from the unit tests. Use configure_transport()."
)
Tests install an httpx.MockTransport instead. A test that forgets fails loudly rather than spending money.
How much does it cover?
The Proxy protects one asset, the provider credential, but that asset is the one with a direct financial consequence and the widest blast radius. A leaked provider key exposes every application that uses that provider account, not just the one that leaked it. With the Proxy, a leaked Smarter key is still serious, since it acts as its owner across Smarter’s API, but what it can reach at the provider is limited by allowedPaths and by budgets, and it can be deactivated without touching any other caller.
On a scale of one to ten, I’d put it at an 8 for any organization where more than a handful of people or programs call an LLM. It doesn’t stop prompt injection, data exfiltration through the model, or abuse by a legitimate caller, and it isn’t meant to. It turns an unmanageable secret into a managed one.
What it costs
Administration
The ongoing work is small. The provider key is one Secret, and rotating it is one apply or one manage.py update_secret, after which every caller is on the new key at the next request. The work that remains is the work you want, which is issuing and revoking Smarter API keys per person or program, and keeping allowedPaths honest when a provider adds endpoints. Smarter API keys can only be created for staff users, so in practice an account admin issues them.
Money
There is no licence and no extra service, since the Proxy is part of the platform. The cost is the compute for one more hop. Requests are relayed rather than buffered, and long streams hold a connection on the Smarter pod for as long as the provider takes to answer. Budgets and per-request charges usually save more than that, because spending becomes visible per caller rather than per provider invoice.
User experience
For developers, nothing changes but two constructor arguments. They keep the provider’s SDK, its types, its retries and its streaming. Errors that Smarter raises (unknown Proxy, path not allowed, budget exceeded) come back in the {"error": {...}} shape that SDKs already parse. The one new thing a developer can trip over is a 403 for a path that isn’t in allowedPaths, and its message lists the allowed paths.
Stability
The Proxy puts Smarter in the request path to the provider. If the platform is down, so is every caller’s access to the model, which is the price of any gateway. Each Proxy has its own timeout (120 seconds by default, 600 at most), a 10-second connect timeout, and no redirects are followed. Smarter’s Django REST Framework throttles apply to the passthrough too, with counters in Redis, so one runaway client can’t take every pod’s capacity.
Takeaways
- Put the provider key behind a gateway, and give each caller a key of your own that you can revoke on its own.
- Accept your key in whatever header the provider’s SDK already uses, so callers change their base URL and key and nothing else.
- Filter incoming headers through a deny list before adding the provider key, and add the provider key last.
- Only let a proxy send a credential that belongs to the proxy owner’s own account.
- Restrict paths with an allow list, and normalize paths before matching.
- Refuse upstream hosts that resolve to private addresses, and back that up with egress policy.
- Check budgets before forwarding, and charge usage after, including on streams.
- Make it impossible for unit tests to reach the real provider.
Closing thoughts
A provider API key behaves less like a password than like a company credit card that nobody signs for. The Proxy’s design is simple. Keep that card in one drawer, and give everyone else a card of their own that you can cancel. Everything else in this post, from the header list and path globs to the host check, budget lock and test guard, is there to make sure the drawer only opens for the requests it should.