Drop in. Stay up.
Swap the base URL and every app you run gets load balancing, mid-stream failover, budgets, audit, and live observability — across 27+ providers, without a code change.
Two environment variables. Zero rewrites.
Trifrost speaks the OpenAI wire format over /v1/*. Point your existing SDK at Trifrost, swap the key for a virtual key, and governance, failover, and accounting apply to every call from that moment on.
- 27+ provider kinds behind one stable API surface
- Reasoning models — reasoning_content / thinking passed through untouched
- Cost in response headers — X-TF-Cost-USD on every call
# before — direct to a provider export OPENAI_BASE_URL=https://api.openai.com/v1 export OPENAI_API_KEY=sk-... # after — governed by Trifrost export OPENAI_BASE_URL=https://api.trifrost.tech/v1 export OPENAI_API_KEY=sk-bf-... # your virtual key # every call now: load-balanced, failover-protected, # budgeted, allowlisted, audited, cost-attributed.
Failover that survives the failure modes you actually see.
Alias-group load balancing (rr_health, weighted, least_latency) fails over on 429/5xx and transport errors — and on the nastiest one: a provider that returns HTTP 200, then truncates the stream before a single token. Trifrost continues the same client SSE stream from the next healthy member and marks the cutter unhealthy despite its 200.
Zero-content stream failover
A stream that ends before any data: payload — clean EOF without [DONE], or an upstream read failure — fails over transparently. Once one content frame reaches the client, the attempt is committed, never retried.
499s never poison the pool
A client that hangs up mid-request is logged as 499 and the failover chain stops — a client disconnect is not upstream evidence. One aborted request can't put a healthy provider into cooldown for everyone else.
Nothing fails silently
Max 3 attempts, each recorded in attempt_trail. Every routing rejection emits exactly one blocked event row with its reason: provider_not_in_group, provider_excluded_all, no_healthy_provider.
See the request while it's still in flight.
The in-flight requests endpoint shows every live request — phase, model, provider, age — with zero DB hits, safe to poll at any cadence. Streaming telemetry captures TTFT and tokens/sec per request, rolled up hourly for provider-performance comparison.
- Live in-flight registry — filter by model, provider, phase, stream, age
- TTFT + tokens/sec on every streaming request
- Provider-performance comparison — pick your primaries on evidence
Pin your region. Cut your latency.
Trifrost runs in AU, APAC, EMEA, UK, and Germany — all live. Serve APAC users from APAC. Keep EMEA data in EMEA. Same price in every region; a new region stood up on request at Enterprise.
Regions & plans →fakellm: test without a vendor bill
A zero-dependency, OpenAI-compatible fake inference server ships with the ecosystem — exact token accounting, configurable latency and failure injection (429/500/503/stream-truncate). Run your entire failover and budget test suite deterministically, for free.
Debug in the playground
The built-in playground runs on the same governed gateway as production traffic — reasoning bubbles, provider routing controls, per-turn cost and latency, and request IDs that jump straight into the event log.