Your inference never goes down because one provider did.
Attach the same alias to models on multiple providers and Trifrost load-balances across them, fails over mid-stream when one cuts out, and brings it back when it recovers. Automatically. On every request.
Watch kimi3 survive a provider outage.
Four providers, one alias. DigitalOcean truncates a stream mid-response — zero content forwarded. Trifrost detects it, marks the provider unhealthy despite its HTTP 200, and continues the same SSE stream from Fireworks. The client's token counter never stops.
1 · Healthy rotation
The kimi3 alias round-robins across OpenRouter, DigitalOcean, Fireworks, and Together. Health is tracked in memory; unhealthy members sit out for a 30s cooldown.
2 · Provider fails mid-stream
DigitalOcean's stream ends with zero content forwarded. Trifrost marks it unhealthy despite the 200 and continues the same SSE stream from the next healthy provider. The client never sees the blip.
3 · Half-open recovery
After cooldown, a probe re-tests DigitalOcean. Healthy → it rejoins rotation. Failed attempts land in attempt_trail — nothing is silent.
Alias groups: one name, many providers.
Attach the alias kimi3 to a model row per provider. Two or more rows with the same name form a load-balancing group. One row stays a pinned route — no surprises.
Round-robin + health
rr_health — the default. Even rotation across healthy members. A member that returns 429/5xx or a transport error sits out for a 30s cooldown, then gets a half-open probe.
Weighted
weighted — bias traffic toward your cheapest or most committed provider with per-member weights. Failover members order by weight descending.
Least latency
least_latency — route to the lowest-EMA-latency member. Unmeasured members get explored first, so the group converges fast.
Failover triggers
Transport errors and upstream 429 / 500 / 502 / 503 / 504 / 529 — for streams, before the first byte — plus zero-content truncation: a stream that ends (clean EOF without [DONE], or an upstream read failure) before any data: payload was forwarded. Once one content frame reaches the client, the attempt is committed — never retried. Max 3 attempts; every failed attempt lands in attempt_trail.
Steer the group per request.
The provider request field carries OpenRouter-style provider preferences, matched by slug. Require members (only), blacklist them (ignore), or soft-preferences them (order) — directives compose, and failover never escapes your constraints. Toggle the group below.
only is strict — a non-member slug fails 400 provider_not_in_group, so a typo never silently re-routes. ignore that empties the group fails 400 provider_excluded_all. order never excludes. All directives are stripped before the request goes upstream.
Client cancels never poison the pool
A client that hangs up mid-request is logged as 499 and the failover chain stops immediately — a client disconnect is not upstream evidence. Unlike a real 429/5xx, it does not trigger the 30s cooldown, so one aborted request can't cause spurious 503 no_healthy_provider for everyone else.
Nothing fails silently
Every rejection — a pin mismatch, an emptied group, an all-unhealthy pool — emits exactly one blocked event row in the audit log with its reason: provider_not_in_group, provider_excluded_all, no_healthy_provider. If routing failed, you can prove when and why.
Manage groups from the dashboard.
Alias groups are first-class in the console: see which models form a group, pick the strategy, tune weights, and watch health state — no config files, no restarts. The load balancer keeps its position and health memory across admin writes.
- Strategy + weights editor per alias group
- Health state — cooldown and half-open recovery visible live
- Org-scoped — another org's alias can never shadow yours
