What we tell users when 9 of our 13 models are down

A gateway fails in two ways. The loud way is a 500. The quiet way is a model that

has been returning `503` for six hours while every dashboard says "active".

We shipped the quiet way twice this month. This is what we changed, and the

contract you can now check before you build against us.

The mistake

We had a health checker that did this, more or less:

for model in catalogue:
    if upstream_returns_2xx():   mark_healthy(model)
    else:                        mark_unhealthy(model)
        if model.unhealthy_for_2_checks(): deactivate(model)

Two assumptions in four lines, both wrong.

**One: "unhealthy" is not always the model's fault.** When the upstream account

runs out of balance, *every* model behind it fails at the same instant, with the

same error. My checker saw eight failures and dutifully deactivated eight models.

The models were fine. The account was empty. Deactivating them made the outage

longer, because now they would not even be retried.

The fix was a distinction the code did not previously have:

if all_checks_failed_with_account_level_error():
    log("SKIP auto-isolate: account-level failure (balance) -- do NOT deactivate")
else:
    isolate_the_single_bad_model()

Now that line exists in the log, and it looks like this:

[auto-isolate] SKIP kimi-k3: account-level failure (balance) -- do NOT deactivate the model

**Two: "2xx from upstream" does not mean "works for this model name".** Some

suppliers route by model name and return a perfectly valid 200 with an error body,

or accept the name in `/models` and reject it in `/chat/completions`. A health

check that only looks at status codes will mark those models healthy forever.

The contract we publish now

If a model is unavailable, the response is not a mystery. It is this, with

`HTTP 503`:

{
  "error": {
    "message": "Service temporarily unavailable: upstream model capacity is exhausted for the requested model. Please retry later or contact support@flintapi.ai.",
    "type": "service_unavailable",
    "code": "upstream_capacity_exhausted",
    "hint": "Other models may still be available right now: https://flintapi.ai/api/health/models",
    "available_models": ["deepseek-v4-pro", "deepseek-v4.1-flash", "glm-5.3", "glm-5.3-flash"]
  }
}

Three properties matter here:

1. **A machine-readable code** (`upstream_capacity_exhausted`), so your retry and

failover logic does not have to parse English.

2. **The list of models that are working right now**, so a 503 is actionable

— you can fall back without a human.

3. **`Retry-After`**, so a naive client does not hammer us in a loop.

And the single source of truth, which is public and needs no auth:

GET https://flintapi.ai/api/health/models

It returns `total` and `healthy` counts joined against the catalogue that we

actually serve. If that endpoint says four models are healthy, then four models

are healthy — we do not keep a second, prettier number anywhere else.

Why we do not "just multiply the model list"

It is tempting to advertise thirty models and route whatever works. That converts

our availability problem into your correctness problem: your temperature, your

token budget, your structured-output assumptions silently run on a different model

than the one you named. We would rather return a loud 503 for a named model than a

quiet 200 from a substitute.

If you *do* want substitution, it is opt-in: send `model: "flint"` and the router

decomposes the prompt and picks per sub-task, and it says which model answered.

The numbers, dated

As of 2026-10-28: **13 models in the catalogue, 4 healthy**, because one upstream

account is at a negative balance of about three cents. This post exists partly

because that is an embarrassing ratio and partly because the interesting part is

not the ratio — it is that the ratio is *published, checkable, and per-model*

rather than hidden behind a green dashboard.

If you are running something similar, two checks are worth adding today:

- Distinguish **account-level** failures from **model-level** failures before any

automated isolation runs. Otherwise one billing problem takes out your whole

catalogue and looks like thirteen separate bugs.

- Return the **currently-available list** inside your error body. It costs one

join and turns every 503 into a self-healing instruction.

*Related: [What Chinese LLM APIs actually cost in 2026](/blog/chinese-llm-api-pricing-2026) and

[Why your health check said "up" while the model was gone](/blog/health-check-said-up-model-was-gone).*