Selling tokens below cost — what a margin audit found in our own gateway
We run a small OpenAI-compatible gateway in front of Chinese LLMs. Today we did
something uncomfortable: we took our own rate card, divided every sell price by the
cost we pay upstream, and looked at the ratios. We expected a narrow spread. What we
found was a range from 0.13x to 12.5x, with five models priced below cost.
The worst offender was also the easiest to measure, because the upstream publishes its
price page and we have our own usage logs.
The arithmetic
`deepseek-v4-pro` is quoted by its vendor, per 1M tokens, in USD:
input (cache miss) output
off-peak $0.66 $1.98
peak $1.32 $3.96Peak hours are 01:00–04:00 and 06:00–10:00 UTC on weekdays; everything else is
off-peak. We were charging **$0.98 in / $1.96 out**.
Then we pulled every usage row for that model from our own database — 23 rows, 800
input tokens and 6,341 output tokens total, split by whether the request fell in peak
hours:
period rows input tok output tok we charged upstream cost margin
off-peak 10 680 6,121 $0.00647 $0.01257 -$0.00610
peak 13 120 220 $0.00030 $0.00103 -$0.00073
total 23 800 6,341 $0.00676 $0.01360 -$0.00683Minus 101%. We were paying roughly two dollars for every dollar we billed, and the
sample was small enough that nobody had noticed. The input side was fine; the output
side was underwater at every hour of the day.
How this happens
Not through one bad decision. Through a series of locally reasonable ones:
1. **The rate card was a collage.** Some models were priced at 1.43x cost (sane), some
at 3.3x (also fine, cheap models), some at 1.00x (no margin at all), and five below
1.0x. Each number was defensible when it was typed. Nobody ever computed the whole
column.
2. **Cost drifted under a fixed price.** The upstream changed its price list at least
once while our sell price sat still. A price you set once is a price you are
accidentally holding against a moving cost.
3. **The cost field itself was untrustworthy.** One model's stored cost implied a 12x
markup would be needed to break even — a strong signal that the stored number was
wrong, not the price. If you multiply a bad cost by any margin, you get a confident
wrong answer.
That third point is the one that generalizes. Before you reprice, audit the
denominator.
What "peak" does to a flat price
The same model costs twice as much between 06:00 and 10:00 UTC on a weekday. A single
flat output price therefore has to cover the peak, or it is only profitable on
weekends. If you publish one number per model, that number is a peak-hours number and
should be set that way — or you should say plainly that you route around the peak.
We have not finished repricing. The honest status is: we know which rows are safely
above cost, which rows are safely below, and which rows are blocked on getting a real
bill from a supplier whose price list we do not have. Guessing the last group would
just move the error.
The four checks we now run
1. **Ratio audit, per row.** `sell / cost` for input and output, weekly. Any row below
1.25x opens a ticket; below 1.0x is an incident.
2. **Official list prices, not memory.** Fetch the vendor's pricing page when auditing;
third-party comparison sites disagreed with the vendor by up to 2x in our sample.
3. **Realized margin from your own logs, not from the rate card.** The rate card says
what you intended. The usage table says what happened. Split by peak/off-peak and by
cached/uncached input if the vendor bills them differently.
4. **Do not reprice off an unaudited cost field.** If the implied markup is absurd in
either direction, the cost is the suspect.
Why publish this
Because the alternative is a quiet number that quietly gets worse. A gateway is a
margin business with a thin spread and a moving denominator; the failure mode is not a
crash, it is a ratio. And because a price article without a date is a rumour — so the
numbers above are dated 2026-10-28, and the live model list is at
[`/api/health/models`](https://flintapi.ai/api/health/models).
If you are running something similar: pull one week of usage, join it to your cost
table, and compute the ratio per row. It takes twenty minutes and it is the most
uncomfortable twenty minutes of the quarter.
*Related: [What Chinese LLM APIs actually cost in 2026](/blog/chinese-llm-api-pricing-2026).*