what inference actually costs

inference is a metered utility, and most teams find that out on the first real bill. the numbers that matter are boring: what a generation costs, how often you run one, and whether anyone needed it to be fresh.

the biggest lever is scheduling. a generation that runs once and gets replayed to every reader costs a fixed amount per day. the same generation triggered on every event costs whatever the event rate says it costs, and you no longer control the bill. the gap between those two modes is often a hundredfold. plenty of products that ask for a real-time llm feature actually want a scheduled one with good replay, because the reader arrives hours after the generation anyway.

the second lever is the output tax. providers price output tokens at a multiple of input tokens, so a system that produces long generations from short prompts has the worst possible cost shape. caching flips it. prompt caching makes repeated context nearly free, which rewards architectures that hold context stable and vary only the small final question.

third, providers are not interchangeable, and the spread is bigger than the marketing suggests. run one prompt across a dozen models and latency varies by an order of magnitude, pricing varies more, and the quality ranking flips depending on the task. the practical conclusion is structural: model choice belongs in config, not in architecture, so that switching providers is an edit and not a rewrite.

the last cost is the one worth paying. language models round numbers to make sentences flow. ask one to narrate figures it was handed and it will eventually turn 83.97% into “80%” because the sentence reads better. the fix is to compute every figure outside the model, pull each number back out of the generated text, and regenerate on any mismatch. regeneration doubles the price of that turn. a reader who catches one wrong number stops trusting all of them, so the doubled turn is cheap.

none of this needs a gpu or a research paper. the wins are scheduling, caching, checking outputs against computed truth, and watching which meter is running.