AI inference uses a trained model to generate predictions or outputs from new inputs. Learn how it works, how to measure cost and latency, and when to route requests.
Start with the core questions, then examine the examples and tradeoffs below.
AI inference is the use of a trained machine learning model to produce an output from a new input. It includes a classifier predicting a label, an embedding model producing a vector, and a language model generating text. Inference can run locally or remotely, online or in batches; it is not limited to live chat or token-priced APIs.
For a hosted language-model API, cost usually depends on input and output tokens, model and endpoint pricing, caching, and batch terms. A self-hosted service also has hardware, utilization, operations, and capacity costs. For an application, count retries, retrieval, tool calls, and failed requests as well as successful responses.
Not necessarily. A lower price does not demonstrate equivalent quality. Compare candidate models on representative tasks, define acceptable errors and latency, and evaluate the complete application. A quality floor is a task-specific acceptance threshold, not a guarantee that every generated answer will be correct.
Separate model development from the service that uses the model.
Training adjusts model parameters using a learning objective and data. Inference applies the resulting model to new inputs. Fine-tuning is an additional training step; inserting retrieved documents into a prompt is not. A retrieval-augmented generation system normally performs retrieval and inference without changing the language model's weights.
Inference happens in prototypes, offline analytics, and production applications. Its lifetime cost may exceed training cost in a heavily used service, but no universal percentage applies. A rarely used model and a high-volume assistant have different economics.
| Stage | What changes | Typical output |
|---|---|---|
| Training or fine-tuning | Model parameters | A model checkpoint |
| Inference | Request context and application state | A prediction, vector, or generated response |
| Evaluation | Our evidence about behavior | Scores, failure cases, and release decisions |
The model is one stage in a larger request path.
An application prepares the input, checks permissions and constraints, selects a model endpoint, and sends the request. The serving system queues work and executes the model. The application then validates the output, records usage, and returns a result or takes a permitted next step.
For an autoregressive language model, prompt processing precedes token-by-token generation. Long inputs and long outputs create different performance pressures. Streaming can improve perceived responsiveness by showing partial output, but it does not make the final response complete sooner by definition.
A successful HTTP response is not proof of a successful task. Schema validation can check whether output is parseable; domain checks and evaluation are needed to assess correctness. Record the model version, relevant configuration, usage, duration, and final task result so changes can be investigated.
Optimize for the constraints that actually matter to your users.
Online inference suits interactive tasks with a response-time requirement. Batch inference suits work that can wait and can be grouped. On-device inference can support offline operation and local processing but is constrained by device capacity. Dedicated or self-hosted serving provides more infrastructure control while adding operational responsibilities.
| Approach | Useful when | Check before choosing |
|---|---|---|
| Hosted API | You want managed access and variable consumption | Quotas, data terms, supported features, endpoint reliability |
| Batch API | Results can arrive later | Completion window, eligible models, failure handling |
| Self-hosted or dedicated | You need infrastructure control or sustained capacity | Utilization, redundancy, operator time, model licence |
| On-device | Local processing or offline use is important | Memory, energy, model size, update mechanism |
Token price is an input to unit economics, not the final metric.
For a text API, estimate request cost as input tokens divided by one million times the input rate, plus output tokens divided by one million times the output rate. Add other billed dimensions where applicable. Keep cached input, reasoning tokens, media, tool fees, and batch discounts explicit rather than hiding them in a universal blended price.
Illustrative arithmetic: 2,000 input tokens at $1 per million and 500 output tokens at $4 per million cost $0.004. These are example rates, not an offer or vendor benchmark. One million such requests cost $4,000 before other charges. If retries add 10% more identical calls, the model cost becomes $4,400.
To compare releases, divide total attributable cost by accepted outcomes. If a cheaper model causes more retries, escalations, or manual corrections, its lower token rate may not reduce this number. Publish assumptions alongside any savings estimate.
A fast average can hide a poor experience for the slowest requests.
Track time to first token, total response time, throughput, errors, timeouts, and accepted task outcomes. Report percentile latency alongside averages. Segment by workload, input length, output length, and route; otherwise changes in traffic mix can look like model improvements.
Create an evaluation set from representative examples with appropriate privacy controls. Include edge cases and costly errors. Keep a held-out set for release decisions, inspect disagreement, and repeat evaluations when prompts, retrieval, tools, or models change. A score from another team's benchmark does not automatically transfer to your application.
Use the least expensive eligible route, then verify the application outcome.
Specify required features, endpoint restrictions, quality criteria, latency limits, and fallback behavior before comparing prices. A route that cannot use the required tool format or context length is not interchangeable with one that can. Test model substitutions against the same task distribution.
o10's routing product uses an OpenAI-compatible endpoint and virtual model aliases. Start with a shadow comparison to inspect proposed alternatives before changing production behavior. Record the baseline and candidate costs separately. Shadow evaluation itself may incur inference cost, which belongs in the experiment budget.
Keep the application responsible for authorization and task-level correctness. Routing and a per-call receipt do not independently prove that an agent completed its job, that its actions were authorized, or that every response passed a human review.
Measure workload volume, input and output usage, errors, latency, and accepted outcomes over a representative period.
Define task-specific checks, material failure cases, and the latency and cost limits you can tolerate.
Evaluate models and serving options against the same cases. Include routing, evaluation, retrieval, and retry overhead.
Use a limited rollout, inspect failures, and keep a known-good route available. Recheck the result when traffic or model versions change.
Technical explanation and illustrative cost arithmetic, not a customer-results study. Product integration details follow the o10 marketing capability specification. Example rates must be replaced with the rates and terms of the endpoint you actually use.
No. Image recognition, forecasting, recommendation, speech recognition, and embedding models also perform inference. Their billing units and serving requirements may differ from those of a text-generation API.
No. Hardware depends on the model, throughput, memory needs, and latency target. CPUs and device accelerators can serve some workloads; larger models may require more capable accelerators or multiple devices.
No. RAG is an application pattern that retrieves information and supplies it to a model. The model then performs inference using that context. Retrieval quality and access control affect the final result.
An agent may perform many inference calls while pursuing one task. It can use a result to choose a tool, inspect the tool output, and decide what to do next. Measure both per-call usage and cost per completed agent task.
No. Routing can restrict candidates using measured criteria, but probabilistic models still make mistakes. Validate outputs and monitor real outcomes; use appropriate human review for consequential actions.
Paste a week of traffic. Get the number that books the audit.
See what you're overpaying →