Source page: https://www.o10.io/routing

Plain-text reference for reading, copying, and citation. Examples are illustrative; use your own credentials for API requests.

Model data: https://www.o10.io/api/models.json
Content index: https://www.o10.io/llms.txt

# LLM routing: choose a model per task with explicit constraints

LLM routing selects a model or endpoint for a request according to a policy. A useful routing policy balances task requirements, measured quality, latency, availability, and total cost.

## Key takeaways

### What is routing?

LLM routing selects a model or endpoint for a request according to a policy. A useful routing policy balances task requirements, measured quality, latency, availability, and total cost.

## Separate access from route selection
A gateway provides a common access layer; routing decides which eligible model or venue should serve a request. A gateway can include routing, and a routing service can expose an API. Compare the actual supported behavior rather than assuming these labels describe mutually exclusive products.
o10 exposes an OpenAI-compatible endpoint with routing aliases. Applications can integrate that endpoint while keeping their own retrieval, tools, permissions, and business logic.

## Define eligibility before price
A candidate must satisfy required features and applicable policy before its price is compared. Define how the system behaves when no route qualifies, an endpoint times out, or a pinned model becomes unavailable.
An aggregate evaluation score does not validate every response. A quality floor describes measured acceptance criteria for a task; output checks and monitoring remain necessary after routing.

## Compare proposed routes before changing production
A shadow comparison can record alternatives without replacing the user-facing response. Include the candidate calls and evaluation overhead in the experiment budget. Segment results by workload so changes in traffic mix do not masquerade as savings.
Release to a limited portion of traffic, track accepted outcomes and errors, and retain a rollback. Revisit the policy after changes to model versions, prompts, tool schemas, or provider behavior.

## Methodology

Cost examples are illustrative. Measure quality and total costs on your own tasks, and check current endpoint terms before implementation.

## Related links

- [AI inference: from model output to production system](https://www.o10.io/ai-inference)
- [AI agents: useful actions, measurable outcomes](https://www.o10.io/ai-agents)
- [AI model selection: capabilities, constraints and evidence](https://www.o10.io/ai-models)

## Source URL

https://www.o10.io/routing