Phase 2 · Months 1 to 3

Learn how models behave and what they cost

Learn the mechanics

Learn how tokens, tokenizers, embeddings, attention, context windows, key-value (KV) caches, sampling, structured output, and inference work in practice. You should be able to explain why long context costs more, why the provider can cache a repeated prompt prefix, and why latency grows with output length.

Learn where the model ends and your application begins. The model predicts text. Your system decides what context the model sees, which tools exist, whether tool arguments are valid, which permissions apply, when to retry, when to stop, and what counts as success.

Route each task to the right model

Send cheap classification and routing tasks to small models and expensive reasoning tasks to large ones. Try a policy where a small model attempts the task first and a larger model takes over only when validation fails. Measure quality and cost instead of assuming the largest model is always best. Apply the same test to your coding agents.

Model the cost of every run

Build a cost model for every run: input tokens, output tokens, cached tokens, embeddings, reranking, paid tool calls, retries, and failed runs. Add a budget for each task and each user. Track the median cost (p50) and the 95th percentile cost (p95) as well as the average, because a few very expensive runs can matter more than the typical one. Test prompt caching, context compression, retrieval instead of long prompts, parallel calls, and limits on output length. Aim for the most successful tasks per dollar and per second, not the fewest tokens.

Build a model router and run it in production

Extend the Phase 1 service into a model gateway that works with more than one provider. Give it at least two profiles, one cheap and fast and one high-quality, and route requests with fixed rules. Then apply the same cost model and routing to your product, where the numbers are real. A chart of cost per task before and after the change, with quality unchanged, is some of the strongest evidence you can produce.

Compare your router with existing gateways

Compare your router with three existing gateways. LiteLLM is self-hosted and supports many providers. Portkey is a managed service with dashboards and guardrails. AWS Bedrock has its own gateway options, which keep traffic inside AWS. Write a one-page decision record that says which you would use for your product and which you would use at a company that runs everything on AWS.

You are done when

Given a workload, you can estimate its cost, name the biggest cost and latency drivers, pick a model tier, and design a fallback. You can explain why a large context window does not mean every request should fill it, and you can show this with your product's own numbers.

Exit checklist