ai-platform/roadmap
All pages

Start here

  1. How to use this roadmap
  2. Two kinds of role
  3. Read code, write it rarely
  4. Target profile
  5. Eight principles
  6. Work on a real product

Phases

  1. 01Direct coding agents and read their code
  2. 02Learn how models behave and what they cost
  3. 03Build retrieval that finds the right evidence
  4. 04Build tool servers with MCP
  5. 05Design agent workflows that survive failure
  6. 06Run agent workloads on AWS
  7. 07Trace every step an agent takes
  8. 08Build evals that catch regressions
  9. 09Run agents reliably in production
  10. 10Secure agents and the code they write
  11. 11Build an internal AI platform
  12. 12Get teams to adopt the platform
  13. 13Build Forge, the capstone platform
  14. Elective: Kubernetes

After the phases

  1. Staff skills
  2. 12-month plan
  3. A typical week
  4. Show your work
  5. Tools to know
  6. What to skip
  7. Final checklist

Phase 9 · Months 8 to 10

Run agents reliably in production

Define reliability in terms the user sees

An agent can return HTTP 200 and still fail the task, so availability is not enough. Define service level indicators (SLIs) for completed tasks, quality, latency, and cost. A service level objective (SLO) is the target you set for an SLI, such as 95% of tasks completed. Examples are the rate of answers with valid citations, the rate of severe false positives, p95 latency, and the cost of each answer.

Plan for partial failure

Learn circuit breakers, retries with limits and random jitter, concurrency limits, load shedding, dead-letter queues, handling of jobs that keep failing, idempotency, and replay. Plan for partial failures explicitly. The model call succeeds but the tool fails. The tool succeeds but the response is lost. The worker dies after it creates a branch but before it records that the job finished.

Stop runaway runs

Give every run hard limits on elapsed time, model calls, tool calls, tokens, dollars, and recursion depth. Add a quota for each tenant and a global kill switch. A reliable agent stops or falls back to a known mode instead of looping forever.

Plan for provider outages

Decide what each workflow does when a model provider goes down. Some workflows can switch to another model. Some should queue the work and wait. Some can return a smaller result from fixed code. Write down what changes for the user in each case. Do not switch to a weaker model without saying so.

Run a game day

A game day is a planned exercise where you break your own system on purpose. Inject model timeouts, rate-limit errors with HTTP 429, malformed output, duplicate deliveries, tool outages, slow database responses, expired credentials, and exhausted budgets into your product's staging environment and the agent runner. Check that alerts fire, that the runbooks work, and that the system recovers. Write a postmortem for at least one of the incidents you caused.

You are done when

You can define an SLO, explain its error budget, run a failure drill, and recover safely. You can show that the platform cannot loop or spend money without limit.

Exit checklist

0 of 6 done