AI Product Architecture: The Reference Design for Production AI Systems (2026)

What are the layers of a production AI product architecture?

Nine: gateway, model router, context assembly, agent runtime, tool/MCP layer, memory, durable execution, evals and observability. A prototype implements three of them: a model call, a prompt, and a UI. The six that are missing are the six where cost, reliability and debuggability live. That is why prototypes are cheap and production systems are not.

This is not a taxonomy exercise. It describes, with the proprietary parts removed, a platform I architected and built solo: AccioMatrix, an AI assessment and interview platform on an event-driven backbone that orchestrates multiple LLMs, AI agents and MCP-based tooling, now serving 20+ enterprise clients. Every layer below has a cost line because I have seen the invoice for it.

The boundaries are not arbitrary either. Each sits where you will eventually swap a vendor. You will change model providers. You will change vector stores. You will change your tool transport: MCP shipped a breaking stateless rewrite on 28 July 2026, and every client written against the old handshake had to move. A boundary that does not match something you might replace is decoration. Something you might replace with no boundary is your next incident.

The most valuable early decision is the least glamorous. Per-request cost attribution, recording tokens, cache status, model version and a price snapshot on every call, takes an afternoon and changes every architectural argument afterwards, because it converts opinions into numbers. The full implementation is in the full cost-per-request breakdown for LLM routing and caching.

What does one request actually do, step by step?

Two things, and they cannot share a code path. A simple completion is seven synchronous hops and resolves in under two seconds. An agentic task is an enqueue, an immediate 202 with a run identifier, then forty or more hops across a worker that may run for four minutes, retry twice, and partially fail. Serve both through one request/response handler and you discover the difference when your load balancer starts killing runs at sixty seconds.

The synchronous path: client, gateway (auth, rate limit, idempotency key), router (policy lookup, cache check), context assembly (retrieve, budget, order for cacheability), model call, response validation, write and return. Seven hops, one model call, one cost line. If this is your whole product, you do not need seven of the nine layers, so do not build them.

The agentic path is different in kind. The gateway writes a run row and returns immediately. A worker picks the run off a queue, assembles context, routes, calls the model, parses a tool call, validates it against a schema, checks an idempotency key, executes the tool through the MCP gateway, bounds the result, appends an observation, journals the step, checks three budgets, and goes round again. Every one of those verbs is a place to fail, and every model call carries a token count and a dollar figure you should be recording.

The habit worth stealing: annotate the sequence diagram with tokens and dollars on every model arrow. Almost nobody does it, and it changes the conversation immediately. A diagram where the third arrow says twelve thousand tokens and three cents is one a CFO can read.

How much does one request cost?

Cost equals uncached input tokens times the input price, plus cached input tokens at roughly a tenth of that price, plus output tokens at five to six times input, plus tools, retrieval, storage and human review. Write it down once, put it in middleware, and stop arguing about it. Below is that equation worked across four request shapes at August 2026 list prices.

Three cost lines surprise everyone: the tool catalogue, the retry, and the human in the loop. Fifty tool schemas at roughly 150 tokens each add 7,500 tokens to every turn. Across a twenty-turn run that is 150,000 tokens of pure catalogue before the agent has done any work. Retries are missing from most teams' cost data entirely, because the failed call never reached a success-path log. And human review, where it exists, is usually the largest line in the system by an order of magnitude.

The key structural fact about agent cost: it grows superlinearly with turns, because the entire history is re-sent every turn. On the model below, doubling a run from twenty turns to forty turns multiplies the cost by 3.3, not by 2. Budget an agent feature linearly in turns and you have under-budgeted it.

The highest-leverage number in the architecture is the cache hit rate on your static prefix, and it is also the easiest to set to zero by accident. A timestamp in the system prompt, a shuffled tool list, a non-deterministic JSON serialiser: each moves the cached prefix boundary to the first token and quietly triples your bill. The arithmetic and the fixes are in the full cost-per-request breakdown for LLM routing and caching.

Where should state live?

In your database, in four distinct kinds, not in your framework. Conversation state, run state, memory and artefacts have different lifetimes, access patterns and privacy obligations. Collapse them into one blob and you get the reason so many AI systems become undebuggable at the exact moment they get customers.

Run state matters most and gets delegated most often. LangGraph checkpointing, Temporal histories and framework-native persistence are fine mechanisms, but none replaces tables you own. The acceptance test is one query: can you say what run 8f3a did at step 7, what it cost, and how long it took, in a single SELECT? If not, your on-call engineer cannot debug production at three in the morning, and neither can you.

The schema below is the one I would start every AI product with. Three details are load-bearing. Cost is stored as bigint micros, never a float, because floats accumulate error across millions of rows and finance will notice. Status is a check constraint, so an unknown state is a write error, not a mystery. And the unique index on idempotency_key is the entire duplicate-side-effect defence: one line that stops a retried step from sending a second email.

The layer above this is the agent runtime itself, and the guards around it deserve their own treatment. That is in how the agent loop actually works in production.

create table agent_run (
  id            uuid primary key,
  tenant_id     uuid not null,
  kind          text not null,
  status        text not null check (status in
                  ('queued','running','waiting_input','succeeded',
                   'failed','cancelled','budget_exceeded','stalled')),
  input         jsonb not null,
  output        jsonb,
  step_budget   int    not null default 25,
  token_budget  int    not null default 400000,
  wall_clock_ms int    not null default 600000,
  tokens_used   int    not null default 0,
  cost_micros   bigint not null default 0,
  trace_id      text   not null,
  created_at    timestamptz not null default now(),
  updated_at    timestamptz not null default now()
);

create table agent_step (
  run_id          uuid not null references agent_run(id) on delete cascade,
  idx             int  not null,
  kind            text not null,          -- model | tool | human | system
  tool_name       text,
  idempotency_key text unique,            -- persisted BEFORE the side effect
  request         jsonb not null,
  response        jsonb,
  error           jsonb,
  input_tokens    int,
  cached_tokens   int,
  output_tokens   int,
  cost_micros     bigint,
  price_snapshot_id uuid,                 -- never compute cost at query time
  started_at      timestamptz not null default now(),
  ended_at        timestamptz,
  primary key (run_id, idx)
);

create index on agent_step (tool_name, started_at desc);
create index on agent_run  (tenant_id, status, created_at desc);

-- the acceptance test for any state design:
select idx, kind, tool_name, cost_micros,
       extract(milliseconds from ended_at - started_at) as ms,
       error
from agent_step
where run_id = '8f3a...'::uuid
order by idx;

Should the architecture be event-driven or request/response?

Event-driven for anything agentic; request/response for anything that resolves in under two seconds with a single model call and no tools. This is the most contrarian position in the post and the one I have lived with longest. AccioMatrix runs on an event-driven backbone because an interview is a long-running, multi-modal, partially-failing workflow that produces a dozen artefacts asynchronously. Modelling that as an HTTP request is the most common architectural mistake in the category.

The argument is about workload shape, not fashion. Agent work is long (seconds to minutes), bursty (a batch import creates a thousand runs at once), retryable (a 429 is normal, not exceptional) and partially failing (step 6 of 20 failed; the first five results are still valuable). That is exactly the workload queues were invented for. Request/response forces you to hold a socket open across an operation that may take four minutes and fail twice, and every layer between client and worker, browser, CDN, load balancer, ingress, has an opinion about how long that is allowed to be.

The cost of the queue is real and gets skipped in most advocacy for it. You now own message ordering, poison messages, dead-letter handling, at-least-once delivery semantics that force idempotency on you, a lease-and-heartbeat protocol so a dead worker's run gets retried rather than lost, and a queue-depth graph someone now has to watch. That is roughly a week of engineering and a permanent increase in the number of things that can page you.

So the rule is conditional, not universal. If your product is a chat box that answers in a second and a half, a queue is over-engineering; do not build one. The moment you add a tool call, a retry, or a step that can outlast your load balancer's idle timeout, the queue stops being optional and becomes the thing you should have had.

How do you make an AI system reliable when the model is not?

With four mechanisms that have nothing to do with prompting: idempotency keys persisted before side effects, retries that mutate context, budgets that terminate into named statuses, and a journal you can query. None get easier with a better model, which is why they belong in the architecture, not the prompt.

The idempotency rule first, because it is the one that costs real money when you get it wrong. Derive a key from the run identifier, the step index, the tool name and a canonical hash of the arguments. Canonical means key-sorted with no whitespace, so two argument objects with the same content but different key order hash identically. Otherwise the whole scheme is decorative. Insert the step row carrying that key before you perform the side effect. On a unique-constraint violation, read the existing row back and return its stored response instead of calling the tool again.

Be clear about the window this does not close. If the worker dies between the successful external call and the response write, the row sits in flight and nobody knows whether the email went out. That gap needs either the provider's own idempotency key on the outbound request or a reconciliation job that queries the provider by key and resolves the row. Anyone who says a database-side idempotency key alone is enough has not had a worker OOM mid-call.

The second rule is one most teams have never heard stated: retrying an LLM call with an unchanged context reproduces the same failure. At low temperature a model is close to deterministic given identical input, so an identical retry of a malformed tool call produces the same malformed tool call and charges you twice. A retry must mutate the context, appending the validation error, the expected schema and the specific offending field. Otherwise it is not a retry, it is a way to pay twice.

import { createHash } from "node:crypto";

function canonicalJson(v: unknown): string {
  if (v === null || typeof v !== "object") return JSON.stringify(v);
  if (Array.isArray(v)) return "[" + v.map(canonicalJson).join(",") + "]";
  const o = v as Record<string, unknown>;
  return "{" + Object.keys(o).sort()
    .map(k => JSON.stringify(k) + ":" + canonicalJson(o[k]))
    .join(",") + "}";
}

export function idempotencyKey(
  runId: string, stepIdx: number, tool: string, args: unknown,
): string {
  return createHash("sha256")
    .update(runId + ":" + stepIdx + ":" + tool + ":" + canonicalJson(args))
    .digest("hex");
}

export async function executeTool(db: Db, ctx: RunCtx, tool: Tool, args: unknown) {
  const key = idempotencyKey(ctx.runId, ctx.stepIdx, tool.name, args);

  const claimed = await db.insertStepIfAbsent({
    runId: ctx.runId, idx: ctx.stepIdx, kind: "tool",
    toolName: tool.name, idempotencyKey: key, request: args,
  });

  if (!claimed) {
    const prior = await db.getStepByKey(key);
    if (prior.response) return prior.response;      // already done, replay it
    return reconcile(prior, tool, key);            // crashed mid-flight
  }

  // provider-side key where supported: closes the crash window properly
  const res = await tool.call(args, { idempotencyKey: key });
  await db.completeStep(ctx.runId, ctx.stepIdx, res);
  return res;
}

What breaks in production, and how would you know?

Ten things, reliably. Below is the failure-mode matrix I would put on the wall for any AI product: blast radius, detection signal and honest time-to-detect for each. Read the cost row twice, where the honest answer to how long detection takes is next month's invoice.

Four signals catch most of it: task success rate, tool error rate by tool, cost per request by request class, and p95 latency. Note what is not on the list. Total spend is not an alert, because it rises when you grow, which trains a team to ignore it within two weeks. Cost per request rising is always a signal, and it is a four-line SQL query away.

Alert on rates and distributions, never on individual traces. An agent that fails one run in two hundred is normal; an agent whose failure rate moved from 0.5% to 4% overnight is an incident, and only the second is worth waking someone for. Output-distribution drift deserves its own alert, because it is the only signal that catches a prompt edit that broke nothing and changed everything.

The three most expensive failures in this table share a property: they succeed. A wrong-but-plausible retrieval, a wrong-but-successful tool selection, and a cache serving a confident answer to a different question all return 200 OK. No error rate moves. That is exactly why trajectory evaluation exists, covered in how to build an eval suite for an agent.

How do you know the system works at all?

Three eval layers, built in a specific order, over about three months. Final-answer evaluation alone is nearly useless for agents, because a right answer reached by a wrong path is a system you can neither trust nor improve. Trajectory evaluation, scoring the ordered sequence of plans, tool calls and handoffs against a golden path, is where agent regressions actually show up.

The ordering matters more than the tooling. Week one: traces and cost attribution, nothing else. Week two: a forty-case smoke suite of purely deterministic assertions that runs in CI in under ninety seconds. Weeks three to six: a golden set mined from production traces, not imagination, targeting several hundred cases. Month two: a calibrated judge, shipped with its calibration report. Month three: online judges on ten to twenty per cent of traffic, plus drift alerts.

Every vendor page shows the full stack. Almost nobody sequences it, and sequencing is what an under-resourced team actually needs: a team with no evals cannot build all five layers at once and will build none if told to.

The budget nobody plans for is the one this section exists to name. Judge calls are a recurring cost line that grows with traffic. Trace storage scales with steps, not requests, so a twenty-step agent run producing twenty to sixty spans puts span volume one to two orders of magnitude above request volume. That is why observability cost ambushes agent teams specifically, and why it belongs in the architecture budget, not a footnote.

What can you skip in year one?

A vector database, multi-agent, a framework, and specialised memory infrastructure. Each recommendation costs me consulting revenue, which is the main reason to believe the rest of this post.

You probably do not need a vector database in month one. Postgres with pgvector handles millions of chunks comfortably, keeps your ACLs in the same transaction as your data (which matters enormously for permission-aware retrieval), and removes a whole system from your operational surface. Reach for a dedicated vector store when you have measured a latency or scale problem, not when you have read about one.

You probably do not need multi-agent. Anthropic's January 2026 guidance is blunt: in their testing, multi-agent implementations typically use three to ten times more tokens than single-agent approaches for equivalent tasks, and they describe teams investing months in elaborate architectures only to find better prompting on a single agent matched them. The decision procedure is in when a multi-agent system is actually worth the token premium.

You probably do not need a framework. LangGraph, the OpenAI Agents SDK, the Claude Agent SDK and CrewAI all give you real things: persistence, handoffs, lifecycle hooks, protocol support. None gives you your budgets, your idempotency keys, your journal schema or your cost attribution, and those are the parts that decide whether the system survives contact with customers. What you do need from day one is cost attribution, and it takes an afternoon.

What does this cost to build, in engineer-weeks?

Ten to fourteen engineer-weeks for a two-person team to reach a production-grade version of the nine layers, of which roughly a third is the eval and observability work that gets cut first and regretted most. Dollars are rarely the constraint for the teams asking. Two senior engineers off the roadmap for a quarter is the constraint, and any estimate given in dollars but not headcount was written by someone who has never had to defend a plan.

The distribution is uneven and worth knowing in advance. The gateway, router and a working synchronous path are about a week. The agent runtime with all its guards is two to three weeks, and the guards are most of it. The tool layer with a gateway, allowlist and schema pinning is a week and a half. Durable execution on a queue plus a journal is a week; on Temporal or Restate it is faster to a demo and slower to a confident production posture, because you have also bought an operational dependency.

Then the part teams delete: three to four weeks of evals and observability. Instrumentation, a CI suite, a golden set mined from real traces, a calibrated judge, and the alerting that makes any of it actionable. This work has no demo, no screenshot and no visible output until the first time it catches a regression, which is exactly why it is the first thing cut when a date moves.

After launch, budget two to four engineer-days a month of steady-state maintenance: provider changes, model deprecations, dependency bumps, cost and latency review, plus an on-call rotation someone has to actually want. If nobody on your team wants that pager, that is a real signal, and it is what AI product development services exist for.

“Dollars are not my constraint. Two senior engineers off the roadmap for a quarter is my constraint, and any post that prices a system in dollars but not headcount was written for someone who has never had to defend a plan.”— The objection this section exists to answer

What do real products built on this look like?

The same nine layers with different boxes filled in, and wildly different cost shapes. That last part is worth internalising: there is no single cost-optimisation strategy, because the largest line item moves with the product.

In an AI interview platform, the dominant cost is real-time voice, not the scoring LLM, the inverse of what almost every team assumes when they budget one. The hard parts are not the models either: signal fusion for proctoring, calibrated rubric scoring with validated evidence spans, and getting the ordering right across three asynchronous artefact producers. The whole system is in the full architecture of an AI interview platform.

In a RAG system over private documents, the dominant architectural risk is permissions, and the dominant cost line is generation output tokens. Post-filtering an approximate-nearest-neighbour result for access control silently destroys recall: a top-50 retrieval where 8% of chunks pass the user's ACL leaves four usable chunks. That is why permission handling belongs in the retrieval design, not a wrapper. The teardown is at RAG over private documents, end to end.

In a voice product, the dominant cost is per-minute transport and speech, and the architecture is judged on tail latency, not throughput. I run production voice agents at about 2.5 cents a minute on a custom LiveKit stack, down from about 10 cents on a managed platform, and the full build-versus-buy arithmetic is in the voice AI build vs buy break-even. The pattern across all three: the reference architecture is stable, the cost profile is not.

AI PRODUCT ARCHITECTURE — REFERENCE DESIGN (Axionry, Aug 2026)
Nine layers. A prototype implements three.

1 GATEWAY          auth, rate limit, tenant, idempotency key, 202+run_id
                   cost ~$0 | fails: holds a socket open across a 4-min run
2 MODEL ROUTER     policy table, cache lookup, 429/5xx fallback
                   cost NEGATIVE (-30..-60%) | fails: silent quality regression
3 CONTEXT ASSEMBLY retrieval, memory read, tool subset, token budget, cache order
                   sets 60-80% of the bill | fails: timestamp zeroes the cache
4 AGENT RUNTIME    loop + step/token/wall-clock budgets + no-progress detector
                   $0.04-$1.05 per run | fails: never terminates
5 TOOL / MCP       gateway, allowlist, schema pinning, credential injection
                   7,500 tok/turn at 50 tools | fails: 40K-token tool result
6 MEMORY           episodic / semantic / procedural, Postgres + pgvector
                   $0.0002-$0.002 per read | fails: stale memory vs fresh retrieval
7 DURABLE EXEC     journal, replay, timers, resumption
                   ~$0.0001 per step | fails: resumed run repeats a paid call
8 EVALS            deterministic CI suite -> golden set -> calibrated judge
                   $0.002-$0.02 per judged trace | fails: stale set passes
9 OBSERVABILITY    OTel GenAI spans + cost meter + price snapshot
                   scales with STEPS not requests | fails: PII past retention

RULE: no architecture claim without a failure mode and a cost number.