Build an AI Agent That Does Real Work: Orchestrator, Specialist Agents and One MCP Harness
- An ops agent that reads a refund ticket, checks the order in Shopify, refunds up to $50 and drafts the reply costs about 3.4 cents a ticket in models and durable execution at September 2026 prices, against about $2.20 of a person's time. Refunds above $50 go to a lead and cost about 58 cents, almost all of it their 90 seconds.
- One orchestrator owns the plan and the state, every specialist agent is stateless, and every tool call passes through one MCP harness that enforces scopes, the $50 limit, approvals, idempotency keys, timeouts, logs and a $0.25 per-task cost cap. A limit written in a prompt is a suggestion; a limit checked on the arguments is a control.
- Choose the orchestrator runtime by how long a run waits: your own state machine for seconds, LangGraph or Google ADK 2.x for minutes, Temporal underneath for hours or days. Below roughly 410 refund tickets a month, a $10,000 build does not pay back within a year on these assumptions, so do not build it.
A helpdesk webhook creates one run row with a stable id, the ticket, the customer and the store. Nothing happens that is not attached to that id.
$0 · fails: duplicate webhooksOwns the plan and the state as a typed state machine: triaged, order checked, refund proposed, gated, refunded, reply drafted, done. For a known intent, no model decides what runs next.
code, not a modelTriage and a reply checker on Claude Haiku 4.5, an order agent and a reply agent on Claude Sonnet 5. Each sees only the context and the one or two tools its step needs, and returns structured output.
$0.032 per ticketThe only path to a tool: pinned registry, per-step allowlists, credentials it holds, argument limits, approvals, idempotency keys, timeouts, retries and cost caps.
fails closedThe Shopify Admin API for orders and refundCreate, the helpdesk API for the reply draft. Both wrapped as MCP tools on a server you own.
store-scoped tokensOne row per model call and tool call with an arguments hash, a bounded result, latency and cost. The same rows feed replay tests, dashboards and the audit trail.
one query per incidentWhat does an AI ops agent that issues refunds actually do?
It reads a customer's refund request, finds the order in Shopify, checks your refund policy, refunds up to $50 on its own, asks a person to approve anything larger, and drafts a reply with the amount and timeline. On the arithmetic in this post it costs about 3.4 cents a ticket, against roughly $2.20 of a person's time.
The buyer is the support or operations lead at an online store who handles a daily stream of 'it arrived broken' and 'where is my refund' tickets. The product is not a chatbot. It is a colleague with a narrow job description: one intent, four tools, one spending limit. A ticket saying the mug in order 1042 arrived cracked becomes an order lookup in Shopify, a policy check (inside 30 days, item damaged, no earlier refund), a refundCreate call for that line item, and a reply waiting in the helpdesk before anyone on the team has opened the ticket.
What it is worth is arithmetic. A person handling the same ticket (open it, find the order, check the policy, refund, write the reply) takes about six minutes, which is $2.20 at $22 an hour fully loaded. The agent closes the under-$50 cases for about 3.4 cents and prepares the rest so a lead approves in about 90 seconds, roughly 58 cents. With one ticket in four needing approval, the blended cost is about 17 cents, so 4,000 refund tickets a month come to roughly $685 against $8,800 handled by hand. These are modelled figures with every input printed, not measurements from a client system.
This is the architecture I start from at Axionry for any agent that writes to another system: one orchestrator that owns the plan and the state, specialist agents that own nothing, and one MCP harness that every tool call passes through. The rest of this post covers why each choice exists, what it costs and where the frameworks fit. The loop mechanics underneath (termination, retries, budgets) are in the agent loop in production.
Why do simple AI agents break in production?
Because a prompt-chained agent keeps its plan, its permissions and its memory inside the model's context, where nothing enforces them. It loops when a tool result is ambiguous, repeats a side effect when the chain is retried, can call any tool its API key reaches, and leaves a transcript, not an audit trail. These are design gaps, not model flaws.
The published evidence points the same way from three directions. The τ-bench paper found that gpt-4o, a leading function-calling agent in 2024, succeeded on under half of its customer-service tasks, and in the retail domain got a task right on all eight of eight runs less than 25% of the time, which is why it introduced the pass^k metric (arXiv 2406.12045). The MAST study of 1,600+ traces across seven multi-agent frameworks catalogued 14 failure modes in three groups: system design, inter-agent misalignment and task verification (arXiv 2503.13657). And Gartner expects over 40% of agentic AI projects to be cancelled by the end of 2027, citing cost, unclear value and inadequate risk controls.
In the refund agent each gap has a price. A loop between order and policy lookups, 40 turns with the history growing about 500 tokens a turn, is roughly 650,000 input tokens or $1.15 on Sonnet 5 before anyone notices. A retried chain that re-sends refundCreate is a second refund and a reconciliation task. A system prompt that says never refund more than $50 is one injected sentence away from refunding $400, the failure OWASP calls excessive agency: too much functionality, permission or autonomy. And a transcript cannot tell you which run refunded order 1042, with whose approval, at what cost.
The fix is to move control out of the context window. The plan becomes data owned by code, permissions become a property of each tool call rather than a line in a prompt, and every side effect gets a key before it happens. The models stay exactly as capable as they were. They stop being the only thing between an angry email and your refunds ledger.
- Who owns the plan
- The model, inside its context
- Tool access
- Every tool the API key can reach
- $50 refund limit
- A sentence in the system prompt
- Retry after a timeout
- Re-run the chain and refund again
- Stuck run
- About $1.15 for a 40-turn loop, then a timeout
- Audit
- A chat transcript
- Who owns the plan
- A typed state machine in code
- Tool access
- One or two tools per step, credentials held by the harness
- $50 refund limit
- Checked on the arguments before dispatch
- Retry after a timeout
- Resume at the failed step with the same idempotency key
- Stuck run
- Six-turn cap, $0.25 cost cap, then a person
- Audit
- One journal row per call, queryable
What is an agent orchestrator, and how does it work with an MCP harness?
An orchestrator is the code that owns a task's plan and state: which step runs next, what each step received and returned, and when to stop. Specialist agents each do one step. The MCP harness sits between every agent and every tool and decides whether a call may run, with which credential, and at what cost.
For a known intent the plan is a template, not a model output: triage, check the order, propose a refund, gate, refund, draft the reply, check the reply. The orchestrator persists each transition, so a crash resumes at the right step rather than at the top. A planner model appears only for intents no template covers, and its plan is validated against the same state machine before any tool runs. This is Anthropic's distinction between workflows and agents applied on purpose: predefined code paths where the path is known, model-directed steps only where it is not.
Specialists are stateless and narrow. The triage agent on Claude Haiku 4.5 extracts intent, order number and risk flags into a schema. The order agent on Claude Sonnet 5 sees the ticket, the order and the policy, may call get_order and get_refund_history, and returns a structured proposal with line items and an amount. The reply agent writes from that proposal and the refund receipt, and a Haiku checker confirms the amount and order number in the reply match the receipt. None of them can call refund_create. Only the orchestrator's refund step can, and only through the harness.
The harness is one MCP client and gateway for every tool call. It wraps the Shopify Admin API and the helpdesk API as MCP tools on a server you own, holds the credentials so no agent ever sees a token, knows which step is calling so an allowlist can say the reply agent may post a draft and nothing else, and writes the journal so every refund traces to a run, a step, an approval and a cost in one query. The gateway mechanics (routing, catalogue caching, schema pinning) are in MCP in production.
- 1Triaged$0.0024
Haiku 4.5 returns intent refund_request, order 1042, item damaged, no risk flags. Unknown intents leave the automated path here.
- 2Order checkedtool call 1
The order agent calls get_order through the harness: fulfilled nine days ago, one mug at $18.00, no earlier refunds.
- 3Refund proposed$0.0193 for two turns
A structured proposal with line item, quantity, amount and reason, validated against a schema and never parsed from prose.
- 4Gated$0
Code, not a model: at or under $50, inside the 30-day window, fewer than two refunds in 90 days. Anything else becomes an approval request with the evidence attached.
- 5Refundedtool call 2
refund_create through the harness, with the idempotency key in the journal before the request leaves. A timeout retries with the same key.
- 6Reply drafted and checked$0.0107, tool call 3
Sonnet 5 drafts from the receipt; Haiku 4.5 and a string match confirm amount and order number before post_draft runs.
- 7Done$0.0336 total
Run closed with total cost in micros, a journal row per step, and the approval id when there was one.
What should an MCP harness enforce on every tool call?
Seven things: a registry of allowed tools with pinned versions, per-step permissions with credentials the harness holds, argument limits checked in code, human approval for risky calls, timeouts and bounded retries, an idempotency key on every write, and a journal with cost caps. If any check cannot run, the call fails closed and the task goes to a person.
The registry pins every tool to a server version and a schema hash, which is not paranoia. In September 2025 an npm package impersonating Postmark's MCP server added one line in version 1.0.16 that BCC'd every email to an outside address. A pinned hash turns that kind of change into a failed call instead of a quiet leak. Permissions are per step rather than per agent, and the harness keeps its own risk labels, because the spec tells clients to treat tool annotations such as destructiveHint as untrusted unless they come from trusted servers.
Approvals, retries and idempotency are one mechanism seen three ways. The harness writes an idempotency key (run, step, order, amount) to the journal before dispatch and passes it to Shopify, whose @idempotent directive is mandatory on refundCreate from API version 2026-04. A timeout retries with the same key. An approval binds to a hash of the exact arguments and expires, so an approved $62 cannot be replayed as $620. And because approvals can wait hours, the harness keeps its own dedupe record: Stripe, for comparison, may prune idempotency keys after 24 hours, which is shorter than a weekend queue.
Logs and cost caps close the loop. Every call is written with its step, an arguments hash, a bounded result, latency and cost, which is what makes replay testing possible later. The per-task cap is $0.25, about seven times a clean run, and a six-turn ceiling per specialist stops the 40-turn loop at about 7 cents instead of $1.15. The MCP spec tells clients to confirm sensitive operations, time out tool calls and log them for audit; the harness is where those recommendations become code, and it is the first part of an agent I would put through a security review.
tool: refund_create
server: shopify-admin # our MCP server wrapping the Admin API
server_version: 1.4.2 # pinned
schema_hash: "sha256:3b1e...9c07" # the call fails if the schema changes
callable_from: [orchestrator.refund_step] # never a model-driven agent
arguments:
order_id: { must_equal: run.order_id } # the model cannot pick another order
amount_usd: { max: order.refundable_total } # never more than was paid
auto_execute:
when: "amount_usd <= 50 and customer.refunds_90d < 2"
approval: # everything auto_execute does not cover
approver_role: support_lead
bind_to: arguments_hash # an approved $62 cannot become $620
expires_after_hours: 24
limits:
max_calls_per_run: 1
timeout_ms: 8000
retry: { attempts: 2, on: [timeout, 502, 503], backoff_ms: [500, 2000] }
idempotency:
key: "{run_id}:{step}:{order_id}:{amount_minor}"
persist_before_dispatch: true
forward_as: shopify_idempotent_directive
budget:
run_cost_cap_usd: 0.25
tenant_daily_cap_usd: 40
log:
arguments: hashed
result: bounded_800_tokens
replayable: true- Registry with pinned server versions and schema hashesStops a changed or impersonated server from changing what your agent can do.
- Per-step allowlists, with credentials held by the harnessStops the reply agent from refunding, and keeps tokens out of every prompt.
- Argument checks in code: the order must match the ticket, the amount must be within limitsStops injected instructions from moving more money than policy allows.
- Human approval bound to an arguments hash, with an expiryStops an approval for one amount being replayed for another.
- Timeouts and bounded retries per toolRetries only on timeouts and 5xx responses, so a slow Shopify call cannot hold a run open.
- Idempotency key persisted before every writeStops a retry or a resumed run from refunding twice.
- Journal row per call, plus per-task and per-tenant cost capsStops runaway spend and answers who refunded what, when, with whose approval.
What changed in the MCP spec in 2026, and what does it mean for your harness?
The current revision is 2026-07-28 and it made MCP stateless: no initialize handshake, no session header, and every request carries its own protocol version and capabilities. For a harness that means ordinary load balancing, routing on Mcp-Method and Mcp-Name headers, cacheable tool lists, and mid-call questions handled by retrying the request instead of holding a stream open.
Two transports are standard: stdio, for a server the client launches as a subprocess, and Streamable HTTP, where each message is a POST to one endpoint (transports). The older HTTP+SSE transport is formally deprecated. Authorization is optional and defined for HTTP: a protected server acts as an OAuth 2.1 resource server and must publish Protected Resource Metadata (RFC 9728), clients must send a resource indicator (RFC 8707) so a token is bound to one server, Client ID Metadata Documents are preferred over the now deprecated Dynamic Client Registration, and clients must check a returned iss parameter (RFC 9207) before redeeming a code (authorization). Servers must not accept or pass through tokens issued for anyone else, the rule that stops a harness becoming a confused deputy.
Three more changes shape the harness directly. Multi Round-Trip Requests replace server-initiated elicitation and sampling: the server returns input_required with its questions and the client retries with answers. Long-running work moved to the io.modelcontextprotocol/tasks extension with poll-based tasks/get. And a formal deprecation policy now sets a twelve-month minimum window, with a narrow expedited exception, and Roots, Sampling and Logging are already deprecated (changelog). I still keep business approvals in the orchestrator rather than in MRTR, because the approver is a support lead in your tool, not the user of the MCP client.
The ecosystem has settled around it. MCP moved to the Linux Foundation's Agentic AI Foundation in December 2025, and the maintainers report close to half a billion Tier 1 SDK downloads a month across TypeScript, Python, Go and C#. A2A reached a 1.0 specification in April 2026 for agents from different organisations to talk to each other. The split is clean: MCP for an agent reaching tools and data, A2A across organisational boundaries. Inside one product the orchestrator is a function call, not a protocol.
| What changed | What the harness does about it | Breaking? | |
|---|---|---|---|
| Sessions and the initialize handshake | Removed; each request carries version and capabilities in _meta | No sticky routing: any harness instance serves any call | Yes |
| Mcp-Method and Mcp-Name headers | Required on Streamable HTTP POSTs | Allowlists and rate limits run on headers before the body is parsed | Yes |
| ttlMs and cacheScope on list results | Tool lists are cacheable and should come back in deterministic order | Cache the catalogue per server; stable order keeps prompt caches warm | No |
| Multi Round-Trip Requests | input_required replaces server-initiated elicitation and sampling | Handle the retry; keep business approvals in the orchestrator | Yes, for those flows |
| Tasks extension | io.modelcontextprotocol/tasks with tasks/get polling | Long tool calls become a durable wait, not a held connection | Moved out of core |
| Authorization | RFC 9728 metadata, RFC 8707 resource indicators, CIMD over DCR, RFC 9207 iss checks | Tokens bound to one server; no user token is forwarded downstream | Yes, for clients |
| Deprecation policy | Twelve-month minimum; Roots, Sampling, Logging and HTTP+SSE deprecated | Migrations go on a calendar instead of into an incident | No |
How much does an AI agent cost per task?
For this refund agent, about 3.2 cents in models and a tenth of a cent in durable execution when it closes the ticket alone, and about 58 cents when a lead approves, nearly all of it their time. The number that moves the business case is the approval rate, not the model price.
Line by line: triage on Haiku 4.5 is 1,800 tokens in and 120 out, $0.0024. The order agent takes two Sonnet 5 turns of 6,000 and 7,200 input tokens with a 3,500-token cached prefix (system prompt, two tool schemas and the 354-token tool-use prompt Anthropic adds on Sonnet 5), for $0.0082 and $0.0111. The reply agent is $0.0078 and the Haiku checker $0.0029, so models total $0.0324. Temporal Cloud at $50 per million actions adds about $0.0013 for 25 actions. Rates are from Anthropic's pricing page, checked 23 September 2026.
The approval path is the expensive one: 90 seconds of a lead at $22 an hour is $0.55. At one approval in four the blended cost is about 17 cents a ticket, and 4,000 refund tickets a month come to roughly $685 against $8,800 handled by hand. The payback line for a $10,000 build sits near 410 refund tickets a month, excluding maintenance. Below it, a helpdesk macro and a person are the better product, and I would tell anyone asking me to build this the same.
Two things move the model line more than model choice does. A runaway loop costs about $1.15 per stuck run, so the six-turn cap is a cost control as much as a safety one. And prompt caching only pays when the cached prefix is stable: tool lists in deterministic order, which the 2026-07-28 spec now asks servers to return, and a system prompt that never embeds the ticket. The routing and caching machinery is in LLM routing and caching, and you can price your own version in the AI product cost estimator.
| Line item | Model or rate | Tokens or units | Auto path | Approval path |
|---|---|---|---|---|
| Triage | Haiku 4.5 at $1 in, $5 out per 1M | 1,800 in / 120 out | $0.0024 | $0.0024 |
| Order agent, turn 1 | Sonnet 5 at $2 in, $10 out, $0.20 cached | 6,000 in (3,500 cached) / 250 out | $0.0082 | $0.0082 |
| Order agent, turn 2 | Sonnet 5 | 7,200 in (3,500 cached) / 300 out | $0.0111 | $0.0111 |
| Policy gate | Code | 0 tokens | $0 | $0 |
| Reply agent | Sonnet 5 | 3,500 in (1,500 cached) / 350 out | $0.0078 | $0.0078 |
| Reply check | Haiku 4.5 plus a string match | 2,500 in / 80 out | $0.0029 | $0.0029 |
| Durable execution | Temporal Cloud at $50 per 1M actions | about 25 actions | $0.0013 | $0.0013 |
| Lead approval | $22 an hour, modelled | 90 seconds | $0 | $0.55 |
| Total per ticket | Blended at 25% approvals: $0.171 | 5 model calls, 3 tool calls | $0.034 | $0.584 |
Which agent framework should you use: LangGraph, OpenAI Agents SDK, Claude Agent SDK, Google ADK or Temporal?
Choose by how long a run waits and what the agent touches. LangGraph and Google ADK 2.x suit plans you want as an explicit graph, the OpenAI Agents SDK suits handoff-shaped flows, the Claude Agent SDK suits agents that need files and a shell, and Temporal sits underneath any of them when waits last hours or days.
LangGraph reached 1.0 in October 2025 and is at 1.2.12 on PyPI this week. Its checkpointer and interrupt() give you approvals that pause a graph, with one sharp edge its docs state plainly: on resume, the node re-runs from its beginning, so any side effect before the interrupt must be idempotent. Google's ADK 2.0 reached general availability for Python on 19 May 2026, Go on 30 June and TypeScript on 21 August, adding graph workflows, human-in-the-loop and collaborative agents, with breaking event-schema changes if you stored 1.x sessions yourself.
The OpenAI Agents SDK is still pre-1.0 (0.22.3 in Python) and a short route to handoffs, guardrails, tracing and MCP in a few files; tools marked needs_approval pause the run into a RunState you can serialise, store and resume after a person decides (human-in-the-loop docs). Non-OpenAI models work through OpenAI-compatible endpoints or LiteLLM and any-llm adapters that the SDK labels best-effort. The Claude Agent SDK is the Claude Code harness as a library: file, shell and web tools, subagents, hooks, MCP, and a permission check that runs hooks, deny rules, ask rules, the permission mode, allow rules and then your canUseTool callback (permissions). It is the right runtime for a specialist that needs a computer, and more machinery than a refund step needs.
Temporal is not an agent framework, and that is its value. It makes the run itself durable: activities retry with their own timeouts, a signal can wait days for an approval, and a crashed worker resumes where it stopped. Its OpenAI Agents SDK integration now ships as a standalone 1.0 package, and its LangGraph plugin entered public preview on 16 July 2026 with a blunt summary of the gap: checkpoints preserve data, not execution. At $50 per million actions, durability for this refund run costs about a tenth of a cent.
| Option | Version on 23 Sep 2026 | What it gives you | What you still build | Use it when | Skip it when |
|---|---|---|---|---|---|
| LangGraph | 1.2.12 Python; 1.0 GA October 2025 | A graph of nodes and edges, checkpointers, interrupt() and resume, streaming | Harness policies, idempotent side effects, cost caps, your own journal | The plan is a graph you want in code and version control | Runs must survive a dead process for days without extra infrastructure |
| OpenAI Agents SDK | 0.22.3 Python; 0.18.0 JS; pre-1.0 | Agents, handoffs, guardrails, sessions, tracing, MCP, needs_approval with a resumable RunState | Durable waits, harness policies, multi-provider routing | Mostly OpenAI models and a handoff-shaped flow | You need a stable 1.0 API or first-class non-OpenAI models |
| Claude Agent SDK | 0.2.158 Python; 0.3.280 TypeScript | Claude Code's loop with file, shell and web tools, subagents, hooks, permission modes, MCP, sessions | The orchestrator around it, business state, the approval UI | A specialist needs files, code or a shell | Each step is a short stateless call; it runs a CLI process per session |
| Google ADK | 2.9.2 Python; 2.0 GA May 2026 | Graph workflows, Sequential, Parallel and Loop agents, human-in-the-loop, MCP tools, A2A, evals with user simulation | Harness policies, idempotency, cost caps | You deploy on Google Cloud or need A2A | You cannot absorb the 2.0 event-schema changes yet |
| Temporal | Python SDK 1.33.0; Cloud at $50 per 1M actions | Durable execution, per-activity retries and timeouts, signals that wait days, replay | Everything agent-specific; it wraps the others | Money moves or approvals wait hours | Tasks finish in seconds and a queue plus Postgres is enough |
| Your own | A Postgres journal, a queue and a typed state machine | Exact control of state, idempotency and audit | All of it, including timers | The plan is mostly fixed and there are 3 to 10 tools | You need parallel sub-agents or durable timers |
The refund agent without approvals fits here, with the least code you do not own.
An explicit graph with interrupts. Make every side effect before an interrupt idempotent.
The run survives crashes and deploys. Works with the OpenAI Agents SDK and, in preview, LangGraph.
Keep it a specialist inside the plan, not the orchestrator.
A framework does not remove the harness work. It only moves where you write it.
How do you test an AI agent before it touches real money?
Test reliability, not one lucky run: replay each labelled ticket several times and require every attempt to pass, assert on the sequence of tool calls as well as the reply, attack it with injected instructions, and run it in shadow mode on live tickets before it may refund anything. After launch, watch rates rather than individual tickets.
Offline, the golden set is 200 real refund tickets with the correct outcome labelled by the support team: refund or not, amount, approval or not. Score pass^k, where all k runs must be right, because single-run accuracy hides inconsistency. Assert the trajectory, not just the text: get_order before any refund, never two refund_create calls in one run, an approval request whenever the amount is above $50. Because the harness logged every tool request and response, those logs replay as fixtures, so a new model or prompt is tested against Shopify's real answers without calling Shopify.
Adversarially, feed it tickets that carry instructions: a customer who writes that the system already approved a $400 refund, an order number that belongs to someone else, a message asking the agent to email a different address. The pass condition is not that the model resists. It is that the harness refuses, whatever the model proposed. Then run two weeks of shadow mode, where the agent proposes and a person decides, and switch on auto-execution only for ticket classes a lead accepts unchanged at least 98% of the time.
Online, four rates tell you most of what matters: approval override rate, refund reversal or chargeback rate on agent refunds, reopen rate within 72 hours, and cost per closed ticket. Alert when a rate moves, not when one ticket goes wrong. The answer-quality side of support automation (confidence gates, citations, escalation briefs) is covered in the AI customer support agent design.
| Failure mode | What you see | Where it is stopped | What catches it |
|---|---|---|---|
| Loop between lookups | Nothing for a minute, then a timeout, about $1.15 spent | Six-turn cap per specialist and a repeated-call detector; the run moves to stalled | Turns-per-run distribution, stalled-run rate |
| Duplicate refund on retry | Two refunds for one order | Idempotency key persisted before dispatch; Shopify @idempotent | Duplicate-key attempts in the journal, finance reconciliation |
| Injected instruction raises the amount | A $400 proposal on an $18 item | Argument check against the refundable total and the $50 auto limit | Adversarial suite in CI, override rate |
| Wrong order refunded | A refund on another customer's order | order_id must equal the order bound to the run | Cross-customer assertions in tests, audit anomalies |
| Approval replayed | An approved $62 becomes a $620 call | Approval bound to the arguments hash, 24-hour expiry | Approval id and hash on every write row |
| Tool server changed underneath you | New behaviour with no deploy on your side | Pinned version and schema hash in the registry | Hash-mismatch alerts |
| Refund succeeded, reply failed | Customer refunded but not told | Orchestrator resumes at the reply step and never re-runs the refund | Runs stuck in the refunded state for over 10 minutes |
| Cost blow-out after a deploy | Spend doubles in an afternoon | Per-task $0.25 cap, per-tenant daily cap, circuit breaker on tool errors | Cost per closed ticket, hourly |
- pass^k on 200 labelled tickets, with k of 4 or moreAll k runs must be right. A single-run pass rate hides the inconsistency τ-bench was built to expose.
- Trajectory assertions on tool order and countsget_order before refund_create, one refund per run, an approval request above $50.
- Replay from production logsRecorded tool responses become fixtures, so new models meet real Shopify answers without touching Shopify.
- Injection suiteTickets that claim approvals, name other orders or redirect the reply. Pass means the harness refused.
- Failure injection on toolsTimeouts, throttling and 5xx responses mid-run. Pass means no duplicate refund and a resumed run.
- Two weeks of shadow modeThe agent proposes and a person decides. Auto-execution switches on per ticket class only above 98% accepted unchanged.
Should you build your own agent orchestrator or use a framework?
Build the harness yourself in every case, because it encodes your policies. For the orchestrator, use your own state machine when the plan is short and fixed, and a framework when you need graphs, long waits or a computer. On my estimate the refund agent is 5 to 6 engineer-weeks, and most of that is harness, evals and rollout.
Frameworks are good at control flow and know nothing about your business rules. None of them knows that refunds over $50 need a lead, that an approval must bind to its arguments, or that your journal must answer which run refunded order 1042. That code is small (the policy file above is most of it), and it is yours whichever framework runs the plan. A team of agents rarely helps either: a refund is short, sequential and single-domain, the shape where one agent per step beats a multi-agent design.
Sequenced, the build runs journal and harness first, then tools and policies, then the orchestrator and specialists, then approvals and evals, then shadow mode. Axionry's published pricing puts an AI feature inside an existing product from $10,000 and a standalone AI-first product from $22,000. A working demo of the happy path starts at $1,000, which is the cheapest way to learn whether your approval rate is 10% or 60% before committing. If refunds are one of several agents on your list, the AI agents worth building in 2026 ranks the others.
If you want this built, Axionry builds it at $0: the work is split into checkpoints with acceptance criteria agreed before work starts, and each checkpoint is invoiced only after you have seen it and accepted it. The service is AI product development.
- Week 1Journal and harness skeleton
Run and step tables, cost in micros, and the MCP harness with registry, allowlists and logging. No agent yet.
- Week 2Tools and policies
get_order, get_refund_history, refund_create and post_draft on an MCP server you own, with argument checks, idempotency, timeouts and the $50 rule.
- Week 3Orchestrator and specialists
The state machine, triage, order agent, reply agent and checker, each with a schema for its output.
- Week 4Approvals and evals
An approval queue bound to argument hashes; the golden set, pass^k, trajectory assertions and the injection suite.
- Weeks 5 to 6Shadow mode and rollout
The agent proposes and people decide; auto-execution switches on per ticket class once acceptance clears 98%.
Agent orchestrators and MCP harnesses: common questions
→What is an MCP harness?
An MCP harness is the single layer every tool call from an agent passes through. It keeps a registry of allowed tools with pinned versions, checks per-step permissions, holds the credentials, enforces argument limits such as a $50 refund cap, requests human approval for risky calls, writes idempotency keys, applies timeouts and cost caps, and logs every call so runs can be audited and replayed.
→How much does an AI agent that issues refunds cost to run?
On September 2026 list prices, about 3.4 cents per ticket when the agent closes it alone: five model calls on Claude Haiku 4.5 and Claude Sonnet 5 plus about 25 Temporal actions. When a lead approves a refund over $50, the modelled cost is about 58 cents, almost all of it 90 seconds of their time at $22 an hour.
→Do I need LangGraph or another framework to build an agent orchestrator?
No. A short, fixed plan runs well on your own state machine with a Postgres journal. Use LangGraph or Google ADK 2.x when you want an explicit graph with pauses, Temporal when approvals wait hours or days, and the Claude Agent SDK when a step needs files or a shell. You build the harness and its policies either way.
→What changed in MCP in 2026?
The 2026-07-28 revision made MCP stateless. The initialize handshake and session header are gone, requests carry their own version and capabilities, Streamable HTTP requires Mcp-Method and Mcp-Name headers, list results are cacheable, and Multi Round-Trip Requests replace server-initiated elicitation and sampling. Authorization now prefers Client ID Metadata Documents over Dynamic Client Registration and requires RFC 9207 issuer checks.
→How do you stop an AI agent from refunding the wrong amount?
Do not rely on the prompt. Check the arguments in code before the call leaves the harness: the order must be the one bound to the run, the amount cannot exceed what was paid, and anything above the auto limit needs a person whose approval is bound to the exact arguments. An idempotency key written before the call stops a retry from refunding twice.
→When should you not build an agent like this?
When volume is low or the policy is unclear. On the modelled numbers, a $10,000 build pays back within a year only above roughly 410 refund tickets a month. Below that, a helpdesk macro and a person are cheaper. And if your refund rules live in people's heads, write them down first, because the agent needs them as code.
Open the article in your assistant with one click and ask it how this applies to your product.
- Build a custom AI agent at $0An agent that does real work in your business tools, inside spending limits, with approvals and a full log.
- Build an AI customer service agent at $0Resolves support tickets in chat and email from your help center and order data, and hands off the rest.
- Build an AI browser agent at $0Works through portals that have no API in a sandboxed browser, and asks before it submits anything.
- Build a ChatGPT app at $0Puts your product inside ChatGPT and Claude: tools over your API, sign-in, rich results and usage tracking.