Build an AI Content Moderation Pipeline: The Cascade, the Uncertain Band and Cost Per Million Items (2026)
- Moderating a million mixed text-and-image items costs about $1,183 on the cascade below — roughly a tenth of a cent each. Adjudicating every item with a language model instead costs about $4,299, and a human-only queue costs about $112,500. The cascade is not an optimisation; it is the product.
- One number decides your entire cost model: the width of the uncertain band, meaning the share of traffic your cheap classifiers cannot confidently resolve. At 5% you spend $883 per million; at 20% you spend $2,682; and above roughly 33% the cascade stops paying and you should adjudicate everything.
- Human reviewers are 59% of the bill even though they see 1.1% of the traffic. Every architectural decision in this pipeline is really a decision about how many items reach a person, which is why threshold calibration is a finance activity as much as a machine learning one.
Perceptual hash against known-violating media, URL and phrase blocklists, account-level rate rules. No inference at all. Resolves the highest-severity content in single-digit milliseconds.
1.5% resolved · $2 per millionA distilled text classifier you host, plus a commodity image moderation API. Produces a calibrated score per policy category. This tier sees essentially all traffic and must be cheap enough that seeing everything is affordable.
92.5% resolved · $213 per millionTwo thresholds per policy category, pinned to a policy version. Below the floor, allow. Above the ceiling, act. Between them, escalate. This is fifteen lines of code and it sets 80% of your bill.
the entire cost modelA cheap model with the policy text in a cached prefix, reasoning over context the classifier cannot see: sarcasm, reclaimed slurs, quoted abuse, satire, medical discussion. Runs on 7.5% of traffic.
6.4% resolved · $207 per millionThe 1.1% the model declines to decide, plus appeals, plus a mandatory random audit sample. Slowest, most expensive, and the only tier that can be held accountable by a regulator.
1.1% of items · $736 per millionEvery decision pins the policy version, the model version, the thresholds and the scores that produced it. Without this you cannot answer why an item was removed, and in the EU you are legally required to.
$25 per million · non-optionalWhat is an AI content moderation pipeline, architecturally?
A cascade. Deterministic matching first, cheap classifiers second, a language model only on the band the classifiers cannot resolve, and humans only on what the model declines to decide. Plus two cross-cutting systems that are not optional: versioned policy, and an audit ledger that can reconstruct any decision.
The word cascade is doing real work here and it is not a synonym for pipeline. A pipeline runs every stage on every item. A cascade runs each stage only on what survived the previous one, and its economics are governed entirely by the survival rate at each step. If your classifiers resolve 92.5% of traffic, the expensive tiers see 7.5%; if they resolve 80%, the expensive tiers see 20% and your bill roughly doubles. That sensitivity is the reason this post is organised around the router rather than around the models.
The scope worth targeting is a mixed feed: text posts and comments, images attached to some of them, and a policy with perhaps a dozen categories — harassment, hate, sexual content, violence, self-harm, illicit goods, spam, impersonation and so on. What makes this hard is not the obvious content. Obvious content is solved: a perceptual hash catches known-violating media in milliseconds and a commodity classifier catches most of the rest. What is hard is the band in the middle, where sarcasm, reclaimed slurs, quoted abuse, satire, medical discussion of self-harm and news reporting of violence all live — and where a wrong decision in either direction is a story.
This is a reference design: how I would build it, and what the arithmetic says it costs. It draws on architecting AccioMatrix, an event-driven AI platform orchestrating multiple LLMs, agents and MCP tooling for 20+ enterprise clients, and on cutting a $200K-a-year cloud bill by more than 70%, which is where the instinct for cascade economics comes from. I have not operated a content moderation system at platform scale. Every figure below is modelled with its inputs printed.
One regulatory fact belongs in the architecture conversation rather than the legal review. Under the EU Digital Services Act, platforms must give users a clear and specific statement of reasons when they restrict content — including whether an automated process found or removed it — and online platforms must submit pseudonymised statements of reasons to the Commission's DSA Transparency Database, which publishes them in machine-readable form. That is not a compliance report you generate quarterly. It is a per-decision data structure that has to exist at the moment of the decision, which is why the audit ledger is in the architecture diagram and not in an appendix.
| Component | What it does | Implementation | Cost per 1M items | Primary failure mode | Skip in v1? |
|---|---|---|---|---|---|
| Ingest + normalise | One item record per piece of content with media extracted, author, surface and context | Queue plus a Postgres item table; media to object storage | ~$2 | Two surfaces produce different item shapes and the router silently treats one as text-only | No |
| Tier 0 deterministic | Perceptual hash against known-violating media, URL and phrase blocklists, rate rules | Hash index in memory, blocklists in Redis, refreshed continuously | In the line above | A stale hash list; the same known-bad media is re-adjudicated by the model at 1,300x the cost | No |
| Tier 1 text classifier | Calibrated per-category scores on every text item | Self-hosted distilled classifier, modelled at $0.02 per 1,000 items | $15.76 | Scores drift after a retrain and the router thresholds no longer mean what they meant | No |
| Tier 1 image classifier | Category scores on every image | Commodity API, modelled at $0.001 per image for the first million per month | $197.00 | Vendor category coverage narrower than your policy, so a category silently never fires | No |
| Cascade router | Two thresholds per category, pinned to a policy version; allow, act or escalate | Fifteen lines of deterministic code plus a versioned threshold table | ~$0 | Thresholds hand-picked rather than calibrated; the uncertain band is 20% and nobody noticed | Absolutely not |
| Tier 2 LLM adjudication | Reasoning over context the classifier cannot see, on the uncertain band only | Haiku-class model, policy text in a cached prefix, structured verdict | $206.85 | Policy text drifts out of the cache prefix and every call costs 4x overnight | No |
| Tier 3 human review | The model's abstentions, appeals, and a mandatory random audit sample | Review tool with an AI brief, priority queue, wellbeing controls | $736.31 | Queue depth grows faster than throughput and the SLA quietly becomes days | No |
| Policy store | Versioned policy text, thresholds, examples and category definitions | Immutable versions with an activation timestamp; every decision pins one | In infra | A threshold changes and last month's decisions become unreconstructable | No |
| Audit ledger | Immutable record: scores, versions, verdict, reasons, actor, timestamp | Append-only Postgres plus object storage for the evidence payload | $25 total | You cannot answer why this was removed, which in the EU is a legal exposure | No |
What does the full architecture look like?
Twelve components across four columns: ingest and policy, the three cascade tiers with the router between them, decision and audit, and the user-facing notice, appeal and transparency surfaces. The load-bearing detail is that the policy store feeds the router rather than the models, and that every decision writes to an immutable ledger before the user is told anything.
Start at the router, because it is the component that everyone underbuilds. It takes a vector of calibrated category scores and, for each category, compares them against a floor and a ceiling drawn from the active policy version. Below the floor the item is allowed. Above the ceiling it is actioned. Between them it escalates. That is genuinely all it does, and it is where your cost, your false-positive rate and your false-negative rate are simultaneously determined. Teams routinely spend three months on model selection and forty minutes picking thresholds, which is exactly backwards.
The two classifier tiers have very different economics and should be reasoned about separately. Text classification is close to free at scale — a distilled model on your own hardware costs a couple of cents per thousand items, which means running it on 100% of traffic is trivially affordable. Image classification is not: at the modelled commodity rate of a tenth of a cent per image, images are 20% of volume and 93% of tier-1 cost. That single asymmetry means image-heavy surfaces need their own tier-0 work — perceptual hashing, dimension and format heuristics, duplicate detection across accounts — in a way text surfaces do not.
Adjudication is where you spend your model budget and where the design choice is to be aggressively cheap. This is not a task that needs a frontier model. It needs a model that can read a policy definition, read an item with its surrounding context, and produce a structured verdict with a cited reason — and a Haiku-class model with the policy in a cached prefix does that for roughly a quarter of a cent. Putting a frontier model here quadruples the second-largest line in your bill to improve agreement on a band where humans themselves disagree.
Everything to the right of decision is compliance surface, and it is real engineering rather than paperwork. The statement of reasons has to be generated per decision, has to say whether automation was involved, and has to offer an appeal route. The audit ledger has to be able to reconstruct the decision from the scores, the thresholds, the policy version and the model version that produced it. The transparency export has to be machine-readable. Build these in week two, not month six, because retrofitting them means backfilling a data structure you did not capture.
Why does the cascade decide the entire cost model?
Because the expensive tiers are between 80 and 5,000 times more expensive per item than the cheap ones, so the only variable that matters is how many items reach them. On the model here, a classifier decision costs $0.0002, an LLM adjudication costs $0.0028, and a human review costs $0.0625 — a 300-fold spread from top to bottom.
Put the three architectures side by side and the argument settles itself. The cascade costs about $1,183 per million items. Adjudicating every item with a language model — the design people reach for because it is simple and the demo is impressive — costs about $4,299, roughly 3.6 times more, for a quality improvement concentrated in a band that the cascade already sends to the model anyway. A human-only queue at 45 seconds an item and $9 an hour costs about $112,500 per million, which is 95 times the cascade and is why no platform above a modest size operates one.
Now the sensitivity that matters. Hold everything else constant and vary only the width of the uncertain band. At 2% the cascade costs $523 per million. At 7.5% — the modelled default — it costs $1,183. At 20% it costs $2,682. And at roughly 33% it crosses the cost of adjudicating everything with a model, at which point the cascade has stopped earning its complexity and the correct engineering decision is to simplify. That crossover is a genuinely useful architectural boundary and almost nobody computes it.
Which reframes what a better classifier is worth. Improving your text classifier is not primarily a quality investment; it is a band-narrowing investment, and its return is measurable in dollars per million items. A retrain that moves the uncertain band from 9% to 6% saves about $360 per million, which at five million items a day is roughly $54,000 a month. That is a substantially better business case than most model-quality arguments, and it is one a finance team can actually evaluate.
The same logic explains why tier 0 is worth building carefully even though it resolves only 1.5% of items. The items it catches are the highest-severity ones, and it catches them for effectively nothing — a perceptual hash lookup against known-violating media costs microseconds. Every item tier 0 resolves is an item that does not consume a classifier call, an adjudication and possibly a human review. The general shape of this reasoning, applied to model selection rather than classifier cascades, is in LLM routing, caching and cost per request.
What happens to one uploaded post, end to end?
Normalise, hash-check, classify text and image in parallel, route on the score vector against version-pinned thresholds, and then either decide immediately — which happens 92.5% of the time in under half a second — or spend about 1.4 seconds on an LLM adjudication, and only rarely queue it for a person. Every path writes to the audit ledger before the user is told anything.
The latency structure is unusual for an AI system and it is worth designing to deliberately. The overwhelming majority of items must be resolved fast enough to be synchronous with publication, because a platform that holds every post for two seconds feels broken. The cheap path — hash, text classifier, image classifier, router — is roughly 470 milliseconds and is dominated by the image API. The expensive path adds an adjudication and lands near 1.9 seconds, which is why items in the uncertain band should publish optimistically and be retracted if adjudication goes against them, rather than being held.
That optimistic-publish decision has a real consequence you should name rather than discover: for a short window, a small fraction of borderline content is visible. For most categories that is the correct trade, because holding all borderline content damages the experience of many more legitimate users than the brief exposure harms. For a small set of categories — child safety, credible threats, terrorist content — it is not, and those categories should be configured to hold rather than publish optimistically. That is a per-category flag in the policy store, not a global architectural stance.
The audit write happens before the user notice, always, and it is worth being strict about the ordering. If the ledger write fails and the notice succeeds, you have told a user their content was removed and have no record of why — which is exactly the scenario the regulation exists to prevent. Write the ledger row first, in the same transaction as the decision if you can, and treat a ledger failure as a reason to abandon the action rather than to proceed without a record.
Note where the cost concentrates in the sequence. The 92.5% fast path costs about two hundredths of a cent per item. The 7.5% that reach adjudication cost about three tenths of a cent. The 1.1% that reach a human cost six and a quarter cents. The bill is a weighted average dominated by its smallest population, which is the signature of every cascade and the reason threshold calibration deserves an engineer rather than an afternoon.
How do you set the uncertain band, and what does the router look like?
Empirically, on a labelled sample, per category, and then re-measured monthly. The floor is the score below which your false-negative rate is acceptable; the ceiling is the score above which your false-positive rate is acceptable; and the gap between them is what you are willing to pay a model and a human to resolve. Hand-picking these numbers is the most common expensive mistake in the category.
The procedure is not complicated but it does require labelled data you have to actually produce. Take a stratified sample of a few thousand items per category, have humans label them against the current policy version, and plot precision and recall against threshold. The ceiling goes where precision is high enough that auto-actioning is defensible — for most categories that means a false-positive rate you would be comfortable defending publicly, because you will have to. The floor goes where recall is high enough that auto-allowing is defensible. Everything in between escalates, and the width of that gap is a number you can now put on a cost model rather than a feeling.
Two properties make the router safe. First, thresholds are per category, not global: harassment and child safety should not share a ceiling, because the acceptable false-positive rate for one is nothing like the other. Second, thresholds are pinned to a policy version and travel with it, so that changing a threshold is a policy change with an audit record rather than a config edit. If a threshold can be changed by an environment variable, then a deployment can silently alter the enforcement outcome for millions of users with no record of who did it or when.
The router also needs an explicit abstain path from tier 2, and this is a design detail teams miss. The adjudicating model must be able to say I cannot decide this from the policy as written, and that verdict must route to a human rather than defaulting to allow or to remove. Forcing a binary from a model on genuinely ambiguous content produces confident coin flips, and confident coin flips are what generate the screenshots. Roughly 15% abstention on the uncertain band is a healthy figure; sustained abstention below 5% usually means the model has learned to guess, and above 30% usually means the policy text is ambiguous and needs rewriting rather than more compute. This is the same principle as the confidence gate in the customer support agent build, where the escape hatch to a human is what makes the automated path safe to run at all, and the wider stack these tiers sit in is described in the AI product architecture guide.
One more property worth building from the start: the router should be able to run in shadow mode against a new policy version while the old one remains in force, so that you can measure what a threshold change would have done before it does it. That capability turns policy changes from a leap into a measurement, and it costs one boolean and a second write to the ledger.
export type Category =
| "harassment" | "hate" | "sexual" | "violence" | "self_harm"
| "illicit" | "spam" | "impersonation" | "csae" | "terrorism";
export type PolicyVersion = {
id: string; // immutable, e.g. "pv_2026_08_11"
activatedAt: string;
thresholds: Record<Category, { floor: number; ceiling: number }>;
severity: Record<Category, number>; // higher wins ties
holdOnUncertain: Set<Category>; // no optimistic publish
};
export type Scores = Partial<Record<Category, number>>;
export type Route =
| { action: "allow"; policyVersion: string }
| { action: "enforce"; category: Category; score: number; policyVersion: string }
| { action: "adjudicate"; category: Category; score: number; hold: boolean; policyVersion: string };
export function route(scores: Scores, pv: PolicyVersion): Route {
let enforce: { category: Category; score: number } | null = null;
let uncertain: { category: Category; score: number } | null = null;
for (const [cat, score] of Object.entries(scores) as [Category, number][]) {
const t = pv.thresholds[cat];
if (!t) continue; // a category with no threshold in this version is inert
if (score >= t.ceiling) {
// Highest severity wins, not highest score. A 0.93 spam score must
// never outrank a 0.91 terrorism score.
if (!enforce || pv.severity[cat] > pv.severity[enforce.category]) {
enforce = { category: cat, score };
}
} else if (score > t.floor) {
if (!uncertain || pv.severity[cat] > pv.severity[uncertain.category]) {
uncertain = { category: cat, score };
}
}
}
if (enforce) {
return { action: "enforce", ...enforce, policyVersion: pv.id };
}
if (uncertain) {
return {
action: "adjudicate",
...uncertain,
hold: pv.holdOnUncertain.has(uncertain.category),
policyVersion: pv.id,
};
}
return { action: "allow", policyVersion: pv.id };
}
/**
* Shadow mode: evaluate a candidate policy version alongside the live one.
* Both routes are written to the ledger; only the live route is enforced.
* This turns a threshold change from a leap into a measurement.
*/
export function routeWithShadow(
scores: Scores,
live: PolicyVersion,
shadow: PolicyVersion | null,
): { live: Route; shadow: Route | null } {
return { live: route(scores, live), shadow: shadow ? route(scores, shadow) : null };
}1.5% of items, resolved in about eight milliseconds for effectively nothing. These are also the highest-severity items on the platform, which is why tier 0 is worth real engineering despite its small share.
Roughly 91% of traffic. Costs about two hundredths of a cent, almost all of it the image API. Nothing else in the pipeline ever sees these items.
About 1.5%. Auto-actioned with a statement of reasons and an appeal route. The ceiling must be calibrated to a false-positive rate you would defend publicly, because you will be asked to.
The 7.5% uncertain band, about a quarter of a cent each. This is where sarcasm, quoted abuse, satire, reclaimed slurs and medical discussion actually get resolved, using context the classifier never had.
About 1.1% of all items at six and a quarter cents each, which is 59% of the total bill. The abstain path is mandatory: a model forced to guess on genuinely ambiguous content produces confident coin flips.
How do policy versioning and the audit trail actually work?
Policy is an immutable versioned artefact containing rule text, category definitions, thresholds and labelled examples. Every decision pins the version that produced it. The audit ledger is append-only and stores the scores, the thresholds applied, both model versions, the verdict, the human-readable reason, whether a person was involved, and the appeal state.
The reason this has to be versioned rather than current is that decisions are challenged after the fact. A user appeals a removal from six weeks ago; a regulator asks about a class of enforcement from last quarter; an internal review asks whether a policy change increased false positives. All three questions are unanswerable if your policy is a document that gets edited and your thresholds live in configuration. All three are a single query if every decision carries a version id and every version is immutable.
Immutability has a practical shape: a new policy version is created, reviewed, activated at a timestamp, and never edited. Changing a single threshold produces a new version. That sounds heavy until you consider the alternative, which is a system where the sentence why was this removed has no reliable answer, and where a threshold change last Tuesday cannot be correlated with a spike in appeals this Monday because nobody recorded that it happened.
The ledger schema matters more than it looks. It must store the score vector, not just the winning category, because the interesting analysis is almost always about the categories that nearly fired. It must store both model versions — classifier and adjudicator — because a classifier retrain shifts score distributions and therefore shifts what the unchanged thresholds mean. And it must store the reason as text a human can read, because that text is what goes into the statement of reasons the user receives and, for EU platforms, into a machine-readable submission to the Commission's transparency database.
Two operational habits make the ledger useful rather than merely present. First, sample-and-review: a small random share of decisions across all outcomes, including allows, goes to a human whose verdict is recorded as agreement or disagreement rather than as a re-decision. This is the only way to measure false negatives, because the items you wrongly allowed never generate a signal. Second, replay: given a stored score vector and a candidate policy version, recompute what would have happened. That turns policy proposals into measurements over real historical traffic at zero model cost, and it is the single highest-return thing you can build on top of a well-designed ledger.
import { z } from "zod";
export const DecisionRecord = z.object({
itemId: z.string(),
decidedAt: z.string(), // ISO 8601, UTC
policyVersion: z.string(), // immutable id, pinned at ingest
classifierVersion: z.string(), // a retrain changes what a threshold means
adjudicatorModel: z.string().nullable(), // null when tier 1 resolved it
tier: z.enum(["t0_deterministic", "t1_classifier", "t2_llm", "t3_human"]),
// The whole vector, not just the winner. The near-misses are the analysis.
scores: z.record(z.string(), z.number()),
thresholdsApplied: z.record(z.string(), z.object({ floor: z.number(), ceiling: z.number() })),
verdict: z.enum(["allow", "limit_reach", "age_gate", "remove", "suspend_account"]),
category: z.string().nullable(),
reasonText: z.string().min(20), // goes to the user verbatim
automated: z.boolean(), // required disclosure under the DSA
humanReviewerId: z.string().nullable(),
appealState: z.enum(["none", "open", "upheld", "overturned"]),
shadowVerdict: z.string().nullable(), // what a candidate policy would have done
});
export type Decision = z.infer<typeof DecisionRecord>;
/** The user-facing notice. Automation disclosure and appeal route are not optional. */
export function statementOfReasons(d: Decision): string {
const lines = [
"We " + verbPhrase(d.verdict) + " your content.",
"",
"Why: " + d.reasonText,
"Policy: " + (d.category ?? "general") + " (version " + d.policyVersion + ")",
d.automated
? "This decision was made by an automated system" +
(d.humanReviewerId ? " and reviewed by a person." : " without human review.")
: "This decision was made by a person.",
"",
"You can appeal this decision. Appeals are reviewed by a person who did not make the original decision.",
];
return lines.join("\n");
}
/**
* Replay: what would a candidate policy version have decided, given the
* scores we already stored? Zero model cost, real historical traffic.
* This is the highest-return thing to build on top of the ledger.
*/
export function replay(
records: Decision[],
candidate: PolicyVersionLike,
): { changed: number; nowEnforced: number; nowAllowed: number; nowEscalated: number } {
let changed = 0, nowEnforced = 0, nowAllowed = 0, nowEscalated = 0;
for (const r of records) {
const next = routeFromScores(r.scores, candidate);
if (next.action === "enforce" && r.verdict === "allow") { changed++; nowEnforced++; }
else if (next.action === "allow" && r.verdict !== "allow") { changed++; nowAllowed++; }
else if (next.action === "adjudicate" && r.tier === "t1_classifier") { changed++; nowEscalated++; }
}
return { changed, nowEnforced, nowAllowed, nowEscalated };
}| Answerable from | Query shape | If you did not store it | |
|---|---|---|---|
| Why was this item removed? | Ledger: reason text, category, policy version | Ledger row + policy version | You have a regulatory exposure, not a bug |
| Was a human involved? | Ledger: automated flag, reviewer id | Ledger row | You cannot produce a compliant statement of reasons |
| What nearly fired on this item? | Ledger: full score vector | Ledger row | Every threshold post-mortem becomes guesswork |
| Did last Tuesday's threshold change cause this spike? | Policy versions with activation timestamps | Version table + ledger join | You correlate a spike with nothing and change something else |
| What would policy version N+1 have done? | Replay over stored score vectors | Ledger + candidate version | Every policy change is a live experiment on users |
| What is our false negative rate? | Random audit sample across all outcomes, including allows | Sampled human verdicts | You measure only the errors that already escalated — the ones you know about |
| How many appeals were overturned, by category? | Ledger: appeal state, category | Ledger aggregate | You cannot report it, and overturn rate is your best precision proxy |
| Which classifier version produced this score? | Ledger: classifier version | Ledger row | A retrain silently redefines every threshold and nothing points at it |
How should the human review queue work?
As a prioritised queue where each item arrives with an AI brief — the scores, the policy text for the relevant category, the surrounding context, and the model's reasoning including why it abstained — so a reviewer decides in about 25 seconds instead of 45. Plus wellbeing controls, plus a mandatory random audit sample that is not enforcement work.
The brief is the single highest-leverage artefact in the tier and it costs almost nothing, because the adjudication call that produced the abstention already generated the reasoning. Surfacing it costs zero additional model spend. Modelled at $9 an hour fully loaded, going from 45 seconds to 25 seconds per item saves five hundredths of a cent per review — which at 11,000 human reviews per million items is about $550 per million, or roughly 47% of the human line. That is the best ratio in the build and it is a UI decision.
Prioritisation should be by severity and by time-to-harm, not by queue arrival order. A credible threat that has been visible for four minutes outranks a spam appeal that has been queued for two days, and a queue that does not model this will process them in the wrong order on the busiest day of the year. Attach an explicit SLA per severity class and alert on the SLA rather than on queue depth, because depth is meaningless without knowing what is in it.
Wellbeing is an architectural constraint in this domain, not an HR footnote. Reviewers seeing the worst content on your platform need rotation limits, blurring and greyscale defaults on media, the ability to make a decision from the brief without opening the media where the classifier scores are unambiguous, and counters that cap consecutive exposures per shift. These have direct engineering consequences — media proxying, thumbnail generation with configurable obfuscation, per-reviewer exposure accounting — and they need to be in the schema from the start.
Finally, keep the random audit sample separate from the enforcement queue and make its verdicts non-binding. Its purpose is measurement, not action. It is the only mechanism that can detect false negatives, because items you wrongly allowed generate no complaint, no appeal and no signal of any kind. A half-percent random sample across all outcomes at six and a quarter cents each is about $310 per million items — roughly a quarter of your human budget spent on knowing whether the other three quarters are working.
- Reviewer reads the item, then opens the policy, then decides
- No visibility into which category nearly fired or by how much
- Media opened at full fidelity by default — a wellbeing problem as well as a speed one
- Inconsistent decisions between reviewers, because each reconstructs the policy from memory
- Scores, thresholds and the specific policy clause rendered inline
- The adjudicator's reasoning and its stated reason for abstaining
- Media blurred by default with one click to reveal, and an exposure counter per shift
- Costs $0 extra — the reasoning was generated by the adjudication call that abstained
What does moderation cost at a million items?
About $1,183, or roughly a tenth of a cent per item, split as $213 of classifiers, $207 of LLM adjudication, $736 of human review including appeals and the audit sample, and $27 of deterministic matching, policy storage and the audit ledger. At five million items a day that is roughly $178,000 a month.
The distribution is the story. Human review is 59% of the bill while touching 1.1% of the items. Image classification alone is $197 — 93% of the tier-1 line and 17% of the whole bill — because images are a fifth of the volume at fifty times the unit price of text. Every language model call in the pipeline combined is 17% of the total. If you arrived at this build expecting to spend your time on prompt engineering, the arithmetic will redirect you within a week.
Three levers move this materially and they are worth ranking honestly. Narrowing the uncertain band is the largest: about $120 per million per percentage point, so a classifier retrain that takes the band from 9% to 6% is worth roughly $360 per million, or about $54,000 a month at five million items a day. Improving the reviewer brief is second: around $550 per million, and it is a front-end change. Reducing image classification cost is third and is mostly a procurement conversation, though tier-0 duplicate detection across accounts genuinely removes a meaningful share of image calls on platforms with reposting behaviour.
The lever that is not on the list is model choice at tier 2. Moving adjudication from a Haiku-class model to a frontier model would take that line from $207 to roughly $900 per million and improve agreement on a band where trained human reviewers themselves disagree at a substantial rate. That is a poor trade in a system where the human tier already exists precisely to handle the cases the model finds hard. Spend that money on narrowing the band instead, where the return is measurable.
One caution on scaling assumptions. Published per-image pricing tiers down with volume — the modelled rate drops from $0.001 to $0.0008 above a million images a month — while human review costs scale linearly and appeal volume scales with enforcement volume, which scales with traffic. So the per-item cost falls modestly with scale and the absolute number does not stop being large. Plan the human tier as a staffing model with a hiring lead time, not as a line item that elastically absorbs a traffic spike.
| Line item | Model / rate | Items processed | Cost per 1M | Share | Note |
|---|---|---|---|---|---|
| Tier 0 deterministic | Hash index + blocklists, amortised infra | 1,000,000 | $2.00 | 0.2% | Resolves 1.5% of items, and the highest-severity 1.5% at that |
| Tier 1 text classifier | Self-hosted distilled model · $0.02 per 1,000 | 788,000 | $15.76 | 1.3% | Cheap enough that running it on everything is not a decision |
| Tier 1 image classifier | Commodity API · $0.001 per image, first 1M/month | 197,000 | $197.00 | 16.7% | 20% of volume, 93% of tier-1 cost. The asymmetry that shapes tier 0 |
| Tier 2 LLM adjudication | Haiku 4.5 · 4.5k in (3.5k cached) / 250 out | 73,875 | $206.85 | 17.5% | $0.0028 each. Policy text in the cached prefix is what makes this viable |
| Tier 3 human review | $9/hr fully loaded · 25s with an AI brief | 11,081 | $692.56 | 58.5% | 1.1% of items, and the majority of the money |
| Appeals | $9/hr · 90s per appeal | 194 | $43.75 | 3.7% | Modelled at 0.4% of actioned items. Overturn rate is your best precision proxy |
| Random audit sample | Included in the human line above | ~5,000 | (in $692.56) | — | 0.5% across all outcomes. The only instrument that sees false negatives |
| Policy store + audit ledger | Postgres + object storage, amortised | 1,000,000 | $25.00 | 2.1% | Non-optional. Cheaper than a single regulatory conversation |
| Total | — | — | $1,182.92 | 100% | $0.00118 per item · modelled, not measured |
- Human review (1.1% of items)$693 · 59%
- LLM adjudication (7.5% of items)$207 · 17%
- Image classification (20% of items)$197 · 17%
- Appeals$44 · 4%
- Ledger, policy, tier 0, text classifier$43 · 4%
- Items per month
- 150,000,000
- Classifier tier
- Not used
- LLM adjudications
- 150M at ~$0.0029
- Human reviews
- 2.2M at $0.0625
- Monthly cost
- ~$644,850
- Cost per item
- $0.0043
- Items per month
- 150,000,000
- Classifier tier
- $31,914
- LLM adjudications
- 11.1M at $0.0028 = $31,028
- Human reviews + appeals
- 1.7M = $110,447
- Monthly cost
- ~$177,440
- Cost per item
- $0.00118
What breaks in production, and how would you know?
Twelve things, and the three most damaging are invisible on every dashboard you would naturally build. A classifier retrain that silently redefines your thresholds, a policy category with no threshold that quietly never fires, and a false-negative rate you cannot see because wrongly allowed content generates no signal at all.
Read the table by its second column. The row that ends products is the false-positive spike on a popular creator: an over-tightened ceiling removes legitimate content at scale, the affected users are exactly the ones with an audience, and the resulting story is about censorship rather than about a threshold. The row that ends companies is the false-negative one, where harmful content stays up because nothing ever escalated it. The asymmetry is worth naming: false positives are loud and recoverable, false negatives are silent and not.
Four signals catch most of it. Uncertain-band width as a daily time series is the primary operational metric, because it is simultaneously your cost driver and your earliest indicator that a classifier or a traffic distribution changed. Appeal overturn rate by category is your best available precision proxy and it needs no labelling. Random-audit disagreement rate, sampled across all outcomes including allows, is the only measurement of false negatives you will ever have. And human-queue SLA attainment by severity class tells you whether the tier that costs 59% of your budget is actually functioning.
Alert on distributions and never on individual decisions. A moderation system that gets one decision in two hundred wrong is a functioning moderation system. A system whose uncertain band moved from 7% to 13% overnight is an incident, and the usual causes are a classifier deployment, a policy version activation, or a change in traffic mix — a viral format, a new surface, a language you were not measuring separately. All three are invisible in model metrics and immediately obvious in band width.
The final row is the one this whole design is organised around. If you cannot reconstruct, for a specific item from two months ago, the score vector, the thresholds applied, the policy version, both model versions, the verdict and the reason, then you cannot answer an appeal, cannot answer a regulator, and cannot tell whether last quarter's policy change helped. That is one append-only table, written before the user is notified, and it is the difference between a system you can govern and a system you can only defend.
| Failure mode | What the user sees | Where to fix it | Detection signal | Cost of getting it wrong |
|---|---|---|---|---|
| Classifier retrain shifts score distribution | Nothing at first, then a wave of removals or a wave of misses | Pin classifier version in the ledger; require threshold recalibration as part of any retrain release | Uncertain-band width as a daily series; score histogram drift per category | Every threshold in the system silently means something different. Systemic and hard to attribute |
| A policy category has no threshold in the active version | A whole class of violating content is never actioned | Router: fail the policy version at validation if any active category lacks thresholds | Per-category enforcement counts — a category at zero for 24h is an alarm | Silent total failure for that category, potentially for months |
| Ceiling set too low on a popular category | Legitimate posts by high-visibility users removed at scale | Policy: calibrate the ceiling to a false-positive rate you would defend publicly | Appeal overturn rate by category; enforcement rate spike per surface | The story becomes about censorship. Loud, fast, and reputationally expensive |
| Floor set too high | Harmful content stays up and nothing escalates it | Policy: floor calibrated on labelled recall, plus a random audit sample over allows | Random-audit disagreement rate on allowed items — the only false-negative signal | The failure with no telemetry. Discovered by journalists or by regulators |
| Stale known-bad hash list | Previously removed media reappears and is re-adjudicated | Tier 0: continuous hash list refresh with a freshness SLA and an alarm | Age of newest hash entry; share of tier-2 verdicts matching known-bad media | You pay 1,300x per item to re-decide something you already decided |
| Policy cache prefix drifts | Nothing visible. The adjudication bill quadruples | Adjudicator: policy text as a stable prefix, invalidated only on version change | Cache hit rate on the adjudication call; cost per adjudication as a series | About $600 extra per million items, discovered in a monthly bill review |
| Model forced to decide instead of abstaining | Confident, arbitrary verdicts on genuinely ambiguous content | Adjudicator: explicit abstain verdict routed to a human, never defaulted | Abstention rate — sustained below 5% means it has learned to guess | Coin-flip enforcement on exactly the content most likely to be screenshotted |
| Human queue SLA silently degrades | Harmful content visible for hours; appeals unanswered for days | Queue: SLA per severity class, alert on attainment rather than on depth | SLA attainment by severity; oldest item age per class | Time-to-harm grows without any dashboard turning red |
| Image API outage or timeout | Either everything publishes unchecked, or nothing publishes | Tier 1: explicit timeout, and a documented fallback that fails safe per category | Image classifier error rate and p99; share of items with a null image score | Whichever way you fail by default is the way you will fail at 3am. Choose it deliberately |
| Appeals reviewed by the original decision-maker | Appeals that uphold at an implausible rate | Queue routing: exclude the original reviewer from the appeal assignment | Overturn rate on appeals; assignment overlap rate — should be exactly zero | The appeal route becomes decorative, which is both a trust and a compliance failure |
| No audit sample over allowed items | Nothing. Ever. That is the problem | Sampling: random across all outcomes including allows, verdicts non-binding | Existence of the sample. Either it runs or your false-negative rate is unknown | You have no idea what you are missing and no way to find out |
| No ledger row before user notice | A user told their content was removed, with no record of why | Write the ledger row in the same transaction as the decision; abandon on failure | Count of notices without a matching ledger row — should be exactly zero | Regulatory exposure under the DSA statement-of-reasons obligation |
- Uncertain-band width, daily, per categoryYour cost driver and your earliest warning simultaneously. A jump from 7% to 13% is an incident, and the cause is a classifier deploy, a policy activation or a traffic-mix change.
- Appeal overturn rate, by categoryThe best precision proxy available without labelling. A category overturning at 30% has a ceiling set too low, and you will find out from users before you find out from a metric.
- Random-audit disagreement rate, across all outcomesThe only false-negative instrument you will ever have. Must include allowed items; a sample drawn from escalations measures the model against itself.
- Human queue SLA attainment, by severity classAlert on attainment, not on depth. Depth is meaningless without knowing what is in the queue and how long it can safely wait.
- Adjudication cache hit rate and cost per adjudicationA silent prefix change quadruples the second-largest line in your bill with no error, no alert and no symptom until the invoice.
- Per-category enforcement counts, with a zero alarmA category that fires zero times in 24 hours is either genuinely absent from your platform or completely broken, and you cannot tell which without looking.
- Reviewer exposure counters and rotation limitsA wellbeing control with direct schema consequences: per-reviewer exposure accounting, media obfuscation defaults, consecutive-exposure caps. Rarely built in v1 and it should be.
What does it take to build, and what should you skip in v1?
About twelve engineer-weeks for a two-person team to a production-grade v1, plus roughly four engineer-days a month of maintenance and a permanent threshold-calibration cadence. Skip video, skip custom classifier training, skip real-time streaming moderation and skip multilingual until the English cascade is calibrated and the ledger is trustworthy.
Week one is ingest, the item model and the audit ledger, which has to exist before any decision is made because it cannot be backfilled. Week two is the policy store with immutable versions and threshold tables. Weeks three and four are tier 0 and tier 1 — hash indexing, blocklists, the text classifier, the image API integration with its timeout and fail-safe behaviour. Week five is the router, which is small code and a large amount of thinking about severity ordering and per-category hold behaviour.
Weeks six and seven are threshold calibration, and this is the phase that gets cut and must not be. It requires a labelled sample — a few thousand items per category, labelled by humans against the current policy — and it produces the precision-recall curves that place your floors and ceilings. Without it you are guessing at the number that sets 80% of your bill and both of your error rates. Week eight is tier 2 adjudication with the cached policy prefix and the abstain path. Weeks nine and ten are the human review tool, the brief, prioritisation, SLA and wellbeing controls. Weeks eleven and twelve are appeals, the transparency export, replay, dashboards and rollout in shadow mode.
Skip video moderation in v1 and be firm about it. Video multiplies cost by frame sampling, multiplies latency by an order of magnitude, and requires a different failure model because a violation may occupy four seconds of a twelve-minute upload. It is a second system, not a feature, and building it before the still-image cascade is calibrated guarantees you calibrate neither. Skip custom classifier training too: commodity classifiers plus a well-placed uncertain band will get you to a working system, and you will not know what to train on until you have several months of ledger data and audit labels — which is exactly the dataset a custom model needs.
What you must not skip: the audit ledger, immutable policy versions, per-category thresholds calibrated on real labels, the abstain path, the random audit sample across all outcomes, and appeals routed away from the original decision-maker. Those six are about four of the twelve weeks and they are the difference between a system you can govern and one you can only apologise for. If you want this architected and built rather than described, that is what AI product development is for; the routing and caching machinery behind the adjudication tier is in LLM routing, caching and cost per request, and the same cascade discipline applied to files rather than posts is in the document processing pipeline build.
- Week 1Item model and audit ledger
One normalised item record, the append-only decision ledger with score vectors and version pins, cost attribution. Cannot be backfilled, which is why it is week one.
- Week 2Policy store
Immutable versions with activation timestamps, per-category thresholds, severity ordering, hold-on-uncertain flags, labelled examples. Policy becomes an artefact, not a document.
- Weeks 3‑4Tier 0 and tier 1
Perceptual hash index with a freshness SLA, blocklists, the text classifier, the image API with an explicit timeout and a per-category fail-safe direction chosen deliberately.
- Week 5The router
Fifteen lines of code and a week of thinking: severity ordering, per-category hold behaviour, shadow-mode evaluation, and validation that fails a policy version missing thresholds.
- Weeks 6‑7Threshold calibration
A few thousand human-labelled items per category, precision-recall curves, floors and ceilings placed on evidence. The phase most often cut and the one that sets 80% of the bill.
- Week 8Tier 2 adjudication
Policy text as a stable cached prefix, structured verdict schema, the mandatory abstain path, and cost instrumentation per adjudication.
- Weeks 9‑10Human review tool
The AI brief, severity prioritisation, SLA per class, media obfuscation defaults, exposure counters and rotation limits. Worth 47% of the human line on speed alone.
- Weeks 11‑12Appeals, transparency, rollout
Appeal routing away from the original reviewer, machine-readable statement-of-reasons export, replay over stored score vectors, dashboards, and two weeks in shadow mode.
Building an AI content moderation pipeline: common questions
→How much does AI content moderation cost per million items?
On the cascade modelled here, about $1,183 per million mixed text-and-image items, or roughly a tenth of a cent each: $213 of classifiers, $207 of LLM adjudication on a 7.5% uncertain band, $736 of human review including appeals and a random audit sample, and $27 of deterministic matching, policy storage and the audit ledger. Sending every item to a language model instead costs about $4,299 per million on the same assumptions, and a human-only queue about $112,500. All of these are modelled from published list rates with the assumptions printed, not measurements of a system I have operated.
→Why use a cascade instead of just sending everything to an LLM?
Because the cost spread between tiers is roughly 300 to one, so the only variable that matters is how many items reach the expensive tiers. A classifier decision is modelled at $0.0002, an LLM adjudication at $0.0028 and a human review at $0.0625. Adjudicating everything is about 3.6 times more expensive and buys quality only on the band the cascade already routes to the model. The cascade does stop being worth its complexity if your classifiers cannot resolve most traffic — on these numbers, above about a 33% uncertain band you should simplify and adjudicate everything.
→How do you set the thresholds for the uncertain band?
Empirically, per category, on a human-labelled sample of a few thousand items measured against the current policy version. The ceiling goes where precision is high enough that auto-actioning is defensible in public; the floor goes where recall is high enough that auto-allowing is defensible; the gap between them is what you pay a model and a human to resolve. Thresholds must be per category rather than global, because the acceptable false-positive rate for spam is nothing like the one for child safety, and they must be pinned to an immutable policy version so that a threshold change is an auditable policy event rather than a config edit.
→Do you still need human moderators if the AI is good?
Yes, for three separate reasons, only one of which is quality. First, some decisions should be made by an accountable person, and regulators increasingly expect that. Second, the model must be allowed to abstain on genuinely ambiguous content, and an abstention has to go somewhere. Third, a random audit sample reviewed by humans is the only mechanism that can measure false negatives, because content you wrongly allowed produces no complaint and no appeal. In the model here humans see 1.1% of items and account for 59% of the cost, which is the correct shape rather than a problem to eliminate.
→What does the EU Digital Services Act require from a moderation pipeline?
In engineering terms, a per-decision data structure. When a platform restricts content it must give the user a clear and specific statement of reasons, including which measure was taken, why, whether an automated process was involved, and how to appeal. Online platforms also submit pseudonymised statements of reasons to the Commission's DSA Transparency Database, which publishes them in machine-readable form close to real time, and very large platforms face independent audits. That means the policy version, the scores, the thresholds, the model versions, the verdict, the reason text and the automation flag all have to exist at the moment of the decision — they cannot be reconstructed later, which is why the audit ledger belongs in week one.
→Which moderation classifier should I use?
Start with a commodity option and spend your effort on the router rather than on model selection. OpenAI's omni-moderation-latest endpoint is free and accepts both text and images, though its image coverage is narrower than its text coverage — violence, self-harm and sexual content for images, with harassment, hate and illicit categories text-only — so map it against your policy before assuming coverage. Amazon Rekognition publishes image moderation at $0.001 per image for the first million per month, but note that AWS stopped offering Batch Image Content Moderation and Streaming Video Analysis to new customers as of 30 April 2026. Hive publishes no list pricing and requires a sales conversation. Custom training is a month-six decision that needs ledger data and audit labels you do not have yet.
→How long does it take to build one?
About twelve engineer-weeks for a two-person team to a production-grade v1, plus roughly four engineer-days a month of maintenance and a permanent threshold-recalibration cadence. The router itself is a week. The audit ledger and policy store are two, threshold calibration is two, the human review tool is two, and appeals plus transparency export plus rollout are two. Video moderation is a separate system rather than a feature and should not be in v1 — it multiplies cost by frame sampling, multiplies latency by an order of magnitude, and needs a different failure model because a violation may occupy four seconds of a twelve-minute upload.