How to Hire AI Developers in 2026: What to Test For (and What to Ignore)
- The job you are hiring for is not machine learning research. It is production systems engineering where one dependency is non-deterministic, so test evaluation, failure handling and cost control, not model architecture trivia.
- US market data in 2026 puts AI and ML engineer averages near $173K–$187K base, with a small frontier-lab tier at multiples of that. You are not competing with the frontier tier and should not price as if you are.
- The single most predictive interview question: walk me through an AI feature you shipped, a specific failure it produced in front of a user, and exactly what you changed. Tutorial-only candidates cannot answer it.
What should you actually test when hiring an AI developer?
Test four things: whether they have shipped an AI feature to real users, whether they can write an evaluation set before writing a prompt, whether they can estimate the per-request cost of their own design, and how they debug a retrieval pipeline that is returning plausible nonsense. Model architecture questions predict almost nothing for this role.
The confusion at the heart of most bad AI hiring processes is a job-title collision. An ML researcher trains models. An AI engineer composes existing models into a product and is accountable when the output embarrasses the company. Those are different jobs with different failure modes, and interviewing the second with questions designed for the first selects for the wrong person confidently.
The failure modes make this role distinct from ordinary backend engineering. A conventional service fails loudly with a stack trace. An AI feature fails quietly, plausibly and intermittently: retrieval drift, a prompt regression no unit test catches, a cost curve that triples after a model upgrade. If a candidate has not been on the receiving end of at least one of those, they have not done the job yet.
I have hired for this and been hired into it. I was promoted from SDE III to Head of Engineering in seven months and built AccioMatrix, an AI assessment and interview platform, solo from zero to 20+ enterprise clients. Everything below is what I actually screen for, including the parts that make candidates uncomfortable.
What does an AI engineer do that a backend engineer does not?
Five things, and they are the whole job. Evaluation, because correctness is statistical rather than binary. Cost control, because every request has a variable price. Failure design, because the dependency returns plausible wrong answers rather than errors. Latency shaping, because streaming changes what users tolerate. And adversarial input handling, because prompt injection is a live attack surface rather than a research curiosity.
Evaluation is the discriminator. A candidate who reaches for an eval set before a prompt has understood the shape of the problem; a candidate who tunes prompts by vibes has not. Ask what they measured, on how many examples, and what threshold decided a change was an improvement. The example count is usually the tell: real eval sets start small and grow from production failures, so a candidate who says "about forty cases, mostly things that broke" is more credible than one who says "a thousand synthetic examples."
Cost control is the second discriminator and it is where I have the most first-hand ground. I took a production voice pipeline from roughly 10¢ to about 2.5¢ per minute through routing, caching and provider selection, and separately cut a $200K+/year cloud bill by more than 70%. The skill that produced both is the same: knowing which line item dominates before optimising anything. A candidate who optimises the model first, when transport or synthesis dominates the bill, will burn a quarter for nothing.
Adversarial input is the one most teams under-weight. If the feature accepts user text and calls a model with any tool or data access, prompt injection is a real vulnerability with real consequences, not a thought experiment. Ask what boundary they put between untrusted input and privileged action. "We told the model not to" is a failing answer.
Can they define correctness before building? Ask for the eval set size, the metric, and the threshold at which they would ship or roll back. Real sets are small and grown from production failures.
test firstCan they estimate per-request cost from token counts and rate cards, and name which line item dominates? Routing, caching and provider choice beat model tuning almost every time.
test secondWhat happens on timeout, on low confidence, on a refusal, on a malformed structured output? Fallbacks, retries with different models, and a deterministic path are the answers you want.
test thirdStreaming, time-to-first-token, and what the interface does while waiting. Perceived latency is a product decision that engineers own in this role.
test fourthPrompt injection, data exfiltration through tool calls, and the boundary between untrusted text and privileged action. Instruction-based defences alone are a failing answer.
test fifthWhat does hiring an AI developer cost in 2026?
The US market has split in two, and knowing which half you are recruiting from prevents most compensation mistakes. Aggregated 2026 reporting puts the AI and ML engineer average around $173,000 on Glassdoor with a 90th percentile near $270,000, and Indeed's average base for US ML engineers around $187,000. Separately, a small frontier-lab cohort commands several times that for nominally the same title.
You are almost certainly not competing with the second group and should not benchmark against it. The engineers you can actually hire are the enterprise tier, and the differentiator you can offer is scope: at a startup, one person owns the whole pipeline from retrieval through evals to cost. That is a genuine attraction for the right candidate and it costs you nothing.
Reported premiums for generative AI and LLM fine-tuning specialisation run around 40–60% above baseline ML compensation in 2026 guides. Worth paying when the model is your product, worth refusing when you are composing APIs, which describes most companies hiring for this. Be honest about which you are.
The contract market is a legitimate arbitrage here. A senior AI engineer on a three-month engagement to build your evaluation harness and cost-routing layer costs materially less than a permanent hire and leaves you with the two assets that keep paying. That is a large share of what my own engineering hiring and dedicated development team engagements actually deliver.
What does a good AI engineering interview loop look like?
Four stages, about four hours of candidate time, and no take-home longer than ninety minutes. The structure matters less than the principle: every stage should be something a tutorial cannot prepare you for, because tutorial preparation is exactly what the current market has an oversupply of.
Stage one is the shipped-feature post-mortem, twenty-five minutes, conversational. Stage two is a live debugging exercise: hand them a retrieval pipeline with a stated symptom (recall is fine but answers cite the wrong document) and watch the method. You are grading whether they isolate retrieval from generation before touching anything, not whether they find the bug.
Stage three is a costed design. Give a concrete workload, say ten thousand conversations a month averaging eight turns, and ask for an architecture with a monthly cost estimate and the single line item they would attack first. Strong candidates ask for token counts and rate cards before answering. Weak ones produce a diagram with no numbers on it.
Stage four is adversarial: how would you attack this feature, and what would you put in the way. Then one honest question about failure: what is an AI feature you built that you would not build again, and why. Unwillingness to answer that last one is itself the answer.
One structural warning. AI-assisted interview cheating is now common enough to change how you design remote loops: one vendor's analysis of 19,368 AI-led interviews between July 2025 and January 2026 flagged 38.5% of candidates for AI-assisted cheating signals, rising to around 48% for technical roles. Design your loop so assistance does not help, which is what the next section is about, and what AI interview proctoring covers in depth.
- Tells a shipped-feature failure story with a date, a stakeholder and a specific fixhardest to fake
- Reaches for an evaluation set before a prompt, and can state its size and thresholdthe discriminator
- Estimates per-request cost from token counts and names the dominant line item
- Isolates retrieval from generation before debugging anythingmethod over answer
- Puts a real boundary between untrusted input and privileged actionnot just instructions
- Bonus: can say what they would not build again, and whyrare, and decisive when present
What signals should you ignore?
Kaggle rankings, certificate courses and the ability to explain attention on a whiteboard. None of them are worthless in absolute terms; all of them are weakly correlated with the job you are hiring for, and all of them are cheap to acquire, which means your pipeline is saturated with them.
Framework name-dropping is the most misleading. Naming four orchestration frameworks tells you what a candidate has read about, not what they have run. The follow-up that separates them: which one did you remove, and why. Engineers who have shipped have removed at least one framework from a production path, and they remember exactly what it cost them.
Years of experience is close to useless for this specific role because the role is younger than most of the numbers on a CV. Someone with eighteen months of hands-on production LLM work will frequently outperform someone with eight years of classical ML and six months of exposure. Screen on artefacts and stories, not on tenure.
The one traditional signal I would keep is ordinary software engineering competence. The most common failure mode I see is not a weak grasp of models. It is an AI engineer who cannot write a reliable background job, structure a schema or reason about idempotency, so the AI feature is fine while everything around it is fragile. This is a systems engineering hire with an unusual dependency, and it should be interviewed accordingly.
| Predictive? | Fakeable? | What to ask instead | |
|---|---|---|---|
| Kaggle / competition rank | Weak | Partly | What broke in production, not on a leaderboard |
| Certificates and courses | — | ✓ | Show me something you shipped that users touched |
| Naming frameworks | — | ✓ | Which one did you remove, and what did it cost |
| Explaining transformer internals | Weak for product roles | Partly | How do you decide a prompt change is an improvement |
| Years of experience | Weak | — | Months of hands-on production LLM ownership |
| A written eval set | ✓ | — | How many cases, and where did they come from |
| A shipped-failure story | ✓ | — | What did the user see, and what did you change |
| Cost estimate from first principles | ✓ | — | Which line item dominates, and how do you know |
How do you run the interview without getting cheated?
Assume assistance is present and design questions that assistance does not help with. That is a cheaper and more reliable posture than trying to detect every tool, and it happens to produce a better interview regardless.
The scale of the problem is now well documented. Fabric's analysis of 19,368 AI-led interviews between July 2025 and January 2026 flagged 38.5% of candidates, with roughly 48% in technical roles against about 12% in sales. In a separate cut across more than 50,000 candidates, the flagged share more than doubled from 15% in June 2025 to 35% by December 2025. Pindrop's 2025 Voice Intelligence and Security Report recorded a 1,300% year-on-year rise in deepfake fraud attempts. Gartner projects that by 2028, one in four candidate profiles worldwide could be fake.
The interview-design response is straightforward. Ask about specific artefacts the candidate produced and follow every answer with a why only the author could answer. Interrupt: real-time assistance is always a turn behind, and a question asked mid-answer collapses a relay setup immediately. And prefer discussion of past work to novel problem-solving, because past work has details no model can invent on the candidate's behalf.
Identity is the separate and more serious problem. If the role is remote and the candidate will hold production access, verify identity properly at offer stage rather than relying on interview-time impressions. The tooling side of that is covered in how candidates cheat AI interviews and in the AI interview proctoring guide.
Should you hire, contract, or buy a pod?
Depends on whether AI is your product or a feature of it, and most companies answer this wrong in the flattering direction. If your differentiation is the model, the data or the evaluation methodology, hire permanently and pay the specialisation premium. If your differentiation is the workflow and the AI is one component, contract the specialist skill and keep the domain knowledge in-house.
The pattern I recommend most often: bring in a senior AI engineer for eight to twelve weeks to build three assets: an evaluation harness, a cost and routing layer and a written failure-mode document. Your existing backend engineers own the feature afterward. Those three artefacts are what a permanent hire would have produced in their first quarter, and they persist after the engagement ends.
Do not hire a permanent AI engineer to do prompt engineering. That is not a role, it is a task, and building a headcount around it produces someone who will leave within a year having built nothing durable. If the honest description of the work is composing three API calls and tuning a prompt, one of your existing engineers should own it with a week of support.
And do not hire before you can evaluate. If nobody in your company can tell a strong AI engineer from a confident one, your first spend should be on the evaluation, not the hire: exactly the gap a short fractional CTO engagement fills.
Pay the 40–60% specialisation premium reported in 2026 guides. This is core capability and it should not sit in a contract.
Buy the eval harness, the routing layer and a written failure-mode document. Your existing backend engineers run it afterwards.
That is a task, not a role. Give it to an existing engineer with a week of senior support and spend the headcount elsewhere.
Hiring before you can assess is how companies acquire a confident engineer instead of a competent one. Fix the assessment, then hire.
Two to four cross-trained engineers with one senior owner. Communication paths stay low and context compounds across features.
Hiring AI developers: common questions
→What skills should an AI engineer have in 2026?
Evaluation design, cost estimation and control, failure handling for non-deterministic dependencies, latency shaping for streaming interfaces, and defences against adversarial input such as prompt injection. Underneath all of it, ordinary systems engineering: schemas, background jobs, idempotency. The most common failure I see is a competent AI feature surrounded by fragile infrastructure.
→How much does it cost to hire an AI developer?
Aggregated 2026 US reporting puts AI and ML engineer averages around $173,000 on Glassdoor and about $187,000 base on Indeed, with a 90th percentile near $270,000. Generative AI and LLM fine-tuning specialists are reported at a 40–60% premium over baseline ML compensation. A separate frontier-lab tier pays several times these figures and is not the pool most companies are recruiting from.
→What is the best interview question for an AI engineer?
Walk me through a generative AI feature you shipped, including a specific failure it produced in front of a user and exactly what you changed afterwards. Candidates who have only built demos answer with an architecture description; candidates who have shipped answer with a story containing a date, a consequence and a fix. The difference is audible within thirty seconds.
→Should I test AI engineers with a coding exercise?
Prefer a live debugging exercise over a take-home. Hand them a retrieval pipeline with a stated symptom and grade the method (whether they isolate retrieval from generation before touching anything) rather than whether they find the bug. Take-homes are the most compromised part of a remote loop, with one vendor flagging 38.5% of candidates across 19,368 interviews for AI-assisted cheating signals.
→Do I need a PhD-level machine learning hire?
Only if you are training or fine-tuning models as your core differentiation. If you are composing existing model APIs into a product, you need a systems engineer who understands statistical correctness, cost curves and failure design. Interviewing a product AI role with research questions selects for the wrong person confidently, which is the most expensive mistake in this category.
→Is it better to hire an AI engineer or contract one?
Contract when AI is a feature and hire when AI is the product. The pattern that works most often is an eight-to-twelve-week engagement to build three durable assets: an evaluation harness, a cost and routing layer, and a written failure-mode document. After that your existing backend engineers own the feature. Those artefacts persist; the contract does not.