Back to blog
6 min read

How to tell whether a candidate can code, or just prompt

aiassessmentsengineeringinterviews

A candidate pastes a broken function into an assistant, gets back something that passes the visible tests, and cannot tell you why it works. That's the actual worry, and it has nothing to do with cheating. The candidate didn't break a rule. They used the tool the way you'd want them to use it on the job. The problem shows up later, in a production incident, when the same person can't debug code that has their name on the commit.

The real worry

Say it plainly: assume every candidate uses an assistant, because on the job they will, and telling them not to during an interview tests a skill they'll never use again. The worry isn't that they use one. It's that a good enough assistant can produce working code for a problem the candidate doesn't understand, and an interview format that only looks at the final answer can't tell the difference between the two people who submitted it.

One of those people can explain every line, would have written something close to it without help, and used the assistant to go faster. The other accepted the first suggestion, ran the tests, and moved on. Both submissions look identical on the page. The cost of not telling them apart shows up three weeks after the offer, not during the interview, which is exactly why it's easy to under-rate.

The tradeoff is that testing for this takes longer than reading a diff. You have to watch the process, not just the output, which means designing exercises around a transcript instead of an answer key. It costs more of your time per candidate. It's also the only way to catch the gap before it's on your team.

Four signals that separate understanding from pasting

None of these require banning the assistant. All four work better with it running.

Can they read code they didn't write

Hand the candidate a function, forty or so lines, that they've never seen. It doesn't matter whether a person or an assistant wrote it. Ask them to explain what it does, what it assumes about its input, and where it would break. Someone who understands code does this in a few minutes and usually finds the edge case unprompted. Someone used to pasting and accepting describes the syntax back to you and stalls on "why."

This takes ten to fifteen minutes and needs no setup beyond picking a function with one planted assumption worth finding.

Can they find the bug the assistant introduced

Ask the candidate to solve a small problem with an assistant, using a prompt or a tool you already know tends to produce a subtly wrong answer: an off-by-one, a comparison that runs backward, a sort that's stable when it needs not to be. Watch whether they ship it or catch it. This is the closest thing to a direct test of the worry itself, since it puts a wrong suggestion in front of them and watches what they do with it.

Budget fifteen to twenty minutes. You need a task where you already know the failure mode the assistant tends to produce, which takes some work to find once, and then you can reuse it.

Can they explain a tradeoff they didn't look up

After they've solved something, ask why the assistant chose one approach over another, live and unplanned: why this data structure and not a hash map, why recursion here and not a loop, what it costs at ten times the input size. A candidate who understood the suggestion has an answer. A candidate who accepted it has a restatement of the code, not a reason for it.

This adds five to ten minutes as a follow-up to whatever they just built. It doesn't work as a standalone exercise, because it needs their own answer to interrogate.

Do they change a suggestion before they accept it

Watch, or read the transcript of, a real working session and check whether they ever reject, edit, or redirect what the assistant gave them. A renamed variable is a weak signal. A rejected suggestion with a reason attached is a strong one. Someone who takes ten suggestions in a row exactly as written either got very lucky or wasn't reading closely.

This isn't a separate exercise. It's what you're watching for during whichever of the first three you run with an assistant in the room.

Why the transcript beats the final code

The final code shows you the destination. The transcript shows you whether the candidate steered or was a passenger. A candidate who writes a clear prompt, gets back something wrong, notices, and redirects the assistant just demonstrated the exact skill you're hiring for. A candidate whose final code is equally clean might have gotten there by accepting five suggestions in a row without reading past the first line of each one. Grade only the diff and both candidates score the same.

This is the part I built into Evaluator directly, because a diff alone wasn't enough to catch it: the AI-collaboration section puts the candidate and an assistant on a task inside the assessment, and it grades the transcript on whether they catch a wrong suggestion, override it, and brief the assistant well. Every question comes back with the reasoning behind the score, not just a number, so you can see what the grader actually noticed.

The cost of grading a transcript instead of a diff is time, plus a rubric that says what "caught it" and "briefed it well" actually look like. Skip the rubric and you'll grade up the candidates who narrate confidently, whether or not the narration matches what they did.

What a false positive and a false negative look like

The false positive is the fluent narrator: someone who talks through code with total confidence, in a tone that sounds like understanding, without you checking whether the explanation matches the mechanism. The bug-finding exercise catches this better than a conversation does, because confidence doesn't fix an off-by-one on its own. Watch for a second version of it too: someone who happens to recognize the specific bug pattern from past experience can look identical to someone reasoning it out fresh, and neither result tells you much about how they'll handle a pattern they haven't seen before.

The false negative is the quiet, careful engineer who thinks before answering and gets marked down for "not explaining enough," when they're just economical with words under pressure. It's also the strong engineer who's unfamiliar with the specific assistant you handed them and loses time fighting the interface instead of the problem. Optimizing hard against the false positive, by assuming everyone is pasting until proven otherwise, is how you lose the second group. The more suspicious your process, the more good, quiet people it filters out alongside the fluent frauds.

Talk about it before the interview starts

Tell candidates outright, before they start, that using an assistant is expected and that you're grading how they use it, not whether they used it at all. Say it in the job posting or on the first screen of the assessment: use whatever you'd use on the job, because judgment is what's being graded, not memory.

The alternative, staying quiet and hoping nobody uses one, doesn't stop anyone from using one. It just adds an incentive to hide it, and a candidate spending effort hiding normal behavior is spending effort you're not measuring anything useful with. Say it plainly and the transcript you get back is the one that actually happened, which is the only version worth grading.

None of these signals need a long interview or an unusual tool. They need you to watch the process instead of grading the output, and to be upfront enough with candidates that the process you watch is the one that actually happened. If you want an assessment built around exactly this, you can generate one from a pasted job description at tryevaluator.com/try, no signup, with the AI-collaboration transcript scored alongside the rest.

Try Evaluator for your next hire

Generate a tailored technical assessment in about 100 seconds. Free plan, no credit card.

Get started free