How to interview engineers who use AI: a rubric you can run this week

Most companies that ban AI tools in interviews are testing for a skill nobody uses on the job anymore: writing code with no help. That's nostalgia dressed up as rigor.

The real question is whether a candidate uses an assistant well. Some engineers brief a tool clearly, read every line it gives back, and catch the one suggestion that's subtly wrong. Others paste the ticket, accept whatever comes back, and ship it without reading it. Those are different engineers, and an interview that hides the tool can't tell them apart.

Why banning AI tests the wrong thing

Ban the assistant and you're left measuring how fast someone can recall syntax under pressure, which was already a weak proxy for the job before AI existed. It rewards whoever drilled algorithm puzzles most recently, not whoever ships reliable code with the tools they'll actually have open every day. If your own engineers use an assistant at their desk, testing candidates without one measures a job that doesn't exist anymore.

Plenty of teams ran interviews this way for longer than they should have: closed tabs, no assistant, just think out loud. It felt rigorous. It said almost nothing about how the same person would work six months later, back at their desk with an assistant open the whole time.

What good AI collaboration looks like

Good use of an assistant has a shape. It starts with a clear brief: stating the goal, the constraints, and the edge cases before typing the first prompt, instead of pasting an error message and hoping. It continues with reading, not skimming, every line the assistant hands back, because the fastest way to ship a bug is to trust code you haven't read.

The clearest signal is what happens when the assistant is wrong. A confident, plausible-looking suggestion that breaks on a null value or quietly ignores a stated constraint gets accepted by a weak engineer and caught by a strong one. Catching it, explaining why it's wrong, and fixing it, or telling the assistant exactly what to change, is the whole skill in miniature.

The last piece is knowing when to stop. Some problems, especially ones that need stepping through actual program state or a judgment call specific to the domain, don't get faster with an assistant in the loop. An engineer who keeps prompting past that point is spending the clock on a tool that stopped helping five minutes ago.

A 45 minute interview format you can run this week

Give the candidate a small, real task, not a puzzle: a bug in a few hundred lines of code, or a small feature in a codebase they've never seen, something with a correct answer you can check in ten minutes. Let them use whatever assistant they use daily, or provide one if they don't have a preference. Screen share, and say up front that candidates are expected to use the tool during the task.

A breakdown that fits in 45 minutes:

• 5 minutes: hand over the task and the constraints, then go quiet.
• 30 minutes: the candidate works and narrates their thinking out loud, while the interviewer watches, takes notes, and asks at most two or three questions, only at natural pauses.
• 10 minutes: debrief. Ask what they'd do differently, where they trusted the assistant, and where they didn't.

The interviewer's job during the 30 minutes is to watch and stay quiet. Every question you ask mid-task risks bailing the candidate out of a decision you should be scoring.

The five row rubric

Score these five things after the session, checking the transcript against your memory rather than trusting your first impression.

1. Briefing the assistant. A 1 types "fix this" and pastes an error. A 3 explains the bug but skips constraints like performance or style. A 5 states the goal, the constraints, the edge cases, and what shouldn't change.
2. Reading the output. A 1 runs whatever the assistant returns without reading it first. A 3 skims for obvious mistakes. A 5 reads every line, checks it against the actual requirement, and can explain what the code does without the assistant's help.
3. Catching a wrong suggestion. A 1 accepts a suggestion that breaks an edge case or introduces a bug. A 3 catches the obvious ones but misses subtler logic errors. A 5 catches the suggestion that looks right but isn't, and can say exactly why.
4. Knowing when to stop. A 1 keeps prompting the assistant on a problem it clearly can't solve, burning the clock. A 3 switches to manual work eventually, but late. A 5 notices within a minute or two that the task needs to be worked by hand and does it without being asked.
5. Narrating the work. A 1 works in silence and leaves the interviewer guessing. A 3 narrates what they're doing but not why. A 5 explains what they're doing, why, and what they'd try next if the first approach failed.

What to do with the transcript afterward

Read it. Don't rely on what you remember from watching, because thirty minutes of someone else typing is long enough that you'll misremember which fix came before which prompt. Go back through the transcript and mark the exact moment that drove each of your five scores: the actual prompt, the actual line of code, the actual point where the candidate caught something or missed it.

Keep the transcript with the rest of the interview record. It's the only artifact from this round that shows your reasoning instead of your impression, and it's what makes calibration between interviewers possible instead of theoretical.

Where this goes wrong

The candidate who accepts everything looks great live. The session moves fast, the demo works, and the interviewer walks away impressed. The rubric is what catches this, specifically the rows on reading output and catching a wrong suggestion, because a candidate who accepts everything scores low on both no matter how smooth the session looked.

The candidate who refuses to use the tool is a different problem. Maybe it's principle, maybe it's fear of looking like they can't code without help, maybe they just don't like being told how to work. Whatever the reason, you haven't tested what you needed to test. Say explicitly, before the task starts, that using the assistant is expected and will be scored. If a candidate still won't touch it, note that and weigh separately whether it matters for the role.

The interviewer who grades speed is the most common failure, and the rubric leaves speed off on purpose. A candidate who lets the assistant drive the whole session will often finish fastest and be the weakest hire. If you notice you're impressed by how quickly someone finished, go back to the rubric before you write down a score.

Calibrating two interviewers

Have two interviewers score the same transcript independently before they compare notes, the way you'd calibrate blind grading on an exam. Where their scores differ by more than a point on any row, don't argue about general impressions. Point to the specific prompt or line of code that drove each score and argue about that instead.

Do this on four or five shared transcripts before trusting either interviewer's solo scoring, and repeat it every few months. Standards drift quietly in one direction or another without anyone deciding to move them.

Evaluator runs a version of this rubric automatically: the candidate works with an assistant inside a generated assessment, and the transcript gets scored on the same dimensions, briefing the tool, catching a wrong suggestion, knowing when to override it. You can see the format at tryevaluator.com/try, no signup.