A few years ago I sat in on a debrief where a strong engineer got rejected because he couldn't produce the textbook definition of a red-black tree on a whiteboard. The interviewer's note read, in full: "Weak on data structures." The candidate had spent the previous forty minutes calmly explaining how he'd untangled a payment reconciliation job that was double-charging a fraction of a percent of customers, and how he'd decided which fix to ship first when he couldn't ship all three at once. We passed on him. He went to a competitor and, from what I can tell, has done very well.
That debrief has bothered me ever since, because it exposed something about how we hire. We test for the recall of facts because facts are easy to grade. Judgment is hard to grade, so we quietly stop testing for it, and then we're surprised when we hire people who know everything and decide nothing.
What trivia actually measures
Let me be precise about what I mean by trivia, because I'm not against rigor. Trivia is any question whose answer you either happen to have memorized or you don't, where memorizing it tells me almost nothing about how you'll behave on my team. The time complexity of quicksort. The difference between a clustered and non-clustered index, recited. Whether you remember that HTTP 409 means conflict. These are real things worth knowing, and I'd expect a good engineer to look most of them up in under a minute. What I can't stand is treating reflex recall as a proxy for competence.
The dirty secret of trivia questions is that they mostly measure how recently someone interviewed somewhere else. The candidate who just spent three weekends grinding a problem set will crush your algorithm round. That candidate is not necessarily better than the one who has spent those weekends shipping. In fintech especially, the most consequential work is rarely a clever algorithm. It's noticing that a retry loop plus an at-least-once queue equals duplicate settlements, and having the instinct to make the operation idempotent before anyone asks.
The idempotency story
Here's a question I've used for years, and it's not clever. I describe a service that charges a customer's card, and I mention offhand that the upstream call sometimes times out even when it succeeds. Then I stop talking and watch.
The trivia-optimized candidate starts describing exponential backoff. Fine, but they've missed the trap. The candidate with judgment goes quiet for a second and says something like: "Wait, if the charge went through but I didn't get the confirmation, retrying charges them twice." That's the moment I'm paying for. It's not knowledge of a pattern. Everyone can define idempotency. It's the reflex to ask what happens to a real person's money when the happy path breaks, before I've prompted them. I've watched senior candidates with immaculate resumes sail right past it, and I've watched a two-year engineer catch it in eight seconds. I know which one I want on the pager.
Judgment is behavior under ambiguity
Judgment isn't a knowledge domain. It's how someone behaves when the problem is underspecified, the trade-offs conflict, and there's no answer key. That's most of the actual job. Requirements are vague, the data is dirty, the deadline is real, and two reasonable senior people want opposite things. What I'm hiring for is the person who can hold all of that and still make a defensible call.
So I've reoriented my interviews around ambiguity on purpose. I give incomplete requirements and watch whether the candidate asks the questions that matter or invents an answer to avoid looking uncertain. I present a scenario with a genuine trade-off and no clean winner. The tell isn't which side they pick. It's whether they can articulate what they're giving up. Someone who says "I'd go with the boring monolith here, and I know that costs us independent deployment later, but we're eleven people and we can't afford the operational tax yet" has shown me more than any correct answer to a puzzle ever could.
The candidate who says "it depends" and then tells me exactly what it depends on has just done the job in front of me. The one who says "it depends" and stops is hiding.
The questions I actually ask
Over time I've settled on a handful of prompts that reliably surface judgment. None of them have a single right answer, and that's the point. I'm listening to the reasoning, not checking a box.
- Tell me about a time you shipped something you knew wasn't quite right. What did you cut, and how did you decide the corner was safe to cut?
- You've got a production incident and two plausible root causes. You can only investigate one first. Walk me through how you choose.
- Describe a technical decision you were wrong about. What was the signal you should have caught earlier?
- Here's a feature request from the head of sales that would take four weeks. How do you find out if it's worth four weeks?
- A junior engineer opens a pull request that works but that you'd never have written that way. What do you do?
That last one is quietly the most revealing question I ask. The engineer who launches into everything wrong with the code is telling me how they'll treat their teammates. The one who asks "does it work, is it tested, is it readable enough for the next person" and then decides whether the difference is worth a conversation is showing me they understand that code review is a relationship, not a correctness gate. I've reversed hiring decisions on the strength of that answer alone.
Enjoying this article?
Get more like it in your inbox — practical engineering leadership, fintech, and AI. No spam, unsubscribe anytime.
The cost of getting it wrong
Hiring for trivia doesn't just miss good people. It actively selects for a specific failure mode: the engineer who is confident, articulate, fast on recall, and completely unmoored from consequences. In a regulated environment that's not a quirk, it's a liability. I've cleaned up after the person who knew every design pattern and applied all of them, turning a service that moved money into a cathedral of abstractions that nobody, including them, could safely change six months later.
The failure is expensive in a way that doesn't show up for a year. A trivia hire passes probation easily because the early work is well-defined and they're sharp. The problem surfaces the first time they hit real ambiguity with real stakes, and they either freeze or, worse, barrel ahead with false confidence. In my world that means a compliance gap, a mispriced risk, or a batch job that quietly corrupts a ledger. The recall skills that got them hired are useless in exactly the moment that matters.
You cannot fake a good war story
The single most efficient signal I've found is asking someone to walk me through a hard problem they actually owned, in detail, and then following the details down as far as they'll go. Real stories have texture. They have the dead end the candidate went down for two days, the metric that finally told them where to look, the argument with a colleague, the fix that was uglier than they wanted. Rehearsed stories are smooth and get vaguer the harder you push.
I push. "Why did you reach for a queue there instead of just a database table with a status column?" "How did you know the latency was in the network and not the database?" "Who disagreed with you, and were they right?" You learn an enormous amount from how someone handles being pressed on their own claimed work. The good ones get more specific and occasionally admit a mistake. The weak ones get defensive or retreat into generalities. I don't need them to have been right. I need them to have been paying attention while they were wrong.
Calibrating the panel
None of this works if the interviewers aren't aligned, and this is where most companies quietly fail. You can write beautiful judgment-based questions and still get garbage signal because five interviewers are grading five different things and nobody wrote down what "good" looks like before the loop. I've been in debriefs that were really just a fight between the person who values shipping speed and the person who values architectural purity, with the candidate as a proxy.
So before a loop I make the panel agree on what we're actually testing, and I make each interviewer commit to a written prediction before the debrief starts, so nobody launders a gut feeling into a rationalization after hearing the room. I also kill the "weak on data structures" style of note. If your feedback can't point to a specific decision the candidate made and explain why it worried you, it doesn't count. Vague negative notes are usually a mask for "they weren't like me," and that's precisely the bias a judgment-based process is supposed to remove.
Where trivia still earns its keep
I want to be fair, because I've overcorrected before and it bit me. There is a floor. If someone is going to write concurrent code that touches money, I do need to know they understand what a race condition is, not as a definition but as a thing they've been burned by. If they're going to own a Postgres database, they should know what an index does to a write, roughly. The knowledge floor is real and pretending otherwise is how you hire a wonderful communicator who ships subtly broken software.
The distinction I hold is this: knowledge is a gate, judgment is the decision. I use a small amount of technical questioning to confirm the person can operate at the level the role needs, and I spend most of the time on how they think. When I've inverted that ratio, I've hired people who interview like champions and then need a senior engineer standing behind them every time a decision gets hard. The floor should be a quick check, not the whole exam.

Conclusion
Here's the thing I'd tell my younger self, the one nodding along in that debrief. The questions that are easy to grade are almost never the questions that predict the job, and the discomfort of grading judgment is not a reason to avoid it. It's the tax you pay for hiring people who'll make good calls when you're not in the room. If your interview process could be passed by someone who memorized a study guide over a long weekend, you're not measuring the thing you actually care about. And somewhere out there is the engineer you rejected for not remembering a tree-balancing rule, now catching the double-charge bug at a company that was willing to watch how he thinks.
Get new posts in your inbox
Occasional, practical notes on engineering leadership, fintech, and building with AI. No spam, unsubscribe anytime.
Comments (0)
Leave a Comment
No comments yet. Be the first to comment!
