Have you ever hired for a job you didn’t understand? It’s scary. If you don’t know how to do the job yourself, how can you assess what a strong candidate looks like?

In this post, we’ll see how to modify a well-researched interview methodology to quickly build a process that helps you hire for the unknown in 4 steps. And, in the spirit of incrementalism, you’ll have something useful after each step.

(This post was also published as a series of articles on LinkedIn. See part one and part two.)

Different People, Same Problem

A couple of weeks ago, I spoke to two very different people who had the same problem. The first was a senior engineering leader at a ~300 person company who needed to hire a data analytics lead, a role that was very unfamiliar to them. The second was a founder who was hiring their first software engineer.

They were pretty stressed about it. Perhaps you can relate.

In my experience, people tend to approach this problem in one of two ways:

1. “We don’t know how to hire for this position, and we could spend a lot of time creating an interview process that doesn’t even work. We should hire based on ‘culture fit’, which we can figure out by getting to know them. Sometimes we have to take risks.”
2. “We don’t know how to hire for this position, It would be very expensive to hire the wrong person. We should create an exhaustive interview that assesses everything needed in this role, since this is a foundational hire. We don’t want to take too many risks.”

These are both natural reactions. Both have clear downsides, and my experience is that both can easily lead to bad hiring decisions. So, perhaps the lesson is that if these are your only options, it’s probably better to take the former!

However, I think we can do better.

With a few hours of work, we can build a process that:

• Reduces hiring risk
• Reduces bias and unfairness
• Creates a good experience for both candidate and the interviewers

Furthermore, you can have something usable—not great, but usable—in under an hour.

How will we do this?

Starting from scratch would take too long. Luckily, there’s a lot of research-based practice that we can hack to make a good first hire without prohibitive effort.

In this article, we’ll look to and oldie-but-goodie as a guide.

Hacking Structured Interviewing

Structured interviewing is a hiring methodology that has been heavily researched and practices for decades. Google has summarized the research and their own experiences with it on re:work, which is a super useful resource that I’ve often referred to.

In their introduction to the topic, the writers state:

Structured interviewing simply means using the same interviewing methods to assess candidates applying for the same job. Research shows that structured interviews can be predictive of candidate performance, even for jobs that are themselves unstructured.

Sounds great! If we’re hiring this role for the first time, the job is likely to be pretty unstructured.

However, in the very next paragraph, we read:

So why don’t more organizations use structured interview questions? Well, they are hard to develop. You have to write them, test them, and make sure interviewers stick to them.

Oof. And here I am assuring you this won’t take long. Maybe it’ll just be easier to go with what you were originally planning. After all, how hard can it be?

Well:

Research has also shown that structured interviews aren’t more frequently used because, in general, interviewers everywhere think they’re good at interviewing and don’t need the help. Surely many of us like to think we’re excellent judges of character.

But when it comes to hiring, don’t trust your gut. Research shows that during first encounters we make snap, unconscious judgments heavily influenced by our existing unconscious biases and beliefs. For example, in an interview context, without realizing it, we shift from assessing the complexities of a candidate’s competencies to hunting for evidence that confirms our initial impression. Psychologists call this confirmation bias.

This cautionary clause helps us define our design problem.

We want to quickly create a hiring process that reduces our confirmation bias, because that will lead to better decisions.

The resources in re:work help with the confirmation bias part, but they don’t show us how to do it quickly. Let’s see how we can hack what they’ve shown us to get some speed out of it.

The Raw Materials

There are a handful of key elements to a structured interview:

1. A small set of competencies that we’re looking for from candidates.
2. Standard questions that test those competencies.
3. Comprehensive feedback gathered by interviewers asking the questions.
4. A rubric that helps interviewers consistently deliver their feedback.

That seems like a lot to consider. However, this short list helps focus our hacking on techniques that are known to work.

We’re going to transform this list into a 4-step recipe for your own structured interview process. In the spirit of incrementalism, you’ll have something useful after each step.

If you follow the whole thing, it should take about 3-4 hours.

Step 1: Define Competencies

Take 15-30 mins. and come up with a list of 3-5 competencies that are important in this role.

It can be tempting to pick more than that, but please start small. If Google can reduce their hiring criteria to 4 competencies, I think we can get there too.

Each one should have a short 1-3 word “slug” that captures its spirit, and a sentence or two to describe what it means in more detail.

If you’re not sure how to start, you can hack this boring-but-effective list:

1. Technically Skilled. Has the hard skills and knowledge required to do the job well.
2. Effective Communicator. Talks about their work in a way that we understand and trust.
3. Has Soft Skills We Value. There are certain strengths or skills that each company uniquely values, so articulate this here.

Each of these higher-level competencies can be broken down into more specific ones, but remember to keep the total to 5 or fewer.

To help with the hacking, let’s explore each of these suggested competencies a bit further.

Technical Skills

“Wait,” you may be thinking. “The whole reason I’m reading this article is because I don’t know how to do this person’s job. Now you’re telling me to figure out how to assess their technical ability? What gives?!”

I know. It sucks. This will likely be the hardest competency for you to define. If you are hiring for a role that is the first of its kind in your company, you may not be able to describe it any more precisely than I already have, and that’s OK. We’ll talk about some ways to manage this in the next step.

Most people in this position will get help from a teammate or other connection who knows the technicals of the job. This is especially true if you’re hiring a new leader to level-up a more junior team that is already doing the work.

Your role is to keep their ideas focused. This is important, because subject-matter experts can have a hard time keeping this list under control. I’ve seen people struggle to pick fewer than 10 separate technical competencies that are important in their jobs.

You can help by finding ways to:

• Coalesce competencies into larger areas of concern, and
• Eliminate things.

If this is nerve-wracking, remember that time is always a constraint. More things on the list means longer interviews, and longer interviews mean less time for other important work. We’re not trying to cover every single thing that is important to the job; we’re trying to assess the most critical things with the time that we have.

Effective Communicator

You may be unsurprised to find this in the list, but I want to highlight why I think this is especially important for a pioneering role.

If you’re creating a new kind of job in your team—even if it already exists somewhere else at your company— it’s really important that you can trust this person to furnish you with information in a way that helps you make effective decisions.

When we’re hiring for something new, we can get so focused on the person’s ability to do the job that we lose sight of how important it is for everyone to meaningfully understand how the work is going. This is especially important if this person is responsible for building an entirely new competency within the company, because it’s hard for new things to build momentum without understanding and trust.

To summarize: It’s important that this person can do the job. It’s also super important that they can explain it to you and others in a way that’s easy to understand. This understanding creates trust, and trust fuels meaningful results.

Has Soft Skills We Value

I once spoke to a founder who was looking for people who were “naturally inquisitive.” That sounded off to me, because it requires more than testing for curiosity; it requires us to determine whether that curiosity is intrinsic.

I asked for a concrete example of what “natural inquisitiveness” looked like. They said: “Well, at the lunch table, we’ll often have big debates about political or social issues. These are really fun, because people won’t just state their thoughts: They’ll try to find the logical arguments or fallacies behind the different positions. It’s not just an emotional conversation. We’re always looking to verify the underlying principles.”

From this starting point, we were eventually able to articulate the “soft skill” that they were looking for: A good candidate would effectively apply the scientific method to everyday problems.

This re-framing improves on the original in several ways:

• It’s easier to assess than “naturally inquisitive”.
• It has less to do with identity and more to do with ability.
• It connects to a unique part of the company’s history, which was founded by scientists out of a university research project.

Because first-time hires tend to be fraught with risk, I find that interviewers naturally want to latch onto something that helps them feel confident about their assessment. This can quickly lead to inspecting who the person is rather than what they can do.

These sorts of identity-focused assessments often fall under the guise of “culture fit.” The debate around the efficacy and perils of “culture fit” and “culture add” is something I won’t treat fully here, but for our purposes, I’d ask you to do this much: If there are things that that you want to know about the candidate’s identity, try to think about how this quality translates to a concrete skill, and then assess that thing directly.

What can I do with this?

Here are some things you can do with your competency list right away.

Share it with your team. Even if your interviewers don’t know what they’re going to ask the candidate yet, this will help them understand what they need to probe for. Having everyone testing for the same things is a useful starting point, even if the way they’re doing it is inconsistent.

Share it with the candidate. Let the candidate know that these competencies are important. This means they’ll focus on the things that matter, and that will save time for everyone.

Focus on it when making a hiring decision. Regardless of who is involved in the hiring process post-interview, make sure to center the conversation around these competencies.

Keeping the competency list front-and-center will make things clearer and more consistent for both your team and or the candidate, even if things get a bit bumpy.

Step 2: Create A Set Of Questions

The core of structured interviewing is using the same questions to assess all candidates for the role, and then consistently evaluating their responses.

Roughly speaking, there are two kinds of questions you can ask in a structured interview. Behavioural questions ask the candidate to describe a past experience. Hypothetical questions present a hypothetical problem that the candidate is likely to encounter on the job and asks them to solve it.

If you’re hiring for a highly technical role, it’s likely that your first instinct will be to use hypothetical questions. Entire books have been published on “cracking” the challenging hypothetical questions in a software engineering interview.

These types of questions are popular because they are really useful. They focus on parts of the job that the candidate may not have experienced before, and the way they handle it can give you a lot of signal on how well they’ll adapt to the challenges of the role.

Unfortunately, it is a lot of work to create hypothetical questions that effectively test the right competencies. If you’re only hiring for this role once, or if you need to build an interview process in a hurry, you may not have the time to do this well.

In this situation, I recommend that you rely mostly on behavioural questions. Using this approach, you can put together a first draft of an interview “playbook” in about 30 mins.

The Recipe

You’ll run the interview in two sessions: One focused on applied skills, and the other on soft skills. If you’re following the rough competency grouping I described in the previous section, you can think of this as:

Session	Competencies Tested
Technical Presentation	Technically Skilled and Effective Communicator
Q&A	Soft Skills

In session 1, you can ask the candidate to describe a past technical project relevant to the role they’re applying for. Make it clear that you want them to (a) explain how they applied the hard skills of interest to solve the problem, and (b) to explain it in a way that you and the other interviewers will understand.

One way to help this go more smoothly is to let the candidate know before the interview that this question be the focus of the session. You may want to ask them to prepare a presentation: Personally, I’m happy to leave that up to the candidate. If it helps them feel more confident, that’s great, but if they prefer to talk their way through it, I think that’s just as good.

You can structure session 2 as a back-and-forth question and answer period. I usually tell the candidate that we’ll trade questions; they can ask anything they want to know about the role, and I’ll be asking questions focused on the specific soft skills articulated in our competency list. This gives them a chance to learn more about us, while still getting the signal we need to make a hiring decision.

The questions you ask in this section will obviously depend a lot on the soft skills you’re looking for. In some cases, it may actually be easier to ask hypothetical questions here. For example, the academic-minded company I mentioned earlier might want to ask the candidate to apply the scientific method to a real-world problem. In general, though, these kinds of skills should be easier to design (or borrow) questions for than the hard-technical stuff.

Open-endedness

The best interview questions are open-ended; there are multiple ways to get to a good solution. However, open-ended does not mean vague! The candidate must understand the situation you’re presenting; and more importantly, they must know what a good solution looks like.

Some people think a good candidate should know to ask or probe for functional requirements. If this is important to you, I recommend making that explicit in your competency list, and then making it clear to the candidate that you’re testing them on requirements-gathering. Making the candidate guess what they should prioritize is rarely a good use of your time.

What can I do with this?

• Share the interview structure with your team. This lets you field any questions/concerns before the interviews.
• Share the interview structure with the candidate. You can do this in advance of the interview, along with any salient details about how they should prepare. The more they know about what you expect from them, the better they’ll be able to demonstrate it to you.
• Get more interviewers involved. Without some structure to follow, you will probably be hard-pressed to have more than a small number of people acting as interviewers for this role. This can be a huge time-sink for them, and their other important work isn’t going to complete itself. A minimal playbook like this one helps get more people involved.
• Feel better prepared. I know this isn’t much structure, but it’s better than what we started with. You should feel more confident about interviewing for this role by now, and you can be sure that the candidate will also feel this.

Step 3: Gather Comprehensive Feedback

By now, you’ve defined the important competencies for the role and created a lightweight set of questions to test them.

However, all of this needs to culminate in a hiring decision. Gathering comprehensive feedback on the candidate can help make that decision more about facts and less about feelings.

A quick way to start is by making a feedback form. The form should prompt for observations about each competency. Some interviewers may not test every competency in each interview session, and that’s OK. They may still have useful observations of things they didn’t intend to test.

Here’s a example feedback form we might to see from a company hiring its first web developer. This hire will pinch-hit on server-side development, so that is also being assessed (but it’s not as important.) The company in question is also looking for people who show resolve when met with difficulty.

Competency	Your Feedback
Web development
Servers
Communication
Stick-to-it-ness

Tell your interviewers to:

Review this form before using it for the first time. This gives them a chance to ask questions before they use it.

Take notes during their interview session. Left to their own devices, many people (like me) will wait until after the interview to write anything down. If something happens and it ends up taking a day or two to fill out the form, my memory of the session will be spotty at best.

It’s helpful to tell the candidate that interviewers will take notes throughout the session. Many people do this on their devices, and this reassures the candidate that they’re not killing time on Reddit and praying for the interview to end.

Complete their interview feedback before talking about it. Some interviewers like doing a ‘retro’ after an interview. This is a good way of quickly identifying areas for improvement; however, it will naturally include thoughts and feelings about the candidate, and it’s better to avoid them biasing each other before they have a chance to write out their own impressions.

State what they observed, not just what they felt. I strongly encourage you to press your interviewers to note specific observations, rather than just the conclusions they drew from those observations. It’s OK for them to state their opinions, but without backing evidence it is difficult to use these notes in a hiring decision.

Continuing our example from before, let’s look at the feedback on ‘Communication’ from two different interviewers for the same web development candidate:

Interviewer A

Communication

The candidate talked about an e-commerce project they were responsible for in their last job. There was a browser glitch that was causing payments to fail randomly once in a while. They knew this because of customer reports, but they weren’t able to reproduce it themselves. Their first step was to introduce monitoring to try to catch the bug in action. This was tricky because the code was understandably written in a hurry: The metaphor they used was like “threading a needle.” I’m not a frontend engineer, but I understand that feeling.

Interviewer B
Communication	The candidate was able to describe their work in a way I could understand. They were good-natured about some hard things they went through. I feel like I could trust them to explain things well to us.

Which feedback is more useful to you?

Interviewer A’s is more substantial and takes longer to parse, but it gives some insight into why this interviewer understood the candidate’s work.

What about Interviewer B? It takes us just a few seconds to read their feedback and know that they understood the candidate. That’s nice, but how about the rest of us? How can we identify the skills the candidate used to do this?

This is the reason I’ve used the word ‘observation’ so frequently in this section. Comprehensive feedback is most helpful when it highlights patterns of candidate behaviour that are relevant to the job.

What can I do with this?

Spot Trends In The Feedback. If you followed the recipe so far, you probably had at least 4 interviewers meeting this candidate across two different sessions. That should give you enough feedback to spot strengths and gaps that seemed consistent across the interviewers (rather than one-off concerns.) These trends are important to consider when making the hiring decision.

Try A Hiring Committee. I understand that ‘committee’ can be a dirty word for many of us who associate it negatively with ‘bureaucracy.’ In this case, it speaks to having one group of people doing the interviewing and another group making the hiring decision. Separating the hire-ers from the interviewers has several benefits, including:

• Reducing individual bias
• Providing a more comprehensive review
• Detecting cases where something went wacky with the interview

If you’re doing this for the first time, I recommend getting 3 people together who are accustomed to making high-discretion decisions. It can help to have one person involved who doesn’t have as much skin in the game, because they can be more dispassionate about the decision.

The hiring committee meeting can be as simple as:

• Everyone takes time to review all the feedback
• Discuss
• Vote yes/no, and discuss seniority if applicable

Because there’s a vote involved, odd-numbered groups are preferable (no ties to break.) Odd-numbered groups also have other advantages, as indicated in re:work. (Their reference link for this is broken, but the relevant paper is pay-walled here. This publicly-available survey paper is also useful.)

In practice, there’s often one person accountable for making the final decision. It should be noted that the research suggests it’s almost always better to take the majority vote, but that’s hard to grapple with when you’re the one who lives or dies by the decision. The votes of the committee can still serve as very useful input into that call. Also, if it goes poorly, you’ll very likely have some empathy from your fellow decision-makers, because they’ll better appreciate the difficulty of the decision :)

You may not have enough people involved in hiring to keep the interviewers and the hiring committee completely separate. Don’t stress too much about that: You’ll get some benefit from delineating “interview mode” from “hiring mode” and focusing on the collective feedback instead of individual opinions.

Share the feedback with the candidate. I’ll write another article about why this can be useful, but suffice it to say: Wonderful things can happen when you’re transparent with the candidate about why they got the job or not. This does take time and some measure of tact, and you may be short on both of those. But if you can even just do this for the occasional candidate—perhaps for those who were very close to getting the job—you’d be surprised at the good things that can happen. I’ve had some very strong referrals from candidates we declined, simply because we took the time to let them know why.

Use the feedback to set the new hire up for success. The feedback you’ve gathered isn’t just useful for making the hiring decision: It can also help you understand if there’s anything you can do to help the candidate hit the ground running when they arrive.

Step 4: Use A Rubric To Grade The Feedback

Structured interviewing improves hiring outcomes partly by making the process more consistent. Consistency means that less noise is contributed by differences in how the interview was conducted, and we get more signal to identify the material differences between candidates.

This final step helps us add some consistency to the feedback-gathering process.

With what we’ve developed so far, interviewers have only gathered free-form observations for each competency. Their personal takes on the importance of each observation—along with the short definition of each competency you’ve given them—will govern how much they emphasize each one.

We can take a step further by asking them to grade each competency based on their observations. However, because a grade summarizes a lot of information into a single label (“Good”, “Mediocre”, “Poor”) we need to give the interviewers some precise guidance on what these labels mean.

A rubric defines how an interviewer can interpret their own feedback and translate it into a “grade”. It tersely describes how to distinguish different levels of performance.

Let’s add a rubric to the ‘Communication’ competency from the feedback sheet in the previous section:

Competency	Good	Fine	Poor	Your Grade	Your Feedback
Communication	Explains difficult topics with nuance and depth. Connects to concepts the audience understands. Creates sense of ease with the problem.	Clearly describes most salient issues. Can recover when struggling to articulate if asked the right questions.	Speaks in a way that misleads or confuses the listener. Presents their thinking without ensuring it is understood.

In addition to the free-form feedback, we’ve added a grid that describes how a candidate can grade what it means to communicate effectively. This doesn’t just help with consistency across interviewers; it also helps with consistency in the same interviewer. If I’m assessing this candidate at 2pm and I haven’t had lunch yet, I may not be very enthusiastic about their performance regardless of what they do. However, when I look at my written feedback and then compare it against the definition of what “poor” entails, I’ll have to admit to myself that this is not it. And even if I don’t catch that, the hiring committee still might. This is where the consistency comes from.

This consistency does come with a bit of a cost. Personally, I find creating the rubric to be surprisingly slow going. Creating a shared understanding among the interviewers precisely enough to write it down in a few short paragraphs is very hard. It also usually takes a few rounds of review before it’s useful to everyone.

However, without the rubric, you end up having these kinds of clarifying conversations anyways during the hiring decision. And without the benefit of the ‘calibration’ that the rubric brings, you may find your hiring committee talking past each other a lot.

Getting rubrics on the grid

I can often drum up a first draft of a rubric in 45mins-1hr. Shopping it around to the interviewers and incorporating feedback can be time-consuming, so I try to put some guard-rails around it: For example, we might do one round of revisions based on the feedback of a few key interviewers with different areas of expertise, and only address each of their top concerns. After that, you can encourage everyone to kick the tires on the rubric in an interview or two before considering further revisions.

Another option is to avoid making competency-specific rubrics at all, and use the generic rubric (“Rating Scale”) found in Section 4 in the United States Office of Personnel Management guide to structured interviewing.

A word of warning to you: When you introduce the grading system, your interviewers may focus more on the grade than on their notes. Their written observations are still absolutely their most important contribution, and the hiring committee should be able to see how their grade follows naturally from what they observed. Without the notes, it is extremely difficult to have a productive hiring conversation.

Here’s how I usually summarize this for interviewers: If I had to pick between comprehensive feedback without a grade, or a very well-thought-out grade without seeing the feedback, I’ll take the feedback every time. The grade is only useful in the context of the notes, whereas the notes are tremendously valuable on their own.

What can I do with this?

Use the grades to help compare candidates. If you’re hiring a role for the first time, it’s pretty likely that you’re only hiring for one of them. You’ll—hopefully!—have a number of candidates competing for the job. While written feedback will still be your most useful tool in the decision, the grades assigned to each candidate can help you “zoom out” and compare them at a higher level. And because you’ve put some effort into calibrating these values, you can feel a bit more confident that this comparison has meaning.

Make concrete trade-offs. It may be difficult to find a candidate who scores “good” across the board. The grades can help you more crisply define what you think is absolutely needed for success in this role, and to have more direct conversations about that during the hiring decision.

Wrapping Up

If you’ve followed this guide to the end, you will have spent a handful of hours building a hiring process that should help your team be more confident in their hiring while providing a better experience for the candidate.

But will it really lead to better hires?

When I wrote the title for this article, I chose the term ‘hacking’ quite deliberately. ‘Hacking’ speaks to taking something that is useful in a related context and pragmatically re-purposing it for your own needs.

A full-fledged structured interview includes many things that we didn’t have time for (e.g. interviewer training) and puts more depth and detail into some of the things we kept (e.g. comprehensive playbooks for each interview question.)

While structured interviewing has been shown to have measurably positive outcomes in the research, what I’ve proposed here doesn’t quite match the methodology that would have been practised in those studies. We can’t just assume that we’ll get the same benefits.

On the other hand, we didn’t throw everything out; if anything, we took a lot of what makes structured interviewing tick and stripped it down for our needs. This includes:

• Clearly defining the important competencies for the job
• Being transparent to the candidate about these competencies
• Standardizing the questions you ask each candidate
• Treating interviewing and hiring decisions separately
• Gathering comprehensive feedback to inform hiring decisions

My belief—and my experience—is that using these principles works better than what most of us come up with unaided. I haven’t explored the research on this to my own satisfaction yet, but I suspect that many of these principles have been studied independently, and that they each contribute some measurable benefit on their own.

At the very least, I think the suggestions in this article can help you feel a lot more comfortable about hiring for an unfamiliar role. I’ve walked many folks through the recipe described here, and I’ve often felt their energy shift from discomfort to optimism within the first half-hour of working with them.

You may not have time to do it all, but I believe you’ll be on the right track even if you only make it through step 1.

Happy hiring :)