Technical Interviews Today and Tomorrow

I would define interviewing as follows:

The process of identifying the individuals who can achieve greatness.

Great companies need to select employees capable of achieving greatness on their own. Then, and only then, can a company be used as a catalyst to achieve that greatness to benefit both the company and the individual.

Remember, employees are selling their most precious assets: their time and intelligence.

I have spoken on interviewing before in my article, What I Look For In An Interview. Now, it is time to review the other side of the fence: what interviewers should look for in candidates.

Granted, I have never interviewed anyone for a technical role myself (Update 03/2017: I’ve now done hundreds of interviews the past few years, and stand by what is laid out in this post). I have been interviewed more than a hundred times and read more than my fair share of how to successfully complete coding / technical interviews. Further, although this may seem a bit strange, I have completed several hundred interviews for guilds in WoW (World of Warcraft).

Although that is moderately embarrassing (playing WoW) and most people won’t believe it when I say the techniques translate to real life, they actually do. WoW taught me a lot about interviewing and allowed me to quickly try out different interview methods with zero risk, honing a method without costing me a dime. It is pretty amazing to consider there are very few other methods of trialing interviewing techniques, without costing significant amounts of money (someone should build a simulator).

Back Story

Over the past six weeks alone, I have completed over a hundred interviews. That may seem like a lot to most people, but many tech companies require new hires to go through somewhere between 5 – 8+ interviews. For example, in one day I flew out to San Francisco, had five interviews for one company, then returned to my hotel and had three more phone interviews for other companies.

During each of these interviews, I spent more time considering how those questions could quantify me as a software engineer and employee rather than attempting to solve the problems. That may sound odd, but to be honest I have little to no intent of working for any company I have interviewed at. I already have an amazing offer, am currently applying to graduate school and applied Y Combinator (though I did not make it in this round). I am, however, interested in “testing the waters,” potentially receiving an amazing offer and/or learning how to give proper interviews for the company I would like to start.

With this in mind, I have compiled a series of notes relating to questions and practices, some I feel quantified me properly and some did not. After rereading all of my notes, reading articles, books, and reviewing my own experiences, it seemed pretty clear to me that many of the technical interviews I (and others) face today are simply inadequate to assess our skill set and is misleading from the interviewer prospective.

Technical Interviews Today

The question, “what makes a successful hire?” may seem easy to answer, however it is much more complex than most people realize.

For example, in business a successful hire should be able to:

• Work exceptionally well in teams
• Present information clearly
• Speak with confidence
• Convince others
• Complete work assigned to them (make decks, fill data sheets, etc.)

Now, let’s imagine the business hire who can:

• Work well in teams
• Present information clearly
• Speak with confidence
• Excellent at convincing others
• Complete most tasks assigned to them
• Can program

Notice the very subtle difference between those two potential business hires. One works exceptionally well in teams, the other just works well in teams; one can program, the other cannot. Most people can bring something to the table, the question is, what?

The key characteristics/skills I would look for generally are the following:

1. What unique skills does the candidate bring
2. Do they have the required skill set
3. Are they competent
4. Will they fit in with the culture
5. Can they create a “moon shot”

What I believe most recruiters/interviewers fail at is identifying the characteristics/skills above, instead focusing on:

1. Smart
2. Get things done

The above is directly from The Guerrilla Guide to Interviewing (version 3.0) by Joel Spolsky, co-founder of Stack Exchange. The thing is, it is a rather open-ended and is very subjective for each interviewer. For example, “getting things done” at various interviewers at the same company I have interviewed at have been anywhere from “hacking a solution quickly” to “thoughtfully engineering a solution.”

The reason I used The Guerrilla Guide to Interviewing as the basis of my critic is because it seems that most companies follow the same pattern mentioned in the guide:

1. Introduction
2. Question about recent project candidate worked on
3. Easy Programming Question
4. Pointer/Recursion Question
5. Are you satisfied?
6. Do you have any questions?

The guide goes on to explain how you can only be sure of the hire after at least six employees complete the above interview pattern. If one or at most two do not think the interviewee is an “instant yes” then they should not be hired.

Although I do agree this is a (somewhat) effective way to hire employees and most of the candidates who become full-time employees will indeed be well vetted, it may not be the most effective way. If the goal is to only hire the candidates who are the “instant yes” type, from 5 / 6 interviews, it leads to an excessive and very costly process (for candidate and company).

Technical Interviews Today – Cost

Let us break down costs for a moment, let’s say $15 for a resume review, $250 per hour phone interview + back checking, $3,500 – $6000 for all day on-sight interviews. Now imagine only 5% – 10% of candidates are hired (given the stringent interview requirements), how much wasted capital is that? Any optimization is a vast improvement.

Imagine a hypothetical position that a company is attempting to fill:

100 applicants apply for a single opening

Total ($) = Current Stage ($)

$1,500 = 100 * $15 (resume screening)

$14,000 = 50 * $250 (half make it through resume screening)

$20,250 = 25 * $250 (half need a second phone interview)

$120,250 = 50 * (1/3) * $6000 (1/3 of phone interviews convert to on sight)

This leaves 16.667 candidates for on-sight interviews, assuming 1/5 are “instant yes” applicants. That leaves roughly three applicants in the pool. Given the cost of narrowing down the candidates, it is likely the company will give offers to all of them, because they probably have other offers as well (and we cross our fingers that one of them accepts).

Meaning, per new hire (given the assumptions above), it costs roughly $120,000 just to fill the position. This does not include training, salary, benefits, sign-on bonus, relocation, etc. Further, it may not even be the best possible employee, only the most vetted (i.e. minimizing false positives, not false negatives)!

This is why clearly quantifying a successful employee is key.

Technical Interviews Today – Defining Success

Imagine if we did all this work and the hire ended up not being the best, always a possibility… which is why this process is so important, and any variance between interviews should be minimized.

To accomplish this, companies often set out a list of goals, for example:

• Every product the company sells is rated as top 3 in the field
• Grow profit at least 10% every six months (for the next 5 years)
• Enter a new market every 3 years (diversify)
• Employ 5% more employees every six months (for the next 5 years)

Note, the above are clearly defined, and seems fairly profitable. By employing only 5% more employees every six months and growing profit 10% every six months, if the company was already profitable, the increase in profits would be dramatic after five years (X * 1.1 * 1.1 * … vs Y * 1.05 * 1.05 * …).

Since the goals of a company are clearly defined for the next five years, it is now possible to generate a mold for a successful new hire:

• Top 10% skill set for specified field
• Capable of learning new skills
• Capable of promotion (i.e. management material)
• Skill set in expansion markets
• Excellent communication skills
• Long-term commitment
• Competent
• Fit in with the culture
• Can create a “moon shot” idea

The above matches the ideal employee for the goals of the company and are always slightly different. However, almost every company enjoys competent, committed, highly skilled employees; you then have to weigh that with the price of this resource.

The best deal on skill vs. price is really what employers are searching for: a successful hire at the lowest cost. That being said, I would like to point out that there really is no maximum price on a good employee. The capacity and imagination to change the world is literally the most important candidate trait, but it is somewhat of a gamble. There is no guarantee an employee will be able to change the world, but if they can, capturing them and paying them their worth is quite possibly the best decision an employer can make.

Critiquing Technical Assessments

Assessing a candidate is really the only point of an interview. The previous section discussed how today’s technical interviews are conducted and how successful hires are quantified and chosen. This section aims to dive a bit deeper into the fabric of what is currently wrong with the way candidates are being interviewed.

The most important aspect of this is perceiving a candidate as a potential resource, rather than as a potential tool.

How does a resource differ from a tool?

Let’s look at the definitions (from Google):

Tool: a device or implement, used to carry out a particular function.

Resource: a stock or supply of money, materials, staff, and other assets that can be drawn on by a person or organization in order to function effectively.

By focusing excessively on what a candidate can do right now, as opposed to their potential, we are viewing them as tools. By instead focusing on what candidates can achieve, we are viewing them as a resource.

This is the largest single point of failure I have seen in all interviews, thus far.

Then, what should be taken into consideration when reviewing a candidate?

Alternative Assessments

After my limited (~6 years) of being an employee, avid book reading (which provides a life-time of condensed experience) and speaking to a few of my managers, here is what I feel the basic tests should consider (in order):

1. How passionate/ambitious are they
2. Do they have a knowledge index of related data
3. Can they solve basic logic problems
4. Will they fit in with the culture
5. Can they learn new skills

The largest difference in my method is it relies very little on how much specialized data they have and rather relies on how passionate they are to learn, their ability to learn, and what they know/believe is possible.

I also mention something called a “knowledge index,” you may be asking yourself, what is that? Well, since the early 2000’s pretty much all knowledge has been accessible through the internet. Better yet, all the information on the internet has been indexed for us (thanks Google!).

Why then, (for instance) should we spend a single second remembering syntax in programming? Obviously, knowing syntax is very helpful, but it is no longer a prerequisite for completing jobs efficiently. Rather, it makes sense for us (as humans not machines), to simply “index” similar to Google. We should know what is possible, and what to look for, rather than know how something works in the moment.

For Example

Say I am in an interview and I know k-nearest neighbor can be used to find the two closest points, given a list of arbitrary points in a graph. Two decades ago, knowing how k-nearest neighbor is implemented would be required to complete jobs efficiently. It would have taken me time to check a book out of a library, or going through a text from school or research paper to find the algorithm.

However, I can now simply know there is a function in python that completes this for me. On the other hand, even if I need to implement it, it is literally as far away as me typing “k-nearest neighbor + <enter>”, 1 click and some scrolling. The time requirement is ~45 seconds, and based off my previous knowledge I can quickly implement an algorithm.

The purpose of assessing the candidate is really assessing their ability to read documents, understand them, and execute a solution. This is why most “whiteboard” interviews are completely ridiculous and ineffective.

Whiteboards are excellent for theory, but lousy for programming, so why is it in practice? Well, I will concede that it is effective for capturing a candidate’s ability to solve basic logic problems and potentially learn new tasks. Whiteboard interviews can only gain limited insight about their index and passion(s).

The Falicy of Assessing Programming Ability

The problem with whiteboards is that most interviewers attempt to use it as a way to assess a candidate’s ability to program and that is simply not possible.

First, just in writing this sentence I used the emacs ctrl+a, ctrl+k, ctrl+y, you may ask how that has anything to do with programming or whiteboards. My brain is programmed to know these commands, associate them with keyboards. Similarly, my skills as a programmer are tied pretty closely with a keyboard, computer, and the internet. However, solving theory or logical problems related to programming are not, which is why theory questions work well on whiteboards.

Second, programming is not syntactically difficult (at least not while using python, ruby, Go, etc.). Problem solving, on the other hand, is. Testing an individual’s ability to program is important, but for the love of god, this can be done off-site either in an online test (such as HackerRank) or in real-time via any shared programming or document space. Reviewing the above costs related to flying out candidates for interviews, the cost of employee time, etc., it should become clear why this is idiotic.

Finally and most importantly, programming is the least important part of the interview process. No candidate should make it through the first resume review without having clear programming experience. Open source projects on github are ideal, but really, any project will do. Then, if they make it through that (as stated before) have them program in real-time while an interviewer watches via the internet. Then, all in-person interviews should assess are the candidate’s ability to problem solve, how he/she will fit into the culture and learn new tasks.

Assessing Tomorrows Candidates

Thus far, I have spoken a lot about the issues of today and only hinted to solutions. Now, it is time to discuss the proper way assessments should be conducted, according to lowly me.

First – Clearly Define Mold

Write out exactly what the goals of the business are, then construct a list of required skills to achieve those goals. After which, add the standard traits related to passion/ambition. A good rule of thumb is if they want to start their own company and they are looking for experience, they are ambitious and passionate. Further, make sure you review your culture and include that in the mold.

The “public” job description should only have a list of required skills. The “private” job description should contain personality qualities, passion/ambition for example, which should be used to assess candidates as well. Just make sure you don’t state so publicly, to avoid conflict.

Second – Resume Review

Review resumes, each candidate receives a timed 10 minutes. During this time, websites/portfolios should be reviewed, courses, school, a quick search, etc. Score them based on what you are looking for. For example, 10/10 if you are searching for a candidate who matches every public and private requirement posted, 8/10 for all of the public, but seemingly only some of the private requirements. Start with the 10/10’s and move down the list.

Save time, the best option is to find candidates from referrals, so ask your co-workers!

Third – Phone Call + Small Task

Call the candidates selected for further interviews, remember only 5 – 20% should make it through the interview process (so call enough people). The phone call should take 10 – 15 minutes and consist of a congratulations and upbeat conversation. During which, the interview process should be explained.

After the phone call the candidate should complete a small programming task/test, it should be timed and done without any person watching. They can use the internet, books, basically their knowledge index. The point being to weed out candidates who just don’t care or do not have a good enough knowledge index. Review the work, make sure they solved/answered appropriately.

Fourth – Phone + Internet Interview: Technical

Make sure the interviewer can speak the given language very well, with little to no accent.

Interviews are hard enough, having someone with a thick accent conduct interviews is damaging, and should be avoided. I cannot emphasize this enough, it happens too often and is a pretty big pet-peeve of mine.

During this interview a programming problem should be presented, something from a CS 101 course. Something as simple as implementing a linked list, string flip, or implementing a function to find a sub string. This problem should be solved in about 10 – 15 minutes, and should be used as the simple programming literacy question. Assuming the resume vetting, as well as the small programming task has been completed, the internet interview should be the final programming check (i.e. make sure they aren’t lying).

If that is completed satisfactory, give the candidate a program to debug. Debugging is what 80% of the developers time is spent on, it should be the primary focus of every programming interview. In this case, there should be a random number of errors, two to four work best. The standard off-by-one, undefined variable, and then one to two more complicated ones.

Note, debugging can be very difficult, and I recommend making it relatively easy to debug, but make sure they can do the tasks at hand. For example, make sure the candidates know to check for memory leaks in C (if they need that for the job). Do not expect every candidate to be perfect at this, as long as they are “satisfactory” make a follow up phone interview, let them know immediately.

By informing the candidates immediately if they passed or not, you are making the process very straight forward and less painful. It also implies that every interviewer has a say in the process.

Fifth – Phone Interview: Culture Fit

This interview should be recorded, or be conducted via a conference call. Only one interviewer should ever speak, but it saves time to have multiple people listen.

At this stage, the candidate’s ability to program should have been assessed well enough. Now it is time to determine if they can/are:

• Seemingly learn skills quickly and well
• Ambitious/passionate
• A cultural fit

This is the classic personality fit section, this is when you discuss projects the candidate has completed, how ambitious they seem, i.e. where do you see yourself in five years? A good question to ask would be, “what don’t you like about our business model?” This ensures that they have done their research, and know what they are getting into, and in turn know the culture (somewhat, at least).

This is the most important interview, if they are smart, ambitious and complete interesting/hard projects, they can probably learn anything they need to learn.

In this case, the candidate leaves the call, then the interviewer(s) discuss the interview with their colleagues and determine if he/she is a good fit. All of the interviewer’s co-workers should agree, if they do not, do not continue with the process.

Within an hour, the candidate should receive an email regarding either an on-sight interview or the decision to not continue with the interview process.

Sixth – On Sight Interview

The on-sight interview should consist of two to four whiteboard style theory/logic based interviews. Essentially, discussing real world case problems. My “everyday algorithms” series are perfect examples of these problems (e.g. pancake sort or elevator allocation). During these interviews the key to search for is problem solving ability, confidence, uniqueness of solution and if they have a “knowledge index” they can apply look it up for them.

Remember, this is all theoretical. However, during this stage big-O notation should be assessed, as well as an algorithm written. Notice, I did not say “code” or “programming” must be completed, only the outline of an algorithm and run-time. It should really be kept as close to a theoretical discussion as possible, act as if it is a philosophical discussion with a friend/colleague.

Seventh – Consensus

At the end of each interview, the interviewer should ask his/herself if he/she would want to work with the candidate. Regardless of the reason, if the interviewer can say, “yes I would like to work with them” then you should say “yes” to the candidate. The same goes for “No,” if you didn’t like them for any reason, if they smell, cannot solve the problems, won’t look you in the eye, etc., say “no” to the candidate.

The interviewers cast all of these votes separately and immediately. After the interviews, the hiring manager should review the votes and if there is a consensus yes vote to the candidate on the spot, offer the candidate a position, compensation and all. They will be off guard and you can pick them up for cheaper (probably).

On the other hand, if it is a split vote, the hiring manager should have a short impromptu meeting with the interviewers. During the meeting a consensus should be reached and explanations should be forwarded. Then the decision/offer should be presented to the candidate at the end of the meeting.

In the case of a no consensus in either of the cases above, let them know immediately.

Overview

Every company is different, but consensus among employees ensure higher quality cultural and technical skills. Remember, the purpose of this interview process is to identify smart, ambitious, hard-workers who can achieve great things. If it takes them a few of months to gain the skills, that is all right, as long as they are a good cultural fit, because they will want to stay (and you can pick them up for cheaper).

Search for the gems among the coals. One “crazy” idea, with the proper ambition and a bit of knowledge behind it, can change the world. Interviewing is the process of identifying the individuals who can achieve greatness.