Assessing wisdom of crowds

Why fit is a quality that requires special treatment

When I wrote earlier about how to conduct a good technical interview, I had only a few things to say about how to assess if the candidate fits in with the team, including this:

This responsibility falls squarely to the hiring manager. You need to have a point of view about how to put together a coherent team, and how a potential candidate fits into that plan. Does the candidate have enough of a common language with the existing team (and with you) that you'll be able to learn from each other? Do they have a background that provides some novel approaches? Does their personality bring something new?

A few commenters have taken issue with the idea that it's solely the hiring manager's responsibility to assess fit, arguing correctly that the whole team should participate in the evaluation and decision. I completely agree. Still, I do think fit is a quality that requires special treatment, because it is the hardest attribute to evaluate.

Unlike the other attributes we look for in an interview candidate (like drive, brains or empathy) fit is not an individual quality. It's caught up in group dynamics. Worse, it has a self-referential quality to it. The very team that is making the assessment is being asked to assess itself at the same time as the candidate. How else can they tell whether the new team that will be created by the addition of this person will be superior to the team as it is presently constituted?

I have found James Surowiecki's book The Wisdom of Crowds particularly helpful in thinking through these issues. This book is Tipping Point-esque, full of anecdotes and interesting social science citations. Its central thesis is that, under the right circumstances, groups of people can be smarter than even their smartest single member. I find his observations compelling, and I feel good recommending the book to you, even though I know there are many among us who find "argument-by-anecdote" irritating. You don't have to buy the argument, but the facts and citations are worth the price of admission.

Let me briefly summarize the part of the book I find most helpful (leaving a few out, which aren't germane to today's topic). Not all crowds are wise. In order to get optimal results from a group-based effort, you need three things: diversity, independance, and an objective method for aggregating results. For example, we conduct elections with a secret ballot, which ensures independance (since nobody can directly influence your vote); we let everyone vote, which ensures diversity (since even extreme opinions can be heard); and we use a numerical count of the results (which, recent experiences notwithstanding, is supposed to mean that everyone's vote counts equally according to an objective formula). Similar mechanisms are at work in the stock market, Google PageRank, and guessing an ox's weight at the state fair.

Remove any of these essential ingredients, and you can find examples of groups gone bad: the Bay of Pigs invasion fiasco, market bubbles, and pretty much every one of Dilbert's team meetings.

Anyway, back to fit. Surowiecki has helped me in two ways:

1. Assessing fit in the context of what makes a good team. In order to improve the performance of a team, it's not enough just to keep adding smart people. You actually need to find a diverse set of people whose strengths and weaknesses complement each other. The problem most teams have with fit, in my experience, is they confuse it with liking. Many great engineers I've worked with (especially the type I talked about in the hacker's lament) seem to think that if they get along well with someone in an interview, they'll be a good addition to the team. This kind of homogeneity can lead to groupthink.
   
   
2. Putting together a process for helping a team assess fit. Another dangerous group dynamic in fit questions is that people are very sensitive to the opinions of their peers when talking about the group itself. In order to make an informed decision, the team needs a process of gathering and combining their opinions without having anyone's voice stifled.
   

For example, let's say you are on a team that prides itself in its all-hands-on-deck-at-all-hours style of working. There are a lot of great teams that work this way, preferring a series of sprints and lulls to a steady pace. Now let's say you're interviewing someone who doesn't seem to like to work that way. They work steady hours, but not for lack of drive; their references say they have exceptional output. Now picture the group meeting to talk about this prospective candidate. First up, the alpha-hacker on the team says something like "we should pass, this candidate is lazy."

Now it's your turn. Even if you are convinced that this candidate would make a great addition to the team, how much courage does it take to say so? You might convince the group, but you might not. And if you don't, will it raise suspicion about how dedicated you are? Will you be implicitly calling into question the validity of the team's values? Will they then be watching you for signs of laziness in the future? What about that vacation you've been planning to take... and so on. In my experience, there are plenty of situations where dissenting voices simply opt out. There's a clear danger to speaking up, but a pretty murky benefit.

If you find this line of reasoning confusing (nobody on my team feels that way) or paranoid (let's just all be rational), you may be surprised what the other people in the room are thinking. Take a look at some of the social science research in this area, like the Asch conformity experiments. In those, a group of people are asked to answer a simple question about their obsevations, one at a time. The first few people are actually actors, and they all give the same patently false answer. The experiment measures the likelihood that the last person in the sequence, the real experimental subject, will conform to what the previous people have said, or dissent. Even though the answer is obvious, and the other people in the room are all strangers, a surprising number of people choose to conform. I have found the pressure is much higher in situations where the answer is unclear, and the other group members are coworkers.

Combatting these tendencies is the real job of the hiring manager. If that's you (or you are on a team whose hiring manager abdicates that responsibility), here are three suggestions that have worked for me to take advantage of the wisdom of crowds in hiring. Each of these are based in changes I've made to my hiring process in response to five whys analysis of previous hiring mistakes.

• Before you meet the candidate, spend time thinking about the strengths and weaknesses of your team. Try to brainstorm some archetypes that probably would interview badly but actually be successful in filling out your team. Having been through this exercise in advance can help you listen carefully to what team members say about the candidate, and see if their objections are simply fearing what is different, or if they have a more serious concern.
  
  One experience I had was with a candidate who was absolutely convinced that our development methodology wouldn't work. He spent an inordinate amount of time grilling us on exactly how we work and why, asking smart but tough questions. He made us nervous, but we took a risk and hired him anyway. After we hired him, he spent weeks driving the team crazy with his critical (but, we had to admit, accurate) eye. Then, all of the sudden, a remarkable thing happened. Another new hire started to complain about the way we worked. Our former critic promptly set him straight, shooting down his complaints with the same ruthless efficiency he had previously devoted to analyzing our work. He had been converted, and from that point on acted as "defender of the faith" far better than I ever could.
  

• Maintain strict independance for each interviewer. Our rule was always that each interviewer was not allowed to talk any other interviewer once their session was concluded. The first time we'd exchange any words at all was during the end of the day assessment meeting. This prevents a previous interview from biasing a later interview. For example, I've seen situations where even a positive comment, like "wow, that candidate is smart!" cause disaster. The next interviewer, armed with an expecation of brilliance, chooses harder questions or becomes disappointed by an "only above average" performance.
  
  
• Aggregate results carefully. When sitting in a room talking about the candidate, I have had success with two precautions. First, we would always share our experiences in reverse-seniority order. That meant that the most junior person would be forced to speak first, without knowing anything about what his or her manager thought. As the hiring manager, I would speak last, and I'd do my best to avoid giving any indication in advance of what I thought.
  
  We'd also have a structured discussion in two parts: in the first part, each person talks only objectively about what happened in the interview, without giving opinions or passing judgment. Others are allowed to ask "clarifying questions" only - no leading questions or comments. Only in the second round does each person give their opinion about whether to hire the person or not. This helps get the facts on the table in an objective way. If our team had ever struggled to do it, I would have insisted on written comments shared anonymously. In other words: do whatever you have to do to get an objective discussion going.

If you've been the victim of groupthink in hiring, or have suggestions for ways to avoid it, please share. All of us are team members, family members, coworkers and leaders. Tell us what you've learned in those different contexts. Maybe it'll help someone else avoid the same mistakes.