The hacker's lament

At their root, all technology problems are human problems

One of the thrilling parts of working and writing in Silicon Valley is the incredible variety of people I've had the chance to meet. Sometimes, I meet someone that I feel a visceral connection with, because they are struggling with challenges that I've experienced myself. In a few cases, they are clearly smart people in a bad situation, and I've written about their pain in The product manager's lament and The engineering manager's lament.

Today I want to talk about another archetype: the incredibly high-IQ hacker who's trying to be a leader. (As always, this is a fictionalized account; I'm blending several people I've known into a single composite. And please forgive the fact that I use male pronouns to describe the archetype. There is terrible gender bias in our profession, but that's a subject for another day. Suffice to say, most of the hackers I've known have been men. As a last disclaimer, please consult the definition of the word hacker if you're not familiar with the controversies surrounding that term.)

It's common to find a hacker at the heart of almost any successful technology company. I know them right away - we can talk high-level architecture all the way down to the bits-and-bytes of his system. When I want to know about some concurrency issues between services in his cluster, he doesn't blink an eye when I suggest we get the source code and take a look. And as soon as I point out an issue, he can instantly work out the consequences in his head, and invent solutions on the fly.

This kind of person is used to being the smartest person in the room. In fact, it's a rare person who can be subjected to recurring evidence of just how stupid the people around them are, and not become incredibly arrogant. Those who have the endurance are the ones that tend to lead teams and join startups, because you just can't be successful in a startup situation without empathy. I would characterize them as intolerant but not arrogant.

When a startup encounters difficult technical problems, this is the guy you want solving them. He's just as comfortable writing code as racking servers, debugging windows drivers, or devising new interview questions. As the company grows, he's the go-to person for almost everything technical, and so he's very much in demand. He throws off volumes of code, and it works. When scalability issues arise, for example, he's in the colo until 2am doing whatever it takes to fix them.

But life is not easy, either. As the company grows, the number of things he's called on to do is enormous, and the level of interruptions are getting intense. It's almost as if he's a country that was immune to the economic theory of comparative advantage. Since he's better at everything, he winds up doing everything - even the unimportant stuff. There's constant pressure for him to delegate, of course, but that doesn't necessarily work. If he delegates a task, and it gets messed up, he's the one that will get called in to deal with it. Better just to take care of it himself, and see that it's done right.

When you're the physical backstop putting dozens of fingers in the damn to prevent it from bursting, you might get a little irritated when people try to "help" you. The last thing you need is a manager telling you how to do your job. You're not very receptive to complaints that when you take on a task, it's unpredictable when you'll finish: "you try getting anything done on schedule when you're under constant interruptions!" Worst of all, your teammates are constantly wanting to have meetings. When they see a problem with the team's process, why don't they just fix it? When the architecture needs modifying - why do we need a meeting? Just change it. And we can't hire new engineers any faster, because you can't be interviewing and debugging and fixing all at the same time!

The picture I'm trying to paint is one of a bright individual contributor stretched to the breaking point. I've been there. Trust me, it's not a lot of fun. And I've also been on the receiving end; and that's not much fun either. Yet, quite often these dynamics play out with ever-increasing amplitude, until finally something drastic happens. Unfortunately, more often than not, it's the hacker who gets fired. What a waste.

What's wrong with this picture?

One of the most exhilarating things about a startup is that feeling of intense no-holds-barred execution. Especially in the early days, you're fighting for survival every day. Every day counts, every minute counts. Even if, in a previous life, you were a world expert in some functional specialty, like in-depth market research or scalable systems design, the compressed timeline of a startup makes it irrelevant. You get to figure things out from first principles all the time, experiment wildly, and invest heavily in what works. From the outside, it looks a lot like chaos. To a hacker, it looks a lot like heaven.

But even a tiny amount of success requires growth. Even with the highest standards imaginable, there's no way to hire just genius hackers. You need a diversity of skills and backgrounds. Suddenly, things slow down a little bit. To me, this is the critical moment, when startups either accept that "process = bureaucracy" or reject that thinking to realize that "process = discipline." And it's here that hackers fall down the most. We're just not naturally that good at thinking about systems of people; we're more comfortable with systems of computers.

If you've ever been abused by a bad manager in your career, it's easy to become traumatized. I think this is the origin of the idea among hackers that managers are idiots who just get in the way. The variations on this theme are legion: the pointy-haired boss, the ivory-tower architect, and of course the infinite variety of marketroids. But whenever groups of people assemble for a common purpose, they adopt process and create culture. If nobody is thinking about it, you're rolling the dice on how they turn out. And, at first, it's OK if the person who's doing that thinking is part-time, but eventually you're going to need to specialize. The alpha-hacker simply can't do everything.

Even in the areas that hackers specialize in, this go-it-alone attitude doesn't work. Building a good application architecture is not just coding. It's more like creating a space for other people to work in. A good architect should be judged, not by the beauty of the diagram, but by the quality of the work that the team does using it. The "just fix it" mentality is counter-productive here. Every bug or defect needs to go through the meta-analysis of what it means for the architecture. But that's impossible if you're constantly fire-fighting. You need to make time to do root cause analysis, to correct the systemic mistakes all of us tend to make.

And taking on too many projects at once is a classic sub-optimization. Sure, it seems efficient. But when there is a task half-done, it's actually slowing the team down. That's because nobody else can work on the task, but it's costly to hand it off. Imagine a team working from a forced-rank priority queue. Naturally, the best person should work on the #1 priority task, right? Not necessarily. If that person is subject to a lot of interruptions, as the people working on the less-important tasks finish, they're forced to keep working down the list. Meanwhile, the #1 task is still not done. It would have been faster for the team as a whole to have someone else work on the task, even if they were much slower. And of course there's the secondary benefit of the fact that as people work on tasks they don't know anything about, they learn and become more capable.

The reason this situation reaches a breaking-point is that it's constantly getting worse. As the team grows, the number of things that can go wrong grows with it. If a single person stays the bottleneck, they can't scale fast enough to handle all those interruptions - no matter how smart they are. And the interruptions themselves make looking for solutions increasingly difficult. Each time you look for solutions, you see a conundrum of this form: you can't hire because you're too busy, but you can't delegate because you can't hire.

All is not lost, though. When I get involved in companies that struggle with this problem, here is the kind of advice I think can help:

• Introduce TDD and continuous integration. This is one of the bedrock practices of any lean startup, and so it's a common piece of advice I give out. However, it's particularly helpful in this situation. Without requiring a lot of meetings, it changes the perspective of the team (and its leadership) from fire-fighting to prevention. Every test is a small investment in preventing a specific class of bugs from recurring; once you've been successful at building this system, it's pretty easy to see the analogy to other kinds of preventative work you could do. It also helps ratchet down the pressure, since so many of the interruptions that plague the typical hacker are actually the same bugs recurring over and over. TDD plus continuous integration works as a natural feedback loop: if the team is working "too fast" to produce quality code reliably, tests fail, which requires the team to slow down and fix them.
  
  
• Use pair programming and collective code ownership. These are two other Extreme Programming practices that are explicitly designed to counteract the problems inherent in this situation. Pair programming is the most radical, but also the most helpful. If your team isn't ready or able to adopt pair-programming across the board, try this technique instead: whenever anyone is becoming a bottleneck (like the proverbial hacker in this post), pass a rule that they are only allowed to pair program until they are not the bottleneck anymore. So each time someone comes to interrupt them, that person will be forced to pair in order to get their problem solved. In the short term, that may seem slower, but the benefits will quickly become obvious. It's another natural feedback loop: as the interruptions increase, so does the knowledge-transfer needed to prevent them.
  
  
• Do five whys. This is a generalization of the previous two suggestions. It requires that we change our perspective, and instead treat every interruption as an opportunity to learn and invest in prevention.
  
  
• Hire a CTO or VP Engineering. A really good technology executive can notice problems like the ones I'm talking about today and address them proactively. The trick is to hire a good one - I wrote a little about this in What does a startup CTO actually do? Sometimes, a great hacker has the potential to grow into the CTO of a company, and in those cases all you need is an outside mentor who can work with them to develop those skills. I've been privileged to have been the recipient of that kind of coaching, and to have done it a few times myself.

At the end of the day, the product development team of a startup (large or small) is a service organization. It exists to serve the needs of customers, and it does this by offering its capabilities to other functions in the company, and partnering with them. That's only possible if those interactions are constructive, which means having the time and space for people of different backgrounds and skills to come together for common purpose. That's the ultimate task for the company's technology leadership.

I strongly believe that all hackers have the innate ability to become great leaders. All that's required is a shift in perspective: at their root, all technology problems are human problems. So, fellow hackers, I'd love to hear from you. Does this sound familiar? Are you ready to try something different?