At their root, all technology problems are human problems
One of the thrilling parts of working and writing in Silicon Valley is
the incredible variety of people I've had the chance to meet.
Sometimes, I meet someone that I feel a visceral connection with,
because they are struggling with challenges that I've experienced
myself. In a few cases, they are clearly smart people in a bad
situation, and I've written about their pain in
The product manager's lament and
The engineering manager's lament.
Today I want to talk about another archetype: the incredibly high-IQ
hacker
who's trying to be a leader. (As always, this is a fictionalized
account; I'm blending several people I've known into a single
composite. And please forgive the fact that I use male pronouns to
describe the archetype. There is terrible gender bias in our
profession, but that's a subject for another day. Suffice to say, most
of the hackers I've known have been men. As a last disclaimer, please
consult
the definition of the word hacker if you're not familiar with the controversies surrounding that term.)
It's
common to find a hacker at the heart of almost any successful
technology company. I know them right away - we can talk high-level
architecture all the way down to the bits-and-bytes of his system. When
I want to know about some concurrency issues between services in his
cluster, he doesn't blink an eye when I suggest we get the source code
and take a look. And as soon as I point out an issue, he can instantly
work out the consequences in his head, and invent solutions on the fly.
This
kind of person is used to being the smartest person in the room. In
fact, it's a rare person who can be subjected to recurring evidence of
just how stupid the people
around them are, and not become incredibly arrogant. Those who have the
endurance are the ones that tend to lead teams and join startups,
because you just can't be successful in a startup situation without
empathy. I would characterize them as intolerant but not arrogant.
When
a startup encounters difficult technical problems, this is the guy you
want solving them. He's just as comfortable writing code as racking
servers, debugging windows drivers, or devising new interview
questions. As the company grows, he's the go-to person for almost
everything technical, and so he's very much in demand. He throws off
volumes of code, and it works. When scalability issues arise, for
example, he's in the colo until 2am doing whatever it takes to fix them.
But
life is not easy, either. As the company grows, the number of things
he's called on to do is enormous, and the level of interruptions are
getting intense. It's almost as if he's a country that was immune to
the economic theory of
comparative advantage.
Since he's better at everything, he winds up doing everything - even
the unimportant stuff. There's constant pressure for him to delegate,
of course, but that doesn't necessarily work. If he delegates a task,
and it gets messed up, he's the one that will get called in to deal
with it. Better just to take care of it himself, and see that it's done
right.
When you're the physical backstop putting dozens of
fingers in the damn to prevent it from bursting, you might get a little
irritated when people try to "help" you. The last thing you need is a
manager telling you how to do your job. You're not very receptive to
complaints that when you take on a task, it's unpredictable when you'll
finish: "you try getting anything done on schedule when you're under
constant interruptions!" Worst of all, your teammates are constantly
wanting to have meetings. When they see a problem with the team's
process, why don't they just fix it? When the architecture needs
modifying - why do we need a meeting? Just change it. And we can't hire
new engineers any faster, because you can't be interviewing and
debugging and fixing all at the same time!
The picture I'm
trying to paint is one of a bright individual contributor stretched to
the breaking point. I've been there. Trust me, it's not a lot of fun.
And I've also been on the receiving end; and that's not much fun
either. Yet, quite often these dynamics play out with ever-increasing
amplitude, until finally something drastic happens. Unfortunately, more
often than not, it's the hacker who gets fired. What a waste.
What's wrong with this picture?
One
of the most exhilarating things about a startup is that feeling of
intense no-holds-barred execution. Especially in the early days, you're
fighting for survival every day. Every day counts, every minute counts.
Even if, in a previous life, you were a world expert in some functional
specialty, like in-depth market research or scalable systems design,
the compressed timeline of a startup makes it irrelevant. You get to
figure things out from first principles all the time, experiment
wildly, and invest heavily in what works. From the outside, it looks a
lot like chaos. To a hacker, it looks a lot like heaven.
But
even a tiny amount of success requires growth. Even with the highest
standards imaginable, there's no way to hire just genius hackers. You
need a diversity of skills and backgrounds. Suddenly, things slow down
a little bit. To me, this is the critical moment, when startups either
accept that "process = bureaucracy" or reject that thinking to realize
that "process = discipline." And it's here that hackers fall down the
most. We're just not naturally that good at thinking about systems of
people; we're more comfortable with systems of computers.
If
you've ever been abused by a bad manager in your career, it's easy to
become traumatized. I think this is the origin of the idea among
hackers that managers are idiots who just get in the way. The
variations on this theme are legion: the pointy-haired boss, the
ivory-tower architect, and of course the infinite variety of
marketroids. But whenever groups of people assemble for a common
purpose, they adopt process and create culture. If nobody is thinking
about it, you're rolling the dice on how they turn out. And, at first,
it's OK if the person who's doing that thinking is part-time, but
eventually you're going to need to specialize. The alpha-hacker simply
can't do everything.
Even in the areas that hackers specialize
in, this go-it-alone attitude doesn't work. Building a good application
architecture is not just coding. It's more like creating a space for
other people to work in. A good architect should be judged, not by the
beauty of the diagram, but by the quality of the work that the team
does using it. The "just fix it" mentality is counter-productive here.
Every bug or defect needs to go through the meta-analysis of what it
means for the architecture. But that's impossible if you're constantly
fire-fighting. You need to make time to do
root cause analysis, to correct the systemic mistakes all of us tend to make.
And
taking on too many projects at once is a classic sub-optimization.
Sure, it seems efficient. But when there is a task half-done, it's
actually slowing the team down. That's because nobody else can work on
the task, but it's costly to hand it off. Imagine a team working from a
forced-rank priority queue.
Naturally, the best person should work on the #1 priority task, right?
Not necessarily. If that person is subject to a lot of interruptions,
as the people working on the less-important tasks finish, they're
forced to keep working down the list. Meanwhile, the #1 task is still
not done. It would have been faster for the team as a whole to have
someone else work on the task, even if they were much slower. And of
course there's the secondary benefit of the fact that as people work on
tasks they don't know anything about, they learn and become more
capable.
The reason this situation reaches a breaking-point is
that it's constantly getting worse. As the team grows, the number of
things that can go wrong grows with it. If a single person stays the
bottleneck, they can't scale fast enough to handle all those
interruptions - no matter how smart they are. And the interruptions
themselves make looking for solutions increasingly difficult. Each time
you look for solutions, you see a conundrum of this form: you can't
hire because you're too busy, but you can't delegate because you can't
hire.
All is not lost, though. When I get involved in companies
that struggle with this problem, here is the kind of advice I think can
help:
- Introduce TDD and continuous integration. This is one of the bedrock practices of any lean startup,
and so it's a common piece of advice I give out. However, it's
particularly helpful in this situation. Without requiring a lot of
meetings, it changes the perspective of the team (and its leadership)
from fire-fighting to prevention. Every test is a small investment in
preventing a specific class of bugs from recurring; once you've been
successful at building this system, it's pretty easy to see the analogy
to other kinds of preventative work you could do. It also helps ratchet
down the pressure, since so many of the interruptions that plague the
typical hacker are actually the same bugs recurring over and over. TDD
plus continuous integration works as a natural feedback loop: if the
team is working "too fast" to produce quality code reliably, tests
fail, which requires the team to slow down and fix them.
- Use pair programming and collective code ownership. These are two other Extreme Programming practices
that are explicitly designed to counteract the problems inherent in
this situation. Pair programming is the most radical, but also the most
helpful. If your team isn't ready or able to adopt pair-programming
across the board, try this technique instead: whenever anyone is
becoming a bottleneck (like the proverbial hacker in this post), pass a
rule that they are only allowed to pair program until they are not the
bottleneck anymore. So each time someone comes to interrupt them, that
person will be forced to pair in order to get their problem solved. In
the short term, that may seem slower, but the benefits will quickly
become obvious. It's another natural feedback loop: as the
interruptions increase, so does the knowledge-transfer needed to
prevent them.
- Do five whys.
This is a generalization of the previous two suggestions. It requires
that we change our perspective, and instead treat every interruption as
an opportunity to learn and invest in prevention.
- Hire
a CTO or VP Engineering. A really good technology executive can notice
problems like the ones I'm talking about today and address them
proactively. The trick is to hire a good one - I wrote a little about
this in What does a startup CTO actually do?
Sometimes, a great hacker has the potential to grow into the CTO of a
company, and in those cases all you need is an outside mentor who can
work with them to develop those skills. I've been privileged to have
been the recipient of that kind of coaching, and to have done it a few
times myself.
At the end of the day, the product development
team of a startup (large or small) is a service organization. It exists
to serve the needs of customers, and it does this by offering its
capabilities to other functions in the company, and partnering with
them. That's only possible if those interactions are constructive,
which means having the time and space for people of different
backgrounds and skills to come together for common purpose. That's the
ultimate task for the company's technology leadership.
I
strongly believe that all hackers have the innate ability to become
great leaders. All that's required is a shift in perspective: at their
root, all technology problems are human problems. So, fellow hackers,
I'd love to hear from you. Does this sound familiar? Are you ready to
try something different?