Productivity gains drive innovation

A software engineer's value has increased three times that of a marketing person

Everyone is more productive these days. This has been a consistent trend for at least the past decade, where productivity gains have been particularly strong within the business sector.  According to data from the U.S. Bureau of Labor Statistics, today's business industry workers are on average 30% more productive than their 1998 counterparts (productivity growth of roughly 2.6% per year).
Within the technology industry, productivity has increased more.  Thanks to smartphones, improved search engines, better CRM software, and ever-increasing bandwidth, salesmen and marketers can find, receive and process information faster than ever.

The most dramatic gains, however, have occurred within software development.

Software engineers today are about 200-400% more productive than software engineers were 10 years ago because of open source software, better programming tools, common libraries, easier access to information, better education, and other factors. This means that one engineer today can do what 3-5 people did in 1999!

The advent of open-source software makes engineers particularly efficient.  One VP Engineering that I talked to gave me an anecdote about one module where they used open source files with about 500,000 lines of code and then wrote 7,000 lines of code to stitch it all together.  Open source software is also free.  In the company I was running in 1999, "software" was a huge budget line item - we had to buy databases, testing suites, libraries, and more.  Today all that stuff is free ... a start-up might spend more money on sodas for the office than it does on software.

We're all familiar with Moore's Law - that the power of computers doubles every 18 months.  In my 15 years of software development, I've seen 5x-10x productivity gains in engineers.  Which could mean that the productivity of a well-trained engineer doubles every five years.  (note that this Law is much harder to prove than Moore's Law - but potentially just as profound).  That would mean that the productivity of an engineer is growing at roughly 14.9% per year!  That's fast ... really fast ... much faster than the 2.6% yearly gains than the population as a whole is making.

This means that today's companies are able to do more software engineering and build more stuff with fewer people.  But should they do more with less? It could be much more prudent for a company, especially for a small company, to do the opposite ... and to double-down on engineering.  You can use the productivity gains in software development as a strategic advantage and invest aggressively in engineers. First, doing so contributes the most to progress and also increases the chance for breakthroughs in innovation. Second, engineers - as opposed to salesmen and marketers - can often hit the ground running (assuming you have a good on-boarding system) and have a positive impact within a few weeks.

Alternatively, many large traditional companies might be able to get by with FEWER but DIFFERENT engineers.  These companies might need to change their approach to engineering to take advantage of the new tools.  The companies that can benefit from fewer engineers are likely ones that haven't changed their technology platforms radically in the last ten years.

Although engineers contribute more to an organization than ever before, their pay - relative to other functions in a company - hasn't followed suit.  I've polled a few dozen companies and have found that over the last ten years, an engineer's pay has held the same relative salary to marketing and sales.  This is odd behavior ... usually when something outputs more, its cost goes up.  So why have engineers' wages in the U.S. stayed constant relative to salespeople and marketers?  Here are two contributing factors that lower demand:

1.    Off-shoring.  Because of new technology and higher bandwidth, more companies are off-shoring their software development.  But this does not fully explain the flat salary phenomenon since firms are also off-shoring sales and marketing (though to a lesser degree).

2.    Need for software engineers has decreased.  Because software engineers are so much more productive then they were ten years ago, many firms are opting to hire fewer of them.  If a company is not doing hard-core engineering, it actually needs fewer engineers as a portion of its total workforce than it did ten years ago.  (I personally think this could be a big mistake ... but I will get to that later).
Both the off-shoring and the decreased need for engineers has led to a lowering of the demand which has likely put a check on wages. 

One problem, of course, is that measuring "output" of an engineer is a really hard thing to do (as opposed to the output of a salesperson) ... so it is really hard to quantify the productivity gains.  And even if you can measure output in engineering, it is sometimes hard to tie that to an increase in profitability.

And, like sales, the quality of engineers varies wildly.  A great engineer is potentially 2-4 times more productive than a good engineer.  Ben Ling from Google pointed out to me that some great engineers are massively compensated - because they tend to be the early hires at a company and get lots of stock (most of Google's first 50 employees were engineers).

Let's recap:  The productivity of a software engineer has increased 2-3 times that of a marketing person in the last 10 years.  Yet their relative compensation has remained about the same.  That means if you are a savvy company, you should stock up on engineers.  In fact, you would want as many great engineers as you can get a hold of.

This engineering productivity boom will only increase and continue to create dislocation and creative destruction.  While the extent of growth and industry makeover are hard to gauge, what is certain is that corporations relying on technology and engineering paradigms from the 1990s or before will find themselves hard-pressed to compete with the new and nimble movers.

(Special thanks to Jonathan Hoffman, Michael Hsu, Ben Ling, Jeremy Lizt, Naghi Prasad, and Dave Selinger for their feedback and edits).
 
(Image source: wunderground)