0 Comments

Software is a unique industry because it is constantly self-improving its own productivity. It acts as an accelerating concept like interchangeable parts or the assembly line, rather than a linear incrementing body of study. The use of software applies that acceleration to everywhere it is used, including itself.

So why don't we have lower skilled people building software by drag/dropping building blocks as many have predicted? After all, we have that self accelerating improvement applied to software over decades. Exponential increases in programmer productivity would suggest that lower skilled people could be employed, dropping average salaries, right? That kind of logic leads toward thinking there is a tech worker shortage.

There is no "tech worker shortage". Companies might not be able to find low priced labor, which they believe they should be able to do based on the reasoning above. But like the linked article quotes: "I'm not sure that qualifies as a shortage, any more than my not being able to find a half-priced TV."

What's wrong with this train of thought?

First off, there are many "building block" type software packages available. It's just hard to see them because the demand for new and more complex software has grown so dramatically. There are website builders, time tracking software, component based development tools galore. They've satisfied an enormous amount of what people wanted to do 20 years ago. But in the meantime, demand and complexity of requirements have grown exponentially. The acceleration of productivity is just barely allowing the software industry to run in place.

At the same time, there is an increasing demand for custom built software that is tailored directly to the data and business logic of the client, particularly in the LOB arena. It used to be too expensive for anything other than massive companies to afford custom development. Now, productivity increases allow even small businesses to have staff or contract programmers building software to enhance their business, again, pushing demand for programmers back up.

Finally, and most importantly, software is getting more complex in general. The requirements are bigger, in features, volume, and reliability. The software enabling productivity increases is sitting on layer upon layer of other software packages. 20 years ago, you'd build a simple desktop app in C++ or VB. Or a CGI app for web. It'd be a couple of components, sitting on a single server (or maybe a cluster of two machines if you were really rich and important), with a simple database. The "cloud" was still a twinkle in Jeff Bezos's eye until less than 10 years ago. Now, you have an MVC app using 20 different packages imported by a package manager running on a load balanced set of web servers, a database sharded across several servers, some memcache servers on the backend to accelerate data access, some varnish caches on the front to accelerate static pages, some CDN to accelerate content. Etcetera etcetera.

A senior software engineer that knows all of the possible tools and can orchestrate them together, someone that knows the concepts from the underlying cpu/cache/memory up to the networked cloud, and can use the right tool (out of 100s) for the right job, getting the job done (not to mention done right, and done fast) -- that person's value and productivity is multiplied by the layer upon layer of tooling developed over the past couple of decades. They can do the work that 10 senior people did 10 or 15 years ago, or 100 junior devs, while the junior dev's productivity has remained about the same.

That dichotomy between productivity of a senior and junior is what is throwing big companies for a loop. Their mental model of staffing is hierarchical, with a couple of seniors at the top, then some mids, and a bunch of juniors filling out the crowd. Their sort of thinking around cost is what leads to outsourcing to low skilled body shops. They can't fathom that it would be cheaper and more effective to hire a bunch of seniors and a couple mids to get the job done right -- they just see the dollar signs on the salary packages and scream "tech shortage".

Software development is becoming another market with winner-take-all dynamics. Sure, anybody can get started easy and for free. But the thousands of hours invested over years that it takes to gain an innate understanding of the innumerable tools and techniques involved? That’s another matter entirely.