It’s time for Software Engineering to grow up

Co-founder of Faros AI — a connected engineering operations platform.

The recent market correction has been a long time coming. For more than a decade, low interest rates and easy access to capital fueled a period of unprincipled growth in Silicon Valley. “Cash-flow positive” had become a distant memory of a bygone era. But like Edward Abbey “Growth for growth’s sake is the ideology of the cancer cell.” He was referring to the erosion of wilderness at the hands of rampant urban sprawl in his beloved Arizona, but the analogy also applies to corporations.

Software engineering organizations in particular experienced rapid growth during this period, out of proportion to other functions. Workforce has always been the primary driver for engineering leaders to significantly increase production. The belief is that more engineers will deliver more software, faster. Recruitment is the solution to everything. Need more features? Hire more engineers. Do the engineers complain? Hire more infrastructure people. Are things moving slowly? Hire more engineering managers, product managers, project managers and recruiters to fill these positions. It’s time to grow up!

Adding more workers to an organization is a costly Band-Aid fix that greatly increases the complexity of the system and often slows it down. The mythical man-month talks about this phenomenon. More engineers means more teams, more dependencies, more interviews and integrations, more processes, more analysis-paralysis, more tech debt, more feature creep and, most exhaustingly, less focus on what really matters. Austen Allred, CEO of the Bloom Institute of Technology, calls it “molasses death spiral.”

On the other hand, headcount has also been a major driver for engineering leaders when it comes time to cut costs, and we’re seeing the effects now.

So why did engineering leaders only have such a blunt tool at their disposal? The answer is a lack of visibility into mechanical operations. Ask your sales or marketing leaders about their metrics—funnel conversion rates, channel effectiveness, sales cycles, forecasted revenue. The answers would be ready. But if you asked engineering leaders for a breakdown of monthly spend, forecasts for the next month, or the dollar impact of an unresolved incident, the answers would require weeks of effort, gathering data from disparate sources, searching logs, writing ad hoc scripts, and more. but. The irony is that for an organization teeming with analytical minds, decisions are based on incomplete data, and intuition is a frequent substitute.

It’s not the engineering leaders fault. They have never been held accountable. The other functions don’t know enough to challenge them. They could spend an entire hour of content in a board meeting without questions. But just because they haven’t been held accountable doesn’t mean they shouldn’t do a better job.

So why is visibility into software engineering operations poor? There are two main reasons. First, it’s simple tough. Engineering data sources are incredibly fragmented and siloed. Organizations use dozens of systems to manage their engineering processes—from source control to task and incident management to continuous integration/delivery, cloud operations, budgeting, HR, and more. These systems do not talk to each other or to any central system, yet responses require data from all of these sources.

The second reason is fear. Fear of alienating a volatile and scarce resource – software engineering. Software engineering is a creative art. Some functional metrics can be “big brother” and will stifle the creativity that drives innovation.

But the result of the adventure is that the mechanical organisms today are blind. Engineering leaders have only one way to grow: hire people. They also only have one way to cut costs: lay people off. They have bloated teams—many of which are overwhelmed by dependencies or tech debt—and don’t have enough visibility to provide timely support. Constant palinodynia is a typical symptom of this dysfunction. It’s time to grow up.

In the interest of keeping the peace, engineering leaders have forgotten that while organizations are made up of people, they should function like well-oiled machines. Especially now. Sticking your head in the sand like an ostrich is a temporary solution to avoid “upsetting” engineers with “metrics”, but it doesn’t solve what the business really needs, what a team’s pain points are, and how to best help them. Constant reorganizations and layoffs do not make engineers happy.

So where do we go from here? The good news is that while visibility into engineering operations is difficult due to the fragmentation and variety of data sources, software teams no longer need to build the necessary instrumentation themselves. There are platforms and tools available to provide this much needed visibility out of the box. At the same time, reference points and frameworks such as DORA and SPACE emerge and gain traction, enabling teams to ascertain how they are doing and room for improvement.

So now, envision a world where engineering organizations have all their business data at their fingertips. The speed and quality of software delivery could really be measured. Bottlenecks in processes could be uncovered and continuously improved. Leaders will know how much time and resources are being spent on important initiatives and whether they are aligned with overall business priorities. Teams could be supported with the resources they need when they need them. More broadly, development could be methodical, needs-driven, and data-driven (eg, which areas really need investment? Which areas would really move the needle?). Course correction could be timely and gradual, avoiding big-bang explosions and layoffs. A focus on speed and quality would lead to practical and technical capabilities that would allow organizations to do much more with much less.

The ongoing technological revolution is changing our world faster than ever before. It gave us the internet, smartphones and artificial intelligence and will, in the near future, give us self-driving cars, private space exploration and more. The technology industry employs some of the brightest minds of our generation, and yet we are nowhere close to realizing their full potential due to the immaturity of our engineering practices. Engineering leaders support this and rely heavily on instinct. It’s time to grow up.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Am I eligible?