Why process stifles innovation

9 min readNov 24, 2022

Lately, I kept reflecting on two topics that share a similar theme. The first topic has to do with the state of California’s public infrastructure and how inefficient it has become over the years. The second is the failure of large tech companies to innovate and execute more rapidly than startup incumbents. Let me provide some perspectives on why these topics are related by first giving you two examples to think about:

Infrastructure: The Golden Gate Bridge took about four years to build (1933–1937) and it cost us ~$500M in today’s dollars. By contrast, the SDS (Suicide Deterrent System), which basically adds a net around the bridge, will take us five years to complete (2017–2023) and is projected to cost us ~$217M. If you do a review of major infrastructure projects in California, you’ll find that for projects completed before the 1950s the work was done faster, cheaper, and with higher quality than it is done today by at least a 5x magnitude, even though today we have more capital, better technology, better raw materials, and a higher absolute number of civil engineers and architects.
Tech Companies: Adobe, despite all its financial and human resources, finds itself having to acquire a competitor for $20B. Whether Adobe overpaid for Figma remains to be seen, but the more interesting question is why did a company with 100x the personnel, 100x the depth of experience in design tooling, and 100x the financial capital fail to win?

In both cases, something went wrong, and it is interesting to examine why. Digging into the infrastructure issue, I stumbled on an interesting explanation that a big problem has to do with the National Environmental Policy Act (NEPA), which although well-intentioned, has created a hyper-restrictive review process that often blocks projects that are meant to improve our impact on the environment. Under current regulations, projects like the Golden Gate Bridge are basically impossible to get through.

Similarly, if you take a look within companies, you might find that as they grow, process and bloat creeps into anything and stifles real work and innovation. In that environment, the employees that are the most eager to create things, the makers, find that it is just more attractive to jump ship to a smaller place where their work does have an outlet to materialize. Both infrastructure failures and company innovation failures are at least partially due to bloat and process creep.

Why does process creep happen?

Every process starts with good intentions.

“We want to make sure all stakeholders have an input into the final work, so all submitted project proposals need to be reviewed by legal, design, product, and engineering teams.”
“We want to make sure that large infrastructure projects don’t cause harm, so we need to perform thorough environmental studies before any work is started.”
“We want to minimize data breaches, so any request for data needs to be done through the IT and Security teams.”

At the outset, these are innocuous and sensible suggestions. No process by itself is a catalyst that prevents work from being done, and some processes improve efficiency and prevent mistakes, yet if you take the cumulative impact of all the rules and requirements within an organization, you find that they ultimately have a cost on execution. Below I have a list of reasons why I think process creep happens and possible resolutions.

The road to perfectionism

I remember hearing a well-intentioned manager tell his direct reports to write software as if it would last 100 years. Although the remark was hyperbolic, the advice echoes something that I’ve heard often in the tech industry through adages like “measure twice and write once” or “if you don’t have time to do it right, do you have time to do it twice?.” This is not bad advice. In some situations, you have to write code as if it will last 100 years, and going headfirst into execution without planning is wasteful. Nonetheless, extreme emphasis on planning can lead to paralysis and failure to recognize the benefits of learning through iteration. If your organization relies on planning everything to perfection before a single line of code is written, you might have a hard time moving in the right direction.

Too much glue

Many years ago I read Tanya Reilly’s blog post on being Glue (highly recommended read, btw). At that point in my career, the content really resonated because I was dealing with a few colleagues who although technically competent, were just not working on the right things. Glue people engage in a process of technical leadership and mentorship that leads to making other people on the team more productive. Some amount of glue is necessary, nonetheless, I found that too much glue creates a set of other problems:

Folks doing glue work prevent other people from improving their non-technical skills. For example, a Tech Lead who does not let their team speak about their work or often acts as a “buffer” is essentially taking opportunities away from ICs to struggle and eventually improve their communication skills.
Extreme and toxic versions of glue behavior sometimes manifest in taking credit for other people’s work or in generating grossly exaggerated attribution (“e.g., without me acting as an intermediary, the project would not succeed”). While Tanya talks about the situations where Glue people often don’t get acknowledged, my experience is completely the opposite. Because Glue people are good communicators, they know how to advertise their work and play politics well. In return, they end up with higher visibility and more promotion opportunities, while the engineer who works on implementation details often gets token acknowledgment or no credit at all.
Lastly, adding too many glue people reduces ownership and accountability for individual engineers. Instead of adding glue people to manage a technically competent team, I would simply let go of engineers that seem to have no business intuition, lack communication skills, or can’t put a paragraph together describing why the work they do is impactful. Suggesting that you need more glue to manage engineers ends up dismissing the general pattern that folks that are technically brilliant also have a high level of general intelligence, read a lot, are lifelong learners, and are very good at business and not just code.

Note that I’m not alone in the opinions above. Here is what Eric Schmidt, former CEO of Google, has to say about glue roles.

When I was at Novell, I had learned that there were people who I call “glue people”. The glue people are incredibly nice people who sit at interstitial boundaries between groups, and they assist in activity. And they are very, very loyal, and people love them, and you don’t need them at all.
At Novell, I kept trying to get rid of these glue people, because they were getting in the way, because they slowed everything down. And every time I get rid of them in one group, they’d show up in another group, and they’d transfer, and get rehired and all that.

Entropy

As organizations grow, institutional knowledge, the number of stakeholders, codebases, and entropy increase. Increased entropy leads to high waste, high cognitive load, disorganization, and chaos.

As software engineers, I think we’ve done a crappy job in managing complexity. For example, for years we’ve heard that monoliths are bad because they generate bloated codebases, make deployments difficult, or create issues with scalability. Yet on the other extreme, where we have to operate in a world of microservices, I find that in order to ship basic things I need to operate across multiple codebases written in different languages, each having their deployment quirks and architecture patterns, with the whole paradigm turning into lambda spaghetti. Stopping and refocusing on what's important has a big impact on reducing entropy. At a low, action-oriented level it can manifest as follows:

Deleting dead code that is not executed in production. I’ve interacted with so many codebases where >50% of the code was dead code and created cognitive load for colleagues. I wish there was something in the dev productivity space that generated code coverage based on lines executed in production. Every quarter you stop and review those reports and simply delete files that had zero execution.
As product managers, we don’t reflect enough on what features should we kill or revert. Yet stopping and reflecting on what should be deleted can be very meaningful in simplifying and improving user experience. I consider myself tech-savvy, yet I find that many of the products created today have been hyper-optimized on a set of existing users, leading to a cluttered and confusing UX experience for new users. Asking at the end of the quarter “What feature can we unship?” can be useful in reducing clutter.

Lastly, I think a big reason for increased entropy is the failure of companies to retain top talent. Again and again I find that there is just no incentive for many folks to stay at a company beyond 2–3 years, even if they are a top performer. Any human capital loss means that institutional knowledge gets lost and new replacements will often engage in an exercise of reinventing the wheel without making any meaningful improvements. If you want reduced entropy, push the same set of people to rapidly iterate on the product for a sustained period of time.

Ego is the enemy

Ryan Holiday suggests in his book Ego is the Enemy, there is a strong “temptation that exists for everyone — for talk and hype to replace action.” Ego is a big reason why progress and innovation can be lackluster in many institutions. Here is how ego manifests itself:

Seeing the same people itching to pitch their thoughts into a group conversation, even though their ideas add no or very little marginal value.
Having people that never engage in actively sponsoring their colleagues or who try to hoard visibility. If you’re a manager or a manager of managers, you can notice this by seeing how often your direct report speaks about other people’s successes or contributions in your 1:1s. Lack of acknowledgment or selfless sponsorship of other folks can be a sign of an unhealthy ego.
Optimizing for the team instead of the overall company or org. I borrowed the diagram from the Staff Engineer’s Path by Tanya Reilly and it perfectly exemplifies this problem where teams optimize for the local maximum. For example, would you be ok with disbanding your team and redistributing employees to other teams if within your domain you’re entering maintenance mode? Most managers would never do that because it would mean losing any social and political capital within the company.

Depending on your role, there is a very strong dichotomy in terms of how you view the value of meetings. “Most powerful people are on the manager’s schedule”. Unfortunately, their perception of the value of meetings creates a cultural override that affects people who prefer to have uninterrupted time to think about a problem and solution. Even for brainstorming purposes, the evidence suggests that group brainstorming is a waste of time. Yet why do so many bloated organizations end up with half of the ICs’ time being spent in meetings?

Overall I don’t think there are sensible solutions to the Ego problem. This is a lot more challenging to fix and often requires having the right people at the top leading through example.

Conclusion

The engine of innovation relies on evolutionary change. Ultimately, if an institution becomes rigid and does not adapt to change, it will die a slow death. In my experience, introducing a process often has unintended costs that lead to reduced velocity. Instead, I would focus on the following:

Favoring iteration and learning from mistakes instead of trying to achieve perfectionism.
Glue people are necessary, but there has to be a limit to their number, so that they can act with high leverage. Simply hiring folks that are both technically competent and have good business intuition can solve the need of overhiring for glue roles.
Lastly I would focus on reducing entropy by constantly asking what can be discarded and rewarding folks for behavior that increases simplicity.