Nothing Special   »   [go: up one dir, main page]

Handling Process Debt in IT

10 December 2023 · 3,137 views · Updated 08 February 2024

I’m sure you’ve worked at companies where you felt that they were moving slowly and it was not even worth putting your best in, and I’m also sure you’ve worked at companies where you felt excited about contributing to the product development. But what separates the former from the latter? The answer is technical process debt.

Process debt is the accumulation of inefficient, outdated, or redundant processes within an organization. It manifests as cumbersome workflows, leading to increased developer frustration and the feeling of “this seems wrong” when you look at how things are done. Similar to technical debt, this kind of debt subtly erodes the operational efficiency of a company, often going unnoticed until it significantly hampers progress.

I also sometimes call this “processes for the sake of processes” — it’s when a startup lacks focus and starts hiring new people because:
a) either the investors are pushing for growth or
b) they’re just throwing people at the problem, hoping it will fix everything.

This eventually leads to processes getting jumbled and new work invented to accommodate new hires.

The Best Process is No Process — the holy grail of maximum efficiency. The only issue with that statement is that you cannot scale without having any processes. So, if you’re a startup of 1 person — you encompass everything in your mind and can do everything without friction. But your mental capacity has a limit. You won’t be able to grow endlessly.

There’s only so much you can do in a day and only so many things you can keep in your short-term memory. Once you cross the point of having too many things to do, either your mental health suffers, your quality of work, or both. This is where you’ll need to start hiring new developers and defining what and how they should work, a.k.a defining the processes.

In 1968, Melvin Conway said: “Organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations". So, assuming you’re a solo developer or a technical co-founder, your system designs will be a copy of perfection, as you understand the complexities of the product best and can also develop it. According to Conway's Law, your organizational process complexity will trickle down into the product you create; the more you grow, the worse your processes get, and the worse your product becomes.

An excellent example of process debt is when the Sales Team has a fantastic CRM Tool but still relies on Excel + Phone + Pen and Paper to track leads and make sales “because that worked for them all these years.” Someone was hired at one point, established some “good enough” process, and then the next hire came in and adopted the same approach.

But it can get worse. The new hire could’ve started doing it differently. And then you have multiple employees doing things in many different ways, and if we extrapolate this over the years — this will become a Frankenstein’s Monster that will be hard to untangle.

🏄 The Best Process Is No Process. Unless you're more than 1 person in a team.

Technical debt, as well as process debt, is inevitable. Once you hire more and more people, you place yourself on the path to process debt. Every link you add to the operational chain is a) extra capacity that allows you to grow and be profitable and b) an increase in human process latency. More people = more discussion = more meetings. As we know from software engineering, the more hops a network request does, the higher the latency.

While technical debt refers to the future costs and complications arising from quick, suboptimal coding decisions made under pressure, process debt pertains to the organizational and workflow inefficiencies that accumulate over time. Slowly. They’re not immediate. It’s even hard to pinpoint where the organizational debt started. Both forms of debt share a common trait: they represent a deferred cost or problem that will need addressing later, usually at a higher expense.

I want to emphasize that EVERY company has processed debts. Over time, it escalates from minor irritations to significant roadblocks as a company scales. That’s why there’s a constant battle in every organization to make sure the processes stay on track.

Bloated Processes

If there’s a manual deployment - that’s technical debt. If there’s a feature development workflow where one person fills out a change request form in an intranet tool, then sends that out per email to another coworker, who prints it out, signs it, and then inputs the data manually into another tool before giving the paper to some other person to sign — that’s compliance process debt.

Let’s make this relatable to all the software developers. Have you ever been stuck in traffic? That's what dealing with bloated processes in software engineering feels like. You know where you need to go, but you're just not moving. You’re eager, you want to be moving, like on the autobahn in Germany. But you’re not. It's the endless meetings that seem to discuss more meetings. It's the convoluted approval process where a simple task requires the digital equivalent of a royal decree.

Then you've got your classic over-engineered code review process, where every line of code is scrutinized by multiple people like it's a matter of national security. (It might be, but probably isn’t) It's a circus of back-and-forth, endless meetings, and bureaucratic hoops that turn what should be a sprint into a never-ending relay race.

Enjoyed the read? Join a growing community of more than 2,500 (🤯) future CTOs.

Then there's the deployment process, layered with so many checks and approvals that you’re practically ready to retire by the time you push something to production. Let's not forget the "agile" stand-ups that have morphed into drawn-out sit-downs, where the coffee gets cold as everyone drones on about what they did yesterday. I have a theory that all these things start as good ideas. They just become bloated and bureaucratic over time.

Then there's the classic: documentation for the sake of documentation. Pages upon pages of reports that nobody reads but everyone insists are essential. It's like building a skyscraper for a lemonade stand. These processes lead to delays, sap productivity, and create a general sense of trudging through mud.

🏄 I’m sure while reading this, you remembered some of your examples of bloated processes that you had to deal with over the years.

These processes grow so complex and intertwined over time that you need a map, compass, and a local guide to get through them. They've lost sight of their original purpose – to streamline work – and have become self-serving.

Growth duality

As you remember, I mentioned above — hiring people before finding work for those people is not optimal. That’s not bad per se. You just need to be careful not to fall into the trap of inventing unnecessary processes for those people. Hiring people before really knowing what they’re going to be doing is one of the problems you will have to solve in a growing startup.

So, imagine you’re a technical co-founder, and you’re growing super fast. Essentially, your job is to figure out what you don’t know, and you do that by hiring people who are way smarter than you and can help you figure it out. The goal of such “abstract” hires is to give you clarity and focus and then, hopefully, help you figure out where you currently lack efficiency as a company and find people to fill those roles.

Scaling a software team is like upgrading from a family sedan to a bus. You can't steer it the same way. Processes that worked for you when you were a tight-knit group of 3 are a disaster at 50. You’re growing from a scrappy bunch of rebels to a structured battalion — you need to think differently. What used to be quick decisions in a chat now require a formal meeting and maybe even a follow-up email to clarify everything discussed so everyone is on the same page.

🏄 You see the irony, right? Processes meant to keep us on track end up derailing us. You can’t have a group chat of 100 people making decisions left and right. That lean methodology we all loved suddenly becomes this bloated beast that no one knows how to tame.

As a startup, our biggest asset is agility – our ability to pivot faster than a caffeinated ballerina. But that comes at the cost of failure, non-compliance, and other nasty things. Startups can allow themselves to make mistakes. The more you grow, the more costly those mistakes get. The solution is the formalized, strict, rigid processes that get set up. But even if complicated processes are a must, things get disproportionally bad as middle management comes into play.

Middle management plays a pivotal role in either mitigating or exacerbating process debt. Their decisions impact how processes are implemented and adhered to within teams. Effective middle management keeps things flowing. Poor middle management can destroy companies; it’s called death by middle management, and Nokia is a good example.

Growing too fast

Here are the things you should remember whenever you’re growing too fast.

Focusing on way too many things at the same time. So much multitasking that nothing moves. You might think it’s good and allows you to do more, but it hinders the business if everyone is doing everything. Just as individuals struggle to juggle disparate tasks simultaneously, a fast-growing business attempting to handle everything at once risks inefficiency and lost focus. The mirage of accomplishing more masks the reality – a chaotic, stressed atmosphere where not much was accomplished. It’s similar to “doing work” vs. “acting busy.” Multitasking, in this case, is “acting busy.”

Great representation of tech debt, but also relates to building processes over time

The next bad thing that can make or break your fast growth is inadequate onboarding for new hires. As discussed, rapid hiring can lead to insufficient onboarding, leaving new employees unclear about their roles and company culture. Rapid growth also means roles are not clearly defined — there might be overlaps or gaps in responsibilities, making everything more confusing. If you don’t give people clarity, they will create it for themselves, and it probably will not match what you want for the company.

The next thing you should forget — is over-reliance on informal communication: Chatting up in Whatsapp and making decisions on the fly is a thing of the past. More people = more need for clarity. I agree that it worked wonders for you and your cofounder when building the MVP. Still, as you scale, reliance on informal communication channels becomes unsustainable, leading to miscommunications and wrong things being built.

As a CTO, you want every team to work similarly, using the same IDEs and development workflows. Why? Without standardized processes, different teams develop their own ways of working — different linters, deployment scripts, and DevOps tools—which brings chaos. This is process debt, not technical debt, as working in the same, standardized way is a management problem, not a coding one.

You should also focus on reducing your bus factor. Startups often neglect to develop strong middle management, which becomes a bottleneck as the organization grows. You cannot make all the decisions alone. Stuff needs to happen without you looking over people’s shoulders. And you do that by building your own middle management that can make decisions autonomously.

Start striving for a good work-life balance culture that you read so much about. The fast-paced environment of a growing startup often leads to employee burnout, which is also in itself a process debt. Flawed processes lead to people doing unnecessary and unfulfilling work. You have the power to reduce that.

Being too corporate

On the other side of the spectrum are things to keep an eye out for when you’ve joined an established company founded in the 90’s.

It’s time to dive deep into why things happen the way they do. There’s highly likely excessive bureaucracy happening all around you. Too many layers of approval slowing down decision-making. A tool for control that eventually became a tool of frustration and leverage. Mapping out the current status quo and understanding the why behind the workflow is the first step to figuring out how to simplify it.

Moving away from synchronous to asynchronous. This is a process debt that I personally don’t like — the over-reliance on meetings. I think this highly correlates with the “acting busy” debt. What could’ve been an email with several people in CC gets promoted to a 30-minute monologue to fill in the working time — an extra meeting with no inherent value.

One of the blockers that you will encounter when investigating organizational debts is legacy systems. The tools have grown over time to adapt to the processes, and not vice versa. Organizations often resist updating these systems due to a) cost concerns and b) change resistance. When people get used to doing things one way, it’s tough to convince them to switch, even if you prove it will be twice as efficient.

Informational Silos in the company. This usually happens when teams operate in isolation and don’t talk across departments. This isolation can lead to double work, miscommunication, and missed opportunities. I once saw a situation where because of misalignment and silos, two separate teams built the same product independently. That was a fun meeting to be in. Breaking down these silos is essential to making sure everyone at the company moves in the same direction.

The next one usually arises from companies with a culture emphasizing blaming when something goes wrong. And what happens when people are afraid to make mistakes? You guessed it, bad things get swept under the rug. This is the Ostrich Problem.

I once found myself working on a project where we definitely were not going to hit our deadlines. I was invited as the tech lead to a meeting with my manager and the C-Level representatives. Before the meeting, my manager told me to lie that we were on track; otherwise, it would look bad for him that we were not meeting our target estimates. During the conversation, when one of the executives asked me point-blank about the likelihood of us not hitting the deadlines — I told him straight that this was definitely a possibility and we should already start making arrangements for this scenario. After the meeting, my direct manager was fuming that I made him look bad, as if “we couldn’t deliver.” It was important for him to seem successful to the management rather than dealing with the realities of us missing the target and working transparently.

🏄 This dance with delusion is not limited to middle management. It can get as bad as the whole company leadership being afraid of showing how bad things really are and spinning up lies to seem successful to the board and investors, but that’s called fraud.

Last but not least, established companies = comfy jobs = no desire to change the status quo = no innovation. Employees who WANT to disrupt the internal status quo get pushed down or pushed out. This aversion to change, coupled with resistance to disrupt the status quo, creates organizational debt. When employees are not given the possibility to influence their processes and the autonomy to make decisions — they leave. Only those accepting the status quo remain, resulting in a bundle of “comfy” ineffective processes.

Conclusion

So here we are. The goal of this essay was to showcase that not only IT suffers from technical debt, but also the management suffers from organizational debt. Engineers' resistance to nonsensical processes highlights the need for transparent and well-communicated workflows. Leaders, such as yourself, must focus on refining and effectively communicating these processes to ensure they add value first and are comprehensible second, not vice versa. As a technical manager or a CTO, you’ll have to work with other C-level executives to ensure the direction is clear and the processes make sense.

Just as code requires refactoring, organizational structures and workflows also need regular evaluation and adjustment to maintain efficiency.

Other Newsletter Issues:

  • Quinn

    It’s funny how we all go through the same challenges with processes in our jobs, right? As the team grows, things can get a bit chaotic, and finding the balance between agility and structure is key. It’s a constant battle to keep things efficient and streamlined, but it’s worth the effort. I’ve definitely had my fair share of experiences with process debt, but it’s all about learning and growing along the way.

  • Elijah Z.

    I witnessed firsthand how quickly “streamlining” turned into a bureaucratic swamp. We started off agile, able to pivot on a dime, but as we scaled, things started to get blocked, delayed with overwhelmed people, layering approval upon approval for the most trivial changes. Our attempts to dodge technical debt by overcomplicating processes ended up being the wrong choice.

  • QuantumHacker

    Cutting down on process bloat felt like decluttering my digital workspace. Started auditing our workflows bi-monthly, focusing on what really slowed us down. Turns out, half the steps we thought were necessary could be scrapped or automated. It took some convincing, but now everyone’s on board, and projects move quicker than before.

  • Anonymous

    Thanks for the good read. I’ve been in corporate dev world for a long time now. The key takeaway for me is the importance of maintaining agility and simplicity in processes, sadly not always possible. As the team grows, this very lack of structure can become a hindrance, leading to inefficiencies and confusion. I’ve had the same experiences listed above. great resource for HR professionals, people managers, and anyone involved in organizational development and culture.

  • Reb

    It’s amusing how these young companies think they can defy the basics of organizational structure. Chaos isn’t a business model. Maybe it’s time to grow up and put some actual systems in place? Or keep playing startup until you crash and burn. People should uderstand that.

  • John

    Insightful read on process debt. It’s a real challenge, especially in fast-growing companies. We often get so caught up in scaling that we forget the importance of efficient processes. Let’s streamline our workflows for better productivity.

  • Anonymous

    Oh, look, another article preaching about process debt. As if we didn’t know! We’re just too busy coding to care about some ‘processes.’ But hey, let’s all pretend we’ll fix it ‘someday.’ Meanwhile, let’s add another layer of bureaucracy to each Pull request and code reviews.