Skip to main content
The Cost of Keeping a Broken Process Alive — Anselm Fowel
Leadership

The Cost of Keeping a Broken Process Alive

9 min read
340 views
Share:

Two years ago I asked a team why our monthly reconciliation still took four people three full days. The answer came back instantly, almost proudly: "That's just how recon works here." Nobody could tell me who designed it, when, or why it needed a shared spreadsheet with 62 tabs and a manual sign-off from someone who had left the company in 2019.

The Cost of Keeping a Broken Process Alive
The Cost of Keeping a Broken Process Alive

That process wasn't malicious. It wasn't even obviously broken from the inside. It was just old, and everyone had learned to route around its worst parts. But it was costing us roughly 40 person-days a month, and it had been quietly doing that for years while we all told ourselves we were too busy to fix it. This post is about what that kind of process actually costs, and why the cost is almost always higher than the fix.

The Invisible Invoice

The reason broken processes survive is that nobody ever gets the bill in one piece. If a vendor sent us an invoice for 40 person-days a month with the line item "reconciliation you could have automated," we'd have escalated it to the board. Instead the cost arrives in fragments: a few hours here, a frustrated engineer there, a payment that settles a day late, a customer who churns for reasons we never quite trace back to the source.

Finance is unusually good at hiding this. Every step in a bad workflow can be justified locally. The spreadsheet exists because the report needed a field the core system didn't expose. The manual sign-off exists because of an incident in 2017. The overnight batch exists because that was the only window the mainframe had free in 2011. Each decision was reasonable. The sum of them is absurd, and the absurdity is invisible until someone forces all the fragments onto a single page.

A broken process rarely fails loudly. It taxes you quietly, in a currency you're not measuring, until the day it fails at the worst possible moment.

Why Nobody Kills It

Here is the uncomfortable part. Broken processes usually survive because someone benefits from them, or at least feels safer with them alive. The person who runs the 62-tab spreadsheet knows it better than anyone, and that knowledge is a kind of job security. The manager who introduced the extra approval step gets to point at it whenever an auditor asks about controls. Killing the process feels like taking something away from real people who did nothing wrong.

There's also plain fear. In regulated payments, "we've always done it this way" is a defensible answer to a regulator, and "we changed it last quarter and something went sideways" is not. So the incentive is to leave working-ish things alone. I understand that instinct. I've felt it. But there's a difference between a control that earns its cost and a ritual we perform because we're afraid of what happens if we stop.

The Day It Actually Broke

Our recon process finally forced the issue on a Friday, which is of course when these things happen. The one person who understood the reconciliation between our ledger and a partner bank's statement was out sick. A settlement file arrived in a slightly different format, one extra column, and the spreadsheet's formulas silently shifted every value one cell to the left. Nobody noticed for six hours because the totals still looked plausible.

We ended up over-reporting a settlement position by a low seven-figure sum for most of a day. No money actually moved incorrectly, thank God, but we had to disclose the control gap, spend a week reconstructing what happened, and explain to a partner why our numbers wobbled. The direct clean-up cost was maybe 15 person-days and a very awkward call. The indirect cost was that a partner who used to trust our operations now asked for extra reporting for the next year.

The lesson wasn't "spreadsheets are bad." Spreadsheets are fine. The lesson was that we had built a load-bearing wall out of something no one had inspected in years, and we only found out when it cracked.

The Competence Trap

The most dangerous broken processes are the ones staffed by genuinely excellent people. When your best analyst can wrestle a terrible workflow into producing correct numbers month after month, you get all the output and none of the pain signal. The process looks healthy on every dashboard. It's only fragile in the exact place you can't see: the single human holding it together with skill and overtime.

I've come to treat individual heroics as a warning light, not a cause for celebration. If a report only comes out clean because Priya stays late on the last business day of every month, then the process has a bus factor of one and I have a problem I'm choosing not to look at. Rewarding the heroics feels good and makes the underlying rot worse, because now the hero is incentivized to keep the mess exactly as it is.

Enjoying this article?

Get more like it in your inbox — practical engineering leadership, fintech, and AI. No spam, unsubscribe anytime.

  • If it only works when a specific person is available, it doesn't work. It's a person doing a job that happens to look like a process.
  • If nobody can explain why a step exists, that step is a liability until proven otherwise.
  • If the documentation is "ask Priya," you don't have documentation, you have a hostage situation you're both too polite to name.
  • If a step's whole purpose is to catch errors created by an earlier step, consider deleting both.

What the Cost Actually Adds Up To

Let me be concrete about the categories, because "it's expensive" is easy to nod at and easy to ignore. The obvious cost is labor: those 40 person-days a month were roughly a third of a headcount we could have redeployed to work customers actually paid us for. At a loaded cost, that's real money, six figures a year for one process.

The larger cost is optionality. Every hour spent feeding a broken process is an hour not spent on the thing that would move the business. While four people babysat reconciliation, we were losing deals to a competitor whose onboarding took two days instead of our nine. I can't prove those two facts are linked with a straight line, but I know where those four people's attention would have gone if it were free, and it wasn't feeding a spreadsheet.

Then there's the compounding cost, the one that scares me most. Broken processes breed. The 62-tab spreadsheet existed because an upstream system wouldn't expose a field, and three other workarounds had grown around that same gap. Fix the process without fixing the root cause and the workarounds regrow within a quarter. You don't inherit a broken process; you inherit an ecosystem of them, each feeding the next.

The Cost of Fixing It, Too

I want to be fair here, because there's a lazy version of this argument that says all old processes are dead weight and any change is progress. That's how you get a CTO who rips out a boring, working system to install a shiny one and creates six new problems. Change has a cost too, and in regulated finance that cost includes risk that's genuinely hard to price.

When we finally automated recon, the migration itself took a quarter, introduced two bugs we had to catch in parallel-running before cutover, and required a control review I would not wish on anyone. For about ten weeks, fixing the broken process was more expensive and more nerve-wracking than living with it. That's normal. The honest comparison isn't "broken versus perfect," it's "the known ongoing tax versus a lumpy, uncomfortable, one-time cost with real execution risk." You have to actually run both numbers, not just assume the new thing pays for itself.

How to Tell a Tax From a Control

The distinction I keep coming back to is whether a step buys you something you'd actually pay for if the price were visible. A control that has caught real errors, that a regulator specifically expects, that reduces a risk you can name and size, earns its cost. A step that exists because of a fear no one can articulate, or an incident so old the underlying system has been replaced twice since, is a tax dressed up as a control.

The test I use is blunt. I ask the team to tell me the last time a given step caught something, and what would happen if we removed it tomorrow. If the answer is a specific save with a date, keep it and maybe strengthen it. If the answer is a vague "well, you never know" and a lot of shifting in seats, we've found a candidate. Not an automatic deletion, but a candidate. The point isn't to be reckless. The point is to stop granting old steps automatic tenure.

Making the Cost Visible

None of this works if the cost stays fragmented, so the real job of a leader here is arithmetic and courage in roughly equal measure. Once a quarter I now make someone put a single number on our worst manual processes: person-hours per month, times loaded cost, times twelve. It's a rough number. People argue about it. That argument is the entire value, because it's the first time the fragments have ever sat on one line.

The first time we did this for reconciliation, the number came out to about 170,000 a year in labor alone, before you counted the Friday incident. That number did in one meeting what two years of my complaining had failed to do. Nobody defends a 170,000 spreadsheet out loud. They'll defend it one justified step at a time forever, but not once it's a single figure with a currency symbol in front of it.

Anselm Fowel, CTO and fintech architect
Anselm Fowel — CTO & fintech architect

Conclusion

The thing I wish someone had told me earlier is that a broken process is not a neutral inheritance you're allowed to ignore. Every month you keep it alive, you're making a purchase, whether or not you meant to. The only question is whether you're the one deciding to buy it, or whether you've handed that decision to a spreadsheet and a tired analyst and a fear from an incident nobody remembers.

So find the one you've been walking past for a year, the one everyone shrugs at, and put a real number on it before the end of the quarter. Not to fix it tomorrow, necessarily. Just to stop pretending it's free. The moment the invoice is visible, the decision gets a lot easier, and it usually turns out you were paying far more than you thought for the privilege of not deciding.

Enjoyed this article? Share it with others!

Share:

Get new posts in your inbox

Occasional, practical notes on engineering leadership, fintech, and building with AI. No spam, unsubscribe anytime.

Comments (0)

Leave a Comment

Comments are moderated and will appear after review.

No comments yet. Be the first to comment!

About the author

Anselm Fowel

Anselm Fowel

Chief Technology Officer & fintech architect. 16+ years leading engineering across AlliancePay, Mondu, Transalliance, Global Accelerex, and Fidelity Bank — writing here about engineering leadership, fintech architecture, and AI in production.

Read next