Skip to main content
Designing for Graceful Degradation Under Load — Anselm Fowel
AI & Technology

Designing for Graceful Degradation Under Load

10 min read
739 views
Share:

At 9:14 on a Monday morning a few years back, one of our payment gateways started returning 502s. Not because it was down, but because a marketing team had launched a promotion nobody warned us about, and traffic tripled inside four minutes. The gateway itself was fine. What killed us was the settlement service behind it, which tried to write every single incoming request to a synchronous audit log, hit connection pool exhaustion, and dragged the whole checkout flow down with it. We lost roughly eleven minutes of transactions. The audit log, ironically, recorded almost none of it.

That morning taught me something I have never forgotten: systems do not fail gracefully by accident. Under load, software does exactly what it was built to do, and most software is quietly built to fall over hard. Graceful degradation is a design decision you make months before the spike, or it is a decision that gets made for you at the worst possible time.

What Degradation Actually Means

People throw around "graceful degradation" like it means "stays up." It does not. It means the system gets worse in a controlled, predictable way instead of collapsing all at once. A checkout page that disables its recommendations widget under load is degrading gracefully. A checkout page that shows a spinner for 40 seconds and then a stack trace is not.

The distinction I care about is between the parts of a system that must work and the parts that would be nice to have. In a payments flow, authorizing a card and recording the transaction must work. Fraud scoring, loyalty points, the "customers also bought" panel, the real-time analytics feed, these are all negotiable when the system is under pressure. The trouble is that in most codebases they are all wired together with the same synchronous calls and the same failure modes, so when one buckles they all buckle. You have to actually separate them before you can shed them.

The Tyranny of the Synchronous Call

Almost every ungraceful failure I have debugged traces back to a synchronous call that should not have been synchronous. Service A calls service B and waits. B calls C and waits. C is slow because a database index got dropped in a migration nobody reviewed, and now A, B, and C are all holding threads open waiting on each other. This is how a slow query in a peripheral service takes down a core one.

The fix is boring and it works: make the non-essential path asynchronous, and make the essential path defend itself with a timeout it can actually survive. If your fraud service normally answers in 80 milliseconds, do not give it a 30-second timeout "just in case." Give it 300 milliseconds and a fallback. When it blows past that, you make a decision with the information you have rather than holding a customer's card hostage while a downstream service catches its breath.

A timeout longer than your customer's patience is not a timeout. It is a way of failing slowly enough that you get blamed for it twice: once for the delay, and once for the eventual error.

Load Shedding Before You Need It

The counterintuitive part of surviving load is that you have to be willing to reject work on purpose. When a system is past its sustainable throughput, accepting more requests does not serve more customers, it just spreads the misery across everyone until nobody gets a response. It is better to serve 80 percent of traffic well and turn away 20 percent cleanly than to serve 100 percent of traffic badly.

Load shedding needs to happen at the edge, before the request has consumed real resources. A rejected request at the load balancer costs almost nothing. The same request rejected after it has opened a database transaction, allocated a fraud-check context, and queued an audit write has already done the damage. We learned to shed at the front door, with a fast queue-depth check, and to return a proper 503 with a Retry-After header rather than letting requests pile up in a thread pool that was never going to drain.

  • Shed the cheapest-to-reject, least-valuable traffic first: health-check spam, aggressive polling clients, and speculative prefetches.
  • Protect authenticated, in-progress transactions over brand-new sessions. A customer halfway through checkout is worth more than one who just landed.
  • Make the shed visible. A silent drop looks like a bug to the client; an explicit 503 with a retry hint looks like a system that knows what it is doing.

Circuit Breakers That Earn Their Keep

Circuit breakers are the pattern everyone knows and few configure correctly. The idea is simple: once a dependency has failed enough times, stop calling it for a while, fail fast, and periodically test whether it has recovered. The value is that you stop throwing threads and connections at something that is already drowning.

The mistake I see constantly is treating the breaker's thresholds as an afterthought. Trip after 50 percent of calls fail over a 60-second window and you have already spent a full minute degrading before the breaker does anything at all. In a payments context that minute is thousands of transactions. I prefer tighter windows and a breaker that opens on latency, not just errors, because a dependency that answers slowly is often more dangerous than one that fails outright. A clean failure you can route around; a slow success ties up the caller and hides the problem.

public async Task<AuthResult> AuthorizeAsync(Payment payment)
{
    if (_fraudBreaker.State == CircuitState.Open)
    {
        // Fraud service is out. Fall back to a conservative rule set
        // rather than blocking the payment entirely.
        return _conservativeRules.Evaluate(payment);
    }

    try
    {
        using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(300));
        var score = await _fraudClient.ScoreAsync(payment, cts.Token);
        _fraudBreaker.RecordSuccess();
        return AuthResult.From(score);
    }
    catch (Exception ex) when (ex is OperationCanceledException or HttpRequestException)
    {
        _fraudBreaker.RecordFailure();
        // Degrade, don't fail. A slightly stricter default beats a hard error.
        return _conservativeRules.Evaluate(payment);
    }
}

Notice the fallback is not "return success" and it is not "throw." It is a conservative decision the business has agreed to in advance. That agreement is the actual work. The code is easy; getting the risk team to sign off on what "degraded fraud checking" means on your busiest day is the hard part, and it needs to happen when everyone is calm, not at 9:14 on a Monday.

Enjoying this article?

Get more like it in your inbox — practical engineering leadership, fintech, and AI. No spam, unsubscribe anytime.

The Database Is Where You Die

Stateless services scale by adding boxes. Databases do not, at least not on the timescale of a traffic spike. Almost every serious outage I have been part of eventually bottomed out at the database: connection pools exhausted, a lock convoy on a hot row, a query that was fine at 500 rows and pathological at 5 million. The database is the shared resource everything else contends for, and it is usually the last thing anyone thinks to protect.

Two habits have saved me repeatedly. First, cap the connection pool per service and mean it, so a single misbehaving service cannot starve the others. Second, separate read paths from write paths so that a reporting query cannot compete with a settlement write for the same locks. When we moved analytics reads to a replica, our p99 write latency during peak dropped by more than half, and it had nothing to do with the writes getting faster. They just stopped waiting behind someone's dashboard refresh.

Then there is what you do when the write path itself is the bottleneck. This is where a durable queue earns its place. If the audit write from my opening story had been a message on a queue instead of a synchronous insert, the settlement service would have kept authorizing payments and the log would have caught up a few seconds later. The lesson is not that queues are magic. It is that not every write needs to be immediately consistent, and pretending otherwise is what turns a busy morning into an incident.

Caches and the Lies They Tell

A cache is a promise that the truth was correct a moment ago. Under load, that promise is often good enough, and being willing to serve slightly stale data is one of the cheapest forms of graceful degradation available. If your exchange rates are 90 seconds old during a spike, most of the business does not care. If your checkout is down, all of the business cares.

The trap is the cache stampede: a popular key expires, and a thousand requests all miss at the same instant and hammer the origin together. I have watched a cache expiry take down a service that ran fine for months, purely because the miss storm arrived faster than the database could answer. The defenses are well known but under-used: stagger your TTLs with a few seconds of jitter so a batch of keys does not all expire on the same tick, serve stale content while a single background request refreshes the value, and never let more than one caller regenerate a given key at once.

Testing the Failure, Not the Success

Here is my mild frustration with how most teams handle this: they load test the happy path and call it resilience testing. They spin up a load generator, ramp to expected peak, watch the graphs stay green, and ship. What they have proven is that the system works when everything works. That was never in doubt.

The test that matters is the one where a dependency is sick. Kill the fraud service mid-test and confirm the fallback fires and payments keep flowing. Add 500 milliseconds of latency to the database and watch whether your timeouts and breakers behave or whether the whole thing quietly backs up. Fault injection feels slightly terrifying the first time you run it against something resembling production, which is exactly why it is worth doing. You want to meet these failures on a Tuesday afternoon with the team watching, not alone at 3am. We run a controlled drill every quarter and do not tell the on-call engineer which dependency we are degrading, because the point is to find out whether the runbook is real. More than once, it was fiction.

Observability When You Are Half-Blind

Everything I have described assumes you can see what is happening. Under load, that assumption breaks in a specific and cruel way: the monitoring system is also under load. The metrics pipeline backs up, dashboards lag by two minutes, and the logs you desperately need are the ones getting dropped because the logging path is saturated. During that eleven-minute outage, our own dashboards were among the last things to update, so for the first several minutes we were flying on customer complaints and gut feeling.

Treat observability as a system that must itself degrade gracefully. Sample logs aggressively when volume spikes rather than dropping them randomly. Keep a small set of high-priority metrics on a separate, cheap path that stays alive when the rich telemetry drowns. And make sure at least one signal, even something as crude as a per-second success counter on the load balancer, is dead simple and always available. When the sophisticated tooling gives up, you want one honest number you can trust.

Anselm Fowel, CTO and fintech architect
Anselm Fowel — CTO & fintech architect

Conclusion

The thing nobody tells you is that graceful degradation is mostly a business conversation dressed up as an engineering one. The code for a circuit breaker, a timeout, a load shedder, a fallback rule, all of it is a weekend of work. Deciding what your product is allowed to do when it cannot do everything, that is the part that takes months and the part that actually saves you. What does checkout look like with fraud scoring degraded? Which customers do you turn away first? Whose dashboard is allowed to go stale? If you have not answered those questions before the spike, your system will answer them for you, and it will pick the worst possible answers. Have the uncomfortable conversation now, while the graphs are still green.

Enjoyed this article? Share it with others!

Share:

Get new posts in your inbox

Occasional, practical notes on engineering leadership, fintech, and building with AI. No spam, unsubscribe anytime.

Comments (0)

Leave a Comment

Comments are moderated and will appear after review.

No comments yet. Be the first to comment!

About the author

Anselm Fowel

Anselm Fowel

Chief Technology Officer & fintech architect. 16+ years leading engineering across AlliancePay, Mondu, Transalliance, Global Accelerex, and Fidelity Bank — writing here about engineering leadership, fintech architecture, and AI in production.

Read next