Understanding Backpressure in Streaming Systems

The first time backpressure bit me hard, it was 2 a.m. and a payment reconciliation pipeline had quietly fallen 90 minutes behind. Nothing had crashed. No alerts had fired. The producer kept happily reading settlement files and shoving events onto a queue, and the consumer that wrote them to the ledger store was doing maybe a third of the throughput it needed. The queue had a retention window measured in hours, and we were burning through it. By the time the on-call engineer noticed, we were one bad afternoon away from silently dropping financial events.

That is the thing about backpressure. It is not a feature you add. It is a property your system either has or doesn't, and if it doesn't, you find out at the worst possible moment. So let me lay out how I actually think about it, and where I've seen teams get it wrong.

What backpressure actually is

Backpressure is the signal a slow consumer sends back up the chain to tell a fast producer to slow down. That's it. When your downstream can only handle 5,000 events a second and your upstream wants to send 20,000, something has to give. Either the producer slows down, the data buffers somewhere, or you throw work away. Those are the only three options, and pretending there's a fourth is how you end up 90 minutes behind at 2 a.m.

Most people first meet backpressure through TCP without realizing it. When a receiver's window fills up, the sender stops transmitting. The whole protocol is built around the receiver dictating pace. The trouble starts the moment we build application-level pipelines on top of that and forget to carry the same idea forward. We stick a queue in the middle, mentally file it under "handled", and move on. A queue is not backpressure. A queue is a place to hide the absence of backpressure until it overflows.

The unbounded buffer trap

The single most common mistake I see is the unbounded buffer. Someone writes code that reads from a source and pushes into an in-memory collection, or a channel with no capacity limit, or a message broker configured with effectively infinite retention. It works beautifully in every test, because tests never sustain a load imbalance for an hour. Then production hits a slow dependency and the buffer grows until the process runs out of memory and the orchestrator kills it. Now you've converted a manageable slowdown into a hard outage, and possibly lost whatever was in memory.

I have a rule I'm fairly stubborn about: every buffer in a system must have a bound, and every bound must have a defined behavior when it's hit. If you can't tell me what happens when the buffer fills, you don't have a design, you have a hope. The bound forces the conversation nobody wants to have, which is what to do when you genuinely can't keep up.

An unbounded queue doesn't remove backpressure. It just moves the moment of failure from "slightly slow" to "completely down", and delays it until you've forgotten the code exists.

The three honest responses to overload

When a consumer can't keep pace, you get exactly three honest responses, and mature systems pick deliberately among them:

• Slow the producer down. Block or pause reading until the consumer catches up. This is the cleanest option when the producer can tolerate being slowed, like reading from a file or a replayable log.
• Shed load. Drop events, sample them, or reject requests with a clear signal like HTTP 429. Painful, but sometimes correct, especially for telemetry where losing 2% of metrics beats losing the whole service.
• Spill and buffer with a hard limit. Absorb bursts in a bounded buffer, on disk if needed, and only then fall back to slowing or shedding. This handles spikes without pretending you can handle infinite load.

The mistake is picking one globally and applying it everywhere. Telemetry should shed. A payment event stream must never silently drop, so it slows or spills. The right answer is per-stream and depends entirely on the cost of a lost message versus the cost of a delay. In fintech that calculus is usually lopsided: latency is annoying, data loss is a regulatory incident.

Pull versus push, and why it matters

Push-based systems are where backpressure goes to die. The producer decides the rate and fires events at the consumer whether it's ready or not. It feels natural and it's easy to code, but the consumer has no lever. All it can do is buffer or drop. Every push system that survives real load eventually grows some clumsy out-of-band flow control, bolted on afterward, and it's always worse than if pull had been the model from the start.

Pull-based systems invert the control. The consumer asks for the next batch when it's ready, so the rate is naturally governed by how fast the consumer can actually go. Kafka's consumer model is pull for exactly this reason: the broker doesn't shove records at you, you poll for them. Reactive Streams codified this with the request(n) protocol, where a subscriber explicitly asks for n elements and never receives more than it requested. It's more ceremony up front, and I'll take that trade every time over debugging a memory leak that only appears under sustained load.

Bounded channels in .NET

Most of my recent work has been on .NET, and the primitive I reach for constantly is System.Threading.Channels. A bounded channel gives you backpressure almost for free, because writing to a full channel awaits until space opens up. That await is the backpressure. The producer's own loop slows down without any explicit coordination code.

var channel = Channel.CreateBounded<PaymentEvent>(new BoundedChannelOptions(10_000)
{
    FullMode = BoundedChannelFullMode.Wait,
    SingleReader = true,
    SingleWriter = false
});

// Producer: WriteAsync blocks when the buffer is full.
// That await IS the backpressure signal.
await foreach (var evt in ReadSettlementFilesAsync(ct))
{
    await channel.Writer.WriteAsync(evt, ct);
}
channel.Writer.Complete();

// Consumer pulls at its own sustainable pace.
await foreach (var evt in channel.Reader.ReadAllAsync(ct))
{
    await _ledger.AppendAsync(evt, ct);
}

The important line is FullMode. The default choices are Wait, DropOldest, DropNewest, and DropWrite. Wait gives you real backpressure. The Drop modes give you load shedding. Both are legitimate, but you have to choose on purpose. I've reviewed code where someone picked DropWrite because it made a flaky test go green, and shipped it into a path that was supposed to be lossless. The channel silently discarded events under load and nobody knew until an auditor asked why a batch total didn't reconcile.

Backpressure across service boundaries

In-process backpressure is the easy case. The hard case is when your producer and consumer are separate services with a broker between them. Now the await trick doesn't work, because the broker gladly accepts everything the producer sends. The producer feels no pain. The pain accumulates as consumer lag, and lag is invisible unless you go looking for it.

This is where consumer lag becomes the single most important metric you're not watching closely enough. For a Kafka-style log, lag is the difference between the latest offset and the consumer's committed offset, per partition. If that number trends up and never comes back down, you are losing the race, full stop. I want lag on a dashboard, alerted on both absolute value and rate of change, because a slow steady climb is often more dangerous than a spike. The spike wakes someone up. The climb of 400 messages a minute waits until Friday night to catch fire.

Across HTTP boundaries, the equivalent of backpressure is honest status codes and client cooperation. A service that's overloaded should return 429 with a Retry-After header, and clients should actually respect it with jittered exponential backoff. A distressing share of production incidents comes from clients that treat 429 as "try again immediately, harder", turning a mild overload into a self-inflicted denial of service against your own infrastructure.

Why unhandled overload cascades

Backpressure that isn't handled doesn't stay local. It propagates. A slow database makes the service in front of it slow, which fills that service's request queue, which makes its callers time out and retry, which multiplies the load, which makes the database even slower. This is the retry storm, and it's how a 20% latency bump on one node metastasizes into a full platform outage in about four minutes. I've watched it happen on a Friday deploy and it is genuinely frightening how fast the graph goes vertical.

The defenses are unglamorous and they work. Bounded queues with fast rejection instead of slow timeouts. Circuit breakers that stop hammering a struggling dependency. Load shedding at the edge, where you reject cheaply before you've spent resources you can't recover. Timeouts that are actually shorter than your client's patience, so you fail before the retry arrives. None of this is exciting. All of it is the difference between a bad ten minutes and an incident review with your compliance team in the room.

You have to test the imbalance on purpose

Here's the uncomfortable truth: backpressure behavior almost never shows up in normal testing, because normal testing runs balanced. Producer and consumer keep pace, the buffer stays shallow, everything's green. The failure modes only appear when you deliberately create sustained imbalance, and that's a test people rarely write because it feels artificial.

So write it anyway. Pin the consumer to half the producer's rate and run it for ten minutes, not ten seconds. Watch memory, watch lag, watch what actually happens when the bound is hit. Does the producer slow gracefully, or does memory climb until the pod dies? Does load shedding kick in where you expected, and drop the right things? I once found a "bounded" queue that was bounded in configuration but the value was parsed from an environment variable that wasn't set in staging, so it silently defaulted to unbounded. We only caught it because someone ran the sustained-imbalance test. That test has paid for itself many times over since.

Conclusion

If I could get one idea to stick, it's this: treat every point where fast meets slow as a decision, not an accident. Somebody, somewhere, has already decided what your system does under overload. The only question is whether that person was you, deliberately, at design time, or whether it was the operating system's OOM killer at 2 a.m., picking for you. I would much rather it be me, awake, with a whiteboard, than a signal handler with no opinion about which payment events matter. Draw the boundaries, bound the buffers, and make the slow path fail loudly and early. The systems that survive aren't the ones that never get overloaded. They're the ones that were told, in advance, exactly how to behave when they do.