Building a Dead Letter Queue That Works

A dead letter queue is where messages go to be forgotten. That is the dirty secret nobody tells you when they draw the tidy architecture diagram with the little box labeled "DLQ" off to the side. In theory it is a safety net for messages that fail processing. In practice, on most teams I have joined, it is a graveyard nobody visits, filling up quietly until the day it causes an incident of its own.

I have opinions about this because I have been burned by it. Early in my career at a payments processor we discovered a DLQ holding eleven thousand settlement messages that had been silently failing for six weeks. Nobody had looked. There was no alarm. The reconciliation team found it before we did, which is the worst possible way to find anything. This post is about building a dead letter queue that earns its place, not one that just hands you a false sense of safety.

What a DLQ is actually for

Strip away the ceremony and a dead letter queue solves one problem: a message failed and you do not want to lose it, but you also do not want it to block everything behind it. In an ordered queue, a single poison message can wedge an entire partition. The consumer picks it up, throws, retries, throws again, and now nothing moves. Shunting the bad message aside lets the good ones keep flowing. That is the whole point.

What a DLQ is not for is retries. I see this conflation constantly. Teams route a message to the dead letter queue on the first transient failure, then build a cron job that drains it back into the main queue. Congratulations: you have invented a retry loop with extra steps and no backoff. Transient failures belong in a retry policy with exponential backoff and jitter, close to the point of failure. The DLQ is for what is left when retries have genuinely given up. Keep those two ideas separate or you will confuse yourself at two in the morning during an incident.

Poison messages versus downstream outages

The single most useful distinction I make when designing failure handling is between a poison message and a downstream outage, because they demand opposite responses. A poison message is bad in itself. Malformed JSON, a null field a downstream call can never accept, a schema version your consumer was never taught to read. No amount of retrying will fix it. Retrying is actively harmful because you are burning throughput on something that will never succeed.

A downstream outage is the opposite. The message is perfectly good; your database is down, or a payment rail is timing out, or a third party is rate-limiting you. Here, dead-lettering is the wrong move. If you shovel a good message into the DLQ because the database blipped for ninety seconds, you now have to replay it manually later, and you have turned a self-healing situation into an operational chore. The trap is that both cases surface as an exception in the same catch block. Telling them apart is the actual engineering work.

The question is never "did this fail?" It is "will this fail again if I try it in five minutes?" If yes, retry. If no, dead-letter. Everything else is detail.

Deciding when to give up

So how do you decide? I lean heavily on exception typing. In .NET this means catching specific exception types rather than a bare catch, and mapping them to an explicit decision. A deserialization failure or a validation failure is non-retryable by definition. A SqlException with a transient error number, an HTTP 503, a socket timeout are retryable. Anything you have not classified, you treat as retryable up to a cap, because assuming a bug is transient is safer than throwing away a message you did not understand.

public FailureDecision Classify(Exception ex) =>
    ex switch
    {
        JsonException            => FailureDecision.DeadLetter,   // poison, never valid
        ValidationException      => FailureDecision.DeadLetter,   // business rule, won't self-heal
        SqlException { Number: 4060 or 40197 or 40501 or 49918 }
                                 => FailureDecision.Retry,        // transient SQL
        HttpRequestException     => FailureDecision.Retry,        // downstream blip
        TaskCanceledException    => FailureDecision.Retry,        // timeout
        _                        => FailureDecision.Retry         // unknown: assume transient, but cap it
    };

The cap matters. An unknown exception retried forever is just a slower poison message. I set a maximum delivery count, usually five, and once a message crosses it regardless of classification, it goes to the DLQ. That way the classifier being wrong costs you a handful of wasted attempts, not an infinite loop. Get the shape of the decision right and the exact numbers are easy to tune later.

The metadata that makes it usable

A message sitting in a dead letter queue with no context is nearly useless. When someone finally opens it, the first thing they ask is: why is this here, when did it fail, and what did it throw? If that information is not attached to the message, they have to correlate logs by timestamp, and at that point most people give up. So I enrich every dead-lettered message before it lands.

The specific fields I attach, learned mostly from not having them:

• The original exception type and message, and the stack trace if it fits.
• The delivery count at time of death, so you can see whether it failed once or fifty times.
• A timestamp of first failure and last failure, which tells you if it is a burst or a slow bleed.
• The consumer version or deployment SHA that gave up on it, because "it started after the Tuesday deploy" is half your investigation done.
• A correlation ID that ties back to the originating request, so support can answer a customer without a forensic dig.

None of this is clever. It is just discipline applied at the moment of failure, when you still have all the context in hand. Five minutes later, in a separate replay process, that context is gone and you cannot reconstruct it. Capture it while it is warm.

Making someone actually look

Here is the rule I now enforce on every team: a dead letter queue with no alarm on it does not exist. It is worse than not existing, because it lets everyone believe failures are being handled when they are quietly piling up. The eleven-thousand-message graveyard I mentioned happened precisely because the queue had no depth alarm and no age alarm. It was invisible until it was a crisis.

I alarm on two things, not one. Depth is obvious: if the DLQ has more than N messages, page someone. But depth alone misses the slow bleed, the one message an hour that never trips a count threshold but represents a real broken code path. So I also alarm on age: if the oldest message in the DLQ is older than, say, fifteen minutes, that is a signal that nobody is draining it. Age catches the trickle that depth ignores. Together they are hard to sneak past.

And the alarm has to route to a human with the authority to act, not to a channel that gets muted by Thursday. I have watched a DLQ alert fire into a Slack channel that had been muted by the entire team because it was too noisy. The queue was doing its job perfectly and telling us clearly. We had trained ourselves not to listen. An alert nobody reads is decoration.

Replay without making it worse

Eventually you fix the bug and you need to replay the messages. This is where good intentions cause outages. Someone writes a quick script that reads the entire DLQ and slams every message back onto the main queue as fast as the loop runs. If there are forty thousand messages and the reason they piled up was a downstream that is only just back on its feet, you have just knocked it over again. I have done this. It is embarrassing.

Replay needs to be a deliberate, throttled, observable operation. Rate-limit it to something the downstream can absorb. Make it idempotent, or at least make the downstream idempotent, because a replayed payment instruction that was actually processed the first time is a real financial event, not a log line. And replay in batches you can stop. I want a big red button that halts the replay mid-flight when the dashboard starts looking wrong, not a fire-and-forget script that I have to race against with a kill command.

One more thing that saved me once: replay to a shadow or staging consumer first when the messages are old and you are not certain the bug is fully fixed. Watching a hundred messages succeed in a safe environment before you touch production is cheap insurance. It feels slow. It is much faster than a second incident.

What I got wrong the first few times

My first serious DLQ design treated it as pure infrastructure, a config setting on the queue and nothing more. Set max delivery count to five, point it at a dead letter target, done. What I missed is that a dead letter queue is a product surface, not a config flag. Someone operates it. That person needs tooling, visibility, and a runbook, and if you have not built those, you have not built a DLQ, you have built a place for problems to hide.

The second thing I got wrong was granularity. I used one big shared DLQ for a service that handled six different message types. When it filled, I had no idea which flow was broken without inspecting every message by hand. Now I lean toward a DLQ per logical message type, or at minimum a message-type tag I can filter and alarm on independently. A payment-capture failure and an email-notification failure are not the same severity, and they should not share an alarm threshold or wake the same person. Splitting them cost almost nothing and made triage obvious.

The numbers that matter

If I inherit a system and want to judge whether its dead letter handling is healthy, I look at a small set of numbers rather than the architecture diagram. What fraction of messages end up dead-lettered? Anything above a fraction of a percent in steady state means the classifier is too aggressive or something is genuinely broken. How long do messages sit before someone acts? If the answer is measured in days, the alarms are not working. How often does replay itself fail? A high replay failure rate usually means the messages were dead-lettered for a reason nobody actually fixed.

These are boring metrics and that is the point. A healthy DLQ is boring. It is nearly empty most of the time, it fills briefly during a real incident, someone gets paged, the underlying bug gets fixed, and it drains under a throttled replay. If your dead letter queue is exciting, something upstream is wrong and the DLQ is just the place where the wrongness became visible.

Conclusion

The best compliment I can pay a dead letter queue is that I forget it exists for months at a time and then, on the one bad afternoon a quarter when I need it, it does exactly what I hoped: it caught the poison, it kept the good traffic moving, it told me clearly and loudly that something was wrong, and it let me put the messages back safely once I had fixed the cause. That is a lot of jobs for one little box in the diagram.

So if you take one thing from this: the queue itself is the easy part. Any broker gives you dead-lettering for free in an afternoon. The hard part, the part that determines whether it is a safety net or a graveyard, is everything around it. Classification, enrichment, alarms that reach a human, and a replay path that will not start a second fire. Build those, and you have a dead letter queue that works. Skip them, and you have a very tidy place to lose eleven thousand settlement messages.