Building a Health Check That Means Something

The worst outage I ever sat through lasted four hours, and for the first two of them every dashboard we owned was green. The load balancer was happily routing traffic to a payment service that had lost its database connection pool twenty minutes earlier. Every instance reported healthy. Every instance was lying. The health check returned 200 because all it ever did was confirm the web server could answer a socket. It could. It just couldn't do anything useful once it had.

That outage cost us real money and a very uncomfortable call with a merchant whose settlement batch went nowhere. It also cured me forever of treating a health check as a checkbox. A health endpoint is a promise your service makes to the machinery around it, and if that promise is empty, you have built a smoke detector with no battery in it.

What a health check is actually for

Start with the audience, because a health check has three different readers and they want different things. Your orchestrator, whether that is Kubernetes or an ECS scheduler or an old-school load balancer, reads it to decide whether to send traffic to this instance or restart it. Your on-call engineer reads it at 3am to decide where to start looking. Your monitoring system reads it on a schedule to decide whether to wake that engineer up in the first place.

The mistake almost everyone makes early is writing one endpoint that tries to serve all three and serves none of them well. A liveness probe that runs a full dependency sweep will restart your pod every time a downstream service hiccups, turning a minor blip into a cascading restart storm. A readiness probe that only checks the process is alive will route payment traffic straight into a black hole. These are not the same question, and conflating them is how you get four green hours during a dead service.

Liveness is not readiness

I insist on this distinction on every team I run, because getting it wrong is expensive in a way that is hard to see until it bites. Liveness answers one question: is this process so broken that the only fix is a restart? That is almost always a narrow question. A deadlock, an unrecoverable out-of-memory state, a thread pool that is wedged. If liveness fails, the platform kills and replaces the instance, so it had better only fail when a restart genuinely helps.

Readiness answers a different question: right now, should this instance receive traffic? A service that has lost its database connection is very much alive. Restarting it will not conjure a database back into existence; it will just add cold-start latency to the pile. What it should do is fail readiness, drop out of the load balancer rotation, and keep breathing so it can rejoin the moment the dependency recovers.

If your liveness probe checks your database, a database blip becomes a fleet-wide restart. You have taken a recoverable problem and given it a hammer.

Deep checks that tell the truth

A readiness check earns its keep by exercising the things the service cannot function without. For a payment authorization service that means the primary database, the message broker it publishes to, and usually a cache or two. Not with a ping. A TCP ping to Postgres tells you the port is open, which is roughly as useful as knowing the lights are on in a building you cannot enter. Run a query. A cheap one, but a real one.

Here is the pattern I standardize on in .NET, using the built-in health check framework because it already handles the plumbing for aggregating results and tagging checks so you can expose different subsets to different probes.

services.AddHealthChecks()
    .AddCheck<LivenessCheck>("self", tags: new[] { "live" })
    .AddNpgSql(
        connectionString,
        healthQuery: "SELECT 1;",
        name: "postgres",
        tags: new[] { "ready" })
    .AddCheck<BrokerHealthCheck>("broker", tags: new[] { "ready" });

app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("live")
});

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready")
});

Two endpoints, one registration. The liveness route only ever looks at the self check, so a wobbling broker will never trigger a restart. The readiness route runs the real dependency queries. This split has saved me more grief than almost any other single piece of operational hygiene.

The timeout you forgot to set

The subtle killer in deep health checks is the one nobody thinks about until it happens: the check itself hangs. Your readiness probe runs a SELECT 1, the database is not down but is desperately slow, and the query sits there for thirty seconds. Meanwhile your probe has no timeout, so it also sits there for thirty seconds. Now your health endpoint is slower than your actual traffic, and the orchestrator, which does have a probe timeout, marks the instance unready for the wrong reason.

Every single dependency check needs its own aggressive timeout, shorter than the probe interval the platform uses to call it. I usually run health checks with a two-second budget per dependency when the probe fires every ten seconds. If a database cannot answer SELECT 1 in two seconds, that is a real signal, not a false alarm. Wrap each check in its own CancellationToken and treat a timeout as a failure, not an exception to be swallowed.

• Set a per-check timeout shorter than the platform probe interval.
• Never let a health check acquire the same connection pool your live traffic depends on, or a pool exhaustion turns your check into a liar.
• Cache expensive checks for a few seconds so a burst of probes cannot become its own load problem.
• Fail closed on unknown state, not open. If you cannot tell, assume not ready.

Do not check what a restart cannot fix

There is a temptation, once you have built a nice health check framework, to shove everything into it. Resist. I have seen readiness checks that verified a third-party fraud-scoring API was reachable. When that vendor had a bad afternoon, every instance across the fleet failed readiness simultaneously and the entire service went dark, even though it could have kept authorizing low-risk transactions with a degraded rule set.

The question to ask for every dependency is blunt: if this is down, is my service genuinely unable to do its job, or merely unable to do part of it? Hard dependencies belong in readiness. Soft dependencies belong in your metrics and your degradation logic, not in the switch that removes you from the load balancer. A payment service that refuses all traffic because an analytics sink is unreachable is a service that has confused its own vital organs with its jewelry.

Make the failure legible to a human

A health check that returns a bare 503 is technically correct and operationally useless. When the on-call engineer curls the endpoint at 3am, they should see which check failed and, ideally, why. The .NET framework lets you write a custom response writer that emits the individual check names, their status, and their duration. Spend the twenty minutes to build it. The version that just returns a status code will cost you an hour of blind poking every time it fires.

I want the readiness payload to read like a triage sheet: postgres healthy in 4ms, broker unhealthy, connection refused, cache degraded at 900ms. That last one matters too. A check that is technically passing but took 900 milliseconds is a warning shot. Expose the duration and you turn your health endpoint into an early-warning system instead of a binary that only speaks up once the building is already on fire.

Startup probes and the cold instance

New instances lie in the opposite direction: they report unready when they are simply not finished waking up. If your service warms a cache or runs migrations or establishes a connection pool on boot, a readiness probe that fires immediately will fail, and an impatient orchestrator will kill the instance before it ever gets to do its job. I once watched a service enter an infinite crash loop purely because its liveness probe had a shorter grace period than its startup time. It never lived long enough to become healthy.

The fix is a dedicated startup probe, separate from both liveness and readiness, with a generous failure threshold. It runs first, and only once it passes do the other two take over. Kubernetes has this as a first-class concept, and you should use it. If you are on something older, at least give your liveness probe an initial delay longer than your worst-case boot time. Measured, not guessed.

Test the thing that is supposed to catch failures

The last piece, and the one I have to nag people about, is that a health check is code, and untested code that runs during outages is a special kind of hazard. Write a test that stops your local database and asserts readiness goes unhealthy. Write one that leaves the database up and asserts it stays healthy. It feels almost too simple to bother with, until the day someone refactors the connection factory and the check silently starts passing against a mock that is always up.

Better still, exercise it in a game day. Kill a dependency in a staging environment and watch whether the instance actually drops out of rotation, whether traffic drains cleanly, whether it rejoins on recovery. A health check you have never seen fail on purpose is a health check you do not actually trust; you just hope it works. Hope is not an operational strategy, and in regulated payments it is not one your auditors will accept either.

Conclusion

Here is the thing I would staple to every code review if I could: a health check is not a formality you add to satisfy a template, but a model of what your service needs to be alive and what it needs to be useful, and those are two different lists. The moment you write it down honestly, you learn something about your own architecture, usually something uncomfortable. If you cannot clearly say what your service must have to do its job, your health check will be vague, and a vague health check is worse than none, because it hands you false confidence at the exact moment you can least afford it.