Building a Config System You Will Not Regret

The worst outage I have signed off on in the last five years started with a single environment variable. Someone flipped a payout batch size from 100 to 10000 in a config file, the change skipped review because config "isn't code," and at 2am the settlement job tried to build one enormous message that blew past our message broker's size limit. Payouts stalled for about forty minutes. No bug in the code. The code did exactly what it was told.

That is the thing about configuration: it is the part of your system you are most likely to change under pressure, with the least ceremony, and the least testing. So I have opinions about how to build a config system, and most of them come from having built bad ones first.

Config Is Code, Whether You Admit It or Not

Start here, because everything else follows from it. A configuration value that changes what your program does is code. It just happens to be written in YAML or stored in a database row instead of C#. Pretending otherwise is how you end up with a "quick config tweak" that takes down settlements. If a value can change behavior, it deserves the same respect as a function: version control, review, and a paper trail.

The counterargument I hear is that config exists precisely so you can change things without a deploy. True, and I like that property. But "without a deploy" should not mean "without accountability." Those are different axes. You can absolutely have a value that is editable at runtime and still recorded, reviewed, and reversible. What you cannot afford is a value that changes silently and nobody can tell you who touched it or when.

If you cannot answer "who changed this, when, and why" for any config value in production within about two minutes, you do not have a config system. You have a pile of switches and a prayer.

Not All Config Is the Same Thing

The single biggest mistake I see is treating all configuration as one undifferentiated bag. It is not. In my experience there are at least three distinct kinds, and they want completely different homes.

• Static deployment config — connection strings, service URLs, feature module toggles that only change with a release. This belongs in version control, baked into the deploy, immutable at runtime.
• Operational config — retry counts, timeouts, batch sizes, rate limits. Things an on-call engineer might tune at 3am. This wants to be changeable without a deploy but still audited and validated.
• Business config — fee schedules, risk thresholds, per-merchant limits. This is data, owned by product and risk teams, and it belongs in a database with a proper edit UI, not a text file an engineer greps for.

When you jam all three into one appsettings file, every one of them ends up served by the worst-fit mechanism. Business users cannot edit fee schedules without a pull request. Engineers cannot tune a timeout without touching a file that also contains merchant limits. Draw the boundaries early. The batch size that took us down should have been operational config with a validated ceiling, not a free-text field.

Strongly Typed, or You Will Pay Later

I have a strong preference here that borders on a rule: config should be bound to typed objects at startup, and the app should refuse to start if the binding fails. The alternative, reading loose strings scattered through the codebase, is a slow leak that eventually floods the boat.

The failure mode is always the same. Somebody reads a value with a fallback default buried three layers deep. The key gets renamed in one place but not the other. Nothing errors. The default silently takes over, and you discover months later that your fraud check timeout has been 30 seconds instead of 3 the whole time because a typo meant the real value never loaded. In .NET this is a solved problem, and it costs you almost nothing to solve it properly.

public sealed class PayoutOptions
{
    [Range(1, 500)]
    public int BatchSize { get; init; } = 100;

    [Range(1, 60)]
    public int SettlementTimeoutSeconds { get; init; } = 10;
}

// Registration: bind, validate on start, fail loud.
services.AddOptions<PayoutOptions>()
    .Bind(config.GetSection("Payout"))
    .ValidateDataAnnotations()
    .ValidateOnStart();

That ValidateOnStart call is the whole point. If the batch size is out of range or missing, the process does not boot. A pod that crash-loops in staging is a Tuesday afternoon. A pod that starts happily with a garbage config is a production incident with your name on the pager. Given the choice I will take the crash-loop every single time.

Validate at the Edge, Not in the Middle

Ranges and required fields are the floor, not the ceiling. The values that hurt you are usually the ones that are individually valid but collectively nonsense. A retry count of 5 is fine. A timeout of 10 seconds is fine. A retry count of 5 with a 10 second timeout inside a job that itself has a 30 second budget is a guaranteed failure, and no single-field validator will catch it.

So I push cross-field and cross-service validation into the load path. When operational config changes, I want a validation pass that understands relationships: total retry time must fit inside the parent timeout, the configured batch size times the average record size must stay under the broker limit, the rate limit must be at least as generous as the downstream provider allows. Write these as explicit checks with real error messages. The person tuning a value at 3am should get told "batch size 10000 would produce a 14MB message, limit is 10MB" instead of watching the job die in a way they cannot connect to their change.

One Layer of Precedence, Written Down

Config systems accrete layers: defaults in code, a base file, an environment-specific file, environment variables, a secrets store, maybe a runtime override table. That layering is fine and useful. What is not fine is when nobody can tell you which one wins.

Every config framework has a precedence order. Most teams have never read it and could not draw it on a whiteboard. That gap is where hours disappear. I once watched two engineers argue for most of an afternoon about why a setting "wasn't taking effect" when an environment variable in the deployment manifest was quietly overriding the file they kept editing. Both of them were right about what the file said. Neither knew the env var existed.

Fix this with two habits. First, write the precedence order in your README in plain language, highest wins to lowest. Second, log the effective, fully-resolved config at startup, with secrets redacted, including where each value came from. That startup log has saved me more debugging time than any dashboard. When someone asks "what is the timeout in prod right now," the answer is a log line, not an archaeology project.

Secrets Live Somewhere Else, Full Stop

This one is short because it is not negotiable in regulated fintech. Secrets are not config. A database password, a signing key, an API credential for your card processor: these go in a dedicated secrets manager with access control and an audit log, and they are injected at runtime. They never sit in a config file, never land in version control, never show up in that startup log I just praised.

I treat any secret that has ever touched a git repository as compromised and rotate it, no exceptions, even if the repo is private. It sounds paranoid until the one time a contractor's laptop backup ends up somewhere it should not. The mechanics of this are cheap now. The reputational cost of a leaked processor key that shows up in a public commit is not something you recover from with an apology.

Every Change Needs a History and a Way Back

For anything editable at runtime, I want three things without exception: an audit record of who changed what and when, a required reason field, and a one-click rollback to the previous value. The reason field feels like bureaucratic friction until the first postmortem where it turns a two-hour investigation into a thirty-second read. "Raised limit for merchant onboarding push, ticket RISK-4412" tells the whole story.

For business config in a database, this is a versioned table, not an in-place update. Never overwrite the row. Insert a new version, keep the old one, and read the current one through a view. Storage is the cheapest thing you own; the ability to answer "what was the fee schedule on the 14th" during a dispute is worth far more than the rows it costs.

-- Fee schedules as an append-only history, never UPDATE in place.
CREATE TABLE fee_schedule_versions (
    id              BIGINT IDENTITY PRIMARY KEY,
    merchant_id     BIGINT       NOT NULL,
    percentage_bps  INT          NOT NULL,   -- basis points
    effective_from  DATETIME2    NOT NULL,
    changed_by      NVARCHAR(128) NOT NULL,
    change_reason   NVARCHAR(512) NOT NULL,
    created_at      DATETIME2    NOT NULL DEFAULT SYSUTCDATETIME()
);

-- "What fee applied to merchant 42 on a given day?" is now a query,
-- not a guess reconstructed from Slack scrollback.

A Config Change Is a Deploy in Disguise

The failure I opened with taught me to treat operational config changes with the same rollout discipline as code. If a value can take down production, then flipping it everywhere at once is reckless, no matter how confident you feel. I want to change it in one region or one cell, watch the metrics for a few minutes, and then let it spread. The same gradual rollout you would never skip for a code deploy, you should not skip for a config change that alters behavior.

This also means config changes belong on the same timeline as everything else. When we do incident review now, the config change history and the deploy history are in one merged view. More than once the culprit was neither a bad deploy nor infrastructure but a config edit that landed forty minutes earlier and looked innocent. If those two histories live in separate systems, you will not connect them under pressure, and pressure is exactly when you need to.

Build for Boring, Not for Dials

Here is the thing I wish someone had told me earlier: the goal of a good config system is not flexibility. Flexibility is easy and mostly a trap. The goal is that a change to production behavior is always visible, always attributable, always reversible, and always validated before it can do harm. If you build for that, you get the flexibility for free and you get to sleep. Build for flexibility alone and you get a machine with a thousand dials, no labels, and a very quiet 2am waiting for you.

Start with one thing this quarter. Add ValidateOnStart and make the app refuse to boot on bad config. It is an afternoon of work, and it converts a whole class of silent production incidents into loud staging failures. That trade is the best deal in this entire discipline, and I have never once regretted taking it.