Why We Log in JSON
Three years ago I spent a Saturday night grepping through 40 gigabytes of plaintext logs trying to work out why a batch of settlement files had gone missing....
Three years ago I spent a Saturday night grepping through 40 gigabytes of plaintext logs trying to work out why a batch of settlement files had gone missing....
The first time an auditor asked me to prove who changed a customer's credit limit on a specific Tuesday eight months earlier, I learned exactly how good our...
The first time I took down a payments platform during business hours, it was for a version bump. Not a migration, not a big-bang rewrite, just a routine deploy...
The worst outage I have signed off on in the last five years started with a single environment variable. Someone flipped a payout batch size from 100 to 10000...
The first thing I do when I inherit a codebase is open the folder tree and look at the top level. Not the README, not the architecture diagram somebody drew...
Last spring one of our payout endpoints started taking eleven seconds at the 95th percentile. Not always. Just often enough that support tickets trickled in...
A few years ago I inherited a payments dashboard that showed merchants their settled balance. It was fast, users loved it, and roughly once a week someone in...
A payments team I worked with once spent three days chasing a phantom. Their ledger showed a merchant balance of 4,812 euros in one screen and 4,798 in...
A dead letter queue is where messages go to be forgotten. That is the dirty secret nobody tells you when they draw the tidy architecture diagram with the...
A few years ago one of my teams shipped a feature that felt trivial in review: when a payment settled, we wrote the settlement record to Postgres and then...
A few years ago I inherited a payments platform where the events team and the billing team had quietly disagreed about what a field called amount meant. One...
The first version of our public API had no rate limiting at all. It survived for about four months, right up until a customer's integration developer wrote a...