Building a Culture Where People Ask for Help
A senior engineer on my team once spent eleven hours trying to reproduce a flaky failure in our settlement reconciliation job. Eleven hours, most of it on a...
Practical notes on engineering leadership, fintech, and building at scale.
A senior engineer on my team once spent eleven hours trying to reproduce a flaky failure in our settlement reconciliation job. Eleven hours, most of it on a...
The first time a webhook retry policy cost us real money, it wasn't because a webhook failed. It was because a webhook succeeded, and then we sent it four more...
A few years ago I sat in a design review where a senior engineer told a junior one that his caching layer was "a mess." That was the whole comment. The junior...
The first time an event schema change took down a downstream service, I was three years into running payments infrastructure and I still thought of events as...
I have killed more projects than I have shipped. That is not a confession, it is a job description. If you run engineering for a payments company, most of your...
The first currency conversion bug I ever shipped cost us about 40 euros. A customer in Poland paid an invoice denominated in euros, we converted to zloty for...
Two of my best engineers spent the better part of a week not speaking to each other properly. Not shouting, nothing dramatic. Just clipped Slack replies,...
A support ticket landed on my desk a few years ago with the subject line "invoice off by one cent." A merchant in the Netherlands had split a €100.00 order...
At 2:14 on a Tuesday afternoon, our payment authorization service started rejecting one in every three card transactions. I know the exact time because the...
A few years ago I spent the better part of a Friday night staring at a payment reconciliation job that had gone from taking ninety seconds to taking forty...
Four years ago I shipped a change to our settlement engine on a Thursday afternoon, right before a long weekend, because I was convinced it was low risk. It...
The worst outage I ever sat through lasted four hours, and for the first two of them every dashboard we owned was green. The load balancer was happily routing...