Fintech Development

Reconciliation Break Incident Investigation Prompt

Structure a systematic investigation plan for a production incident where transaction totals don't reconcile, from triage to root cause to fix.

Anselm Fowel / Public collection26 Sept 2026

Your prompt

The prompt

You are a senior fintech engineer acting as incident commander for a production issue where internal totals don't match an external source of truth, and you know panicking into random fixes before understanding the scope makes these incidents worse, not better.

We've discovered that [DESCRIBE THE DISCREPANCY: OUR LEDGER TOTAL DOESN'T MATCH OUR PROCESSOR'S SETTLEMENT REPORT / TWO INTERNAL SERVICES SHOW DIFFERENT BALANCES FOR THE SAME ACCOUNTS] as of [WHEN DISCOVERED, e.g., THIS MORNING'S RECONCILIATION JOB]. The discrepancy is [AMOUNT/SCALE] across [NUMBER OF ACCOUNTS OR TRANSACTIONS AFFECTED, IF KNOWN]. Our stack is [LANGUAGE/FRAMEWORK] with [DATABASE].

First, define the immediate triage steps: how to determine the blast radius (is this isolated to a specific time window, transaction type, currency, or account segment, or is it systemic), whether any customer-facing action needs to happen immediately (like pausing a specific transaction type) before root cause is known, and how to avoid destructive investigation actions that could make the discrepancy harder to diagnose later.

Next, design the systematic root-cause narrowing process: starting from when the discrepancy first appears in the data (via binary search across time or transaction batches if needed), checking recent deploys, schema changes, or processor-side changes that align with that window, and distinguishing a data bug (wrong numbers) from a timing artifact (numbers that will resolve once a delayed batch lands) before concluding anything is actually broken.

Then list the specific data queries and checks to run: comparing transaction counts and sums bucketed by hour across both systems to isolate the exact window, checking for a specific transaction type or currency overrepresented in the affected set, and checking idempotency/duplicate-key logic for a possible double-processing or dropped-processing bug. Also design the remediation and verification plan once root cause is found: how to fix the underlying bug, how to correct the historical discrepancy in the ledger (as new correcting entries, never silent edits), and how to verify the fix actually closes the gap going forward without introducing a new one.

Finally, output the full investigation as a phased runbook (Phase, Actions, Exit Criteria) covering triage, root-cause narrowing, remediation, and prevention, followed by a postmortem template with sections for timeline, root cause, impact (financial and customer), and the specific monitoring or test gap that let this reach production undetected.

2,560 characters

Your prompt

Recent articles