The first time I load tested a payment API properly, I did it wrong. We fired 5,000 requests per second at a shiny new authorization endpoint, watched it hold up beautifully, and shipped with confidence. Two weeks later a mid-size merchant ran a flash sale and the same service fell over at maybe 300 real transactions per second. The test had been generating the same card, the same amount, the same merchant, over and over. Production sent 300 different cards to 300 different acquirer routes, and every one of them hit a cold path we had never exercised.
That gap between the traffic you generate and the traffic you actually get is where most payment load tests go to die. A payment API is not a content endpoint. It writes money-moving records, it calls out to acquirers and card networks you do not control, and it has correctness guarantees that survive under load or the whole exercise is pointless. So this is less a tutorial on wiring up a load generator and more a set of hard lessons about testing the thing that fails the way payments actually fail.
Why payments break the usual playbook
Most load testing advice assumes your bottleneck is CPU, memory, or a database you own. For a payment API, the bottleneck is almost always something on the other side of a network call: an acquirer sandbox with a 20 request-per-second cap, a card network simulator that rate-limits you without telling you, or a fraud service that adds 400ms under load. You can scale your own pods to the moon and it will not matter, because the slowest thing in the chain is a partner endpoint whose real capacity nobody ever tells you.
The other difference is that a payment request is not idempotent by nature but must behave as if it is. A dropped connection followed by a retry cannot charge a card twice. That single requirement shapes everything: how you generate load, what you assert on, and what counts as a pass. A test that hits 10,000 RPS but double-charges 0.1% of retried requests has not passed. It has found the most expensive bug you will ever ship.
Model the traffic you actually get, not round numbers
Nobody sends you a clean 1,000 RPS. Real payment traffic is spiky, uneven, and correlated in ways that hurt. A retailer's checkout traffic follows the working day, then detonates during a promotion. Payroll runs cluster on the 25th and the last business day of the month. Subscription billing all fires at midnight UTC because everyone's cron job is lazy and defaults to zero.
Before you write a single script, pull a week of real production logs and look at the actual shape. I care about four things when I build the traffic model:
- Peak-to-average ratio. If your average is 200 RPS but your peak minute hits 1,800, you test for 1,800 plus headroom, not 200.
- The mix of request types. Authorizations, captures, refunds, and status lookups have wildly different cost profiles. Refunds often touch more tables and more downstream systems than a plain auth.
- Cardinality. Different cards, different merchants, different amounts, different currencies. Uniqueness defeats caching and exposes lock contention that a single hot card never will.
- The retry storm. Model what happens when a downstream slows down and every client retries at once. That secondary spike is frequently larger than the primary one.
If you skip the cardinality point you will get a beautiful, meaningless result. I have seen a team celebrate 8,000 RPS on an auth endpoint where every request used the same idempotency key, so the service short-circuited on a cache hit after the first one. They load tested their cache. Congratulations.
Choose a tool that speaks your language
I have used JMeter, Gatling, Locust, and k6, and I keep coming back to k6 for payment work. Not because it is magic, but because the tests are JavaScript, they live in the same repo as the service, and engineers actually maintain them instead of treating them as a mysterious XML artifact one person understands. Gatling is excellent if your team is comfortable in Scala; most are not. JMeter is powerful and I would rather not.
Whatever you pick, the load generator has to run from somewhere with realistic network distance and enough outbound bandwidth to not become the bottleneck itself. Running a 2,000 RPS test from a single laptop over office wifi tells you about your wifi. Run it from cloud instances in the region that mirrors your real client geography, and run enough of them that no single generator is above 70% CPU. A saturated generator underreports latency and you will chase a ghost for a day.
Test the idempotency layer on purpose
This is the part most teams forget, and it is the part I check first. Your API almost certainly accepts an idempotency key so that a client can safely retry. Under load, the race conditions in that layer come out to play. Two identical requests arriving 3ms apart, before the first has committed its record, is exactly the scenario a normal functional test never produces and a load test produces constantly.
So build a scenario whose entire job is to hammer the same key with concurrent requests and assert that exactly one charge results. Here is the shape of the server-side logic I want to see holding up under that pressure, using a database-backed lock rather than an in-memory one:
Enjoying this article?
Get more like it in your inbox — practical engineering leadership, fintech, and AI. No spam, unsubscribe anytime.
public async Task<PaymentResult> AuthorizeAsync(PaymentRequest req)
{
// Reserve the idempotency key atomically. If a row already
// exists, another request owns this key.
var inserted = await _db.ExecuteAsync(
@"INSERT INTO idempotency_keys (key, status, created_at)
VALUES (@key, 'in_progress', @now)
ON CONFLICT (key) DO NOTHING",
new { key = req.IdempotencyKey, now = DateTime.UtcNow });
if (inserted == 0)
{
// Someone else is handling or has handled this key.
// Return the stored result; never call the acquirer again.
return await WaitForStoredResultAsync(req.IdempotencyKey);
}
var result = await _acquirer.AuthorizeAsync(req);
await StoreResultAsync(req.IdempotencyKey, result);
return result;
}
The load test's assertion is not about latency here. It is a correctness check: send N concurrent copies of the same key, then query the acquirer's sandbox and confirm exactly one authorization landed. If two land, you have a bug that costs real money, and no throughput number can buy that back.
Respect the systems you do not own
Here is where load testing payments gets genuinely awkward. You cannot point a 2,000 RPS test at a real card network. You will get your sandbox credentials revoked, and rightly so. The acquirer's test environment is shared, throttled, and slower than production, so it lies to you in both directions.
My approach is to run two distinct tests. The first uses a mock acquirer that you control, tuned to the real latency distribution you have measured in production: not a flat 50ms, but a realistic long tail with the occasional 2-second outlier. This tells you how your service behaves when the downstream is healthy but variable. The second, smaller test uses the real sandbox at a rate the partner has agreed to, purely to validate that your integration handles their actual error codes, timeouts, and connection resets. Never conflate the two.
The mock has to lie the way production lies. A downstream that always responds in exactly 50ms is not a test double, it is a fantasy, and it hides every queueing and thread-pool problem you have.
Measure percentiles, and watch the tail
Average latency is worse than useless for a payment API; it is actively misleading. If your mean auth latency is 120ms but your p99 is 4 seconds, a merchant's checkout is timing out for one customer in a hundred, and that customer abandons the cart. I look at p50, p95, p99, and p99.9, and I care most about the last two, because that tail is where the thread-pool exhaustion and the connection-pool starvation show up first.
Beyond latency, track the numbers that tell you why. Error rate broken down by type, not lumped together: a 429 from your rate limiter means something completely different from a 504 from the acquirer. Database connection pool utilization. Thread-pool queue depth. And crucially, the downstream call latency measured separately, so when your p99 spikes you can immediately see whether it was you or the acquirer. A test that only reports "requests per second and average latency" cannot answer the only question that matters when things go wrong, which is where the time went.
Push it until it breaks, then read the wreckage
A load test that passes tells you almost nothing. A load test that fails tells you exactly where your ceiling is and how the system behaves at that ceiling, which is the information you actually need. So I always run a ramp that keeps climbing past the expected peak until something gives, then I study the failure mode carefully.
The failure mode is the whole point. Does the service shed load gracefully, returning clean 429s and keeping already-accepted payments consistent? Or does it fall into a death spiral where requests pile up, latency climbs, clients retry, and the retries add more load until it stops responding entirely? I once watched a service that had no concurrency limit accept every request, exhaust its database connections, and then hold thousands of half-finished transactions in memory until it OOM-killed itself. It would have been far healthier to reject 20% of traffic at the door and keep the other 80% correct. Graceful degradation is a feature you have to test for, because it never happens by accident.
Keep a small synthetic drip in production
Pre-release load testing catches the big structural problems. It does not catch the slow drift: the new fraud rule that added 80ms, the connection pool someone shrank in a config change, the acquirer who quietly degraded their own capacity. For that you want a small, continuous stream of synthetic transactions running against production around the clock, against a test merchant with a known card, at a volume low enough to be invisible in your real numbers.
Ten transactions a minute is plenty. What matters is that it is constant, it exercises the full real path including the real acquirer, and it alerts the moment the p99 or the success rate moves. This is the difference between finding out your latency doubled from a dashboard at 2am and finding out from a furious merchant at 9am. I would give up a great deal of pre-release tooling before I gave up the synthetic drip.

Conclusion
If you take one thing from this, make it the correctness assertion, not the throughput number. The temptation in load testing is to chase a big RPS figure you can put in a slide, but a payment API that does 10,000 RPS and occasionally double-charges is a liability, not an achievement. Test the idempotency layer under real concurrency, model traffic with actual cardinality, treat the systems you do not own with suspicion, and read the failure mode instead of the pass. Do that, and the flash sale that would have taken you down becomes a Tuesday. Skip it, and you will learn the same lesson I did, except the classroom will be a production incident and the tuition will be a very uncomfortable call with a merchant.
Get new posts in your inbox
Occasional, practical notes on engineering leadership, fintech, and building with AI. No spam, unsubscribe anytime.
Comments (0)
Leave a Comment
No comments yet. Be the first to comment!
