Idempotency is a security control, not a reliability nicety

The bug

A payment connector timed out after the gateway had already accepted the refund but before it returned a settlement reference. Our retry logic saw a failed step, replayed it, and issued a second refund for the same order line. The gateway accepted both, because from its side these were two well-formed requests with different idempotency keys.

The keys were different because we derived them from the run id. A replay is a new run, so it got a new key. That is a defensible choice for a step that sends an email and an indefensible one for a step that moves money, and the same code path handled both.

Why we filed it as a security issue

The instinct is to call this a reliability bug: a retry misbehaved, money moved twice, reconciliation would have caught it. We filed it under security instead, and the distinction turned out to matter more than the label.

A reliability bug is something you fix and move on from. A security bug forces you to ask what an attacker could do with it deliberately. Once we asked that question, the answer was uncomfortable: anyone able to induce a timeout on a payout step — by exhausting a rate limit, for instance — could cause repeated payouts without ever touching an approval gate. The bug was not that a retry was unsafe; it was that the amount of money a workflow could move was bounded by nothing except how many times it could be made to fail.

That reframing produced a different fix. A reliability fix keys idempotency on the external reference and stops. The security fix also caps what the connector's credential is permitted to refund, so a future logic error cannot exceed a ceiling that lives outside the logic.

What we changed

Payout steps now key idempotency on the external reference — the order line, the settlement id, the invoice number — never on anything derived from the run. A replayed run resolves to the original settlement and reports it, rather than creating a second one.

Automatic retries on payment steps are off by default. A failed payout parks with its full input payload and waits for an operator to replay it deliberately. Every finance team we described this to assumed it already worked that way, which is usually a sign the default was wrong.

And every payout path now has two independent ceilings: what the workflow will approve, and what the credential can execute. A bug in the first cannot exceed the second. That redundancy is the part we would keep if we could only keep one thing.