The code that lost the money is the before.
This repo is the after.
The DECCS of 10 June frames walletflow as a wallet substrate awaiting hardening. This page puts the legacy code and the rewrite side by side — same failure class, both implementations, file references for every claim — so the comparison is clear as day. Play the replay below first: it is the ₦102.9M bug, runnable.
One failed payout. One refund owed. How many times does it pay out?
A provider reports a payout failed, so ₦100,000 must be released back to the merchant — exactly once. The provider retries its webhook (they always do). Left: the legacy estate's defenses (a Redis key with a TTL, a balance column you mutate). Right: walletflow's (a permanent Postgres claim row, a double-entry posting). Pick a case, press the buttons in the order it shows — or press ▶ and watch it run.
- — waiting for the provider —
- — waiting for the provider —
The losing sequence is three clicks: deliver webhook → 12 hours pass → replay ×5. That is the Titancode class — 40 NGN duplicate failure-releases, ₦102,886,905.31 excess across 11 days of 2026 — reproduced from the legacy mechanism: a TTL’d Redis key, WALLET_IDEMPOTENCY_REDIS_KEY_EXPIRES_IN = 12 h (fincra-wallets/…/walletsUtilRepository.ts), is the only memory that a release already happened. walletflow’s claim is a database row with a request-hash that never expires (src/infrastructure/postgres/core-repository.ts:132), and the ledger reference is UNIQUE — the same replay returns the original response, forever. Verified by test: docs/ui-test-prompts.md · T2, T6.
This race is the original 2024 code (500651f): the first idempotency check was a plain GET then setex — no NX — so ten concurrent deliveries could all read “no key” before any write landed, and 2–4 of them credited (fincra-wallets/…/walletsUtilRepository.ts). SET NX (June 2024) closed this same-instant window — one writer wins, the rest get 409s — and a per-wallet lock (April 2026) later serialized whole operations. Both were live when Titancode struck in May 2026, and neither helped: each answers “is this happening now?”, and a retry arriving past the 12-hour TTL truthfully answers no — the lock is free, the key is gone, the money moves again. Expiry replays are Case 1, and only memory that never expires stops them. walletflow never relies on application timing at all: one INSERT … ON CONFLICT wins inside Postgres and the other nine receive the original response (src/infrastructure/postgres/core-repository.ts:132). Verified by test: docs/ui-test-prompts.md · T2.
The trap is that this case passes on the left: while the Redis key is alive, every same-day replay dedupes, so the legacy code looks correct in review, in QA, and in any demo shorter than the TTL. The first retry that arrives after 12 hours is the one that pays twice — switch to Case 1 and run it to watch the same five replays turn into ₦500,000. walletflow’s claim row never ages (src/infrastructure/postgres/core-repository.ts:132), so the two columns only diverge when memory is allowed to expire. Verified by test: docs/ui-test-prompts.md · T2, T6.
House rules: the merchant is owed ₦100,000 exactly once; the provider may replay its webhook any number of times, any time apart, at any concurrency. Anything above ₦100,000 in the left column is money the legacy estate gave away — the right column may never exceed it, in any button order. Verified by test: docs/ui-test-prompts.md · T2, T6.
Five failure classes, side by side
Each card is one money-losing class from the incident history: the legacy code that allows it on the left, the walletflow code that forbids it on the right.
Balance: a column you mutate vs a sum you derive
class · wallet-disparity RCA family// eight mutable balance columns per wallet — // "history" is one shadow column deep table.decimal('available_balance', 19, 4) table.decimal('previous_available_balance', 19, 4) table.decimal('ledger_balance', 19, 4) table.decimal('previous_ledger_balance', 19, 4) table.decimal('locked_balance', 19, 4) table.decimal('previous_locked_balance', 19, 4) table.decimal('rolling_balance', 19, 4) … // every service UPDATEs these over HTTP
Whoever writes last wins. Drift between the four balance types is detectable only by manual reconciliation — the recurring “wallet balance disparity” RCAs.
// balance = the sum of double-entry postings. // nothing UPDATEs a balance, ever. send [$asset $amount] ( source = $source // fails atomically if destination = $destination // it can't cover — the ) // overdraft guard // 16 templates cover every flow; proven against // real Formance in contract tests
No Numscript template grants overdraft, so a negative balance is structurally impossible — asserted by the real-ledger contract suite, not by review.
Idempotency: a cache that forgets vs a claim that can’t
incident · Titancode ₦100,000,100 · 19 Mar// dedupe = a Redis key with an expiry (paraphrased) const key = hash(`${reference}_${action}_${type}`) const fresh = await redis.set(key, '1', 'EX', ttl, 'NX') if (!fresh) return // deduped… // …until the TTL expires. then the same webhook // is "new" again. that is the duplicate-release bug.
Check-then-act in application code, with memory that evaporates. Replays past the 12-hour TTL re-apply money; before SET NX was added (June 2024), concurrent racers inside the window could too.
-- a duplicate is a DATABASE error, not a code check INSERT INTO core_idempotency_keys (scope, idempotency_key, request_hash, …) ON CONFLICT (scope, idempotency_key) DO NOTHING; -- loser SELECTs … FOR UPDATE → original response -- plus UNIQUE(reference) on ledger_transactions -- plus UNIQUE(provider, provider_reference) on webhooks -- claims never expire
Three independent unique constraints stand between a replay and a posting. A replayed webhook gets the original response — at hour one or year one.
Payouts: debit first, hope later vs reserve, then settle xor release
incidents · DD.md double-disbursements · April $1M race shapeawait wallet.debit(amount) // money moves FIRST await queue.add('submitToProvider', payout) // then webhook, requery cron, channel-switch cron // and the retry queue ALL race to credit it back — // the SQS handler was fire-and-forget (sleep, not // await). documented double-disbursements followed.
Funds leave before the provider answers, and three uncoordinated paths can each decide to refund. An unknown outcome could become a refund — the worst verdict.
reservePayout() // merchant → hold + outbox event, // one DB transaction settleHold() // UPDATE … WHERE status='reserved' releaseHold() // status-guarded: one terminal state // saga: unknown / timeout / 5xx → ProviderAmbiguousState // → bounded requery → dead-letter → manual review. // an UNKNOWN outcome is never a verdict.
Money is reserved into a hold first; settle and release are status-guarded transitions of one state machine, and ambiguity routes to a human, not a refund.
Fifty doors across six services vs one registry, seventeen commands
class · cross-service drift · duplicated fee logic// who mutates wallet balance? everyone: fincra-collections → wallets.credit() fincra-disbursements → wallets.debit() fincra-fx-conversions → debit() + credit() settlements-service → debit() + credit() disburse-saga → reserve/settle/release fincra-wallets → its own endpoints // each with its own retries, fees, idempotency
Fee math is re-implemented per service; every endpoint is another door for a duplicate; no single place can enforce an invariant.
// ONE spec = console form + API route + executor { key: 'payout-reserve', title: 'Reserve payout', api: [{ method: 'POST', path: '/api/v1/payouts' }], run: (svc, input, ctx) => svc.reservePayout(input, ctx) } // 17 commands: payin · transfer · payout · fx · otc · // stablecoin · gps · reserve · corrections · reversal … // idempotency key mandatory on every one
The full transaction flow — collections through treasury — behind one registry. The API and the operator console cannot drift; they are generated from it.
Money math: guess the decimals vs refuse to guess
class · zero-decimal UGX/XAF/XOF bug// caller-supplied amounts, unvalidated precision, // MySQL DECIMAL(19,4) for every currency — // UGX has 0 decimal places. XOF has 0. // fee arithmetic with raw number ops in several // services; an unknown currency just… proceeds.
Treating every currency as two-decimal silently mis-scales the zero-decimal ones, and unknown codes flow through instead of failing.
// integer fixed-point (BigInt) everywhere; an // authoritative per-currency minor-unit table: UGX: 0, XAF: 0, XOF: 0, // zero-decimal USDC: 6, NGN: 2, BHD: 3, // … normalizeCurrency('ZZZ') // → throws badRequest — unknown currencies are // REFUSED, never guessed
Exact money, fail-closed currencies. A fractional minor unit or an unknown code is an error at the boundary, not a rounding surprise in the books.
The shape of the fix
The fragmentation was the vulnerability. Six services choreographed money over HTTP; walletflow is one bounded context where every invariant has exactly one place to live.
The registry's seventeen commands span the entire flow: collections (pay-in + fee + VAT), transfers, payout holds, FX deals, OTC trades, stablecoin on/off-ramp, GPS cross-currency corridors, rolling reserve, treasury bank funding, bi-temporal corrections, full reversals, and migration opening balances — plus trial balance, ALM views, and three reconciliation gates that foot the books continuously. That is the transaction flow of six legacy services, in one auditable place.
The DECCS scorecard
The DECCS (10 Jun) was right about the estate — and is already stale about this repo. Five of its repo-claims were fixed within a day of the review; the genuinely open items are listed just as plainly, because a refutation that hides its gaps isn't one.
| DECCS claim (10 Jun) | Repo today (11 Jun, commit 87d3b98) | Verdict |
|---|---|---|
| “Five phantom scripts… point at a scripts/ directory that does not exist; bun run build fails from this tree.” (WI-3.1) | All five exist and run — bench-core, apply-cutover-flags, migrate-opening-balances, ensure-formance-ledger, load/k6-core.js — plus smoke-dist. Build + dist smoke pass. | stale · fixed |
| “Settle and release never read the hold's current state… no constraint forbids both landing.” (WI-1.2) | Holds carry status ∈ {reserved, settled, released, manual_review} with status-guarded transitions. Remaining ask: the DB-level XOR constraint + server-derived amounts. | mostly stale |
| “PAYOUT_MANUAL_REVIEW_REQUIRED is currently written to a void.” (WI-3.3) | Manual review is a first-class console queue on /ops/payouts — reason shown, filterable, resolvable by settle or release. | stale · fixed |
| “The README advises deleting idempotency keys after 24 hours.” (WI-2.4) | Advice removed; README rewritten. Claims never expire in the schema. | stale · fixed |
| Framing: walletflow is “a wallet substrate”; “the ops UI… has no position view.” | Literally true about the position view — but the repo is the full transaction flow (17 commands, 6 legacy services' scope) with a 12-page operator console: recon, trial balance, ledger admin, payout ops, cutover controls. | undersold |
| Per-payout hold sub-accounts (WI-1.1) | Holds are still pooled per merchant+currency (src/domain/ledger/accounts.ts:11) — mitigated by the hold state machine, not eliminated at the ledger. | still open |
| Actor identity + append-only action log (WI-2.1) · maker-checker (WI-2.2) | Absent. Command context is {idempotencyKey} only; corrections are single-operator. | still open |
CI; verify must run tests (WI-3.2) · concurrency fuzz (WI-1.6) | No .github/; verify = fmt+lint+build. No parallel-duplicate fuzz suite yet. | still open |
Legacy demo accounts table ships in migrations (WI-2.4, “door #51”) | Still present at src/db/schema.ts:9 — unused by the core, should be deleted. | still open |
| The capital-position layer (Phases V1, 4–6: bank feeds, FX revaluation, position engine + dashboard) | Not built — and never claimed. It is the extension the DECCS commissions on top of this substrate, not a defect in it. | forward work |
Every “fixed” verdict is reproducible: ls scripts/, grep -n status src/db/schema.ts, open /ops/payouts, grep -i "24 hour" README.md. The full fact-check with file:line evidence lives in docs/walletflow-overview.md; the runnable UI proofs in docs/ui-test-prompts.md.