Cut over without ever being wrong.
Replacing a live wallet is a trust exercise. The plan is deliberate: stand up the ledger, load the chart of accounts, migrate opening balances, then prove the books foot four different ways before a single real payment is allowed through — and only decommission the old paths after repeated zero drift.
—
—
What has to be running
walletflow is one stateless service plus a ledger, a database, and an async fabric. Everything scales independently; the service holds no balances, so it can be replicated freely.
| Service | Role |
|---|---|
| walletflow (Bun/Hono) | API, use cases, read models, outbox relay, statement worker |
| Walletflow Postgres | command state, idempotency, holds, outbox, COA, read models, flags |
| Formance Ledger + its Postgres | immutable postings — the system of record |
| AWS SNS | non-saga domain events |
| AWS Step Functions + SQS/DLQ | payout saga orchestration |
| Provider HTTP endpoints | payout submit / status rails |
Production Formance is deployed via Helm + the operator (infra/formance); the payout saga is a CDK app (infra/payout-saga). The two workers — outbox:relay and statement:worker — run as long-lived processes or scheduled jobs.
The ramp
Traffic is moved in three phases, one currency and one provider path at a time. The old system keeps running until repeated zero-drift checks justify turning its write paths off.
fail_closed so only explicitly-enabled scopes transact. One deliberate exception to fail_closed: saga completions (holds.settle / holds.release) bypass the gate by default. By the time a settle fires, the provider has usually already moved the money — the settle only records reality. Blocking it doesn't stop a payout; it makes the ledger lie, strands funds in holds:*, breaks provider reconciliation, and trips the holds-recon gate. Releasing a failed payout back to the merchant is never what a cutover flag means either. For a genuine hard freeze (suspected compromise, regulator order) set CORE_WALLET_COMPLETION_GATING=gate: the same flag rules then apply to completions, which return 409 and are retried by the saga once the freeze lifts.
Four ways to prove it foots
Reconciliation is not one check but four, each looking at the books from a different angle. All four must read zero before write cutover, and repeatedly before decommissioning the legacy paths.
| Endpoint | Checks | Pass |
|---|---|---|
| /internal/reconciliation/drift | read-model vs ledger balance | 0 drift |
| /internal/reconciliation/trial-balance | debits = credits per asset | balanced |
| /internal/reconciliation/holds | reserved holds vs ledger hold accounts | 0 drift |
| /internal/coa/drift | COA / GL / template coverage | ok |
When a back-dated adjustment is needed, POST /internal/corrections with an effectiveAt posts a bi-temporal correction — recorded now, effective then — so the audit trail stays honest.
The same posting data drives /internal/reporting/trial-balance and /internal/reporting/alm (provider cash position, merchant obligations, in-flight payouts, currency exposure) — all derived, never stored.
The env contract
A handful of variables decide safety-critical behaviour. The defaults are chosen so an un-configured deploy is safe; production overrides tighten them further. Expand each.
LEDGER_POSTING_MODEdefault · script
send from a non-world source fails atomically on insufficient funds — the overdraft guard. Raw postings mode has no guard and is only for the balance-enforcing in-memory test double. Never run postings mode against Formance in production. CORE_WALLET_FLAG_MODEdefault · fail_open
fail_closed denies any scope without an explicitly-enabled core_wallet_enabled flag — set it during cutover so un-flagged merchants, currencies and providers are blocked by default. CORE_WALLET_COMPLETION_GATINGdefault · allow
holds.settle/holds.release) always go through, because the provider may have already moved the money — blocking the recording strands reserved holds. gate extends the flag rules to completions for a hard freeze; blocked completions 409 and the saga retries them. CURRENCY_SCALESJSON override
PROVIDER_REGISTRYper-rail routing
PROVIDER_HTTP_BASE_URL for anything unlisted. FORMANCE_LEDGER_*ledger connection
bun run formance:ensure-ledger. OUTBOX_* / AWS_*async fabric
Verification & observability
The riskiest behaviours — overdraft rejection, zero-amount Numscript sends — are not verified by hope. They run automatically against a real Formance ledger booted by Testcontainers, and they're watched in production by Prometheus alerts.
The suite spins up an ephemeral Postgres and a real Formance container, wires the production adapter into the service in script mode, and proves the whole path end to end: pay-in with fee/VAT, a zero-fee/VAT pay-in (zero-amount sends), transfer/reserve/settle, a Numscript overdraft rejection, bank funding feeding a stablecoin onramp/offramp round trip, and a footing trial balance. Opt out with SKIP_FORMANCE_TESTCONTAINERS=true.
# offline gates
bun test # 157 tests, incl. real Formance + Postgres
bun run ready # fmt + lint + typecheck + build
# live gates (staging)
bun run formance:contract # all 18 templates, direct
BASE_URL=… bun run load:k6 # p95 < 200ms · < 1% errors · ~1000 TPSAlerts that matter
| Signal | Means |
|---|---|
| Formance health down | ledger unreachable — stop writes |
| ledger tx failures ↑ | template or balance problem |
| outbox pending age ↑ | relay stalled — events not dispatching |
| DLQ depth > 0 | payouts stuck — manual review |
| reconciliation drift ≠ 0 | books don't foot — investigate now |
| template mismatch ≠ 0 | active DB templates ≠ bundled |