# ADR 0004: Idempotency and Locking Strategy for Money Movement

- Status: Accepted
- Date: 2026-03-11
- Owners: Financial Domain Team, Architecture Group
- Related:
  - `docs/adr/0002-eventing-and-outbox.md`
  - `docs/adr/0003-financial-ledger-invariants.md`
  - `docs/non-functional-slo.md`
  - `docs/phase-tracker.md`

## Context
Transfer and posting flows operate under retries, duplicated client submissions, network timeouts, and concurrent requests. Without a deterministic idempotency and locking strategy, duplicate debits, inconsistent transfer states, and deadlocks become likely.

This ADR defines one consistent mechanism for request deduplication, conflict control, and lock lifecycle across `wallet_transfers`, `wallet_ledger`, and `wallet_state`.

## Decision
Adopt a layered strategy:
1. API-level idempotency key required for all money movement write endpoints.
2. Domain-level business reference uniqueness for transfer/posting commands.
3. Short-lived distributed lock for critical transfer transitions.
4. Database transactional constraints as final correctness boundary.

Idempotency and locking must be explicit in command handlers and cannot be optional.

## Idempotency Contract
Required request headers/fields:
- `Idempotency-Key` (client-generated UUID recommended)
- `X-Correlation-ID` (trace propagation)

Idempotency key scope:
- Key uniqueness is per `(tenant_id, actor_id, route_signature)`.
- Reuse with different payload hash is rejected.

Persistence model (in `wallet_state`):
- `idempotency_key`
- `scope_hash`
- `payload_hash`
- `first_seen_at`
- `expires_at`
- `status` (`in_progress`, `succeeded`, `failed`)
- `result_code`
- `result_body_hash`
- `transfer_id` (optional)

Rules:
1. First valid request inserts key with `in_progress`.
2. If a matching succeeded record exists, return prior semantic result.
3. If payload hash differs for same key and scope, reject with conflict.
4. Keys expire by policy after retention window.

Retention defaults:
- Transfer writes: 72 hours.
- Ledger posting writes: 7 days.

## Locking Strategy
Lock domain:
- Transfer lifecycle transition locks for operations that can race:
  - reserve
  - complete
  - fail
  - cancel

Lock key format:
- `transfer:{transfer_id}` or `account:{account_id}:debit` when account-level serialization is required.

Lock rules:
1. Acquire lock before state transition and posting command orchestration.
2. Lock TTL must exceed expected transaction time with safety margin.
3. Lock renewal allowed only by lock owner.
4. Always release lock in `after`/finally semantics.
5. On lock timeout, return retryable conflict outcome.

Store choices:
- Primary: DB row-level lock + transactional update where possible.
- Supplementary distributed lock: Redis/Mnesia-backed lock when cross-node coordination is needed.

## Database Constraints (Mandatory)
1. Unique index on transfer business reference.
2. Unique index on ledger posting `reference_id`.
3. State transition check constraints (or guarded transition table).
4. Foreign key consistency between transfer and posting references.

## State Transition Rules
Allowed transitions for transfer aggregate:
- `initiated` -> `reserved`
- `reserved` -> `completed`
- `reserved` -> `failed`
- `initiated` -> `canceled`

Forbidden transitions:
- Any transition out of terminal states (`completed`, `failed`, `canceled`) except explicit compensation flow.

## Failure Handling
1. Unknown outcome (timeout after server processing):
- client retries with same idempotency key.
- system returns canonical existing result.

2. Handler crash after lock acquisition:
- lock auto-expires by TTL.
- idempotency record remains `in_progress` until recovery worker reconciles state.

3. Partial downstream failure after posting:
- do not re-post.
- execute compensating workflow if business outcome requires reversal.

## Security Considerations
- Idempotency payload hash must avoid storing plaintext sensitive data.
- Key entropy must prevent brute-force key collisions.
- Abuse controls required for high-rate duplicate submissions.

## Observability Requirements
Mandatory metrics:
- idempotency hit ratio
- idempotency key conflicts
- in-progress key timeout count
- lock acquisition latency
- lock contention rate
- lock timeout count

Mandatory logs/traces:
- include `idempotency_key`, `correlation_id`, `transfer_id` in write-path traces.

## Test Requirements
1. Duplicate submit test: same key, same payload returns same result.
2. Key conflict test: same key, different payload rejected.
3. Concurrent transition test: only one transition succeeds under race.
4. Crash-recovery test: in-progress keys reconciled correctly.
5. Load test: contention remains within SLO thresholds.

## Consequences
Positive:
- Prevents duplicate debits under retries.
- Deterministic behavior under partial failures.
- Clear operational metrics for contention and replay.

Trade-offs:
- Additional persistence and lock management complexity.
- Requires disciplined client and gateway behavior for idempotency keys.

## Acceptance Criteria
1. All money movement write APIs enforce idempotency key validation.
2. Duplicate requests do not create duplicate transfer or ledger entries.
3. Transition races are serialized and auditable.
4. Lock contention and idempotency metrics are visible and alertable.
5. Phase 4 retry and concurrency tests pass in CI.