# Phase 5 Execution Checklist (Async, Settlement, Reconciliation, Notifications)

Reference artifacts:
- `docs/wallet-implementation-plan-apps-mode.md`
- `docs/phase-tracker.md`
- `docs/adr/0002-eventing-and-outbox.md`
- `docs/adr/0007-observability-and-audit-traceability-standard.md`
- `docs/adr/0008-integration-adapter-contract-and-failure-policy.md`
- `docs/adr/0012-testing-strategy-and-quality-gates.md`

## 1. Phase Objective
Move non-critical side effects off synchronous request paths and establish resilient async
processing for settlement, reconciliation, and user notifications.

Phase status:
- Start: `not-started`
- Completed: `done` (2026-03-13)
- Test result: 99 tests, 0 failures (56 `wallet_settlement` + 43 `wallet_notifications`)
- Boundary check: 0 violations (`scripts/check_boundaries.sh`)

## 2. Scope
In scope:
- `wallet_settlement` and `wallet_notifications` OTP app foundations.
- Oban queue topology with criticality-based isolation (config + CI-compatible in-process queue).
- Settlement batch workflow, reconciliation pipeline, and exception record/resolution lifecycle.
- Notification queueing, channel dispatch, preference model, and delivery tracking.
- Audit event emission and domain event publishing for all commands.

Out of scope (deferred):
- External payment/core-banking adapter production integration → Phase 6.
- Real-time async health dashboards and operational runbooks → Phase 8 (SRE gate).
- Production Oban/Postgres queue setup → Phase 8 (infrastructure gate).
- Full compliance/risk maturity workflows → Phase 7.
- Performance/load impact tests (sync latency under async load) → Phase 8.

## 3. Work Breakdown

## Track A: App Foundations

1. Create `wallet_settlement` OTP app with public interfaces.
- Owner: Settlement Team
- Artifacts:
  - `apps/wallet_settlement/mix.exs`
  - `apps/wallet_settlement/lib/wallet_settlement/application.ex` (supervision tree: `SettlementStore`, `ReconciliationStore`, `ExceptionStore`, `JobQueue`)
  - `apps/wallet_settlement/README.md`
- Status: done

2. Create `wallet_notifications` OTP app with public interfaces.
- Owner: Messaging Team
- Artifacts:
  - `apps/wallet_notifications/mix.exs`
  - `apps/wallet_notifications/lib/wallet_notifications/application.ex` (supervision tree: `NotificationStore`, `PreferenceStore`, `JobQueue`)
  - `apps/wallet_notifications/README.md`
- Status: done

3. Define async command/event boundaries.
- Owner: Architecture + Settlement + Messaging
- Decision: synchronous path ends once a batch/notification record is persisted and a job is
  enqueued; all downstream processing (delivery, settlement execution, reconciliation) is async via
  worker `perform/1`. Workers emit domain events and audit records on completion.
- Status: done

## Track B: Oban Queue Topology and Worker Isolation

1. Define queue classes and priorities.
- Owner: Platform + SRE
- Artifacts:
  - `apps/wallet_settlement/lib/wallet_settlement/queue_config.ex` — `WalletSettlement.QueueConfig`
  - Documents 5 queues: `:settlement_high` (p1, limit 5, max_attempts 3),
    `:settlement_normal` (p2, limit 10, max_attempts 5), `:settlement_low` (p3, limit 20,
    max_attempts 3), `:notifications_high` (p1, limit 10, max_attempts 3),
    `:notifications_low` (p3, limit 30, max_attempts 5).
  - Full Oban `config/runtime.exs` snippet included in module doc.
- Status: done

2. Implement Oban configuration and worker modules.
- Owner: Platform + Domain Teams
- Artifacts:
  - `apps/wallet_settlement/lib/wallet_settlement/job_queue.ex` — `WalletSettlement.JobQueue`
    (ETS-backed GenServer; replaces Oban in CI; `perform/1` contract compatible with Oban workers)
  - `apps/wallet_notifications/lib/wallet_notifications/job_queue.ex` — `WalletNotifications.JobQueue`
    (independent ETS-backed queue scoped to notification queues; avoids cross-app dependency)
  - `apps/wallet_settlement/lib/wallet_settlement/workers/settlement_worker.ex`
  - `apps/wallet_settlement/lib/wallet_settlement/workers/reconciliation_worker.ex`
  - `apps/wallet_notifications/lib/wallet_notifications/workers/notification_worker.ex`
- Note: production Oban wiring (Ecto repo + plugins) deferred to Phase 8 infrastructure gate.
- Status: done

3. Implement retry/backoff and poison-message policy.
- Owner: Platform + SRE
- Artifacts:
  - `WalletSettlement.QueueConfig.backoff_ms/1` — exponential backoff formula: `15_000 + 2^attempt * 1_000 ms`.
  - `WalletSettlement.JobQueue` + `WalletNotifications.JobQueue` — on failure before
    `max_attempts`: job status set to `:available` (retryable); on `attempts >= max_attempts`:
    status set to `:discarded` (dead-letter).
  - Dead-letter escalation threshold documented in `QueueConfig` moduledoc.
- Status: done

## Track C: Settlement Workflow

1. Implement settlement batch lifecycle.
- Owner: Settlement Team
- Artifacts:
  - `apps/wallet_settlement/lib/wallet_settlement/settlement_batch.ex` — `SettlementBatch`
    struct + state machine (`:pending` → `:running` → `:completed` | `:partial` | `:failed`);
    terminal-state guards; `start/1`, `complete/2` (with function header for default opts),
    `fail/2`, `terminal?/1`.
  - `apps/wallet_settlement/lib/wallet_settlement/settlement_store.ex` — `SettlementStore`
    ETS GenServer with status index.
  - `apps/wallet_settlement/lib/wallet_settlement/commands/run_settlement_batch.ex` —
    `RunSettlementBatch` (creates pending batch → persists → enqueues `SettlementWorker` on
    `:settlement_high` → emits audit).
  - `apps/wallet_settlement/lib/wallet_settlement/workers/settlement_worker.ex` —
    `SettlementWorker.perform/1` (start batch → process transfers via `transfers_module`
    adapter → complete/partial/fail → emit events + audit).
  - Stub transfers adapter wired via `Application.get_env(:wallet_settlement, :transfers_module)`.
- Status: done

2. Implement settlement status queries.
- Owner: Settlement Team
- Artifacts:
  - `apps/wallet_settlement/lib/wallet_settlement/queries/get_settlement_batch.ex`
  - `apps/wallet_settlement/lib/wallet_settlement/queries/list_settlement_batches.ex`
    (filter by status or list all)
- Status: done

3. Emit settlement events.
- Owner: Settlement Team
- Artifacts:
  - `apps/wallet_settlement/lib/wallet_settlement/events/settlement_batch_started.ex` — `SettlementBatchStarted.v1`
  - `apps/wallet_settlement/lib/wallet_settlement/events/settlement_batch_completed.ex` — `SettlementBatchCompleted.v1`
  - `apps/wallet_settlement/lib/wallet_settlement/events/settlement_exception_raised.ex` — `SettlementExceptionRaised.v1`
  - All implement `WalletEvents.DomainEvent` behaviour; published via Phoenix PubSub
    (`wallet_settlement:events` topic).
- Status: done

## Track D: Reconciliation and Exception Handling

1. Implement reconciliation job.
- Owner: Settlement + Financial Team
- Artifacts:
  - `apps/wallet_settlement/lib/wallet_settlement/reconciliation_run.ex` — `ReconciliationRun`
    struct (`:pending` → `:running` → `:completed` | `:failed`) with result counters:
    `transfers_checked`, `ledger_entries_checked`, `mismatches_found`, `exceptions_raised`.
  - `apps/wallet_settlement/lib/wallet_settlement/reconciliation_store.ex`
  - `apps/wallet_settlement/lib/wallet_settlement/commands/run_reconciliation.ex` —
    `RunReconciliation` (creates pending run → persists → enqueues `ReconciliationWorker` on
    `:settlement_normal`).
  - `apps/wallet_settlement/lib/wallet_settlement/workers/reconciliation_worker.ex` —
    `ReconciliationWorker.perform/1` (start run → detect_mismatches → complete → emit events
    + audit). Phase 5: mismatch detection returns empty list (no external CBS feed yet);
    Phase 6 injects real ledger data via `wallet_integrations` adapter.
  - `apps/wallet_settlement/lib/wallet_settlement/queries/get_reconciliation_run.ex`
  - `apps/wallet_settlement/lib/wallet_settlement/events/reconciliation_completed.ex` — `ReconciliationCompleted.v1`
- Status: done

2. Implement exception record model.
- Owner: Settlement + Compliance
- Artifacts:
  - `apps/wallet_settlement/lib/wallet_settlement/exception_record.ex` — `ExceptionRecord`
    struct; mismatch types: `:amount_mismatch`, `:missing_transfer`, `:missing_ledger_entry`,
    `:duplicate_posting`, `:status_mismatch`, `:manual`; severities: `:critical` / `:high` /
    `:medium` / `:low`; lifecycle: `:open` → `:investigating` → `:resolved` | `:escalated`
    → `:resolved`; `investigate/2`, `resolve/2`, `escalate/1` with arity-correct fallback clauses.
  - `apps/wallet_settlement/lib/wallet_settlement/exception_store.ex` — `ExceptionStore`
    ETS GenServer with batch-id and status indexes.
  - `apps/wallet_settlement/lib/wallet_settlement/queries/list_exceptions.ex`
    (filter by batch_id, status, or list all)
- Status: done

3. Implement exception resolution command.
- Owner: Settlement Team
- Artifacts:
  - `apps/wallet_settlement/lib/wallet_settlement/commands/resolve_settlement_exception.ex` —
    `ResolveSettlementException` (load exception → `ExceptionRecord.resolve/2` → persist →
    emit `ExceptionResolved` event + audit record with `resolved_by` actor).
  - `apps/wallet_settlement/lib/wallet_settlement/events/exception_resolved.ex` — `ExceptionResolved.v1`
- Status: done

## Track E: Notification Service

1. Implement notification queueing and dispatch flow.
- Owner: Messaging Team
- Artifacts:
  - `apps/wallet_notifications/lib/wallet_notifications/notification.ex` — `Notification`
    struct; delivery lifecycle: `:queued` → `:sending` → `:sent` | `:failed`;
    `:failed` → `:queued` (requeue); `:permanently_failed` (max_attempts exceeded, terminal);
    `mark_sending/1`, `mark_sent/1`, `mark_failed/2`, `requeue/1`, `terminal?/1`.
  - `apps/wallet_notifications/lib/wallet_notifications/notification_store.ex` —
    `NotificationStore` ETS GenServer with user index and idempotency key index
    (per-user + key scoping, consistent with ADR 0004/0005).
  - `apps/wallet_notifications/lib/wallet_notifications/commands/queue_notification.ex` —
    `QueueNotification` (create notification → idempotency check → persist → enqueue
    `NotificationWorker` on `:notifications_high` or `:notifications_low` based on type →
    emit `NotificationQueued` event + audit; forwards `max_attempts` option).
  - `apps/wallet_notifications/lib/wallet_notifications/commands/send_notification.ex` —
    `SendNotification` (synchronous dispatch; delegates to `NotificationWorker.perform/1`).
  - `apps/wallet_notifications/lib/wallet_notifications/workers/notification_worker.ex` —
    `NotificationWorker.perform/1` (check preference → soft-suppress or mark-sending →
    dispatch via channel adapter → mark-sent or mark-failed → emit event + audit).
  - `apps/wallet_notifications/lib/wallet_notifications/adapters/stub_adapter.ex` —
    `StubAdapter` (Phase 5 CI; `deliver/1` returns `:ok` by default; overridable via
    `Application.put_env(:wallet_notifications, :stub_adapter_result, ...)` for failure tests).
  - Channel adapter resolved per-channel via `Application.get_env(:wallet_notifications,
    :<channel>_adapter, WalletNotifications.Adapters.StubAdapter)`.
- Status: done

2. Implement preference model.
- Owner: Messaging Team
- Artifacts:
  - `apps/wallet_notifications/lib/wallet_notifications/preference.ex` — `Preference` struct;
    per-user, per-channel; `enabled` boolean; `suppression_rules` map supporting
    `:opt_out_types`, `:quiet_hours`, `:daily_limit` keys; `allowed?/2` pure predicate.
  - `apps/wallet_notifications/lib/wallet_notifications/preference_store.ex` —
    `PreferenceStore` ETS GenServer keyed by `{user_id, channel}`; `upsert/1`, `get/2`,
    `list_by_user/1`.
  - `apps/wallet_notifications/lib/wallet_notifications/commands/update_preferences.ex` —
    `UpdatePreferences` (upsert preference → emit audit record).
- Status: done

3. Implement delivery tracking.
- Owner: Messaging Team
- Artifacts:
  - Delivery status persisted in `NotificationStore` on each lifecycle transition
    (`mark_sending`, `mark_sent`, `mark_failed`).
  - `attempts` counter incremented on each send attempt; `sent_at` timestamp on success;
    `failure_reason` string on failure.
  - `correlation_id` propagated from queue command → worker → events for end-to-end traceability.
  - `apps/wallet_notifications/lib/wallet_notifications/queries/get_notification.ex`
  - `apps/wallet_notifications/lib/wallet_notifications/queries/list_user_notifications.ex`
    (filter by status and/or channel).
- Status: done

## Track F: Observability and Runbooks

1. Implement async health dashboards.
- Owner: SRE + Observability Team
- Status: deferred — Phase 8 (SRE + Observability gate).
- Note: `WalletSettlement.JobQueue` and `WalletNotifications.JobQueue` expose `list_jobs/2`
  for queue depth inspection; telemetry events emitted via `:telemetry.execute/3` on all
  audit paths. Dashboard wiring (Prometheus/Grafana) deferred to Phase 8.

2. Implement settlement/reconciliation dashboards.
- Owner: SRE + Settlement Team
- Status: deferred — Phase 8.
- Note: `SettlementStore.list_by_status/1`, `ExceptionStore.list_by_status/1`, and
  `ReconciliationStore.list_all/0` provide the data surfaces required for dashboards.

3. Publish operational runbooks.
- Owner: SRE + Domain Teams
- Status: deferred — Phase 8.
- Note: `WalletSettlement.QueueConfig` moduledoc includes dead-letter escalation criteria and
  retry policy reference. Full runbooks (queue incident, settlement delay, reconciliation
  mismatch, notification outage) to be published in Phase 8.

## Track G: Testing and Quality Gates

1. Worker behavior tests.
- Owner: QA + Platform
- Artifacts:
  - `apps/wallet_settlement/test/wallet_settlement/settlement_worker_test.exs` — 15 tests:
    `RunSettlementBatch` creates batch + enqueues job; `SettlementWorker.perform/1` happy path
    (stub completed transfers → batch status `:completed`); missing-transfer path (exceptions
    raised, status `:partial`); error cases (missing_batch_id, not_found); `drain_queue`
    processes multiple jobs; dead-letter after max_attempts (`:discarded` status after 4 drains).
  - `apps/wallet_settlement/test/wallet_settlement/reconciliation_test.exs` — 13 tests:
    `RunReconciliation` creates run + enqueues job; `ReconciliationWorker.perform/1` completes
    run with 0 mismatches (Phase 5 baseline); error cases; `drain_queue` processes multiple
    jobs; `QueueConfig.backoff_ms/1` monotonically increasing, base ≥ 15 000 ms.
- Status: done

2. Settlement/reconciliation integration tests.
- Owner: QA + Settlement + Financial
- Artifacts:
  - `apps/wallet_settlement/test/wallet_settlement/settlement_batch_test.exs` — 16 tests:
    `new/4` defaults, prefix, options; `start/1` pending → running, rejects re-start;
    `complete/2` `:completed` vs `:partial` (failed_count/exception_count), completed_at set,
    rejects from non-running; `fail/2` running → failed, rejects from pending;
    `terminal?/1` across all statuses.
  - `apps/wallet_settlement/test/wallet_settlement/exception_test.exs` — 13 tests:
    `new/4` defaults and prefix; `investigate/2` open → investigating, rejects on resolved;
    `resolve/2` from open/investigating/escalated, rejects re-resolve; `escalate/1` from
    open/investigating, rejects on resolved.
  - `apps/wallet_settlement/test/wallet_settlement/resolve_exception_test.exs` — 8 tests:
    `ResolveSettlementException.execute/3` resolves open exception, persists state, returns
    not_found for missing, returns error on re-resolve; `ListExceptions` all/by batch/by status.
- Status: done

3. Notification delivery tests.
- Owner: QA + Messaging
- Artifacts:
  - `apps/wallet_notifications/test/wallet_notifications/queue_test.exs` — 12 tests:
    `QueueNotification` happy path (status, prefix, channel); queue routing (high for
    transaction_alert/security_alert, low for promotional); idempotency replay (same key +
    same user returns existing notification); per-user key isolation (same key different users
    get separate notifications); `GetNotification`/`ListUserNotifications` with channel filter
    and status filter (including post-send verification).
  - `apps/wallet_notifications/test/wallet_notifications/send_test.exs` — 10 tests:
    `SendNotification` happy path → `:sent`, `sent_at` set, `attempts` == 1; failure path →
    `:failed` with `failure_reason`; permanently_failed when `max_attempts` exceeded;
    `Notification.requeue/1` clears failure and re-queues, rejects requeue on permanently_failed;
    `drain_queue` delivers all high-priority notifications; failed jobs counted in drain result.
  - `apps/wallet_notifications/test/wallet_notifications/preference_test.exs` — 13 tests:
    `Preference.new/3` defaults; `Preference.allowed?/2` — enabled, disabled, opt_out_types
    suppression; `UpdatePreferences` sets preference, upsert overwrites, channel independence;
    preference enforcement during send: allowed, soft-suppressed (channel disabled → marked
    `:sent` not `:failed`), opt_out type suppressed, non-suppressed type passes through;
    `PreferenceStore.list_by_user/1` all prefs for user, empty for unknown user.
  - `apps/wallet_notifications/test/wallet_notifications/delivery_tracking_test.exs` — 8 tests:
    full lifecycle queued → sent with attempt count and sent_at; failure records failure_reason
    and increments attempts; retry clears failure_reason and re-queues; independent tracking
    across multiple notifications per user; traceability IDs (correlation_id, ntf_ prefix);
    `Notification.terminal?/1` across queued/sent/permanently_failed/failed statuses.
- Status: done

4. Performance impact tests.
- Owner: QA + SRE
- Status: deferred — Phase 8.
- Note: ETS-based stores have O(1) lookup; all sync command paths (InitiateTransfer,
  PostJournalEntry) are unaffected by Phase 5 workers. Formal latency measurement under
  concurrent async load deferred to Phase 8 load/chaos testing gate.

## 4. Deliverables

| # | Deliverable | Status | Artifact Path |
|---|---|---|---|
| 1 | `wallet_settlement` app integrated into umbrella | done | `apps/wallet_settlement/` |
| 2 | `wallet_notifications` app integrated into umbrella | done | `apps/wallet_notifications/` |
| 3 | Oban queue topology documented + CI-compatible queue implemented | done | `queue_config.ex`, `job_queue.ex` (both apps) |
| 4 | Settlement + reconciliation workflows with exception management | done | `settlement_batch.ex`, `reconciliation_run.ex`, `exception_record.ex`, workers, commands |
| 5 | Notification preference and delivery tracking model | done | `notification.ex`, `preference.ex`, workers, commands |
| 6 | Async operational dashboards and runbooks | deferred | Phase 8 — SRE gate |

## 5. Entry and Exit Criteria

Entry criteria:
- [x] Phase 4 transfer/lifecycle foundation completed (2026-03-11).
- [x] ADR 0002 (eventing/outbox) and ADR 0008 (adapter contract/failure policy) available.

Exit criteria:
1. [x] Phase 5 milestone checklist in `docs/phase-tracker.md` completed.
2. [x] Queue behavior tests pass (retry/backoff/dead-letter) — `settlement_worker_test.exs`,
       `reconciliation_test.exs`, `send_test.exs`.
3. [~] Settlement/reconciliation variance reporting generated daily — `ReconciliationWorker`
       implemented and tested; daily scheduling deferred to Phase 8 cron wiring.
4. [x] Notification delivery and preference tests pass — all 43 `wallet_notifications` tests passing.
5. [~] Core sync transaction latency within SLO targets under async load — ETS paths unaffected;
       formal measurement deferred to Phase 8 load gate.

## 6. Risks and Mitigations

1. Risk: queue saturation impacts critical processing.
- Mitigation: queue isolation (`:settlement_high` vs `:notifications_low` fully separate),
  concurrency caps in `QueueConfig`, alert threshold documented in dead-letter escalation policy.

2. Risk: reconciliation mismatch backlog grows unmanaged.
- Mitigation: `ExceptionRecord` severity tiers (`critical`/`high`/`medium`/`low`) drive SLA
  ownership; `ResolveSettlementException` requires explicit resolution note for audit trail;
  `ListExceptions` supports status-filtered backlog views.

3. Risk: duplicate async side effects.
- Mitigation: `NotificationStore` idempotency key index (per-user scoped, consistent with ADR
  0004/0005); `QueueNotification` replay returns existing notification without re-enqueueing.

4. Risk: notification failures become silent.
- Mitigation: `mark_failed/2` captures `failure_reason` and increments `attempts`;
  `permanently_failed` status flags max-attempts breach; `NotificationFailed.v1` event emitted
  on each failure for downstream monitoring.

## 7. Sprint Execution Summary

Sprint A (completed 2026-03-13):
- Created `wallet_settlement` and `wallet_notifications` apps with full supervision trees.
- Implemented `QueueConfig`, `WalletSettlement.JobQueue`, `WalletNotifications.JobQueue`.
- Implemented settlement batch lifecycle and notification queueing baseline.

Sprint B (completed 2026-03-13):
- Implemented reconciliation run and exception record/resolution.
- Completed notification preference, delivery tracking, and stub channel adapter.
- Full test suite passing: 99 tests, 0 failures.

## 8. Evidence Checklist

- [x] App scaffolding and interface docs — `apps/wallet_settlement/README.md`,
      `apps/wallet_notifications/README.md` (2026-03-13).
- [x] Queue topology config — `WalletSettlement.QueueConfig`: 5 queues, priorities,
      concurrency limits, backoff formula, dead-letter escalation criteria documented (2026-03-13).
- [x] Worker behavior test report — retry/backoff/dead-letter coverage in
      `settlement_worker_test.exs` and `reconciliation_test.exs`; drain_queue returning
      `:discarded` after max_attempts confirmed (2026-03-13).
- [x] Settlement batch lifecycle test report — `settlement_batch_test.exs` (16 tests),
      `settlement_worker_test.exs` (15 tests) passing (2026-03-13).
- [x] Reconciliation test report — `reconciliation_test.exs` (13 tests) passing; Phase 5
      baseline (0 external mismatches); CBS feed hook stubbed for Phase 6 (2026-03-13).
- [x] Exception backlog/resolve flow evidence — `exception_test.exs` (13 tests),
      `resolve_exception_test.exs` (8 tests) passing (2026-03-13).
- [x] Notification preference and delivery tracking test report — `queue_test.exs` (12 tests),
      `send_test.exs` (10 tests), `preference_test.exs` (13 tests),
      `delivery_tracking_test.exs` (8 tests) passing (2026-03-13).
- [ ] Async health dashboard and runbook links — deferred to Phase 8 (SRE + Observability gate).
- [x] Phase 5 exit approval — 99 tests passing (56 `wallet_settlement` + 43
      `wallet_notifications`), 0 failures, 0 boundary violations (2026-03-13).
      Settlement + Messaging Teams exit criteria met.