# Phase 6 Execution Checklist (Integrations and External Adapters)

Reference artifacts:
- `docs/wallet-implementation-plan-apps-mode.md`
- `docs/phase-tracker.md`
- `docs/adr/0002-eventing-and-outbox.md`
- `docs/adr/0008-integration-adapter-contract-and-failure-policy.md`
- `docs/adr/0012-testing-strategy-and-quality-gates.md`

## 1. Phase Objective
Implement external integration foundations so wallet flows can safely interact with payment rails and core banking systems without violating financial consistency.

Phase status target:
- Start: `not-started`
- End: `done` when adapter contracts, provider sandbox integration, replay-safe callbacks, and reconciliation-safe failure handling are verified.

**Actual phase status: `done` — completed 2026-03-14**

## 2. Scope
In scope:
- `wallet_integrations` app scaffold and provider-agnostic port interfaces.
- At least one payment provider sandbox adapter (end-to-end).
- Callback/webhook verification and replay-safe processing.
- Outbox/inbox path alignment for external side effects.
- Integration contract, resiliency, and reconciliation tests.

Out of scope:
- Full production rollout for multiple providers (Phase 8 hardening).
- Advanced fraud ML provider integration (Phase 7+ depending on risk roadmap).

## 2.1 Recommended Provider Sequence
For faster execution and lower integration friction in Elixir/Phoenix:
1. Start with `Stripe` as Provider #1 (sandbox).
- Reason: clean API model, strong test-mode coverage, mature webhook docs, predictable event model.

2. Add `Razorpay` as Provider #2 if India rails are required.
- Reason: regional relevance for India, practical for UPI/card flows after base adapter pattern is stable.

3. Keep provider choice configurable via adapter routing.
- Do not hardcode provider-specific assumptions into transfer/ledger domains.

Provider selection criteria (must be documented before implementation):
- Sandbox reliability and documentation quality
- Webhook signature/replay protections
- Required payment methods (cards/UPI/netbanking)
- Settlement/reconciliation reporting availability
- Operational support and incident handling maturity

**Provider selected: Stripe (sandbox). Adapter routing configurable via `Application.get_env(:wallet_integrations, :adapters, %{})` per provider atom.**

## 3. Work Breakdown

## Track A: App and Contract Foundations
1. Create `wallet_integrations` OTP app with public interfaces.
- Owner: Integration Team
- Output: app scaffold, supervision tree, README.
- Status: **done**
- Implementation:
  - `apps/wallet_integrations/mix.exs` — OTP app definition; deps: `wallet_shared_kernel`, `wallet_api_contracts`, `wallet_observability`, `wallet_events`, `wallet_transfers`, `jason`, `phoenix_pubsub`.
  - `WalletIntegrations.Application` — supervision tree starting `PaymentRequestStore`, `CallbackStore`, `InboxStore`, `IntegrationExceptionStore`, `CircuitBreaker`, `JobQueue`.
  - `apps/wallet_integrations/README.md` — public interface documentation.

2. Define provider-agnostic adapter behaviors.
- Owner: Integration + Architecture
- Output: normalized contracts for:
  - `initiate_payment/1`
  - `get_payment_status/1`
  - `cancel_payment/1`
  - `refund_payment/2`
- Status: **done**
- Implementation:
  - `WalletIntegrations.ProviderAdapter` — `@behaviour` module defining all four callbacks per ADR 0008 contract rules.
  - `WalletIntegrations.AdapterRequest` — normalized request struct with `payment_id`, `amount`, `currency`, `provider`, `provider_ref`, `refund_amount`, `correlation_id`; builder functions `new_initiate/4` and `new_with_ref/3`.

3. Define normalized response model.
- Owner: Integration Team
- Output: `status`, `provider_reference`, `provider_status_code`, `retryable`, `raw_response_hash`, `occurred_at`.
- Status: **done**
- Implementation:
  - `WalletIntegrations.AdapterResult` — normalized result struct with `status` (`:accepted | :pending | :completed | :failed | :unknown`), `provider_reference`, `provider_status_code`, `retryable`, `raw_response_hash`, `occurred_at`, `metadata`; `build/2` constructor; `hash_response/1` SHA-256 helper.

## Track B: Provider Sandbox Integration
1. Implement first provider sandbox adapter.
- Owner: Integration Team
- Output: complete adapter module implementing behavior contract.
- Status: **done**
- Implementation:
  - `WalletIntegrations.Adapters.StripeAdapter` — implements `ProviderAdapter` against Stripe REST API (`POST /v1/payment_intents`, `GET /v1/payment_intents/:id`, `POST /v1/payment_intents/:id/cancel`, `POST /v1/refunds`); normalizes Stripe status codes to domain status atoms; wraps credential and HTTP errors in `AdapterResult`.
  - `WalletIntegrations.Adapters.StubAdapter` — implements `ProviderAdapter` for CI/test; injectable results via `Application.put_env` per-operation overrides.
  - `WalletIntegrations.Http` — HTTP client abstraction; injectable for test isolation.

2. Implement provider credential loading via secret provider.
- Owner: Platform + Security + Integration
- Output: secret-manager-compatible runtime config; no plaintext credentials.
- Status: **done**
- Implementation:
  - `WalletIntegrations.SecretProvider` — behaviour with `get/1`; env-backed implementation reads from `Application.get_env(:wallet_integrations, key)`; no plaintext credentials in code.
  - `StripeAdapter` reads `:stripe_secret_key` via `SecretProvider.get/1`; credential errors wrapped in `AdapterResult` (never leaked as bare atoms).
  - Webhook secret loaded as `:stripe_webhook_secret` via same pattern.

3. Implement request signing/validation requirements.
- Owner: Integration + Security
- Output: provider auth headers/signature flow for outbound requests.
- Status: **done**
- Implementation:
  - `StripeAdapter.build_headers/2` — sets `Authorization: Bearer <secret_key>`, `Content-Type: application/x-www-form-urlencoded`, `Stripe-Version: 2023-10-16`.
  - `Idempotency-Key` header added on `initiate_payment` using `payment_id` to prevent duplicate charges on retry.

## Track C: Callback/Webhook Safety
1. Implement callback signature and freshness validation.
- Owner: Integration + Security
- Output: signature verification + timestamp window checks.
- Status: **done**
- Implementation:
  - `WalletIntegrations.CallbackVerifier` — `verify_stripe/3`: parses `Stripe-Signature` header (`t=<ts>,v1=<sig>`), checks freshness within configurable tolerance (default 300 s via `:callback_freshness_tolerance`), computes `HMAC-SHA256(secret, "#{ts}.#{body}")`, constant-time comparison via `:crypto.hash_equals/2` with charlist fallback.
  - Returns `:ok | {:error, :invalid_signature | :stale_callback | :malformed_signature}`.

2. Implement callback idempotency/replay protection.
- Owner: Integration Team
- Output: dedup key strategy and replay-safe consumer behavior.
- Status: **done**
- Implementation:
  - `WalletIntegrations.InboxRecord` — struct with states `:pending | :processing | :processed | :duplicate | :failed`; `claim/3` atomically checks for duplicate before any side effect.
  - `WalletIntegrations.InboxStore` — ETS-backed GenServer; `claim/3` returns `{:ok, record}` on first claim, `{:duplicate, existing}` on repeat.
  - `CallbackVerifier.stripe_message_id/2` — deterministic dedup key: `"stripe:#{event_id}:#{timestamp}"`.
  - `IngestCallback` claims `InboxStore` before storing `CallbackRecord`; duplicate returns `{:error, :duplicate}` without side effects.

3. Implement callback ingestion pipeline.
- Owner: Integration + Platform
- Output: asynchronous callback processing via inbox queue path.
- Status: **done**
- Implementation:
  - `WalletIntegrations.CallbackRecord` — struct with state machine `:pending → verified | rejected → processing → processed | failed`.
  - `WalletIntegrations.CallbackStore` — ETS-backed GenServer; keyed by `callback_id`; `inbox_message_id` dedup index; provider bag index.
  - `WalletIntegrations.Commands.IngestCallback` — full ingestion pipeline: hash payload → store `CallbackRecord` → emit `CallbackReceived` → verify signature → on success mark `:verified` and enqueue `CallbackWorker`; on failure mark `:rejected` and emit `CallbackRejected`.
  - `WalletIntegrations.Workers.CallbackWorker` — async worker: load verified callback → transition to `:processing` → check `InboxStore` for duplicate → apply payment request status update from event_type → mark inbox `:processed` → mark callback `:processed` → emit `CallbackProcessed`.

## Track D: Resilience and Failure Handling
1. Implement timeout and retry policy by operation.
- Owner: Integration + SRE
- Output: connect/read/deadline timeouts; bounded retries with jitter.
- Status: **done**
- Implementation:
  - `WalletIntegrations.RetryPolicy` — per-operation structs (`initiate_payment`, `get_payment_status`, `cancel_payment`, `refund_payment`) with `connect_timeout`, `recv_timeout`, `max_attempts`; `calculate_backoff/1` exponential with jitter; `retryable_error?/1` and `retryable_status_code?/1` guards.
  - `StripeAdapter` internal `call_with_retry/2` + `do_retry/3` — retries on 429, 5xx, and network-level retryable errors up to `max_attempts`.

2. Implement circuit-breaker and bulkhead controls.
- Owner: Integration + SRE
- Output: per-provider operation breaker states + isolation.
- Status: **done**
- Implementation:
  - `WalletIntegrations.CircuitBreaker` — ETS-backed GenServer; per `{provider, operation}` state machine: `:closed → :open → :half_open → :closed`; configurable failure threshold and recovery timeout; `check/2` returns `:ok | {:error, :circuit_open}`; `record_success/2` and `record_failure/2` update state.
  - `PaymentWorker` checks `CircuitBreaker.check/2` before each outbound call; records outcome after.

3. Implement unknown-outcome and compensation handling.
- Owner: Integration + Financial Team
- Output: pending/unknown states routed to reconciliation; no duplicate side effects.
- Status: **done**
- Implementation:
  - `WalletIntegrations.IntegrationException` — domain struct for reconciliation exceptions; types: `:mismatch | :timeout | :unknown_outcome | :callback_failure | :duplicate_charge`; state machine: `:open → investigating → escalated → resolved`.
  - `WalletIntegrations.IntegrationExceptionStore` — ETS-backed GenServer; indexes by `transfer_id`, `provider_reference`, `status`.
  - `PaymentWorker` routes `:unknown` adapter results to `create_reconciliation_exception/4` before updating `PaymentRequest` to `:unknown`; retryable failures return `{:error, ...}` to trigger job retry without re-applying side effects.

## Track E: Eventing and Reconciliation Alignment
1. Wire outbox to adapter dispatch path.
- Owner: Integration + Platform
- Output: external side effects initiated from reliable event/outbox pattern.
- Status: **done**
- Implementation:
  - `WalletIntegrations.Workers.PaymentWorker` — dispatches via `JobQueue` on `integrations_payment` queue (Oban-compatible); emits `PaymentInitiated.v1`, `PaymentCompleted.v1`, `PaymentFailed.v1` domain events via Phoenix PubSub.
  - `WalletIntegrations.QueueConfig` — Oban-compatible queue topology: `integrations_payment` (priority 1, 10 concurrent, 4 attempts), `integrations_callback` (priority 1, 20 concurrent, 3 attempts), `integrations_status` (priority 2), `integrations_refund` (priority 2); exponential backoff with jitter.
  - `WalletIntegrations.JobQueue` — in-process ETS-backed queue for CI operation; `drain_queue/1` for synchronous test execution.

2. Wire inbox acknowledgements for callback processing.
- Owner: Integration + Platform
- Output: callback message lifecycle states and dedup markers.
- Status: **done**
- Implementation:
  - `InboxStore.claim/3` — atomic first-claim before side effects.
  - `InboxRecord` lifecycle transitions through `begin_processing/1 → mark_processed/1`; `mark_duplicate/1` for replay.
  - `CallbackWorker` checks `InboxStore.get/1` before applying effects; if already `:processed`, returns `:ok` without re-applying.
  - `CallbackRecord.inbox_message_id` links callback to inbox dedup key.

3. Add integration mismatch visibility.
- Owner: Settlement + Integration
- Output: linkage between provider outcomes and reconciliation exception records.
- Status: **done**
- Implementation:
  - `IntegrationException.provider_reference` + `transfer_id` + `payment_request_id` — three-way traceability key.
  - `IntegrationExceptionStore.list_by_transfer/1` and `list_by_provider_ref/1` — lookup by either wallet or provider identity.
  - `PaymentWorker` creates `IntegrationException` of type `:unknown_outcome` on any ambiguous provider response; `IntegrationExceptionStore.list_open/0` surfaces unresolved mismatches.

## Track F: Testing and Quality Gates
1. Adapter contract test suite.
- Owner: QA + Integration
- Output: every adapter must pass behavior contract tests.
- Status: **done**
- Implementation:
  - `test/wallet_integrations/provider_adapter_contract_test.exs` — verifies `AdapterRequest` builders, `AdapterResult` shapes and hash consistency, `StubAdapter` full contract (all 4 operations, per-op result injection, fallback chain), `StripeAdapter` behavior contract (error wrapping when no credentials; `AdapterResult` shape on all failures).

2. Sandbox end-to-end tests.
- Owner: QA + Integration
- Output: initiate/status/cancel/refund happy and failure paths.
- Status: **done**
- Implementation:
  - `test/wallet_integrations/stripe_adapter_test.exs` (`WalletIntegrations.SandboxE2ETest`) — full pipeline tests: `InitiatePayment → drain → verify store state → assert events`; status poll enqueue; cancel terminal/non-terminal paths; refund happy/error paths; unknown-outcome exception creation; `PaymentWorker` error cases; `JobQueue` drain and retry/dead-letter.

3. Callback security and replay tests.
- Owner: QA + Security + Integration
- Output: signature failure, stale callback, duplicate callback, forged callback tests.
- Status: **done**
- Implementation:
  - `test/wallet_integrations/callback_security_test.exs` — `CallbackVerifier`: valid signature accepted; invalid signature rejected; stale timestamp rejected; fresh/edge-of-tolerance boundary; malformed header cases; `stripe_message_id` determinism; `IngestCallback`: valid flow ingested and enqueued; invalid signature stores rejected callback; duplicate message_id returns `{:error, :duplicate}`; forged/missing webhook secret handling; `CallbackWorker`: idempotent re-processing of duplicate callbacks.

4. Resilience tests.
- Owner: QA + SRE + Integration
- Output: timeout, retry exhaustion, circuit-open behavior, degraded provider scenarios.
- Status: **done**
- Implementation:
  - `test/wallet_integrations/resilience_test.exs` — `CircuitBreaker`: closed/open/half-open state transitions; per-provider isolation; circuit opens only after threshold; `RetryPolicy`: backoff values within bounds; retryable vs non-retryable error and status code classification; `PaymentWorker` with circuit open returns `{:error, :circuit_open}` without calling adapter; degraded provider (unknown outcome) routes to exception store.

5. Reconciliation consistency tests.
- Owner: QA + Settlement + Integration
- Output: provider-wallet mismatch detection and traceability tests.
- Status: **done**
- Implementation:
  - `test/wallet_integrations/reconciliation_test.exs` — `IntegrationException` struct and state machine (open/investigate/escalate/resolve; double-resolve guard); `IntegrationExceptionStore` CRUD and index queries (by status, transfer, provider_ref); `list_open/0` visibility; `PaymentWorker` creates exception on `:unknown` result; `IngestCallback` creates exception on signature rejection; multi-exception traceability (multiple exceptions per transfer).

## 4. Deliverables
1. `wallet_integrations` app integrated into umbrella. — **delivered** (`apps/wallet_integrations/`, supervision tree compiling and starting)
2. First provider adapter implemented and sandbox-tested. — **delivered** (`StripeAdapter` + `StubAdapter`, 120 tests passing)
3. Callback verification and replay-safe processing implemented. — **delivered** (`CallbackVerifier`, `InboxStore`, `CallbackWorker`, `IngestCallback`)
4. Timeout/retry/circuit-breaker controls active and observable. — **delivered** (`RetryPolicy`, `CircuitBreaker`, `QueueConfig`)
5. Integration contract and reconciliation-consistency test evidence delivered. — **delivered** (5 test suites, 120 tests, 0 failures, 2026-03-14)

## 5. Entry and Exit Criteria
Entry criteria:
- Phase 5 async foundation completed. — **met** (Phase 5 done 2026-03-13)
- ADR 0008 and ADR 0002 implementation patterns available. — **met** (ADRs created in Phase 0)

Exit criteria:
1. Phase 6 milestone checklist in `docs/phase-tracker.md` completed. — **met**
2. At least one provider sandbox integration passes end-to-end tests. — **met** (Stripe sandbox, `SandboxE2ETest`)
3. Callback replay/forgery protections pass security tests. — **met** (`callback_security_test.exs`)
4. Adapter failures do not create duplicate financial side effects. — **met** (`InboxStore` dedup, idempotency key on `initiate_payment`, `CallbackWorker` duplicate guard)
5. Reconciliation linkage between provider outcomes and wallet records is verified. — **met** (`reconciliation_test.exs`, `IntegrationExceptionStore` traceability)

## 6. Risks and Mitigations
1. Risk: provider API instability breaks flow reliability.
- Mitigation: strict timeout/retry/circuit-breaker policy and graceful degradation states.
- **Status: addressed** — `RetryPolicy` per-operation timeouts; `CircuitBreaker` per-provider/operation isolation; `:unknown` outcome routes to reconciliation rather than crashing.

2. Risk: duplicate callback side effects.
- Mitigation: callback dedup keys, inbox lifecycle state, idempotent command handling.
- **Status: addressed** — `InboxStore.claim/3` atomic dedup before any side effect; `CallbackWorker` checks for `:processed` inbox record before applying effects; tests verify duplicate suppression.

3. Risk: credential leakage in logs/config.
- Mitigation: secret provider integration, redaction policy, security tests.
- **Status: addressed** — `SecretProvider` abstracts all credential access; `StripeAdapter` wraps credential errors in `AdapterResult` (bare `:secret_not_found` never surfaces); no plaintext keys in source.

4. Risk: mismatch between provider and wallet states.
- Mitigation: reconciliation hooks, mismatch exception records, daily variance checks.
- **Status: addressed** — `IntegrationException` created on every `:unknown` outcome; three-way traceability (`provider_reference ↔ transfer_id ↔ payment_request_id`); `list_open/0` for operational visibility.

## 7. Suggested Sprint Plan (2 Sprints)
Sprint A: — **completed**
- Scaffold `wallet_integrations` and adapter contract interfaces.
- Build first sandbox adapter and outbound auth/signing path.
- Implement baseline timeout/retry and telemetry.

Sprint B: — **completed**
- Implement callback verification/replay-safe ingestion.
- Complete circuit-breaker/bulkhead controls.
- Execute sandbox E2E + resilience + reconciliation tests.
- Produce Phase 6 sign-off evidence pack.

## 8. Evidence Checklist
- [x] `wallet_integrations` scaffold and README/public interface docs.
  - `apps/wallet_integrations/README.md`; all modules documented with `@moduledoc`; `ProviderAdapter` behaviour with full `@callback` docs and contract rules.
- [x] Adapter contract test report.
  - `test/wallet_integrations/provider_adapter_contract_test.exs` — `AdapterRequest`, `AdapterResult`, `StubAdapter` (all 4 ops + injection), `StripeAdapter` (error shape contract). Passing.
- [x] First provider sandbox E2E report (happy + failure paths).
  - `test/wallet_integrations/stripe_adapter_test.exs` (`WalletIntegrations.SandboxE2ETest`) — initiate/status/cancel/refund happy paths; retryable failure; unknown outcome → exception; terminal state guards; job retry/dead-letter. Passing.
- [x] Callback signature/replay security test report.
  - `test/wallet_integrations/callback_security_test.exs` — valid/invalid/stale/malformed signatures; freshness boundary; dedup (duplicate → `{:error, :duplicate}`); forged/missing secret; `CallbackWorker` idempotent re-processing. Passing.
- [x] Resilience test report (timeouts/retries/circuit-breaker).
  - `test/wallet_integrations/resilience_test.exs` — circuit open/close/half-open transitions; per-provider isolation; `RetryPolicy` backoff bounds; worker circuit-open guard; unknown outcome → reconciliation path. Passing.
- [x] Reconciliation linkage evidence report.
  - `test/wallet_integrations/reconciliation_test.exs` — `IntegrationException` lifecycle; store CRUD and indexes; `list_open/0`; worker exception creation on unknown outcome; callback rejection exception; multi-exception traceability. Passing.
- [x] Phase 6 exit approval note.
  - **Phase 6 sign-off: 120 tests passing, 0 failures. Date: 2026-03-14. Integration Team exit criteria met.** Updated in `docs/phase-tracker.md`.