# ADR 0010: Release Strategy and Migration Rollback Policy

- Status: Accepted
- Date: 2026-03-11
- Owners: Platform Team, SRE Team, Architecture Group
- Related:
  - `docs/wallet-implementation-plan-apps-mode.md`
  - `docs/adr/0003-financial-ledger-invariants.md`
  - `docs/adr/0007-observability-and-audit-traceability-standard.md`
  - `docs/adr/0009-data-partitioning-and-retention-policy.md`
  - `docs/non-functional-slo.md`
  - `docs/phase-tracker.md`

## Context
The platform will evolve from a single Phoenix app toward apps mode, then progressively enable security, ledger, transfer, and integration capabilities. Release and schema changes in a financial system require deterministic rollback paths and explicit cutover controls to avoid data integrity issues.

This ADR defines release stages, deployment safety controls, database migration patterns, and rollback policy.

## Decision
Adopt progressive delivery with backward-compatible migrations and controlled cutover gates.

Core decisions:
1. Use phased releases: dark launch -> internal canary -> limited production -> full rollout.
2. Require backward-compatible application and schema changes before cutover.
3. Use expand-migrate-contract migration pattern.
4. Rollback strategy prioritizes traffic rollback over destructive schema rollback.
5. Financial posting path releases require additional change-approval gate.

## Release Strategy
Release tiers:
1. Tier 0: local/dev validation
2. Tier 1: staging and integration sandbox
3. Tier 2: internal canary (small traffic slice)
4. Tier 3: limited production cohort
5. Tier 4: full production rollout

Promotion criteria:
- SLO threshold compliance for current tier
- No unresolved Sev-1/Sev-2 defects impacting critical path
- Mandatory dashboard and alert checks green
- Required migration verification checks passed

## Feature Flag and Kill-Switch Policy
- Critical new flows are behind feature flags.
- Flags support tenant/cohort/percentage targeting.
- Every critical flag must have a documented kill-switch owner and procedure.
- Kill switch must be executable without redeploy where possible.

## Database Migration Policy
Migration modes:
1. Expand:
- Additive schema changes only (new tables/columns/indexes).
- Keep old reads/writes functioning.

2. Migrate:
- Backfill data with idempotent jobs.
- Validate data consistency with reconciliation checks.

3. Contract:
- Remove deprecated schema only after full rollout and verification window.

Rules:
- No destructive migration in same release as feature activation.
- Large backfills require throttling and observability.
- Migration scripts must be rerunnable or safely resumable.

## Financial Path Change Controls
For ledger, transfer, and posting engine changes:
- Mandatory peer review by financial domain maintainers.
- Pre-release invariant replay tests on production-like snapshots.
- Post-release drift checks against baseline invariants.
- Immediate rollback criteria documented before deploy.

## Rollback Strategy
Application rollback:
- First option: route traffic back to previous stable version.
- Keep schema backward compatibility to support immediate app rollback.

Data rollback:
- Avoid destructive rollback of financial records.
- Use compensating operations for business corrections.
- Use forward-fix migration for schema issues when safe.

Emergency controls:
- Disable newly introduced write paths via kill switch.
- Switch to degraded read-only mode for impacted domains if required.

## Release Readiness Checklist (Minimum)
1. Migration risk assessment completed.
2. Rollback/forward-fix runbook attached to release.
3. Feature flags and kill switches verified in staging.
4. SLO and alert thresholds validated for new scope.
5. On-call staffing and release comms plan confirmed.

## Post-Release Verification
Within defined observation window:
- compare key KPIs to baseline
- inspect error-code distribution shifts
- verify queue backlogs and dead-letter rates
- verify reconciliation mismatch rates
- confirm audit event completeness

If thresholds exceeded:
- trigger rollback or kill-switch policy
- start incident workflow and root-cause analysis

## Roles and Responsibilities
- Release Manager: coordinates promotion decisions and comms.
- Domain Owner: approves domain-specific readiness.
- SRE: validates observability and rollback readiness.
- Compliance/Security: sign-off for regulated and security-sensitive changes.

## Test and Verification Requirements
1. Blue/green or canary promotion test in non-prod.
2. Migration dry-run on production-like dataset.
3. Rollback drill at least once per major release train.
4. Feature-flag kill-switch drill verification.
5. Invariant/regression test suite must pass before promotion.

## Consequences
Positive:
- Lower risk of catastrophic release failures.
- Safer schema evolution with controlled rollback paths.
- Improved operational confidence during high-impact changes.

Trade-offs:
- Slower release cadence for critical domains.
- Additional planning and operational overhead.

## Acceptance Criteria
1. Progressive delivery process documented and adopted.
2. Expand-migrate-contract pattern enforced in migration reviews.
3. Rollback and kill-switch runbooks exist for critical flows.
4. Financial-path releases pass additional readiness gate.
5. Phase 8 production readiness gate includes release-drill evidence.