# ADR 0014: Business Continuity and Disaster Recovery

- Status: Accepted
- Date: 2026-03-11
- Owners: SRE Team, Platform Team, Security Team, Compliance Team
- Related:
  - `docs/adr/0010-release-strategy-and-migration-rollback.md`
  - `docs/adr/0013-environment-and-infrastructure-topology.md`
  - `docs/adr/0007-observability-and-audit-traceability-standard.md`
  - `docs/adr/0009-data-partitioning-and-retention-policy.md`
  - `docs/non-functional-slo.md`
  - `docs/phase-tracker.md`

## Context
The wallet platform supports regulated financial and identity workflows that require high availability and provable recovery capability. Infrastructure outages, region failures, data corruption, and security incidents can disrupt operations and impact users and compliance obligations.

A formal business continuity and disaster recovery (BC/DR) policy is required to define targets, recovery procedures, and validation cadence.

## Decision
Adopt a tiered BC/DR model with defined recovery objectives, tested runbooks, and evidence-based readiness.

Core decisions:
1. Classify services by criticality and assign RTO/RPO targets.
2. Use multi-zone/region resilient architecture for critical services where feasible.
3. Maintain tested backup and restore procedures for all stateful systems.
4. Use incident command and continuity workflows for service degradation scenarios.
5. Require periodic DR drills with measurable outcomes and corrective actions.

## Criticality Tiers and Objectives
Tier 1 (critical financial path):
- Examples: ledger posting, transfer orchestration, auth core, event reliability core.
- Target RTO: <= 30 minutes.
- Target RPO: <= 5 minutes.

Tier 2 (important but not immediate financial finality):
- Examples: notifications, non-critical reporting, operational dashboards.
- Target RTO: <= 4 hours.
- Target RPO: <= 30 minutes.

Tier 3 (deferred/recoverable without immediate customer impact):
- Examples: batch analytics and auxiliary services.
- Target RTO: <= 24 hours.
- Target RPO: <= 24 hours.

Note:
- Final targets require organization and regulatory sign-off.

## Disaster Scenarios In Scope
1. Region or zone outage.
2. Primary database unavailability.
3. Data corruption or accidental destructive change.
4. Message/event backlog accumulation due to downstream failure.
5. Secret/key compromise requiring emergency rotation.
6. Critical third-party provider outage.

## Recovery Strategy Baseline
1. Application continuity:
- Support traffic failover to healthy zones/regions.
- Prioritize Tier 1 services during degraded operations.

2. Data continuity:
- Regular backups with integrity checks.
- Point-in-time recovery for financial databases where supported.
- Immutable backup copies and secure storage.

3. Event continuity:
- Preserve outbox/inbox durability.
- Rebuild/replay strategy for delayed or interrupted processing.

4. Security continuity:
- Emergency key rotation and token revocation procedures.
- Controlled credential rollover for compromised secrets.

## Degraded Mode Operations
When full functionality is not possible:
- Maintain safe read-only or restricted-write mode for protected domains.
- Block non-essential high-risk operations until consistency is confirmed.
- Provide transparent service-status communication to stakeholders.

## Runbooks and Command Structure
Mandatory runbooks:
- Regional failover runbook
- Database restore and validation runbook
- Event replay and queue recovery runbook
- Key compromise response runbook
- Third-party outage compensation/reconciliation runbook

Incident command requirements:
- Declared incident roles (commander, technical lead, communications lead, scribe).
- Timeline and decision log captured during incident.
- Exit criteria for incident closure defined per scenario.

## Backup, Restore, and Integrity Validation
- Backup schedules documented per data tier.
- Restore testing cadence:
  - Tier 1: monthly sample restore validation
  - Tier 2: quarterly restore validation
  - Tier 3: semi-annual restore validation
- Post-restore integrity checks required:
  - ledger invariant validation
  - reconciliation checkpoints
  - audit continuity verification

## Communication and Stakeholder Management
- Incident status updates on defined cadence during major disruptions.
- Regulatory/compliance notification workflow if threshold conditions met.
- Internal and external communication templates maintained and versioned.

## DR Drill Policy
- At least two formal DR exercises per year for Tier 1 scope.
- At least one cross-functional tabletop exercise per quarter.
- Drill reports include:
  - achieved vs target RTO/RPO
  - failure points
  - corrective actions with owners and due dates

## Testing and Verification Requirements
1. Failover simulation tests in staging/pre-prod.
2. Database restore + invariant/reconciliation validation tests.
3. Event replay and backlog recovery tests.
4. Key compromise and credential-rotation drill tests.
5. Communications workflow drill validation.

## Governance and Evidence
- BC/DR evidence repository must include runbooks, drill logs, outcomes, and action closure status.
- Audit trail required for all emergency actions and privileged operations during incidents.
- Open corrective actions tracked and reviewed until closure.

## Consequences
Positive:
- Increased resilience against major outages and incidents.
- Better compliance confidence through tested recovery evidence.
- Faster and more predictable recovery for critical wallet services.

Trade-offs:
- Increased operational overhead for drills and documentation.
- Additional infrastructure and tooling costs for redundancy.

## Acceptance Criteria
1. Criticality tiers and RTO/RPO targets are approved.
2. Mandatory DR runbooks are implemented and accessible.
3. Backup/restore and integrity validation drills executed with evidence.
4. Cross-functional DR exercises produce tracked corrective actions.
5. Phase 8 readiness includes BC/DR sign-off with drill outcomes.