# ADR 0007: Observability and Audit Traceability Standard

- Status: Accepted
- Date: 2026-03-11
- Owners: Platform Team, SRE Team, Compliance Team
- Related:
  - `docs/non-functional-slo.md`
  - `docs/adr/0002-eventing-and-outbox.md`
  - `docs/adr/0003-financial-ledger-invariants.md`
  - `docs/adr/0006-security-and-key-management-baseline.md`
  - `docs/phase-tracker.md`

## Context
The platform requires end-to-end traceability for wallet transactions, security events, and compliance audits. Without a standardized observability model, troubleshooting, incident response, and audit evidence become inconsistent across domain apps.

This ADR defines one telemetry and audit standard for all OTP apps.

## Decision
Adopt a unified observability model with mandatory structured logging, distributed tracing, metric standards, and immutable audit event requirements.

Core decisions:
1. Every request and domain command must carry correlation and request identifiers.
2. Every critical business action emits an audit event with standardized fields.
3. Metrics and traces use common naming and label policies.
4. Dashboards and alerts are mandatory for phase gates.

## Trace and Correlation Standard
Required identifiers:
- `request_id`
- `correlation_id`
- `causation_id` (for event-driven chains)
- `actor_id` (where permitted)
- `tenant_id` (where applicable)

Propagation rules:
1. `wallet_web` creates/accepts `request_id` and `correlation_id`.
2. Downstream calls and events propagate both IDs unchanged.
3. Consumers create `causation_id` links to source event/message id.
4. Missing IDs are generated at ingress and logged as generated.

## Structured Logging Standard
Required log fields:
- `timestamp`
- `level`
- `service` (otp app name)
- `request_id`
- `correlation_id`
- `event` or `command`
- `status`
- `error_code` (if present)
- `duration_ms` (for timed operations)

Rules:
- JSON structured logs only in non-dev environments.
- No secrets/tokens/PII in plaintext logs.
- Redaction policy mandatory for sensitive keys and payload fields.

## Metrics Standard
Metric naming:
- Prefix: `wallet.`
- Example: `wallet.api.request.count`, `wallet.transfer.authorize.latency.p95`

Required metric groups:
1. API health:
- request count, success/failure rates, latency by endpoint and status.

2. Auth/security:
- login success/failure, OTP challenge outcomes, token refresh results.

3. Financial core:
- posting attempts/success/failure, invariant violations, reversal counts.

4. Eventing:
- outbox backlog, dispatch latency, retry counts, dead-letter counts.

5. Async workers:
- queue depth, queue wait time, job success/failure rates.

6. Reconciliation:
- mismatch counts, mismatch rate, settlement completion latency.

Label policy:
- allowed: `service`, `endpoint`, `status_code`, `error_code`, `queue`, `event_name`
- avoid high-cardinality labels in metrics (IDs belong in traces/logs, not metric labels).

## Audit Event Standard
Mandatory audit event fields:
- `audit_id`
- `occurred_at`
- `actor_type` (`user|service|admin|system`)
- `actor_id`
- `action`
- `resource_type`
- `resource_id`
- `outcome` (`success|failure|denied`)
- `reason_code` (if failure/denied)
- `request_id`
- `correlation_id`
- `metadata` (non-sensitive contextual attributes)

Mandatory audited actions:
- authentication and session lifecycle actions
- token issue/refresh/revoke
- financial posting and reversal operations
- transfer state transitions
- policy/rule changes
- privileged/admin operations
- key rotation and secret-access failures

Storage and retention:
- audit stream is append-only and tamper-evident.
- retention and access controls follow compliance policy.

## Dashboards and Alerts (Minimum)
Required dashboards:
1. API reliability and latency
2. Auth and security health
3. Transfer/ledger critical path
4. Event bus and outbox/inbox health
5. Worker queues and settlement/reconciliation
6. Audit event volume and failure trends

Required alert classes:
- SLO burn-rate alerts (warning, critical)
- invariant violation alert (critical)
- dead-letter growth alert
- queue saturation alert
- audit pipeline failure alert

## Data Access and Compliance
- Production observability data access is least privilege.
- Audit log access requires role-based approval.
- Query access and export operations are themselves auditable.

## Test and Verification Requirements
1. Trace propagation tests across API -> command -> event -> consumer path.
2. Log schema tests for required fields and redaction.
3. Metrics emission tests for critical endpoints and workflows.
4. Audit event tests for all mandatory audited actions.
5. Failure-injection tests validating alerts and dashboards.

## Consequences
Positive:
- Faster root-cause analysis and better incident response.
- Strong evidence trail for compliance and audits.
- Measurable and enforceable reliability goals.

Trade-offs:
- Increased telemetry ingestion/storage cost.
- Additional implementation overhead for schema consistency.

## Acceptance Criteria
1. All OTP apps emit required trace/log fields.
2. Mandatory dashboards and alerts are live in non-prod.
3. Mandatory audit actions are captured with complete fields.
4. SLO-linked telemetry from `docs/non-functional-slo.md` is queryable and alertable.
5. Phase 5 and Phase 8 observability gates pass with evidence.