# Operations Runbooks

Purpose: provide execution steps for common operational procedures before production rollout.

---

## 1. Deployment Runbook

1. Verify CI is green for target release.
2. Confirm readiness gate P0 items are complete.
3. Deploy to staging and run smoke checks:
- /health/live
- /health/ready
- one sync transaction flow
- one async job flow
4. Promote to production with rolling strategy.
5. Monitor error rate, latency, and circuit breaker metrics for 30 minutes.

Rollback trigger examples:
- error rate above threshold for 5 minutes
- sustained readiness failures
- critical auth failure pattern

---

## 2. Rollback Runbook

1. Stop further traffic shift.
2. Roll back to previous stable release.
3. Validate /health/ready and critical flow checks.
4. Record incident summary and root-cause owner.

---

## 3. Incident Response Runbook

1. Classify severity (SEV1-SEV3).
2. Assign incident commander and communications owner.
3. Collect trace_id samples, failing endpoint/topic, and impacted tenants.
4. Apply mitigation:
- disable affected route rule
- open maintenance response for impacted endpoints
- increase timeout only if validated safe
5. Publish incident updates every 15 minutes until stable.

---

## 4. JWT/API Key Rotation Runbook

1. Generate and store new key material in secret manager.
2. Activate overlap window for verification of old and new keys.
3. Rotate signing/verification config.
4. Revoke compromised keys immediately when needed.
5. Confirm auth success/failure metrics stabilize after rotation.

---

## 5. DLQ Operations Runbook

1. Review DLQ depth and failure categories.
2. Validate if payloads are safe and reprocessable.
3. Re-queue in controlled batches.
4. Discard only with approval and incident record.
5. Create permanent fix issue for recurring error signatures.

---

## 6. On-Call Ownership

| Area | Primary Team | Secondary Team |
|------|---------------|----------------|
| North gateways | Team North | Team Core |
| Core pipeline and auth | Team Core | Team Infra |
| South adapters | Team South | Team Core |
| Infra and telemetry | Team Infra | Team South |

---

## 7. Operational Evidence Retention

Keep for each incident/deployment:
- deployment ID and timestamp
- impacted endpoints/topics
- trace IDs used for diagnosis
- metrics snapshots
- final resolution and follow-up action items