# Runbook: Health Check and SLO Monitoring

**Service:** WalletWeb Admin Console + Platform Observability
**Owner:** SRE Team
**Escalation:** Platform Team → Architecture Team

---

## Overview
Procedures for monitoring system health, SLO (Service Level Objective) compliance, and responding to degradation alerts.

## Prerequisites
- Admin console access with `sre` or `admin` role
- Alert notification access (PagerDuty/Slack)
- Grafana/observability dashboard access (if configured)

## Health Check Dashboard

### Accessing Health Status

1. Log in to `/admin/login`
2. Navigate to `/admin/platform/health`

3. Dashboard shows:
   - **Overall Health**: Aggregate status (healthy, degraded, critical)
   - **Component Health**: Individual service/dependency statuses
   - **Last Check Time**: Timestamp of most recent health probe
   - **Recent Incidents**: Open health-related incidents

### Health Check Components

| Component | Checked | Healthy Criteria |
|---|---|---|
| Database | Connection pool | >80% available connections |
| Redis/Cache | Ping + SETEX test | <100ms response |
| External APIs | HTTP health endpoint | 2xx response in <5s |
| Queue/PubSub | Pub + sub echo test | Message delivered <1s |
| Ledger Balance | Sample balance query | Returns without error |
| Auth/Session | Token generation test | JWT sign + verify succeeds |

### Interpreting Status

- 🟢 **Healthy**: All checks passing, system operating normally
- 🟡 **Degraded**: 1-2 non-critical checks failing, partial functionality impacted
- 🔴 **Critical**: Core checks failing (DB, auth), major functionality unavailable

---

## SLO (Service Level Objective) Monitoring

### Accessing SLO Dashboard

Navigate to `/admin/platform/slo`

### Defined SLOs

| SLO | Target | Measurement Window | Alert Threshold |
|---|---|---|---|
| **Transfer Success Rate** | ≥99.5% | Rolling 7 days | <99.0% |
| **Login Success Rate** | ≥99.9% | Rolling 24 hours | <99.5% |
| **API Latency (p95)** | ≤500ms | Rolling 1 hour | >750ms |
| **Page Load Time (p95)** | ≤2s | Rolling 1 hour | >3s |
| **Uptime** | ≥99.95% | Rolling 30 days | <99.9% |

### SLO Violation Workflow

1. **Alert Received** (via PagerDuty/Slack)
   - Alert includes: SLO name, current value, target, time window

2. **Acknowledge**
   - Log in to `/admin/platform/slo`
   - Locate violated SLO
   - Click "View Details" for violation timeline

3. **Assess Impact**
   - Check if violation is:
     - **Active**: Currently below target (urgent)
     - **Historical**: Was below target, now recovered (review needed)
   - Check dashboard graphs for trend:
     - Sudden drop → incident (investigate immediately)
     - Gradual decline → degradation (investigate root cause)

4. **Investigate**
   - Click "View Related Incidents" to see if incident already created
   - Check health dashboard for correlated component failures
   - Review error logs for time window of violation

5. **Mitigate**
   - If active incident exists: follow [Incident Response Runbook](./incident-response.md)
   - If degraded component identified:
     - Apply circuit breaker (see [Service Controls](./service-controls.md))
     - Scale up capacity (if resource exhaustion)
     - Roll back recent deployment (if correlation exists)

6. **Document**
   - Create/update incident record in `/admin/platform/incidents`
   - Link SLO violation to incident
   - Document root cause and remediation

7. **Resolve and Review**
   - After SLO returns to target:
     - Mark incident as resolved
     - Schedule post-incident review (PIR)
     - Update runbooks if new failure mode discovered

---

## Common SLO Violations

### Transfer Success Rate Drop

**Possible Causes:**
- External payment gateway outage
- Database connection pool exhaustion
- Ledger balance calculation errors

**Investigation:**
1. Check health dashboard → payment gateway circuit state
2. Check transaction inquiry dashboard → filter by status "failed"
3. Review failure reasons distribution

**Actions:**
- If gateway outage: open circuit breaker, disable external transfers
- If DB pool exhaustion: scale up DB connections or reduce load
- If ledger errors: escalate to Platform Team

### Login Success Rate Drop

**Possible Causes:**
- Auth service degradation
- Session store (Redis) failure
- Credential stuffing attack (high invalid login rate)

**Investigation:**
1. Check health dashboard → auth + Redis status
2. Check login attempt logs for error patterns
3. Check rate of failed login attempts per IP

**Actions:**
- If auth/Redis failure: restart service, check resource limits
- If credential stuffing: apply stricter rate limits (see Service Controls)

### API Latency Spike

**Possible Causes:**
- Database slow query
- Memory pressure (GC pauses)
- Dependency timeout (network issue)

**Investigation:**
1. Check health dashboard → DB query times
2. Check application metrics → heap usage, GC frequency
3. Check external dependency response times

**Actions:**
- If DB slow: identify slow queries via EXPLAIN, add indexes
- If memory: increase heap size or scale horizontally
- If dependency timeout: adjust timeout config or apply circuit breaker

---

## Scheduled Health Checks

SRE team should perform manual health verification:
- **Daily**: Review SLO dashboard during business hours start
- **Weekly**: Export SLO violation report for trend analysis
- **Monthly**: SLO compliance review meeting with stakeholders

---

## Alerts Configuration

Health and SLO alerts are configured via:
- `apps/wallet_resilience` → SLO specs and violation detection
- PagerDuty integration (production)
- Slack webhooks (staging + production)

**Alert Escalation Policy:**
1. On-call SRE (immediate)
2. SRE Lead (15 minutes)
3. Platform Team Lead (30 minutes)

---

## Troubleshooting

### Health Dashboard Shows Stale Data
- **Check**: Last check timestamp (top right of dashboard)
- **Action**: Click "Refresh Now" to trigger manual health check
- **If Still Stale**: Health check scheduler may be down → restart wallet_resilience app

### SLO Shows Violation but No Impact Observed
- **Possible**: False positive due to measurement window lag
- **Action**: Wait 5 minutes and refresh
- **If Persists**: Check SLO calculation logic in wallet_resilience codebase

### Cannot Access Health Dashboard
- **Check**: Verify you have `sre` or `admin` role
- **Check**: `:require_sre` pipeline in router
- **Escalate**: If role is correct but access denied → security misconfiguration

---

## Audit and Compliance
- SLO violation records retained for 12 months
- Monthly SLO compliance reports generated for:
  - Executive team (summary)
  - Architecture team (detailed)
- SLO targets reviewed quarterly and adjusted based on business requirements

---

## Related Runbooks
- [Service Controls and Circuit Breakers](./service-controls.md)
- [Incident Response](./incident-response.md)
- DR Rehearsal (SRE internal)