# Runbook 8.5 — Chaos Test: Redis Failure

## Goal
Verify the platform degrades gracefully (rules-only scoring, no crash) when
Redis becomes unavailable.

## Expected Behaviour
| Component           | Redis up      | Redis down (circuit blown) |
|---------------------|---------------|---------------------------|
| Feature hydration   | Full velocity | Type-A only (payload)     |
| Sanctions check     | Redis + ETS   | ETS fallback (stale OK)   |
| ML scoring          | Enabled       | Enabled (ETS model)       |
| VelocityPipeline    | Writes Redis  | Drops update, logs warn   |
| RedisTier.cmd/1     | {:ok, result} | {:error, :circuit_open}   |

## Circuit Breaker Parameters (infra_feature_store)
- Melts after 5 failures within 10s
- Resets after 30s

## Step-by-Step

### 1. Baseline check
```bash
curl -s http://localhost:4000/health/ready | jq .
# Expect: {"status":"ok"}
```

### 2. Send a healthy transaction and record baseline risk_score
```bash
curl -s -X POST http://localhost:4000/api/v1/transactions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"amount":100,"currency":"EUR","from_account":"4111111111111111","to_account":"merchant_1","ip_address":"10.0.0.1"}' | jq .risk_score
```

### 3. Kill Redis
```bash
docker compose stop redis
```

### 4. Send transactions — verify degraded-mode scoring
```bash
for i in $(seq 1 20); do
  curl -s -X POST http://localhost:4000/api/v1/transactions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer <token>" \
    -d '{"amount":100,"currency":"EUR","from_account":"4111111111111111","to_account":"merchant_1","ip_address":"10.0.0.1"}' | jq '{risk_score: .risk_score, risk_decision: .risk_decision}'
done
```

**Pass criteria:**
- All 20 requests return HTTP 200
- `risk_score` and `risk_decision` are present (rules-only)
- Velocity features are all 0.0 (no Redis data)
- No 500 errors, no BEAM crash

### 5. Check logs for expected warnings
```bash
grep -E "(circuit_open|redis_unavailable|degraded)" _build/prod/rel/mw_core/log/erlang.log
```
Expect: `[warning] RedisTier: circuit open — skipping Redis read`

### 6. Restore Redis and verify self-healing
```bash
docker compose start redis
sleep 35   # Wait for circuit reset (30s + buffer)

curl -s -X POST http://localhost:4000/api/v1/transactions \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"amount":100,"currency":"EUR","from_account":"4111111111111111","to_account":"merchant_1","ip_address":"10.0.0.1"}' | jq .risk_score
```
**Pass criteria:** `risk_score` returns a non-zero velocity-enriched score.

## Rollback
If the circuit does not reset: `docker compose restart redis` and restart the app.