# Runbook 8.6 — Chaos Test: MySQL Write Failure

## Goal
Verify the scoring pipeline handles MySQL write failures gracefully:
- Async risk_score writes drop silently (Oban retries)
- Synchronous transaction response is not blocked by DB failures
- No BEAM crash; health endpoint returns healthy

## Expected Behaviour
| Write path            | MySQL up    | MySQL down                        |
|-----------------------|-------------|-----------------------------------|
| risk_scores insert    | Async Oban  | Oban job retries (up to 3×)      |
| risk_cases insert     | Async Oban  | Oban job retries                  |
| Transaction response  | Synchronous | Not affected (write is async)     |
| Health /ready         | ok          | degraded (DB check fails)         |

## Step-by-Step

### 1. Baseline
```bash
curl -s http://localhost:4000/health/ready | jq .
```

### 2. Kill MySQL
```bash
docker compose stop mysql
```

### 3. Send transactions — responses should still succeed
```bash
for i in $(seq 1 10); do
  curl -s -X POST http://localhost:4000/api/v1/transactions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer <token>" \
    -d '{"amount":250,"currency":"EUR","from_account":"4111111111111111","to_account":"merchant_42"}' | jq '{status: .status, risk_decision: .risk_decision}'
done
```

**Pass criteria:**
- HTTP 200 on all requests
- `risk_decision` present
- No 500 errors

### 4. Check Oban queue for failed jobs
After MySQL comes back, verify retries succeeded:
```bash
# In IEx:
import Ecto.Query
InfraRepo.Repo.all(from j in Oban.Job, where: j.queue == "risk_scoring" and j.state == "retryable")
```

### 5. Restore MySQL
```bash
docker compose start mysql
```

### 6. Verify /health/ready returns ok
```bash
sleep 15
curl -s http://localhost:4000/health/ready | jq .
```

### 7. Verify Oban drained the retryable jobs
```bash
# After ~2 minutes Oban should have retried and completed
InfraRepo.Repo.aggregate(from(j in Oban.Job, where: j.queue == "risk_scoring" and j.state == "retryable"), :count)
# Expect: 0
```

## Log signatures to watch
```
[warning] Oban.Engines.Dolphin: job failed attempt=1 reason="DBConnection.ConnectionError"
[info]    Oban job risk_scoring:N completed after retry
```

## Rollback
If Oban jobs exhausted retries (`state = "discarded"`): manually re-enqueue or
accept data loss for that window (decision was already returned to the caller).