# Runbook 8.14 — 5000 TPS Load Test (P99 < 100 ms)

## Goal
Sustain 5000 transactions per second with P99 latency below 100 ms and
error rate below 0.1%.

## Infrastructure Requirements
| Component  | Minimum for 5000 TPS              |
|------------|-----------------------------------|
| App nodes  | 2× (horizontal, behind HAProxy)   |
| MySQL      | RDS db.r6g.2xlarge or equivalent  |
| Redis      | ElastiCache r6g.large (cluster mode optional) |
| VerneMQ    | 2-node cluster (if MQTT enabled)  |
| k6 runner  | 16 vCPU, 32 GB RAM                |

## Pre-test Checklist
- [ ] Release build deployed (`MIX_ENV=prod`)
- [ ] Redis `maxmemory` set to 4 GB with `allkeys-lru` eviction
- [ ] MySQL `max_connections=500`, connection pool configured
- [ ] Oban queues tuned: `risk_scoring: 100`, `risk_velocity: 200`
- [ ] `RISK_SCORING_ENABLED=true`, `RISK_MODEL_SERVING=true`
- [ ] Monitoring dashboards open

## k6 Script

```javascript
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    ramp_up: {
      executor: 'ramping-arrival-rate',
      startRate: 500,
      timeUnit: '1s',
      stages: [
        { duration: '2m', target: 5000 },
        { duration: '10m', target: 5000 },
        { duration: '1m', target: 0 },
      ],
      preAllocatedVUs: 1000,
      maxVUs: 2000,
    },
  },
  thresholds: {
    http_req_duration: ['p(99)<100', 'p(95)<50'],
    http_req_failed:   ['rate<0.001'],
  },
};

const HOSTS = ['http://app1:4000', 'http://app2:4000'];
const HEADERS = { 'Content-Type': 'application/json', 'Authorization': 'Bearer <token>' };

export default function () {
  const host = HOSTS[Math.floor(Math.random() * HOSTS.length)];
  const res = http.post(`${host}/api/v1/transactions`, JSON.stringify({
    amount: 10 + Math.random() * 990,
    currency: 'EUR',
    from_account: '41111111' + String(Math.floor(Math.random() * 100000000)).padStart(8, '0'),
    to_account: 'merchant_' + (Math.floor(Math.random() * 500) + 1),
    ip_address: `${Math.floor(Math.random()*254)+1}.${Math.floor(Math.random()*255)}.0.1`,
  }), { headers: HEADERS });

  check(res, {
    'ok':         (r) => r.status === 200,
    'has_score':  (r) => !!JSON.parse(r.body).risk_score,
  });
}
```

## Running

```bash
k6 run --out json=results.json load_test_5000.js
# Parse results:
k6 inspect results.json | jq '.metrics | {p99: .http_req_duration.values["p(99)"], p95: .http_req_duration.values["p(95)"], err_rate: .http_req_failed.values.rate}'
```

## Pass / Fail Criteria

| Metric              | Pass      | Fail        | Action                              |
|---------------------|-----------|-------------|-------------------------------------|
| P99 latency         | < 100 ms  | ≥ 100 ms    | Profile Redis pool / DB write path  |
| P95 latency         | < 50 ms   | ≥ 50 ms     | Profile ModelServer / RuleCache     |
| Error rate          | < 0.1%    | ≥ 0.1%      | Check circuit breaker / DB pool     |
| Redis miss rate     | < 5%      | ≥ 5%        | Increase ETS cache size             |
| Oban queue depth    | < 1000    | ≥ 1000      | Scale risk_velocity concurrency     |

## Tuning Knobs (if P99 fails)

```elixir
# config/prod.exs
config :mw_risk, risk_velocity_pipeline: true

config :infra_repo, Oban,
  queues: [risk_scoring: 100, risk_velocity: 200, risk_model: 10]

config :infra_feature_store,
  ets_max_size: 200_000,   # was 50_000
  ets_default_ttl_ms: 60_000
```

## Post-test Artefacts
- k6 HTML report: `k6 run --out web-dashboard load_test_5000.js`
- Store signed report: `docs/fraud-extension/security/load-test-<date>.pdf`