# Runbook 8.1 — P95 Latency Profiling at 2000 TPS

## Goal
Confirm the scoring pipeline stays below 50 ms P95 under 2000 sustained TPS.

## Prerequisites
- Docker Compose stack running: MySQL, Redis, VerneMQ
- `RISK_SCORING_ENABLED=true`, `RISK_MODEL_SERVING=true`
- `k6` (or `vegeta`) installed on the load-driver machine

## Test Setup

```bash
# 1. Compile a release build (eliminates Mix overhead)
MIX_ENV=prod mix release

# 2. Start the release (single node)
_build/prod/rel/mw_core/bin/mw_core start

# 3. Pre-populate seed data (rules, model)
_build/prod/rel/mw_core/bin/mw_core eval "InfraRepo.Repo.seed()"
```

## k6 Script (save as load_test.js)

```javascript
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    sustained: {
      executor: 'constant-arrival-rate',
      rate: 2000,
      timeUnit: '1s',
      duration: '3m',
      preAllocatedVUs: 400,
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<50'],
    http_req_failed:   ['rate<0.01'],
  },
};

const BASE = 'http://localhost:4000';
const HEADERS = { 'Content-Type': 'application/json', 'Authorization': 'Bearer <test_token>' };

export default function () {
  const payload = JSON.stringify({
    amount: Math.random() * 1000,
    currency: 'EUR',
    from_account: '411111XXXXXX' + Math.floor(Math.random() * 10000),
    to_account: 'merchant_' + (Math.floor(Math.random() * 100) + 1),
    ip_address: `10.0.${Math.floor(Math.random() * 255)}.${Math.floor(Math.random() * 255)}`,
  });

  const res = http.post(`${BASE}/api/v1/transactions`, payload, { headers: HEADERS });

  check(res, {
    'status 200': (r) => r.status === 200,
    'has risk_score': (r) => JSON.parse(r.body).risk_score !== undefined,
  });
}
```

## Running the Test

```bash
k6 run load_test.js
```

## Interpreting Results

| Metric               | Pass Threshold |
|----------------------|---------------|
| P95 response time    | < 50 ms       |
| P99 response time    | < 100 ms      |
| Error rate           | < 1%          |
| Redis P95 read time  | < 3 ms (check via `redis-cli INFO stats`) |

## Flamegraph (if P95 fails)

```bash
# Attach async-profiler (JVM not applicable — use Erlang observer)
# In IEx on the running node:
:observer.start()
# Navigate to Load Charts → Processes → ScoringPipeline
# Or use recon_trace for function-level tracing:
:recon_trace.calls({MwRisk.ScoringPipeline, :score, 2}, 100, [])
```

## Known Bottlenecks to Investigate First
1. `RedisTier.pipeline/1` — every transaction makes 5–8 Redis calls; pool contention at > 1500 TPS
2. `ModelServer.predict/2` — IsolationForest pathScore is O(n_trees * depth); profile with Scholar models
3. MySQL async write queue — VelocityPipeline Broadway batcher; increase `:risk_velocity` Oban concurrency