# TMS heartbeat / auto-push load test

Simulates a fleet of TMS terminals heartbeating on `tms/status/{serial}`
every ~5s, a fraction of which report an outdated version, to verify the
heartbeat/push path holds up under load: no connection drops, and every
version gap gets exactly one auto-push within a reasonable latency. See
`docs/HEARTBEAT_PUSH_SCALING.md` for the design and the numbers it produced.

## Design notes (read this first)

- **Why not 10,000 real TCP connections by default.** The bug being
  validated is the TMS server's *own* single MQTT connection getting
  backlogged and dropped — the broker fans messages from however many
  publishers into that one subscription, so what stresses the TMS side is
  aggregate message *rate*, not how many sockets the simulator itself opens.
  By default the simulator multiplexes 10,000 virtual devices over ~200 real
  connections (`LT_DEVICES_PER_CONNECTION=50`), which produces the identical
  load pattern on the TMS side while being much lighter on the shared
  broker. Set `LT_DEVICES_PER_CONNECTION=1` if you specifically also want to
  stress the broker's own concurrent-connection handling.
- **Why only emv_config/application get "gap" devices**, not
  parameter/keys_config: a parameter push needs a real `pos_terminal`/
  `pos_merchant` row in the MMS DB (synthetic LoadTest serials don't have
  one — it would just fail loudly for no useful signal), and a keys_config
  push calls the real `rki_endpoint` over HTTP. `PushGate` skips a type
  entirely when there's no active `config_file_versions`/`parameter_template`
  target for it, so simply not seeding those two keeps them out of the test
  cleanly (see `seed.py`'s docstring).
- **Isolation from real fleet data:** all synthetic terminals use
  `serial_number LIKE 'LOADTEST-%'` and `vendor = 'LoadTest'`
  (`model = 'mf919'` so it exercises the real MF919 push path —
  `MQTTCommandBuilder` routes on model, not vendor). `cleanup.py` purges
  everything scoped by those two, and touches nothing else.

## Prerequisites — do these before running anything

1. **Use an MQTT client id nobody else is using.** The app connects with
   `:mqtt_client_id` from `config/dev.exs`, and the default broker is shared.
   Two connections with the same client id keep kicking each other off, so
   the app cycles `connected` → `Socket closed before we handed it to the
   receiver` → `disconnected`, with empty SUBACKs, and receives nothing — the
   run then measures nothing. Set a unique id for the run and restore it
   afterwards.
2. **Bump the DB pool size.** `config/dev.exs` currently hardcodes
   `pool_size: 2` for both `DaProductApp.Repo` and `PlatformCore.Repo` — at
   even a few hundred devices/sec that will bottleneck immediately on
   `device_status`/`device_keys_push`/`push_lane` DB access and you'll be
   measuring the pool, not the fix. Bump both to something like `20` for the
   duration of the test, and set back to `2` (or whatever it was) afterward
   — this file has other people's local overrides in it, so don't leave it
   changed.
3. Make sure the app actually starts cleanly and connects
   (`Device <your client id> connected` in the logs, exactly once, and
   `TargetVersionCache: loaded targets for N vendor/model pairs`).

## Runbook

```bash
cd tools/terminal-heartbeat
python3 -m venv .venv && ./.venv/bin/pip install -r requirements.txt   # first time only
source .venv/bin/activate   # or just use ./.venv/bin/python everywhere below

# seed.py / cleanup.py need the MySQL password; it has no default on purpose
export LT_DB_PASSWORD=...

# 1. Seed the LoadTest emv_config + application target versions
./seed.py

# 2. Make sure the app's cache has picked the seed up — either restart the
#    app now (cleanest), or wait up to 60s for the periodic refresh, or
#    from an iex session attached to the running app:
#      TmsCore.TerminalManagement.TargetVersionCache.refresh()

# 3. ALWAYS smoke-test first (50 devices, 2 minutes) before the full run
./simulator.py --smoke

# 4. Check results/report-*.json and the app's own logs (see "What to watch"
#    below). If it looks healthy, run the full 10k pass:
./simulator.py                          # 10,000 devices, 30 min, default settings
# or, to size it yourself:
./simulator.py --devices 10000 --duration 1800

# 5. Clean up afterward
./cleanup.py
```

## What to watch during the run

**In the app's own logs:**
- No `ping_timeout` lines, no repeated `Device ... connected` (that would
  mean the connection is cycling — exactly the bug this is meant to catch).
- `PushLane`/`StatusIntake` error lines, if any.
- `HeartbeatBuffer: ... failed` lines — almost always DB pool exhaustion.
  Heartbeats no longer go through Oban, so the `device_status` queue should
  stay empty; if it grows, something is still enqueueing heartbeat work.
- `device_keys_push` queue depth isn't climbing unboundedly (keys_config /
  application pushes still complete through Oban).

**In `results/report-*.json`** (also printed at the end of the run):
- `disconnects` / `connect_failures`: should be ~0.
- `duplicate_pushes`: should be 0 — more than one push for the same
  serial/config_type within a run means the cooldown/gate isn't suppressing
  a repeat as it should.
- `gap_devices_never_pushed`: should be 0 by the end of the run — every
  seeded gap device should have received its push.
- `latency_seconds_by_type`: p50/p95/p99 time from "device started
  reporting a gap" to "push received". Single-digit seconds is the
  expectation post-fix; anything creeping into minutes suggests backlog is
  building again somewhere.

## Tuning (env vars, see `config.py` for the full list)

| Var | Default | Meaning |
|---|---|---|
| `LT_MQTT_HOST` / `LT_MQTT_PORT` | `demo.ctrmv.com` / `1883` | must match the running app's config |
| `LT_DEVICE_COUNT` | `10000` | total virtual devices |
| `LT_DEVICES_PER_CONNECTION` | `50` | real MQTT connections = device_count / this |
| `LT_HEARTBEAT_INTERVAL` | `5` | seconds between a device's reports |
| `LT_GAP_FRACTION` | `0.05` | fraction of devices that start non-compliant |
| `LT_DURATION` | `1800` | test length in seconds |
| `LT_CONNECT_RATE` | `50` | connections/sec during ramp-up |
| `LT_DB_NAME` | `shukria_transactions` | only used by `seed.py`/`cleanup.py` |
| `LT_DB_PASSWORD` | *(none — required)* | only used by `seed.py`/`cleanup.py` |
| `LT_TMS_REPO_ROOT` | the checkout this rig lives in | where the fixture files resolve from |

## Artifact modes

`l3_config_artifact_mode` defaults to `:shared` (one file per
`config_file_versions` row, reused by every terminal), which is what this
run exercises for emv_config. `application_artifact_mode` is still
`:per_device`; to validate `:shared` for it too, set it in
`config/config.exs`, restart the app, and re-run.

## Files

- `config.py` — all settings (env-var overridable)
- `seed.py` — creates the two `config_file_versions` rows the test needs
- `simulator.py` — the load generator; `--smoke` for a 50-device sanity check
- `cleanup.py` — purges everything the test created (asks for confirmation)
- `results/` — JSON reports land here (created on first run)

Fixture files this relies on, elsewhere in this repo (harmless, tiny, not
real device artifacts): `priv/static/loadtest/l3config_source.xml`,
`priv/static/loadtest/agent_app.apk`.