# Heartbeat & push scaling — what changed, and what is still partial Written for engineers picking up the terminal heartbeat / auto-push path. This records the state after `perf/heartbeat-ingest-conflation`, and — more importantly — the parts that were deliberately left unfinished, so that if something misbehaves later you know where to look first. ## Why any of this exists At ~1000 terminals on a 5s heartbeat the system was arriving at ~200 heartbeats/sec and draining far slower. The visible symptom was push latency (p95 ~20s, p99 ~26s), but that was downstream of the real problem: every heartbeat did several DB round trips and wrote 7 rows, so the `device_status` Oban queue grew without bound (45,133 rows measured, draining at ~4.7 jobs/sec) and starved the connection pool that the push path also needs. An indexed lookup on a 15-row table measured 0.74ms standalone and ~100ms through the app while that backlog drained. A heartbeat is a *current-state* signal. Once a newer report for the same terminal arrives, the older one carries nothing anyone reads back. That is the assumption the whole design now rests on — see "If this assumption ever changes" below. ## Measured outcome (1000 devices, same rig, same duration) | | before | after | |------------------|----------|---------| | push p50 | 4.077s | 2.081s | | push p95 | 20.635s | 5.082s | | push p99 | 25.780s | 5.390s | | duplicate pushes | 14 | 0 | | `oban_jobs` backlog | 45,133 | 0 | | `status_log` rows written | ~120,000 | ~2,000 | Reproduce with the rig in `tools/terminal-heartbeat/` — its README has the full runbook (`./seed.py`, `./simulator.py --smoke`, then `./simulator.py --devices 1000 --duration 600`). Two things it insists on, both of which invalidated runs while this was being measured: raise `pool_size` in `config/dev.exs` for the duration (at the default of 2 you are measuring the pool, not the change), and use an MQTT client id nobody else holds on the shared broker. ## What is still partial — check these first if pushes misbehave ### 1. `application` and `keys_config` never got the treatment `emv_config` did Three of the changes were applied only to `parameter` and `emv_config`: - **Target threading.** `PushGate` resolves each gap's target from `TargetVersionCache` and passes it down the gap tuple. `push_type/7` in `push_lane/worker.ex` uses it for `parameter`/`emv_config` and discards it (`_target`) for `keys_config`/`application`. Those two still call `ConfigFileVersion.find_active/3` per push — see `auto_push_service.ex` (currently around the `reserve_keys_push` / `reserve_application_push` paths and `push_regular_config`). Four such call sites remain. - **Shared artifacts.** `l3_config_artifact_mode` is now `:shared`; `application_artifact_mode` is still `:per_device`, so the application package is staged (copied + hashed) per terminal. The `:shared` machinery already exists for it — the config comment notes the two modes are independent precisely so one could move ahead of the other. - **Artifact size/checksum memoisation.** `artifact_meta/1` only memoises paths under `/ota/l3/`, so the application artifact is re-stat'ed and re-SHA256'd on every push. **Why it was left:** both types are disabled in this dev environment (`keys_push_enabled: false`, `application_push_enabled: false` in `config/dev.exs`) because `keys_config` calls a real RKI HTTP endpoint and `application` needs a staged APK that isn't present locally. Neither could be exercised or verified here, and the load-test rig deliberately doesn't seed them either. They are also *already off the hot path* — both are reserved inline and completed by `SlowPushWorker` on the Oban `device_keys_push` queue (concurrency 3), so their per-push DB lookup costs far less than it did on the synchronous lane. **When this will bite:** a mass application rollout. At that point every device in the rollout pays a `find_active` query plus a per-device APK copy and hash. If you see the `device_keys_push` queue backing up or the pool saturating during an upgrade campaign, this is the cause, and the fix is the same three changes applied to these two types. ### 2. Retention / partitioning for status history — not started Write volume is down ~98%, but nothing prunes `tms_terminal_status_logs` or `tms_terminal_status_items`, and there is no partitioning. At 10k terminals the residual write rate is manageable; the *table growth* is still unbounded. Time-partitioning by date and dropping old partitions is the intended fix — a `DELETE` on a table that size is an outage. ### 3. `nil` vs `""` in `StatusReport.extract_versions/1` — decided: leave as is **Decision (confirmed with the team):** real devices always send every version itemkey, so the current behaviour stays. Only revisit this if a firmware change starts sending partial payloads. The analysis below is kept so that decision can be re-checked quickly if that ever happens. `extract_versions/1` returns `nil` for an itemkey the device didn't send, and `PushGate` treats `nil` identically to `""`: ```elixir {target, reported} when reported in [nil, ""] -> [{type, :missing, target}] ``` So **"the device didn't report this field" is indistinguishable from "the device has no version installed"**, and a partial payload triggers a push of *all four* config types. This was reproduced deliberately: a heartbeat carrying only a `status` itemkey produced four `:missing` gaps. Changing it would be a device-facing semantic decision: if any firmware omitted a version field when it had nothing installed, treating absent as "no information" would silently stop pushing to those devices. The in-flight guard (below) bounds the damage if a partial payload ever does arrive — which is how a hand-built `mosquitto_pub` test with only a `status` itemkey behaves: all four types push, once. ### 4. The in-flight guard is per-node, in-process state `push_lane/worker.ex` keeps `%{{serial, type} => timestamp}` in the worker's own GenServer state, which is sound because `PushLane` partitions by serial. But it is **not** shared across nodes and does not survive a restart. On a multi-node deployment two nodes could each push the same missing config once. The `:outdated` path's cooldown is DB-backed and does survive, which is why it was left as-is rather than folded into the same mechanism. ### 5. `Oban.Plugins.Lifeline` still absent Jobs `executing` when a node dies are never rescued — 370 such rows were found, the oldest over a day old. This matters much less now that heartbeats no longer go through Oban, but `device_keys_push` still does. ## If this assumption ever changes Everything above is built on: **heartbeat history is operational, not regulatory — losing past heartbeats is acceptable, current state is what matters.** That was confirmed explicitly when the design was chosen. If an audit or compliance requirement later demands every heartbeat be retained verbatim, the conflating buffer is the wrong shape and should be revisited first — it discards superseded reports by design, and `HeartbeatBuffer.stats/0` reports how many (`:conflated`) it has absorbed. ## Operational signals worth watching - `HeartbeatBuffer.stats/0` — `:conflated` rising sharply means flushes are falling behind ingest; `:pending` is bounded by fleet size by construction; `:requeued` climbing means persists keep failing transiently and being retried; `:logs_written` / `:items_written` are the durable write volume. - `HeartbeatBuffer: ... retrying next tick` in the log — a transient DB failure (almost always pool exhaustion); the reports were put back and will be written on the next flush. Occasional is fine; continuous is not. - `HeartbeatBuffer: dropping status for , rejected by the DB` — that terminal sent a value the schema won't accept. It's retried only at the snapshot interval, so a repeat every few minutes points at the device. - `device_status` queue depth should now stay at zero. If it grows, something is still enqueueing heartbeat work through Oban.