# TMS Load-Test Findings for Application Developers

**Date:** 2026-09-26  
**Scope:** Heartbeat ingestion and automatic EMV/application pushes against the isolated `shukria_transactions_loadtest` database. The MQTT broker was `demo.ctrmv.com:1883`. Phoenix, MySQL, and the load generator ran on the same four-core host.

## Executive Summary

The 1k pass completed cleanly. At 5k, transport remained healthy, but the application did not keep up with push work: median gap-to-push latency was 182 seconds, 244 expected config pushes were not observed within five minutes, and duplicate deliveries occurred. At 10k, the simulator reported substantial MQTT publish errors and almost all gaps remained unresolved. These measurements include contention from running Phoenix, MySQL, and the generator on the same host.

The slow `device_keys_push` Oban jobs are not the primary explanation for the 5k delay: all 128 application jobs in that run began within one second on average and completed in about 0.2 seconds on average. The stronger lead is backlog or contention earlier in MQTT status intake / `PushLane`, amplified by same-host CPU and database I/O pressure. The available reports do not measure per-stage queue age, so this remains a hypothesis to instrument, not a proven single root cause.

The simulator counted duplicate push deliveries, but the cloned DB has only one push-log row per RERUN5K terminal/config-type pair. That argues against duplicate app-level publish attempts. QoS 1 redelivery is plausible, but the test report does not record MQTT DUP flags or correlate repeated deliveries by `requestId`, so the exact cause is unconfirmed.

## Test Results

All runs used a five-minute duration and 5-second heartbeat cadence. The 5k and 10k reruns used 100 and 200 simulator MQTT connections respectively (50 virtual devices per connection). CPU is reported on a four-core aggregate scale where 400% is full host capacity.

| Devices | Heartbeats | Publish errors | Mid-run disconnects | Pushes received | Duplicate deliveries | Gaps still unpushed | CPU peak |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 1,000 | 59,991 | 0 | 0 | 100 | 0 | 0 | 344.7% / 400% |
| 5,000, rerun | 300,038 | 0 | 0 | 278 | 22 | 244 | 347.5% / 400% |
| 10,000, rerun | 327,223 | 272,188 | 1 | 121 | 64 | 943 | 385.9% / 400% |

Reports: `tools/terminal-heartbeat/results/report-20260926-173810.json` (1k), `report-20260926-181346.json` (5k), `report-20260926-182525.json` (10k).

At 5k, gap-to-push latency was 182.4 seconds p50 and about 300 seconds p95 for both EMV and application. The simulator received 278 messages, including 22 repeats; only 256 distinct terminal/config-type combinations were delivered by the five-minute deadline. The cloned DB contained one publish-log row per pair: 128 rows per type. Only 49 of those 256 rows had reached `acknowledged` status at inspection time; this needs ACK-correlation investigation because the simulator counts ACK publish attempts, not broker-confirmed or DB-recorded ACKs.

The 10k run connected all 200 simulator workers and had no connection-attempt failures, but publish errors reached 272,188. This makes file-descriptor exhaustion an unlikely explanation for that run; it does not rule out broker-side publish-rate limits or host saturation.

## Application Path and Evidence

1. `DaProductApp.MQTT.Handler` receives `tms/status/+` and dispatches reports to `StatusIntake`. `StatusIntake` is partitioned 8 ways by serial; its worker decodes the report, updates the in-memory heartbeat buffer, calculates version gaps, and casts gaps to `PushLane`.
2. `PushLane` is partitioned 16 ways by serial. Its worker processes each partition sequentially. It checks the in-memory in-flight map, and for outdated gaps also checks the 30-minute DB cooldown before publishing.
3. EMV pushes are handled inline in the push-lane worker. Application pushes are reserved there and handed to `SlowPushWorker` on the `device_keys_push` Oban queue, configured at concurrency 3.
4. In the 5k run, the 128 application slow-push jobs started within 1 second on average and completed within 2 seconds maximum. Their queue delay does not explain the approximately 182-second p50 end-to-end latency.
5. During the tests, `PushLane` status logs showed dozens of gaps still in flight with oldest gaps over 50 seconds at 5k and over 100 seconds under higher load. This is evidence of unfinished push work, but no message-queue depth or per-stage timing was captured.
6. The 10k run averaged about 1,091 heartbeats/sec over five minutes versus a nominal 2,000/sec at the configured cadence; this was observed while the app, MySQL, and generator shared the same host and the simulator itself reported many publish errors.

Relevant implementation: `apps/da_product_app/lib/da_product_app/mqtt/handler.ex`, `apps/tms_core/lib/tms_core/terminal_management/status_intake/worker.ex`, `apps/tms_core/lib/tms_core/terminal_management/push_lane.ex`, `apps/tms_core/lib/tms_core/terminal_management/push_lane/worker.ex`, `apps/tms_core/lib/tms_core/terminal_management/heartbeat_buffer.ex`, `apps/tms_core/lib/tms_core/terminal_management/workers/slow_push_worker.ex`.

## Duplicate Delivery Analysis

At 5k, the simulator observed 278 push messages, but SQL found just 128 application and 128 EMV publish-log rows, each for a distinct terminal/type pair. Thus the 22 extra messages were not accompanied by repeat app push-log inserts for those pairs. The app's in-flight guard has a 60-second default TTL; outdated gaps also have a DB-backed 30-minute cooldown for pending/sent pushes.

The simulator uses QoS 1 for its subscriptions. MQTT QoS 1 is at-least-once and permits duplicate deliveries. The simulator currently increments `duplicate_pushes` by serial/config type and does not expose packet DUP flags or repeated `requestId` values. Therefore this evidence favors broker/client redelivery over repeated app publishing, but does not prove it. The app ACK handler routes `ota/ack/+` payloads through `OtaService.handle_ota_acknowledgment`; the gap between simulator ACK attempts and acknowledged DB rows should be investigated independently.

## Recommended App-Side Follow-Up

1. Add low-overhead queue-age and mailbox-depth measurements for all `StatusIntake` and `PushLane` partitions. Capture time from MQTT receipt to intake handling, gap detection, lane handling, publish acceptance, and simulator receipt. Use histograms/counters rather than per-heartbeat INFO logs.
2. Expose the existing `PushLane.status/0` summary during a run, and correlate oldest in-flight age with partition mailbox lengths. Also capture `HeartbeatBuffer.stats/0` (`pending`, `requeued`, `conflated`, `logs_written`, `items_written`) and Repo pool checkout/queue-time metrics.
3. Log/correlate each publish and ACK by `(serial, config_type, requestId)`. For duplicates, record broker/client DUP metadata if available. Change the simulator report to distinguish duplicate deliveries with the same request ID from new commands for the same serial/type.
4. Reconcile the 5k run's 278 simulator ACK publish attempts with the 49 `parameter_push_logs` rows marked acknowledged. Verify the ACK arrived at the app, matched the intended push-log row, and completed before shutdown; do not treat the simulator's local `acks_sent` counter as a broker-confirmed ACK.
5. Repeat capacity measurements with the generator on a separate machine. Keep Phoenix and its cloned DB on the target host so app capacity is measured independently of load-generator CPU, memory, and disk I/O.
6. After obtaining per-stage timings, evaluate whether to tune the 8 `StatusIntake` / 16 `PushLane` partitions, the shared TMS/MySQL pool size, or heartbeat persistence settings. Avoid raising partition counts or Oban concurrency blindly; the current 5k evidence does not identify the Oban worker queue as the bottleneck.

## Current Test-Safety State

After the runs, the Phoenix process was stopped, `application_push_enabled` was restored to `false`, and synthetic rows were retained in the cloned DB as requested. The 5k/10k test outcomes are from a same-host setup and should not be used as isolated broker capacity figures.