# MomentPay Monitoring System — Implementation Plan

**Reference:** [jPOS-monitoring](https://github.com/jpos/jPOS-monitoring)  
**Monitoring repo:** [momentpay/momentpay-monitoring](https://github.com/momentpay/momentpay-monitoring)  
**Goal:** Build a production-grade monitoring stack equivalent to jPOS-monitoring, tailored to the Mercury ISO 8583 switch (Elixir/Phoenix/BEAM).

---

## Two-Repository Strategy

Mirrors the jPOS-monitoring approach exactly — the application and the monitoring stack are maintained independently.

```
┌──────────────────────────────────────────┐     ┌─────────────────────────────────────────┐
│  mercury_device_middlelayer              │     │  momentpay-monitoring                    │
│  github.com/momentpay/mercury-device-*  │     │  github.com/momentpay/momentpay-monitoring│
│                                          │     │                                           │
│  Phase 1 — Elixir metrics code          │     │  Phase 2 — Docker Compose stack           │
│  ├── SwitchTelemetry                    │     │  ├── compose.yaml                         │
│  ├── SwitchMetrics (GenServer)          │  ←  │  ├── prometheus/                          │
│  ├── PrometheusExporter (extended)      │     │  └── grafana/                             │
│  └── GET /monitoring/metrics            │     │                                           │
│                                          │     │  Phase 3 — Grafana Dashboards             │
│  Must co-deploy with the switch          │     │  ├── dashboards/transactions.json         │
│  (BEAM node co-location required)        │     │  ├── dashboards/networks.json             │
│                                          │     │  ├── dashboards/beam_vm.json              │
│                                          │     │  └── dashboards/reversal_dashboard.json   │
│                                          │     │                                           │
│                                          │     │  Deploy independently:                    │
│                                          │     │  docker compose up -d                     │
└──────────────────────────────────────────┘     └─────────────────────────────────────────┘
```

**Why this split:**
- Phase 1 Elixir code must run on the same BEAM node as the switch — it cannot be deployed separately
- Phase 2 + 3 (Docker containers + JSON files) are pure infrastructure — they point at any running Mercury instance and have no Elixir dependency
- Ops team can update dashboards and alert rules without touching application code
- Matches jPOS-monitoring exactly: the monitoring stack is a standalone, independently versioned project

---

## What jPOS-monitoring provides vs. Mercury current state

| jPOS Concept | Mercury Equivalent | Current Status |
|---|---|---|
| ISO message throughput, TPS per MTI | `iso8583_*` metrics (0200/0210/0400/0410/0800) | **Missing** |
| MUX / QServer connection metrics | AsyncCorrelator per-network health & response times | Collected in ETS, **not exported to Prometheus** |
| TransactionManager participant timing | Per-processor latency histograms | **Missing** |
| JVM dashboard (heap, GC, threads) | BEAM VM dashboard (memory, processes, schedulers, GC) | **Missing** |
| Docker Compose monitoring stack | `compose.yaml` — Prometheus + Grafana + AlertManager | **Missing** |
| Grafana auto-provisioning | `provisioning/datasources/` + `provisioning/dashboards/` | **Missing** |
| `grafana/dashboards/jPOS.json` | `transactions.json` + `networks.json` | **Missing** |
| `grafana/dashboards/JVM.json` | `beam_vm.json` | **Missing** |

The existing Mercury monitoring covers only the **reversal subsystem**. This plan extends it to cover the full switch.

---

## Architecture Overview

```
┌─────────────────────────────────────────────────────────────────────┐
│              mercury_device_middlelayer (Elixir/Phoenix)             │
│                                                                       │
│  ISO 8583 Messages                                                    │
│       │                                                               │
│       ├─→ SwitchTelemetry.emit_*()  ←──── new (Phase 1)             │
│       │         │                                                     │
│       │         ↓                                                     │
│       │   SwitchMetrics (GenServer)  ←──── new (Phase 1)             │
│       │         │                                                     │
│       │   AsyncCorrelatorTelemetry   ←──── existing (tap into)       │
│       │         │                                                     │
│       │   ReversalMetrics            ←──── existing                  │
│       │         │                                                     │
│       └─────────↓                                                     │
│         PrometheusExporter  ←──────────── extended (Phase 1)         │
│         GET /monitoring/metrics  (port 4000)                         │
└──────────────────────────────────────────────────────────────────────┘
              │
              │ scrape every 10s
              ▼
┌──────────────────────────────────────────────────────────────────────┐
│              momentpay-monitoring  (Docker Compose)                   │
│                                                                       │
│  ┌─────────────────────┐        ┌────────────────────────────────┐   │
│  │     Prometheus       │        │           Grafana               │   │
│  │   localhost:9090     │───────▶│        localhost:3000           │   │
│  │                      │        │   4 auto-provisioned dashboards │   │
│  └─────────────────────┘        └────────────────────────────────┘   │
│                                                                       │
│  ┌─────────────────────┐                                             │
│  │    AlertManager      │                                             │
│  │   localhost:9093     │                                             │
│  └─────────────────────┘                                             │
│                                                                       │
│  docker compose up -d   ←── independently deployable                 │
└──────────────────────────────────────────────────────────────────────┘
```

---

## Phase 1 — Elixir Application Metrics

> **Repo:** `mercury_device_middlelayer`  
> **Branch:** `feature/switch-metrics` (merge to `main` when complete)  
> **Deliverable:** New metric families published at the existing `/monitoring/metrics` endpoint.  
> **Analogous to:** jPOS's built-in `-mp` metrics endpoint providing ISO/JVM metrics.

### 1.1 `SwitchTelemetry` module

**File:** `lib/da_product_app/mercury_iso8583/switch_telemetry.ex`

Emit `:telemetry` events at every ISO 8583 processing boundary. Pattern follows existing `ReversalTelemetry`.

**Events to emit:**

| Event | Measurements | Metadata | Where to call |
|---|---|---|---|
| `[:da_product_app, :switch, :transaction_received]` | `%{system_time: t}` | `%{mti: "0200", network: "jaywan", channel: pid}` | Message ingress point |
| `[:da_product_app, :switch, :transaction_processed]` | `%{duration_ms: n}` | `%{mti: "0200", network: "jaywan", outcome: :approved, response_code: "00"}` | After processing completes |
| `[:da_product_app, :switch, :transaction_routed]` | `%{routing_ms: n}` | `%{mti: "0200", target_network: "jaywan", rule: "issuer_bin"}` | After routing decision |
| `[:da_product_app, :switch, :network_connected]` | `%{system_time: t}` | `%{network: "jaywan", host: "192.168.1.1", port: 9100}` | Channel connect event |
| `[:da_product_app, :switch, :network_disconnected]` | `%{system_time: t}` | `%{network: "jaywan", reason: :timeout}` | Channel disconnect event |
| `[:da_product_app, :switch, :network_reconnecting]` | `%{attempt: n}` | `%{network: "jaywan", backoff_ms: 5000}` | Reconnect attempt |
| `[:da_product_app, :switch, :echo_test_sent]` | `%{system_time: t}` | `%{network: "jaywan"}` | 0800 echo sent |
| `[:da_product_app, :switch, :echo_test_received]` | `%{rtt_ms: n}` | `%{network: "jaywan", success: true}` | 0810 received |

**Key helper functions:**
```elixir
def emit_transaction_received(mti, network, channel_pid)
def emit_transaction_processed(mti, network, outcome, response_code, duration_ms)
def emit_network_connected(network, host, port)
def emit_network_disconnected(network, reason)
def event_names()  # Returns list of all events (for attach_many)
```

---

### 1.2 `SwitchMetrics` GenServer

**File:** `lib/da_product_app/telemetry/switch_metrics.ex`

Single GenServer that collects three metric families. Registered in `application.ex` after `ReversalMetrics`.

#### 1.2.1 ISO 8583 Transaction Metrics (analogous to jPOS TransactionManager metrics)

**Counters:**
- `iso8583_transactions_received_total{mti, network}` — messages entering the switch
- `iso8583_transactions_processed_total{mti, network, outcome}` — completed (outcome: approved/declined/error/timeout)
- `iso8583_transactions_routed_total{mti, target_network, rule}` — routing decisions taken
- `iso8583_echo_tests_total{network, success}` — 0800/0810 echo test results

**Gauges:**
- `iso8583_transactions_per_second{network}` — rolling 60s TPS per network (recalculated on `:calculate_tps` timer)
- `iso8583_active_transactions{network}` — in-flight transactions (increment on received, decrement on processed)

**Histograms:**
- `iso8583_processing_duration_seconds{mti, network}` — end-to-end processing time
- `iso8583_routing_duration_seconds{network}` — routing decision latency

#### 1.2.2 Network Channel Metrics (from AsyncCorrelatorTelemetry ETS, analogous to jPOS MUX/QServer metrics)

Pull from `AsyncCorrelatorTelemetry` ETS tables on each Prometheus scrape (or on a 10s timer).

**Gauges:**
- `network_connections_active{network}` — currently connected upstream channels
- `network_avg_response_time_ms{network}` — rolling average RTT per network

**Counters (from AsyncCorrelator ETS):**
- `network_requests_sent_total{network, instance}` — total requests transmitted
- `network_responses_received_total{network, instance, outcome}` — outcome: success/timeout/error
- `network_reconnects_total{network}` — total reconnect attempts since start
- `network_load_balance_decisions_total{network, instance}` — routing choices by instance

**Histograms:**
- `network_response_time_seconds{network}` — response time histogram (sampled from AsyncCorrelator stats)

#### 1.2.3 BEAM VM Metrics (analogous to jPOS JVM dashboard)

Collected on a 15s timer via `Process.send_after/3`.

**Gauges:**
- `beam_memory_total_bytes` — `:erlang.memory(:total)`
- `beam_memory_processes_bytes` — `:erlang.memory(:processes)`
- `beam_memory_binary_bytes` — `:erlang.memory(:binary)`
- `beam_memory_ets_bytes` — `:erlang.memory(:ets)`
- `beam_memory_atom_bytes` — `:erlang.memory(:atom)`
- `beam_process_count` — `:erlang.system_info(:process_count)`
- `beam_process_limit` — `:erlang.system_info(:process_limit)`
- `beam_run_queue_length` — `:erlang.statistics(:run_queue)`
- `beam_scheduler_count` — `:erlang.system_info(:schedulers_online)`
- `beam_port_count` — `:erlang.system_info(:port_count)`
- `beam_uptime_seconds` — `:erlang.statistics(:wall_clock)` element 0 / 1000

**Counters:**
- `beam_gc_count_total` — `:erlang.statistics(:garbage_collection)` element 0
- `beam_gc_words_reclaimed_total` — `:erlang.statistics(:garbage_collection)` element 1

**Gauges (scheduler utilization):**
- `beam_scheduler_utilization{scheduler_id}` — from `:scheduler_wall_time` (sampled 1s apart)

---

### 1.3 Extend `PrometheusExporter`

**File:** `lib/da_product_app/telemetry/prometheus_exporter.ex` (modify existing)

```elixir
def generate_prometheus_metrics do
  reversal_snapshot = ReversalMetrics.get_metrics_snapshot()
  switch_snapshot   = SwitchMetrics.get_metrics_snapshot()   # new
  timestamp = DateTime.utc_now() |> DateTime.to_unix(:millisecond)

  [
    generate_reversal_metrics(reversal_snapshot, timestamp),           # existing (renamed)
    generate_iso8583_metrics(switch_snapshot.transactions, timestamp), # new
    generate_network_metrics(switch_snapshot.networks, timestamp),     # new
    generate_beam_vm_metrics(switch_snapshot.beam_vm, timestamp),      # new
  ]
  |> Enum.join("\n")
end
```

### 1.4 Wire into supervision tree

**File:** `lib/da_product_app/application.ex` (modify)

```elixir
# 2.4. Switch Metrics Collector: ISO 8583 + network + BEAM VM observability
DaProductApp.Telemetry.SwitchMetrics,
```

### 1.5 Call sites — where to place `SwitchTelemetry.emit_*`

| Caller file | Function | Event to emit |
|---|---|---|
| Channel ingress handler (incoming listener) | On message receipt | `emit_transaction_received/3` |
| Message processor / router | After routing | `emit_transaction_routed/4` |
| Response handler | On response sent to device | `emit_transaction_processed/5` |
| Network connector (upstream supervisor) | On TCP connect | `emit_network_connected/3` |
| Network connector | On TCP disconnect/crash | `emit_network_disconnected/2` |
| Echo test handler | On 0800 send / 0810 recv | `emit_echo_test_sent/1`, `emit_echo_test_received/3` |

**Note:** Identify exact module names during implementation by grepping for message ingress and network lifecycle callbacks.

### 1.6 Phase 1 files summary

| File | Action |
|---|---|
| `lib/da_product_app/mercury_iso8583/switch_telemetry.ex` | **Create** |
| `lib/da_product_app/telemetry/switch_metrics.ex` | **Create** |
| `lib/da_product_app/telemetry/prometheus_exporter.ex` | **Modify** — add 3 new metric generators |
| `lib/da_product_app/application.ex` | **Modify** — add SwitchMetrics to supervision tree |

### 1.7 Phase 1 acceptance criteria

- [ ] `curl localhost:4000/monitoring/metrics` returns `iso8583_*`, `network_*`, and `beam_vm_*` families alongside existing `reversal_*`
- [ ] BEAM VM gauges update every 15 seconds
- [ ] TPS gauge reflects real transaction rate during load testing
- [ ] No regressions to existing reversal metrics

---

## Phase 2 — Docker Compose Monitoring Stack

> **Repo:** `momentpay-monitoring` (https://github.com/momentpay/momentpay-monitoring)  
> **Deliverable:** Clone the repo, run `docker compose up -d`, monitoring is live — no manual config.  
> **Analogous to:** jPOS-monitoring's `compose.yaml` + `prometheus/` + `grafana/provisioning/`.

### Repository structure

```
momentpay-monitoring/
├── compose.yaml
├── .env.example                          # MERCURY_HOST, MERCURY_PORT, GF_ADMIN_PASSWORD
├── prometheus/
│   ├── prometheus.yaml
│   └── alert_rules/
│       ├── reversal_alerts.yml           # ported from mercury_device_middlelayer
│       ├── switch_alerts.yml             # new
│       └── beam_vm_alerts.yml            # new
├── alertmanager/
│   └── alertmanager.yml
├── grafana/
│   ├── provisioning/
│   │   ├── datasources/
│   │   │   └── datasource.yaml
│   │   └── dashboards/
│   │       └── dashboards.yaml
│   └── dashboards/
│       ├── reversal_dashboard.json       # ported from mercury_device_middlelayer
│       ├── transactions.json             # new — Phase 3
│       ├── networks.json                 # new — Phase 3
│       └── beam_vm.json                  # new — Phase 3
├── justfile                              # convenience commands (same as jPOS-monitoring)
└── README.md
```

### 2.1 `compose.yaml`

Services:
- **prometheus** — `prom/prometheus:v2.51.0`, port `9090:9090`, mounts `./prometheus` config + persistent volume
- **grafana** — `grafana/grafana:10.4.0`, port `3000:3000`, mounts `./grafana`, env vars from `.env`
- **alertmanager** — `prom/alertmanager:v0.27.0`, port `9093:9093`

Network: `momentpay-monitoring` bridge.  
Volumes: `prometheus_data`, `grafana_data` (persistent across restarts).

**Target host configuration via `.env`:**
```env
MERCURY_HOST=host.docker.internal   # Docker Desktop (Windows/Mac)
                                     # Use host-gateway on Linux
MERCURY_PORT=4000
GF_SECURITY_ADMIN_PASSWORD=changeme
```

`host.docker.internal` resolves to the Docker host machine — Prometheus can reach the Phoenix app without network changes.

### 2.2 `prometheus/prometheus.yaml`

```yaml
global:
  scrape_interval: 15s
  external_labels:
    project: momentpay
    environment: production

scrape_configs:
  - job_name: mercury-switch
    metrics_path: /monitoring/metrics
    scrape_interval: 10s
    static_configs:
      - targets: ['${MERCURY_HOST}:${MERCURY_PORT}']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'iso8583_.*'
        target_label: component
        replacement: switch
      - source_labels: [__name__]
        regex: 'network_.*'
        target_label: component
        replacement: upstream-networks
      - source_labels: [__name__]
        regex: 'beam_.*'
        target_label: component
        replacement: beam-vm
      - source_labels: [__name__]
        regex: 'reversal_.*'
        target_label: component
        replacement: reversal-system

  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
```

### 2.3 `prometheus/alert_rules/switch_alerts.yml`

| Alert | Condition | Severity |
|---|---|---|
| `SwitchTPSDropped` | TPS drops >50% vs 5m baseline | warning |
| `SwitchHighErrorRate` | Error outcomes >5% of processed (2m) | critical |
| `SwitchHighLatencyP95` | p95 processing time >10s (3m) | warning |
| `NetworkConnectionDown` | `network_connections_active == 0` for any network (1m) | critical |
| `NetworkResponseTimeDegraded` | avg RTT >5s for any network (2m) | warning |
| `NetworkHighTimeoutRate` | timeout responses >10% of sent (2m) | critical |
| `EchoTestFailing` | Echo test failure count >0 for any network (5m) | warning |

### 2.4 `prometheus/alert_rules/beam_vm_alerts.yml`

| Alert | Condition | Severity |
|---|---|---|
| `BEAMHighMemory` | Total memory >2GB (adjust to deployment) | warning |
| `BEAMHighProcessCount` | Process count >90% of `beam_process_limit` | warning |
| `BEAMHighRunQueue` | Run queue length >100 sustained 2m | warning |
| `BEAMSchedulerSaturated` | Any scheduler utilization >90% (3m) | warning |

### 2.5 `grafana/provisioning/datasources/datasource.yaml`

```yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true
    access: proxy
    jsonData:
      httpMethod: GET
      timeInterval: "10s"
```

### 2.6 `grafana/provisioning/dashboards/dashboards.yaml`

```yaml
apiVersion: 1
providers:
  - name: MomentPay Switch Dashboards
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards
```

### 2.7 `justfile` (convenience — same pattern as jPOS-monitoring)

```just
# Start monitoring stack
up:
    docker compose up -d

# Stop monitoring stack
down:
    docker compose down

# View logs
logs:
    docker compose logs -f

# Validate prometheus config
check:
    docker run --rm -v ./prometheus:/prometheus prom/prometheus:v2.51.0 \
        promtool check config /prometheus/prometheus.yaml

# Validate alert rules
check-rules:
    docker run --rm -v ./prometheus:/prometheus prom/prometheus:v2.51.0 \
        promtool check rules /prometheus/alert_rules/*.yml

# Open Grafana in browser
open:
    open http://localhost:3000

# Open Prometheus in browser
prometheus:
    open http://localhost:9090
```

---

## Phase 3 — Grafana Dashboards

> **Repo:** `momentpay-monitoring`  
> **Files:** `grafana/dashboards/`  
> **Deliverable:** Four production dashboards, auto-provisioned on `docker compose up`.  
> **Analogous to:** jPOS-monitoring's `jPOS.json` + `JVM.json`.

All dashboards share:
- Dark theme, 30s auto-refresh
- `$network` template variable (multi-select, populated from metric labels)
- Time range: last 1 hour default
- UID pattern: `momentpay-{type}`
- Tags: `momentpay`, `iso8583`, `production`

---

### 3.1 `transactions.json` — ISO 8583 Transaction Dashboard

**Analogous to:** jPOS `jPOS.json` (TransactionManager + ISO throughput panels)

**Row 1 — Overview (Stat panels):**
- Total TPS — sum of `iso8583_transactions_per_second` across all networks
- Overall success rate % — approved / (approved + declined + error + timeout), 5m window
- Active in-flight transactions — sum of `iso8583_active_transactions`
- p95 processing latency — from `iso8583_processing_duration_seconds`

**Row 2 — Throughput (Time Series):**
- TPS per network — `rate(iso8583_transactions_received_total[$__rate_interval])` by `network`
- Processed outcomes — stacked area: approved / declined / error / timeout
- Active transactions over time

**Row 3 — Latency (Time Series):**
- Processing p50 / p95 / p99 by MTI type
- Routing duration p95 by network

**Row 4 — Per-MTI breakdown (Table):**
- Columns: MTI | Network | Received | Processed | Success% | p95 Latency
- Color threshold on Success%: red <95%, yellow 95-99%, green ≥99%

**Row 5 — Echo Tests (Stat + Time Series):**
- Echo test success rate per network (Stat with threshold coloring)
- Echo test RTT over time

**Template variables:**
- `$network` — `label_values(iso8583_transactions_received_total, network)`, multi-select
- `$mti` — static: 0200, 0210, 0400, 0410, 0800, All

---

### 3.2 `networks.json` — Network Connections Dashboard

**Analogous to:** jPOS `jPOS.json` MUX / QServer panels

**Row 1 — Connection Overview (Stat panels):**
- Total active connections across all networks
- Networks with 0 active connections (red if >0)
- Total reconnect attempts since start
- Average RTT across all networks

**Row 2 — Per-Network Status (Table):**
- Columns: Network | Active Connections | Requests Sent | Success Rate | Avg RTT | Reconnects
- Row color: green = connected + low RTT, yellow = degraded RTT, red = disconnected

**Row 3 — Request Flow (Time Series):**
- Requests sent rate per network
- Responses by outcome — stacked: success / timeout / error
- Reconnect events over time (bar marker)

**Row 4 — Response Time (Time Series):**
- p50 / p95 / p99 RTT per network
- Timeout rate % per network

**Row 5 — Load Balancing (Time Series):**
- Load balance decisions per instance
- Per-instance request share % (pie or stacked bar)

**Template variables:**
- `$network` — from `network_requests_sent_total` label
- `$instance` — from `instance` label, filtered by selected `$network`

---

### 3.3 `beam_vm.json` — BEAM VM Dashboard

**Analogous to:** jPOS `JVM.json` (heap, GC, threads, class loading)

**Row 1 — Memory Overview (Gauge panels):**
- Total memory
- Process memory
- Binary memory
- ETS memory
- Atom memory

**Row 2 — Memory Over Time (Time Series, stacked):**
- All memory categories over time
- Memory growth trend line

**Row 3 — Processes & Ports (Stat + Time Series):**
- Process count vs. process limit (% used, red if >90%)
- Port count
- Run queue length (red if >100)

**Row 4 — Scheduler Utilization (Time Series):**
- One line per scheduler (`beam_scheduler_utilization{scheduler_id}`)
- Useful for detecting uneven load distribution or a hot scheduler

**Row 5 — Garbage Collection (Time Series):**
- GC rate — `rate(beam_gc_count_total[$__rate_interval])`
- Words reclaimed rate
- GC pressure indicator (GC time % if available)

**Row 6 — System Info (Stat):**
- Uptime (formatted as human-readable duration)
- Scheduler count
- Process limit

---

### 3.4 `reversal_dashboard.json` — Reversal System Dashboard (ported)

- Port from `config/grafana/reversal_dashboard.json` in `mercury_device_middlelayer`
- Add navigation links to the three new dashboards (dashboard links panel at top)
- Add `$network` template variable
- Update UID to `momentpay-reversals`

---

## Implementation Order

```
Phase 1  (mercury_device_middlelayer — branch: feature/switch-metrics)
  1.  SwitchTelemetry module
  2.  SwitchMetrics GenServer — transactions + BEAM VM metric collection
  3.  SwitchMetrics — network metrics section (tap AsyncCorrelatorTelemetry ETS)
  4.  Extend PrometheusExporter with 3 new generators
  5.  Wire SwitchMetrics into application.ex supervision tree
  6.  Add SwitchTelemetry.emit_* call sites across the codebase
  7.  Test: curl /monitoring/metrics and verify all families present

Phase 2  (momentpay-monitoring — new repo)
  8.  Create repository at github.com/momentpay/momentpay-monitoring
  9.  compose.yaml with prometheus + grafana + alertmanager
  10. prometheus/prometheus.yaml (scrape + relabeling)
  11. prometheus/alert_rules/ — switch_alerts.yml + beam_vm_alerts.yml
  12. Port reversal_alert_rules.yml from mercury_device_middlelayer
  13. grafana/provisioning/ — datasource.yaml + dashboards.yaml
  14. .env.example + justfile + README
  15. Test: docker compose up -d → Prometheus targets UP

Phase 3  (momentpay-monitoring — same repo, same branch as Phase 2)
  16. grafana/dashboards/transactions.json
  17. grafana/dashboards/networks.json
  18. grafana/dashboards/beam_vm.json
  19. grafana/dashboards/reversal_dashboard.json (ported + updated)
  20. Test: all 4 dashboards appear in Grafana without manual import
```

---

## Key Design Decisions

| Decision | Choice | Reason |
|---|---|---|
| Repo split | Phase 1 in app repo, Phase 2+3 in `momentpay-monitoring` | Matches jPOS-monitoring; ops can deploy monitoring independently |
| Metric collection style | In-memory GenServer (same pattern as ReversalMetrics) | No new deps; consistent with existing codebase |
| Network metrics source | AsyncCorrelatorTelemetry ETS | Already collected there; avoids double-counting |
| BEAM metrics collection | Direct `:erlang` calls on 15s timer | Most accurate; no library needed |
| TPS calculation | Rolling 60s window in GenServer | Avoids Prometheus `rate()` unreliability on low-traffic counters |
| Grafana provisioning | File-based auto-provisioning | Same as jPOS-monitoring; zero manual UI setup on `docker compose up` |
| Prometheus storage | Docker named volume | Persists across restarts; same as jPOS-monitoring |
| Target host config | `.env` file with `MERCURY_HOST` / `MERCURY_PORT` | Operators can point at any Mercury instance without editing YAML |
| Dashboard UIDs | `momentpay-{type}` prefix | Consistent naming; avoids Grafana UID collisions |

---

## Acceptance Criteria

### Phase 1
- [ ] `curl localhost:4000/monitoring/metrics` returns `iso8583_*`, `network_*`, and `beam_vm_*` families alongside existing `reversal_*`
- [ ] BEAM VM gauges update every 15 seconds
- [ ] TPS gauge reflects real transaction rate during load
- [ ] No regressions to existing reversal metrics or endpoints

### Phase 2
- [ ] `git clone github.com/momentpay/momentpay-monitoring && docker compose up -d` completes with no errors
- [ ] Prometheus at `localhost:9090/targets` shows `mercury-switch` target as UP
- [ ] `just check-rules` passes with no errors for all alert rule files
- [ ] AlertManager UI accessible at `localhost:9093`

### Phase 3
- [ ] Grafana at `localhost:3000` shows all 4 dashboards without any manual import
- [ ] `transactions.json` TPS panel shows non-zero values during load testing
- [ ] `beam_vm.json` memory panel shows live BEAM memory breakdown
- [ ] `networks.json` shows per-network connection status with correct labels
- [ ] `reversal_dashboard.json` navigation links reach the other three dashboards