# CloudLayer Prometheus Metrics

CloudLayer exposes Prometheus-format metrics at:

```
GET /monitoring/metrics
```

No authentication pipeline is applied to this route (matches Mercury's
`mercury_device_middlelayer` convention) — restrict access at the network/proxy
layer in production.

## Implementation

Hand-rolled exporter following Mercury's pattern — **no external Prometheus
hex dependency was added**. Uses only `Plug`, `:telemetry`, `telemetry_metrics`
and `telemetry_poller`, all of which were already dependencies of CloudLayer.

| Component | File | Role |
|---|---|---|
| Emitters | `lib/da_product_app/telemetry/business_telemetry.ex` | `:telemetry.execute/3` calls for payment and external-API events |
| Collector | `lib/da_product_app/telemetry/metrics_collector.ex` | Supervised `GenServer`; attaches to telemetry events, aggregates counters/histograms, polls BEAM VM every 15s |
| Exporter | `lib/da_product_app/telemetry/prometheus_exporter.ex` | `Plug`/controller that renders the collector's snapshot as Prometheus text format (`text/plain; version=0.0.4`) |

The collector is started as an ordinary child in `DaProductApp.Application`'s
supervision tree. The route is registered in `DaProductAppWeb.Router` outside
any auth pipeline.

## Metrics

All label sets are bounded/low-cardinality (fixed set of providers, HTTP
methods, route patterns from the router, status codes, and `:success`/`:failure`
result tags) — no raw IDs, emails, or free-text values are used as labels.

All four duration metrics (`payment_duration_seconds`, `external_api_duration_seconds`,
`http_request_duration_seconds`, `db_query_duration_seconds`) are true Prometheus
histograms — `_bucket{le="..."}`, `_sum`, `_count` — using the fixed boundaries
`[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, +Inf]` seconds.

### Application

| Metric | Type | Labels | Description |
|---|---|---|---|
| `application_info` | gauge | `version` | Always `1`; `version` carries the app's `mix.exs` version |
| `application_build_info` | gauge | `version` | Always `1`; same version, exposed under the conventional build-info metric name (no git SHA/build timestamp is captured at build time, so only version is available) |
| `application_uptime_seconds` | gauge | — | Seconds since the application supervisor started |

### Payment (`QRMiddleLayerController.processTransaction/2`)

| Metric | Type | Labels | Description |
|---|---|---|---|
| `payment_requests_total` | counter | `provider`, `result` | Total payment/QR provider flows completed (`result`: `success`\|`failure`) |
| `payment_duration_seconds` | histogram | `provider` | Time from provider-call start to flow completion |

Emitted once per completed provider call in
`QRMiddleLayerController.process_validated_transaction/1`, covering the main
QR-generation payment flow (Alipay + AANI). The provider-call span (`call_provider/2`
through `handle_provider_response/7`) is wrapped in `try/rescue`: if provider-response
handling raises, a `result="failure"` sample is recorded exactly once and the original
exception is re-raised unchanged (`reraise/2` with the original stacktrace) — the crash
still propagates and the HTTP response is unaffected, only the metric now exists.
Earlier validation branches (device/provider not found, invalid amount, DB failure
creating the transaction) are unchanged and still don't emit — only the provider-call
span is instrumented.

### External provider APIs (Alipay/AANI)

| Metric | Type | Labels | Description |
|---|---|---|---|
| `external_api_requests_total` | counter | `provider`, `operation`, `result` | Total calls to Alipay/AANI HTTP APIs (`operation`: `generate`\|`inquire`\|`cancel`\|`refund`\|`status`) |
| `external_api_duration_seconds` | histogram | `provider`, `operation` | External HTTP call latency |

Emitted from `instrumented_post/4` in `DaProductApp.QRProviders.Alipay` and
`DaProductApp.QRProviders.Aani`, wrapping every `HTTPoison.post/3` call. A
call is `:success` when the HTTP status is 2xx, `:failure` otherwise
(non-2xx, transport error, or `HTTPoison.post/3` raising — the raise case
records `result="failure"` and re-raises the original exception unchanged).

### HTTP (Phoenix router)

| Metric | Type | Labels | Description |
|---|---|---|---|
| `http_requests_total` | counter | `route`, `method`, `status` | Total requests dispatched by the router, by matched route pattern (e.g. `/api/getCardTransactionById/:id`) |
| `http_request_duration_seconds` | histogram | `route`, `method` | Router-dispatch duration |
| `http_requests_active` | gauge | — | Requests currently in flight through the endpoint |

Sourced from Phoenix's built-in `[:phoenix, :router_dispatch, :stop]` (route,
method, status, duration) and the existing `Plug.Telemetry` plug already
configured in `DaProductAppWeb.Endpoint` (`[:phoenix, :endpoint, :start]`/`:stop`,
for the active-request gauge). No new Plug was added.

### Ecto / database

| Metric | Type | Labels | Description |
|---|---|---|---|
| `db_queries_total` | counter | `repo` | Total Ecto queries executed, by repo |
| `db_query_errors_total` | counter | `repo` | Queries whose driver result was `{:error, _}`, by repo |
| `db_query_duration_seconds` | histogram | `repo` | Query duration, by repo |

`repo` is `"main"` for `DaProductApp.Repo` (database `shukria_transactions`) or
`"shukria_mms"` for `DaProductApp.Repos.ShukriaMmsRepo` (database
`shukria_mms_new_local`) — the only two Ecto repos in the app, sourced from their
respective built-in telemetry events (`[:da_product_app, :repo, :query]` and
`[:da_product_app, :repos, :shukria_mms_repo, :query]`). No query-level label is
used to avoid unbounded cardinality from ad-hoc query sources.

### BEAM VM

| Metric | Type | Labels |
|---|---|---|
| `beam_memory_total_bytes`, `beam_memory_processes_bytes`, `beam_memory_binary_bytes`, `beam_memory_ets_bytes`, `beam_memory_atom_bytes` | gauge | — |
| `beam_process_count`, `beam_process_limit`, `beam_port_count` | gauge | — |
| `beam_run_queue_length`, `beam_scheduler_count` | gauge | — |
| `beam_uptime_seconds` | gauge | — |
| `beam_scheduler_utilization` | gauge | `scheduler_id` |

Sampled every 15s via `:erlang.memory/0`, `:erlang.statistics/1` and
`scheduler_wall_time`, following the same sampling approach as Mercury's
`SwitchMetrics` collector.

## What is intentionally out of scope (follow-up)

- Per-route/per-provider gauges beyond what's listed above.
- HTTP metrics for requests that don't match any router (raw 404s) — these
  don't emit `router_dispatch` events; only `http_requests_active` reflects them.
- Distinguishing DB error *types* (constraint violation vs. timeout vs.
  connection failure) — `db_query_errors_total` is a single counter today.
- CPU percentage — the BEAM does not expose this directly; `beam_scheduler_utilization`
  is the standard proxy metric.