# ADR-007 — OpenTelemetry for Distributed Tracing

**Date:** 2026-04-26
**Status:** Accepted
**Deciders:** Architecture Team

---

## Context

A single request through MW-Core can touch:
- A gateway (gateway_api)
- The auth layer (mw_auth)
- The routing pipeline (mw_router)
- One or more adapters (adapter_banking)
- The audit log (mw_audit)
- The database (infra_repo)

Without correlation, debugging production issues requires manually stitching together log
lines from multiple apps. With clustering (multiple nodes), this becomes practically impossible.

We need distributed tracing that:
1. Correlates all processing stages for a single request into one trace
2. Shows timing breakdowns per stage
3. Exports to a vendor-neutral backend
4. Adds minimal overhead to the hot path

---

## Decision

Adopt **OpenTelemetry (OTel)** with OTLP export. Implemented in `infra_telemetry`.

Libraries:
- `opentelemetry_api` — span creation API (zero-cost when no exporter configured)
- `opentelemetry_sdk` — SDK and batch exporter
- `opentelemetry_exporter_otlp` — OTLP gRPC/HTTP export to Jaeger, Tempo, Honeycomb
- `opentelemetry_phoenix` — auto-instrument all Phoenix requests (one span per request)
- `opentelemetry_ecto` — auto-instrument all Ecto queries (one span per DB query)
- `opentelemetry_finch` — auto-instrument all Finch HTTP calls (adapter outbound spans)

---

## Trace Structure

A full transaction trace looks like:

```
[gateway_api] POST /api/v1/transactions                          250ms
  ├── [mw_auth] jwt.verify                                         2ms
  ├── [mw_router] rate_limit.check                                 1ms
  ├── [mw_transform] inbound.map                                   3ms
  ├── [mw_router] route.resolve                                   <1ms
  ├── [adapter_banking] banking.send                             180ms
  │     ├── [finch] HTTP POST https://cbs.internal/tx            175ms
  │     └── [adapter_banking] response.transform                   3ms
  ├── [mw_transform] outbound.map                                  2ms
  ├── [mw_audit] event.write (async)                               8ms
  └── [infra_repo] INSERT audit_events                             6ms
```

---

## Trace Context Propagation

Every `MwKernel.Context` struct carries `trace_id` and `span_id` as OTel context.

```elixir
defmodule MwKernel.Context do
  defstruct [
    :trace_id,    # binary, from OTel span context
    :span_id,
    # ...
  ]
end
```

When `mw_router` hands off to an adapter, the OTel context is propagated via process
dictionary (`:otel_ctx` key) — standard OTel Elixir pattern. `opentelemetry_finch`
automatically injects `traceparent` / `tracestate` headers into outbound HTTP calls,
enabling trace propagation into internal systems that also support OTel.

---

## Metrics (Telemetry)

In addition to traces, `infra_telemetry` defines Telemetry.Metrics that feed Prometheus:

```elixir
counter("mw_router.request.count", tags: [:adapter, :status])
summary("mw_router.request.duration", unit: {:native, :millisecond}, tags: [:adapter])
counter("mw_auth.failure.count", tags: [:reason])
counter("adapter_banking.circuit.open.count")
last_value("infra_queue.dlq.depth")
```

Prometheus scrapes `/metrics` every 15s.
Grafana dashboards are templated and provisioned via ConfigMap.

---

## Sampling Strategy

| Environment | Sampling Rate | Rationale |
|-------------|--------------|-----------|
| Development | 100% | Full visibility during development |
| Staging | 100% | Full visibility for integration testing |
| Production | 10% (head-based) + 100% for errors | Balance cost vs visibility |

Error traces are always sampled via a tail-based sampler configured in the OTel Collector.

---

## Structured Logging

All log lines include `trace_id` as a structured field:

```elixir
Logger.metadata(trace_id: context.trace_id, tenant_id: context.tenant_id)
Logger.info("adapter dispatch", adapter: adapter, message_type: message.type)
```

Log aggregation (Loki, Elasticsearch) can correlate logs with OTel traces via `trace_id`.

---

## Alternatives Considered

| Option | Rejected Because |
|--------|-----------------|
| Custom correlation ID only | Loses timing breakdown; no span parent-child relationships |
| Datadog APM (`dd-trace`) | Vendor lock-in; commercial only |
| New Relic agent | Same — vendor lock-in |
| Manual span creation only | OTel auto-instrumentation for Phoenix/Ecto/Finch eliminates most boilerplate |

---

## Consequences

### Positive
- Vendor-neutral: switch from Jaeger to Tempo to Honeycomb by changing exporter config
- Auto-instrumentation covers Phoenix, Ecto, Finch with zero application code
- OTel `trace_id` becomes the universal correlation key across logs, traces, and audit events
- `opentelemetry_api` is a no-op when no SDK is configured — zero prod overhead if exporter is off

### Negative
- OTel SDK adds ~5MB to the release
- OTLP export adds network traffic (mitigated by batching in the SDK)
- OTel Elixir libraries are still maturing; some API changes between minor versions
  — pin library versions in `mix.exs`