# Phase 6 — Production Hardening

**Duration:** Weeks 15–16
**Status:** ✅ Complete — 2026-04-26
**Goal:** Enterprise-grade reliability, security, observability, and zero-downtime operations.

---

## Deliverable

1. OpenTelemetry traces exported to Jaeger/Tempo with full span coverage
2. Prometheus `/metrics` endpoint with Grafana dashboards
3. Multi-node clustering via `libcluster` + `Horde`
4. Zero-downtime deploys with health check + drain
5. Secrets management via environment-encrypted runtime config (or Vault)
6. Load testing results with defined SLA targets
7. Security review checklist complete

---

## Tasks

### 1. OpenTelemetry — Full Instrumentation

**Dependencies to add:** `opentelemetry`, `opentelemetry_api`, `opentelemetry_exporter_otlp`,
`opentelemetry_phoenix`, `opentelemetry_ecto`, `opentelemetry_finch`

```elixir
# config/runtime.exs
config :opentelemetry_exporter,
  otlp_protocol: :http_protobuf,
  otlp_endpoint: System.get_env("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4318")

config :opentelemetry,
  span_processor: :batch,
  traces_exporter: :otlp,
  resource: [
    service: [
      name: "mw-core",
      version: System.get_env("RELEASE_VERSION", "dev")
    ]
  ]
```

Add custom spans at adapter dispatch:

```elixir
defmodule MwRouter.Dispatcher do
  require OpenTelemetry.Tracer, as: Tracer

  def dispatch(context, adapter_module) do
    Tracer.with_span "adapter.dispatch", %{attributes: [{"adapter", inspect(adapter_module)}]} do
      adapter_module.send(context.adapter_state, context.request)
    end
  end
end
```

Verify in Jaeger: full trace from gateway → auth → router → adapter → audit.

### 2. Prometheus Metrics + Grafana

**Dependencies to add:** `telemetry_metrics_prometheus_core`, `plug_cowboy` (metrics server)

```elixir
# apps/infra_telemetry/lib/infra_telemetry/application.ex
{TelemetryMetricsPrometheus, metrics: InfraTelemetry.Metrics.metrics(), port: 9568}
```

Grafana dashboards (provisioned as JSON ConfigMaps):
- **Overview:** requests/sec, error rate, P99 latency by adapter
- **Adapters:** circuit breaker state, adapter health, timeout rate
- **Queue:** Broadway throughput, DLQ depth, batch latency
- **Infrastructure:** BEAM memory, GC pauses, process count, scheduler utilization

Alerting rules (Prometheus Alertmanager):
```yaml
- alert: CircuitBreakerOpen
  expr: mw_router_circuit_open_total > 0
  for: 1m
  labels: {severity: warning}

- alert: DLQDepthHigh
  expr: infra_queue_dlq_depth > 100
  for: 5m
  labels: {severity: critical}

- alert: HighErrorRate
  expr: rate(mw_router_request_count{status="error"}[5m]) / rate(mw_router_request_count[5m]) > 0.05
  for: 2m
  labels: {severity: warning}
```

### 3. Multi-node Clustering

**Dependencies to add:** `libcluster`, `horde`

```elixir
# config/runtime.exs
config :libcluster,
  topologies: [
    mw_core: [
      strategy: Cluster.Strategy.Kubernetes.DNS,
      config: [
        service: System.get_env("K8S_SERVICE_NAME", "mw-core-headless"),
        application_name: "mw_core"
      ]
    ]
  ]
```

ETS routing table replication across nodes:

```elixir
# On route table update:
Phoenix.PubSub.broadcast(MwCore.PubSub, "route_table:updated", {:reload, message_type})

# Each node subscribes and reloads its own ETS:
def handle_info({:reload, message_type}, state) do
  MwRouter.RouteTable.reload_rule(message_type)
  {:noreply, state}
end
```

File watcher (singleton) via `Horde.Registry` + `Horde.DynamicSupervisor`:
Only one node runs `AdapterFile.FileWatcher` at a time. If that node fails, Horde
re-starts it on another node automatically.

### 4. Zero-downtime Deploys

Health check endpoint:

```elixir
# GET /health/live  → 200 always (liveness)
# GET /health/ready → 200 when all adapters healthy and DB connected (readiness)
defmodule GatewayApiWeb.HealthController do
  def ready(conn, _) do
    checks = [
      db: InfraRepo.Repo |> Ecto.Adapters.SQL.query("SELECT 1", []),
      banking: AdapterBanking.health_check(get_state()),
      dw: AdapterDw.health_check(get_state())
    ]

    if Enum.all?(checks, fn {_, r} -> match?(:ok, r) or match?({:ok, _}, r) end) do
      json(conn, %{status: "ready"})
    else
      conn |> put_status(503) |> json(%{status: "degraded", checks: format(checks)})
    end
  end
end
```

Kubernetes deployment:
```yaml
readinessProbe:
  httpGet:
    path: /health/ready
    port: 4000
  initialDelaySeconds: 10
  periodSeconds: 5

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]  # drain connections before SIGTERM
```

### 5. Secrets Management

Production secrets via runtime environment variables (minimum viable):

```elixir
# config/runtime.exs
config :infra_repo, InfraRepo.Repo,
  url: System.fetch_env!("DATABASE_URL"),
  pool_size: String.to_integer(System.get_env("POOL_SIZE", "10"))

config :mw_auth, :jwt_secret, System.fetch_env!("JWT_SECRET_KEY")
```

Vault integration (if required by compliance):

```elixir
defmodule InfraRepo.VaultConfig do
  def fetch_db_config do
    {:ok, secret} = Vault.read("secret/data/mw-core/db")
    secret["data"]
  end
end
```

### 6. Load Testing

Tools: `k6` for HTTP, custom WebSocket scripts for `gateway_ws`.

Target SLAs:

| Metric | Target |
|--------|--------|
| REST API P99 latency (excl. CBS) | < 50ms |
| REST API P99 latency (incl. CBS sandbox) | < 300ms |
| WebSocket connection establishment | < 100ms |
| Broadway file throughput | > 5,000 rows/sec |
| Maximum concurrent WebSocket connections | > 10,000 |
| Error rate under 2x normal load | < 0.1% |

Chaos scenarios:
- Kill `adapter_banking` mid-request → circuit breaker opens, 503 returned, recovers
- Kill one cluster node → ETS reloads on remaining nodes, FileWatcher restarts
- SFTP server unavailable → FileWatcher retries with backoff, no data loss

### 7. Security Review Checklist

- [ ] All endpoints require authentication (no anonymous routes except `/health`)
- [ ] JWT keys rotated procedure documented
- [ ] API keys stored as Argon2 hashes only
- [ ] No secrets in application code or config files
- [ ] SQL injection impossible (all queries use Ecto parameterisation)
- [ ] No raw user input interpolated into log messages (XSS in log viewers)
- [ ] TLS enforced in production (Bandit SSL + HSTS header)
- [ ] CORS policy restricts to known origins
- [ ] Rate limiting applied to auth endpoints (prevent brute force)
- [ ] Audit log is append-only (no update/delete on `audit_events` table)
- [ ] DLQ entries include sanitised data (no raw PII in error messages)
- [ ] OTel traces do not include payment card data in span attributes

---

## Acceptance Criteria

- [ ] OTel trace visible in Jaeger for every REST request end-to-end
- [ ] Prometheus scrapes `/metrics` with all defined counters/summaries present
- [ ] Grafana overview dashboard shows live data
- [ ] Two-node cluster starts; route table change on node A reflects on node B within 1s
- [ ] `GET /health/ready` returns 503 when banking adapter is down
- [ ] k6 load test: P99 < 300ms at 500 concurrent users, error rate < 0.1%
- [ ] All security checklist items completed and signed off
- [ ] `mix release` produces single artifact deployable to Kubernetes