# Agentic AI for TMS — Architecture & Implementation Plan

**Status:** Draft for review
**Scope:** `tmsuat_apps` umbrella (tms_core / settlement_core / risk_core / platform_core / platform_web)
**Supersedes:** the Mercury Switch monitoring proposal in [agentic-ai.md](agentic-ai.md) and [Architecture.md](Architecture.md)
**Re-homes:** the mw-core investigation design in [investigation_agent_design_copied.md](investigation_agent_design_copied.md)
**Companion docs:** [investigation_agent_design.md](investigation_agent_design.md) (A6 detail) · [entity-resolution.md](entity-resolution.md) (cross-system identity, §1.8) · [mcp-servers.md](mcp-servers.md) (tool contracts, D8) · [ui-ux-plan.md](ui-ux-plan.md) (Agent Console screens) · [implementation-plan.md](implementation-plan.md) (sprint-level execution) · [business-brief.md](business-brief.md) (non-technical summary)

---

## 1. Why the copied *monitoring* design does not transfer as-is

The inherited proposal was written for **Mercury Switch** — a stateless-ish ISO 8583 switch observed through Prometheus. Its five agents (Transaction Performance, Network Resilience, VM Health, Reversal System, Incident Commander) all reason over **scrape-interval time series** and act on **infrastructure knobs** (connection pool size, GC, restart).

| Assumption in the copied design | Reality in TMS |
|---|---|
| Primary signal = Prometheus time series | Primary signal = **domain state in MySQL** + **MQTT event stream**. No Prometheus exporter exists in this repo (`mix.lock` has `telemetry` but no `prom_ex` / `telemetry_metrics_prometheus`). |
| Actions = infra knobs (pool size, GC, restart) | Actions = **business operations** (push a config, roll back an app package, hold/release a transaction) |
| Outcome = "did latency drop in 30s?" — noisy, self-correcting | Outcome = **explicit, durable, ground-truthed** — the device ACKs or it doesn't; the version converges or it doesn't |
| Incident unit = a service | Incident unit = **a terminal, a merchant, a rollout cohort, a settlement date** |
| Blast radius = one VM | Blast radius = **a fleet of payment terminals in the field** — much higher stakes, much stricter gating |
| Success metric = MTTR | Success metric = **convergence rate, first-push success, exception aging, SLA attainment** |

Two things carry over cleanly and should be kept: the **observe → reason → act → learn** loop, and the **risk-tiered approval gate**. Almost everything else needs re-grounding.

### What TMS gives you that the switch did not

This is the actual argument for building it here:

1. **Real actuators already exist as plain functions.** `AutoPushService.trigger_config_push/4`, `AppPackageService.rollback_package/2`, `OtaService.send_merchant_config_update/2`, `RemoteLogService.start_log_session/3`, `ReconciliationEngine.hold_transaction/2`. An agent needs a tool layer, not a new control plane.
2. **Ground-truth outcome labels are free.** `parameter_push_logs.status` transitions `pending → sent → acknowledged | failed`, and the *next heartbeat* tells you whether the version actually converged. That is a supervised training signal the Prometheus system could never produce.
3. **Durable orchestration is already installed.** Oban with dedicated queues (`parameter_push`, `app_package`, `file_download`, `reconciliation`) and a Cron plugin.
4. **Human-in-the-loop UI already exists.** LiveView with `version_compliance_live`, `application_upgrade_status_live`, `alerts_live`, `audit_compliance_live`. The approval console is an addition, not a greenfield build.
5. **Audit tables already exist.** `risk_audit_logs`, `settlement_file_audit`, `parameter_push_logs`. Regulator-facing traceability is a schema extension, not a new discipline.
6. **There are already three hand-written "proto-agents"** that are rule-based and brittle — they are the natural first migration targets:
   - `VersionComplianceChecker` — fixed 30-minute cooldown, no notion of *why* a push failed last time ([version_compliance_checker.ex:24](../../apps/tms_core/lib/tms_core/terminal_management/version_compliance_checker.ex#L24))
   - `OfflineMonitor` — fixed 5-minute tick, fixed 10-minute offline threshold, no correlation ([offline_monitor.ex:12](../../apps/tms_core/lib/tms_core/terminal_management/offline_monitor.ex#L12))
   - `SlaMonitorWorker` — fixed cron checkpoints, emails on breach, no diagnosis

---

## 1.5 The mw-core investigation design — and why its blocker doesn't exist here

The second inherited document ([investigation_agent_design_copied.md](investigation_agent_design_copied.md)) is from **mw-core**, a different Elixir system (`MwKernel.*` / `InfraRepo.*` / `GatewayWebWeb.*`). It is a much better-reasoned document than the monitoring proposal, and three of its decisions should be adopted wholesale:

1. **Investigation first, because it is read-only.** Low blast radius, useful without approval gates, and it exercises the same tool-calling runtime that later remediation agents need. This is the right sequencing argument and it applies here too.
2. **The read-only invariant is enforced in code, not in the prompt** — a registry flag checked before invocation, with no write tools defined at all.
3. **Evidence, not just prose** — return the raw tool results alongside the narrative so the operator can verify. "Never trust the summary."

### Its hard prerequisite — and why TMS clears it

mw-core §5 identifies the **correlation key** as the blocker, and is blunt about it: *"the largest and most cross-team item — treat it as a prerequisite work-stream, not part of the agent code."* The problem is that mw-core owns almost none of the truth. Its `transactions` table is a risk-oriented shadow copy; authoritative auth state lives in the Switch (keyed by RRN/STAN), settlement state in the Settlement system (batch reference), payout state in the Payout system (disbursement id). §6 therefore requires **three external teams to build new read APIs** before the flagship scenario works at all.

**In this repo that work is already done, and the correlation is materialised in the schema.** `settlement_core` owns the whole chain in one MySQL database reachable from one Repo:

```
switch_dump_records          rrn, tid, auth_number, stan, response_code,
  │                          match_status, matched_at, core_transaction_id ─┐
  │                                                                        │
qr_scheme_dump_records       rrn, tid, auth_number, stan,                   │
  │                          scheme_reference_no                            │
  │                                                                        │
  └──────────────────────────────────────────────────────────────────────► │
core_transactions        ◄───────────────────────────────────────────────── ┘
  │   id, rrn, tid, auth_number, stan, invoice_number, batch_number,
  │   merchant_mid, settlement_batch_id, settlement_status,
  │   switch_settled_date, risk_hold, source_type, source_ref_id
  │
  ├─► settlement_mis_items      core_transaction_id (FK), settlement_mis_id,
  │                             rrn, tid, auth_number, gross/mdr/vat amounts
  │        │
  │        └─► payout_items     settlement_mis_id, payout_batch_id,
  │                             merchant_mid, payout_amount, status,
  │                             bank_transfer_ref
  │                                  │
  │                                  └─► payout_batches → bank confirmation
  │
  └─► reconciliation_exceptions  core_transaction_id, switch_dump_record_id,
                                 qr_dump_record_id, rrn, tid, auth_number,
                                 exception_type, sla_due_at, status
```

`ReconciliationEngine` already performs the switch-dump ↔ core-transaction match and **writes the failure to `reconciliation_exceptions` when it doesn't match.** So the mw-core flagship scenario — *"the switch says approved, the merchant says it's missing from their settlement report; where did it stop?"* — is answerable **by SQL joins in a single Repo**, and in the common case the system has already recorded *why* as an exception row.

This is the single strongest argument for building the agentic layer in this repo rather than mw-core: **mw-core needed a cross-team correlation-key programme plus three new external read APIs before its primary use case could work at all. Here the join path already exists and is indexed.**

### What still differs from mw-core

| mw-core | Here |
|---|---|
| OpenAI is the only wired provider (`MwKernel.LlmConfig`); Claude/Llama are placeholders | **Nothing is wired.** A scan of `mix.lock` and every `.ex`/`.exs` finds no LLM provider at all — provider choice is genuinely open, not inherited |
| `call_llm/2` is single-shot text→text; needs a new `call_with_tools/3` | Same gap, but greenfield — build the tool-calling loop once, correctly, with no legacy single-shot path to keep working |
| Entry points: chat LiveView + Alertmanager webhook | Chat LiveView + **MQTT/Oban/DB triggers, plus AlertManager for the infra/switch domain** (see §1.6 and §9) |
| Investigation only; remediation explicitly out of scope | Investigation **plus** the six acting agents, because the actuators already exist here |

---

## 1.6 The full connected-systems landscape — four applications, two databases

Everything in §1–1.5 was scoped to `tmsuat_apps` alone. That undersells the reach this agentic layer actually has, because **TMS is not the only system sharing its database.** [config/dev.exs](../../config/dev.exs) and the RFP feature docs in [connected-apps/](connected-apps/) show four separate applications resolving down to **two physical MySQL databases**:

```
┌──────────────────────────────────────────────────────────────────────────┐
│  Database: shukria_transactions                                          │
│  (config/dev.exs — DaProductApp.Repo AND PlatformCore.Repo both point    │
│   here: "PlatformCore.Repo shares the same database as DaProductApp.Repo")│
│                                                                            │
│  ┌──────────────────────┐  ┌───────────────────────┐  ┌────────────────┐ │
│  │ TMS (this app)       │  │ Cloud Layer            │  │ jPOS Switch    │ │
│  │ tms_core /           │  │ (da_product_app, SAME  │  │ (jpos-main,    │ │
│  │ settlement_core /    │  │ Elixir umbrella,       │  │ external Java  │ │
│  │ risk_core            │  │ apps/da_product_app)   │  │ process)       │ │
│  │                      │  │                        │  │                │ │
│  │ tms_terminals,       │  │ transactions,          │  │ WRITES its own │ │
│  │ parameter_push_logs, │  │ cloud_transactions,    │  │ tables directly│ │
│  │ core_transactions,   │  │ pos_terminals, stores, │  │ into this same │ │
│  │ reconciliation_      │  │ custom_events,         │  │ DB:            │ │
│  │ exceptions, ...      │  │ software, ...          │  │ pos_transaction│ │
│  │                      │  │                        │  │ pos_temp_      │ │
│  │                      │  │ ALREADY reads jPOS     │  │   transaction  │ │
│  │                      │  │ tables via             │  │ pos_failed_    │ │
│  │                      │  │ DaProductApp.          │  │   transaction  │ │
│  │                      │  │ PosTransactions.       │  │ pos_transaction│ │
│  │                      │  │ PosTransaction         │  │   _reversal    │ │
│  │                      │  │ (in-repo Ecto schema)  │  │ acquirer_      │ │
│  │                      │  │                        │  │   terminal_*   │ │
│  └──────────────────────┘  └───────────────────────┘  └────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────────────┐
│  Database: shukria_mms_new_local  ("mms db")                             │
│  config/dev.exs: PlatformCore.ShukriaMmsRepo — comment reads             │
│  "read-only connection to shukria_mms (Merchant Management System DB)"   │
│                                                                            │
│  ┌────────────────────────────────────────────────────────────────────┐  │
│  │ Merchant Management System (epay-main, external Laravel/PHP app)   │  │
│  │                                                                     │  │
│  │ transaction_rules, user_metadata, kyc_requests,                    │  │
│  │ merchant_configuration, users                                      │  │
│  │ — KYC status, LSEG/World-Check screening, merchant risk category,  │  │
│  │   credit score, MCC risk profile, MDR template, contract status    │  │
│  └────────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────────┘
```

### Why this matters for the agent design

**1. The correlation-key finding in §1.5 gets stronger, not just confirmed.** The join path there used `switch_dump_records` — a post-hoc EOD reconciliation mirror populated by dump-file ingestion. But **the live jPOS switch table is already one query away**: `DaProductApp.PosTransactions.PosTransaction` (schema: `pos_transaction`) carries `s_tid_stan` (POS STAN), `reference_no` (RRN), `acquirer_reference_no`, `scheme_reference_no`, `approval_code`, `response_code`, `masked_card_no`, and `mcc_code` — the actual switch record, not a same-day dump copy. `A6.trace_transaction/1` should hit this table **first**, before falling back to `switch_dump_records` for anything the live table doesn't carry (e.g. post-settlement scheme fields). This closes the mw-core correlation-key gap even further than §1.5 stated: mw-core needed a new Switch read API; here the switch's own transaction table is a local Ecto query.

**2. Merchant risk/KYC context becomes available to every agent, read-only.** `PlatformCore.ShukriaMmsRepo` already exists and is explicitly commented as read-only. That means A5 (Risk Triage) and A6 (Investigation) can pull **real** merchant context — KYC completion status, LSEG screening result, computed risk category, MCC risk tier, active MDR template — instead of reasoning only from `risk_core`'s local rule hits. A dispute investigation that currently stops at "the transaction is missing from settlement" can now also surface "this merchant's KYC review has been overdue for 40 days" or "LSEG flagged this merchant's UBO last week" as contributing context, when relevant.

**3. Two new domains exist that the original roster (§3) didn't cover.** Both are now in reach:

| Domain | Reachability | What it adds |
|---|---|---|
| **Cloud Layer** (QR orchestration, Rules Engine, ERP integration) | **In-process** — same Elixir umbrella, `apps/da_product_app`, same DB | QR payments stuck in `pending` past the Alipay polling window (12 × 10s), Rules Engine decline storms (a BIN/velocity/daily-total rule mis-tuned and blocking legitimate traffic), MQTT device-offline-before-QR-push failures |
| **jPOS Switch** transaction data | **Read, in-process** (`pos_transaction` family in the shared DB) | The live switch record for any RRN/STAN — no need to wait for `switch_dump_records` ingestion |
| **Merchant Management System** (epay-main) | **Read-only, cross-DB** (`ShukriaMmsRepo`) | KYC status, LSEG/World-Check screening, credit score, risk category, MCC risk, MDR template |

**4. What is genuinely still out of reach.** Two things do not become available just because the databases are shared:

- **epay-main's write path** — onboarding approval, KYC decision, credit score override, contract signing. `ShukriaMmsRepo` is read-only by its own design comment; **no tool should ever attempt a write through it.** This is a T4 boundary enforced the same way as key material in §4 — by absence, not by policy.
- **The jPOS switch's live in-memory / session state** — `acquirer_terminal_state` (idle/busy/reversal-pending) and the Q2 service container's runtime health are not something a DB read reconstructs perfectly; they reflect the switch's state at last write, not its current in-memory truth. Useful as a signal, not authoritative for "is this terminal mid-transaction right now." **This is exactly the gap §1.7 addresses.**

**5. Scope discipline.** This finding is an argument for **what A6 can read**, not an argument to add a Cloud Layer agent or a Switch agent to the Phase 0–4 roster in §10. Expanding the acting-agent roster into Cloud Layer's QR/rules domain is a legitimate Phase 5+ candidate once A2–A5 have proven the pattern — but doing it now would dilute an already substantial plan. The recommendation for this phase is: **widen A6's read reach to include these three sources; do not widen the acting-agent roster.**

---

## 1.7 What §1.6 doesn't cover: BEAM health, jPOS process health, and network health

§1.6 closes the *transaction data* gap between TMS and its neighbours — everything reachable as a database row. It does not close a separate gap: **process and network health, which has no database row at all.**

Checked directly against this repo:

- `PlatformWeb.Telemetry` already defines `vm.memory.total` and `vm.total_run_queue_lengths.*` as `Telemetry.Metrics` — but the only consumer is LiveDashboard. No `prom_ex`, no `telemetry_metrics_prometheus`, no reporter beyond a commented-out `ConsoleReporter` in `apps/platform_web/lib/platform_web/telemetry.ex`. BEAM health is **live-only** today: visible if someone has the dashboard tab open, gone the moment they don't. There is no way to ask "has memory been climbing for the last 45 minutes" — nothing persists it.
- jpos-main has **zero footprint** in this repo — no config, no client code, nothing. Its own RFP feature doc ([connected-apps/device-middle-layer.md](connected-apps/device-middle-layer.md) §11, §13) describes real internal telemetry that already exists on the Java side: the Q2 system monitor (memory, thread count, transaction rate, service status), `ReversalMetrics` (retry counts, success rate, MANUAL_REVIEW backlog), and bank-response-latency logging. None of it is exported anywhere TMS — or an agent — can see.
- Acquirer network health (YSP/Fiserv SSL channel state, connection drops, TLS handshake failures) is the same story: it exists as jPOS log lines and Q2 service state (`YSP Channel Worker`, `YSP Network Management` heartbeat), not as anything queryable from this repo.

None of this is covered by "query current DB state," because there is no state to query — these are time-series phenomena (a trend, a rate, a backlog growing) that a point-in-time read can't see, and that is exactly the class of problem Prometheus + Grafana + AlertManager were built for. The original monitoring proposal wasn't wrong that these matter; it was wrong about which of TMS's problems they solve. §9 revises the stance from "optional, secondary" to "primary sensor for this one domain, still not the backbone for the rest," and adds a dedicated agent (A8, §3) whose sensors are metrics and alerts rather than DB rows or MQTT events.

**Two different dependency profiles, worth being honest about separately:**
- **BEAM export is fully within this repo's control** — adding `prom_ex` (or `telemetry_metrics_prometheus`) to `platform_web` and `da_product_app` is Phase 0/1-sized work, no external team involved.
- **jPOS/network export is not** — it requires the Java side to expose something scrapeable (a JMX-to-Prometheus bridge, or extending the existing Q2 `99_sysmon.xml` monitor). This is a real cross-team dependency, smaller in scope than mw-core's correlation-key programme (§1.5) but real, and it should be scoped and asked for early rather than discovered as a blocker in Phase 3.

---

## 1.8 Cross-system entity resolution — verified, not assumed

§1.6 mapped *transaction lineage* (an RRN walks to a settlement/payout record). A separate, more foundational question — *which merchant is this, and what is their identity in every other system* — needed its own check, because the natural next request is exactly what got raised: **MMS merchant → jPOS MID/TID → live transaction**, and **MMS → merchant → TID → device serial number → TMS devices.**

Full detail, with file:line citations, in [entity-resolution.md](entity-resolution.md). Headline findings:

1. **MMS merchant identity resolution is real and already wired** — `SettlementCore.Mms.MerchantMetadata` (table `user_metadata`, key field `merchant_refrence_number`) and `SettlementCore.CoreTransactionSync.get_merchant_metadata/1` are tested, in-production code. A5/A6 tools should call through this, not invent a new direct `ShukriaMmsRepo` query.
2. **jPOS TID/MID mapping is real, differently named than the RFP doc claims** — `pos_terminal` → `pos_terminal_acquirer_terminal` → `acquirer_terminal` → `acquirer_merchant`, all in `apps/da_product_app`.
3. **`ShukriaMmsRepo`'s read-only guarantee is stronger than §1.6 stated** — it's `read_only: true` at the Ecto adapter level, not just a naming/comment convention. Ecto refuses non-`SELECT` statements through this Repo outright.
4. **Correction to §1.6:** the `merchant_configuration` table named there (sourced from the Cloud Layer RFP doc) doesn't exist in this codebase. The verified table is `merchant_stores`.
5. **The important finding:** two features documented in the Cloud Layer RFP doc — the "group → brand → store → device" merchant hierarchy, and the `checkDuplicateTidMid`/`updateShukriaProviderMidTid`/`updateShukriaYspMidTid` endpoints — have **zero code backing them** anywhere in this repo. The RFP docs in [connected-apps/](connected-apps/) describe a target feature set for a proposal; they are not a substitute for reading the schema. Every claim sourced from them needs the same verify-before-build check applied in the entity-resolution doc, before any agent tool depends on it.
6. **The gap that matters most:** the walk *device serial number → TMS terminal → merchant → MID/TID* is **not a single verified query today.** `tms_terminals.serial_number` and `pos_terminal.serial_number` are the same column name in two different apps with no code joining them; `tms_terminals.merchant_id` (a bare string) has no code path to `pos_terminal.pos_merchant_id` (an integer) or to `MerchantMetadata.merchant_refrence_number`. Recommended: a half-day, read-only data-quality spike in Phase 0 to measure the actual match rate before any tool assumes this join holds. If it doesn't hold reliably, `trace_transaction`-style tools must report that hop as `:unresolved`, the same way a broken settlement hop reports `:absent` — never silently guessed.

---

## 2. Target architecture

New umbrella app: **`apps/agent_core`** (`AgentCore.*`), following the existing `tms_core` / `risk_core` / `settlement_core` convention. Web console lives in `platform_web`.

```
┌───────────────────────────────────────────────────────────────────────────┐
│ L6  AGENT CONSOLE  (platform_web / LiveView)                              │
│     Incident feed · Decision cards · Approval queue · Playbook editor      │
│     Kill switch · Audit export                                            │
└───────────────────────────────┬───────────────────────────────────────────┘
                                │
┌───────────────────────────────┴───────────────────────────────────────────┐
│ L5  GOVERNANCE                                                            │
│     Policy engine (risk tier × confidence × blast radius)                  │
│     Approval workflow · Rate limits & circuit breakers · Immutable audit   │
│     Global + per-agent kill switch                                        │
└───────────────────────────────┬───────────────────────────────────────────┘
                                │
┌───────────────────────────────┴───────────────────────────────────────────┐
│ L4  ACTUATORS — one MCP server per domain, not one monolith (D8)          │
│     tms-mcp · settlement-mcp (incl. verified Mms.* resolution) ·          │
│     switch-mcp (read-only pos_transaction) · risk-mcp · infra-mcp         │
│     Every tool declares: risk_tier, reversible?, blast_radius, rate_limit  │
│     Wrapping EXISTING services: AutoPushService · OtaService ·            │
│     AppPackageService · RemoteLogService · ParameterPushService ·         │
│     ReconciliationEngine · MqttCommandBuilder                             │
└───────────────────────────────┬───────────────────────────────────────────┘
                                │
┌───────────────────────────────┴───────────────────────────────────────────┐
│ L3  REASONING  (multi-provider tool-use loop — D6)                        │
│     AgentCore.LLM.Provider behaviour: Anthropic · OpenAI · Ollama         │
│     Agent runtime: GenServer per agent + Oban job per reasoning turn       │
│     Structured output → AgentCore.Decision                                │
└───────────────────────────────┬───────────────────────────────────────────┘
                                │
┌───────────────────────────────┴───────────────────────────────────────────┐
│ L2  MEMORY                                                                │
│     Episodic  → agent_incidents + agent_decisions (what happened)          │
│     Semantic  → agent_playbooks (curated, versioned, human-editable)       │
│     Procedural→ agent_learned_params (thresholds/cooldowns per fingerprint)│
└───────────────────────────────┬───────────────────────────────────────────┘
                                │
┌───────────────────────────────┴───────────────────────────────────────────┐
│ L1  OBSERVATION STORE  (agent_observations — normalised, append-only)      │
└───────────────────────────────┬───────────────────────────────────────────┘
                                │
┌───────────────────────────────┴───────────────────────────────────────────┐
│ L0  SENSORS                                                               │
│  MQTT           tms/status/{sn}, ota/ack/{sn}, ota/{pk}/{sn}/logpush       │
│  DB state       tms_terminals, parameter_push_logs, app_upgrade_*,         │
│                 reconciliation_exceptions, risk_rule_hits, pos_transaction │
│  Oban           job failure / retry-exhausted telemetry                    │
│  Phoenix        :telemetry events (already wired in PlatformWeb.Telemetry) │
│  Prometheus     BEAM health (Phase 1, in-repo) + jPOS/network health       │
│                 (Phase 3+, needs Java-side export — see §1.7)             │
│  AlertManager   webhook trigger source — see §9, §6 step [2]              │
└───────────────────────────────────────────────────────────────────────────┘
```

### Key architectural decisions

**D1 — Event-driven, not poll-driven, for the TMS-native domains.** The copied design polls Prometheus every 15s for everything. TMS's business-domain agents (A1–A5) should hang off the MQTT handler and the Ecto write path, which already fire on every heartbeat and ACK. Poll only for aging/absence detection ("no heartbeat in N minutes", "dump file never arrived"). This does **not** apply to the infra/switch domain (§1.7, A8) — BEAM/jPOS/network health has no event to hang off; it is inherently a polled, trend-based signal, and that's fine — it's a different sensing modality for a different problem, not a violation of D1's intent.

**D2 — The LLM proposes; deterministic Elixir disposes.** No tool executes because the model emitted a tool call. Every tool call passes through the policy engine, which is ordinary pattern-matched Elixir with unit tests. The model's output is a *proposal record*, not a command.

**D3 — Reasoning turns are Oban jobs.** Durable, retryable, rate-limitable, and observable with the tooling you already run. A crashed node does not lose an in-flight incident.

**D4 — Start with structured similarity, not a vector DB.** The copied design assumes a vector store. Your DB is MySQL; there is no pgvector. Incidents in TMS have naturally *discrete* fingerprints:
`{vendor, model, config_type, failure_mode, firmware_band}`. Exact and partial fingerprint match over `agent_incidents` will retrieve good precedents on day one, with an index instead of an embedding service. Add embeddings in Phase 3 only if fingerprint recall proves insufficient — and then as a bounded candidate re-rank, not a primary index.

**D5 — Shadow mode is a first-class runtime mode, not a config flag.** Every agent runs in one of `:shadow | :suggest | :auto_low_risk | :auto`. Mode is per-agent and per-environment, stored in DB, changeable from the console without a deploy.

**D6 — Multi-provider from day one: `AgentCore.LLM.Provider` behaviour with Anthropic, OpenAI, and Ollama backends.** Not "Claude, swappable later if forced" — three real implementations, because the reasons to want each are concrete, not hypothetical:

```elixir
defmodule AgentCore.LLM.Provider do
  @callback complete(messages :: list(), tools :: list(), opts :: keyword()) ::
              {:ok, AgentCore.LLM.Response.t()} | {:error, term()}
end

# AgentCore.LLM.Providers.Anthropic   — Req + Messages API (primary/recommended)
# AgentCore.LLM.Providers.OpenAI      — Req + Chat Completions / Responses API
# AgentCore.LLM.Providers.Ollama      — Req against a local Ollama server, OpenAI-
#                                        compatible /v1/chat/completions endpoint
```

| Provider | Why it's in the design, not just theoretically swappable |
|---|---|
| **Anthropic (Claude)** | Recommended default for reasoning-heavy, high-stakes work — A2's push decisions, A6's investigations. No official Elixir SDK exists (Anthropic ships Python/TS/Java/Go/Ruby/C#/PHP), so this adapter is a thin `Req`-based wrapper over `POST /v1/messages`, not a port of an SDK. |
| **OpenAI** | Secondary option — different tool-calling JSON shape (function-calling `tools: [{type: "function", ...}]`), different reasoning-model handling (o-series reasoning tokens vs. Claude's `thinking` blocks). Useful for cost/latency comparison per agent, not a special case beyond having its own adapter. |
| **Ollama (local)** | **This is what makes §12 item 1's data-residency fallback real rather than a footnote.** If UAE payment-ops regulation blocks sending terminal/settlement telemetry to a US-hosted API, Ollama is the sovereignty-compliant path — no external network call at all. Designed in from day one, not bolted on if the residency question fails. |

**Anthropic adapter specifics** (the provider most likely to be default; the constraints below are easy to get wrong from memory):

```elixir
# headers: x-api-key, anthropic-version: 2023-06-01
%{
  model: "claude-opus-4-8",
  max_tokens: 16_000,
  thinking: %{type: "adaptive"},          # NOT budget_tokens — removed on 4.7+, returns 400
  output_config: %{effort: "high"},       # low | medium | high | xhigh | max
  system: [%{type: "text", text: stable_prompt,
             cache_control: %{type: "ephemeral"}}],   # see caching note below
  tools: AgentCore.ToolRegistry.specs(),  # translated from MCP tool defs — see D8
  messages: messages
}
```

- **No `temperature` / `top_p` / `top_k`.** Removed on Opus 4.7+ — sending any of them returns a 400. Steer with prompting.
- **No assistant prefill.** A trailing `role: "assistant"` message returns a 400. Use `output_config.format` (structured outputs) when a fixed response shape is needed.
- **Prompt caching is a prefix match**, and the minimum cacheable prefix on Opus 4.8 is **4096 tokens** — below that it silently doesn't cache. Render order is `tools` → `system` → `messages`, so the tool catalogue and the stable system prompt cache together. **Never interpolate a timestamp, incident ID, or terminal serial into the system prompt** — that invalidates the whole prefix on every call. Volatile context goes in `messages`, after the last breakpoint. Verify via `usage.cache_read_input_tokens`; if it's zero across repeated calls, something in the prefix is varying.
- **Stream anything with `max_tokens` above ~16K**, or the request will hit an HTTP timeout.

**Routing policy, not a single global choice.** Extend `agent_policies` (§5) with a `provider` column, set per agent/per environment: Claude by default; Ollama mandatory if data residency (§12) requires it, or for any tool touching cardholder-adjacent fields ahead of the redaction layer being proven; OpenAI available where a team wants a second opinion or a cost comparison. This is the same shape as the existing risk-tier policy — a deterministic gate, not a prompt instruction.

**Be honest about Ollama's cost.** Local hosting isn't free — it needs real GPU/CPU infrastructure, and open-weights models available through Ollama lag Claude/GPT-tier models on tool-use reliability today. Use it where sovereignty or cost genuinely dictates, not as a blanket default that quietly degrades every agent's judgment.

**D7 — "Skills" means portable playbooks, not the provider-native Skills feature.** Anthropic's literal Agent Skills mechanism (`SKILL.md` folders, progressive disclosure) requires the code-execution container tool (`container.skills` + `code_execution_20260521` on the Messages API, or the `skills` array on Managed Agents) — it's tied to a sandboxed execution container this design doesn't use (D6 is a manual tool-use loop, not Managed Agents), and it has no equivalent on OpenAI or Ollama. Building on it natively would break D6's multi-provider goal outright.

What "skill" means here instead is **`agent_playbooks`** (§5) — versioned, human-editable markdown matched by fingerprint, loaded at control-loop step [3] RECALL (§6). This is provider-agnostic (it's just context content — identical regardless of which of the three providers answers the call) and is exactly the concept the mw-core-inspired design already had; this makes it explicit and per-agent:

| Agent | Skill boundary (tool subset + playbook domain) |
|---|---|
| A1 Fleet Health | `tms-mcp` (diagnostics, offline marking) + `agent_playbooks.domain = "fleet"` |
| A2 Config Compliance | `tms-mcp` (push/regenerate) + `domain = "config_compliance"` |
| A3 Rollout | `tms-mcp` (retry/pause/rollback) + `domain = "rollout"` |
| A4 Settlement Ops | `settlement-mcp` + `domain = "settlement"` |
| A5 Risk Triage | `risk-mcp` + `settlement-mcp` (read-only Mms.* resolution, §1.8) + `domain = "risk"` |
| A6 Investigation | **all read-only servers** (tms-mcp, settlement-mcp, switch-mcp, risk-mcp, infra-mcp) + all playbook domains, read-only |
| A7 Orchestrator | reads every agent's decisions/outcomes; no domain tools of its own |
| A8 Infra & Switch Health | `infra-mcp` only + `domain = "infra"` |

Each agent's system prompt only ever sees its own tool subset — this is what keeps prompts focused instead of one giant do-everything catalogue, and it's a second, independent enforcement layer alongside the T1–T4 risk tiers (§4): a tool an agent's role doesn't need isn't just gated, it isn't in scope at all.

**Where literal Anthropic Skills could be added later, opportunistically:** report-heavy output — A4's formatted SLA-breach packets, A6's exportable Investigation Reports as `.xlsx`/`.pptx` — genuinely matches Anthropic's prebuilt `xlsx`/`docx`/`pptx` skills. This is a Phase 5+, Claude-only enhancement, not core, and it needs its own risk-tier thinking before adoption: code-execution-container access is a materially larger blast radius than this design's narrow read/write tool model.

**D8 — Every domain is an MCP server, not a monolithic tool registry.** L4 (Actuators) is restructured from one `AgentCore.Tool` registry into discrete, independently-versioned MCP servers, one per domain:

| Server | Wraps | Used by |
|---|---|---|
| `tms-mcp` | `AutoPushService`, `OtaService`, `AppPackageService`, `RemoteLogService`, `MqttCommandBuilder` | A1, A2, A3 |
| `settlement-mcp` | `ReconciliationEngine`, `ParameterPushService`, plus the verified `SettlementCore.Mms.*` merchant-resolution path (§1.8) | A4, A5, A6 |
| `switch-mcp` | Read-only `pos_transaction`/`pos_failed_transaction`/`pos_transaction_reversal` (§1.6) | A6 |
| `risk-mcp` | `risk_rule_hits`, redacted | A5, A6 |
| `infra-mcp` | Prometheus + AlertManager reads, once §1.7's export work lands | A8 |

Why this is worth the extra structure over one registry:

1. **Reuse beyond `AgentCore`'s own loop.** A developer can point Claude Code, or any other MCP-aware client, at `settlement-mcp` directly to debug a reconciliation issue — without going through agent orchestration at all.
2. **It's the actual mechanism that reconciles D6 and D7 with D8.** MCP tool definitions are provider-agnostic. Claude consumes them via its native MCP connector (`mcp_servers` + `mcp_toolset`, beta `mcp-client-2025-11-20`); OpenAI's Responses API has its own remote-MCP tool support; a manual loop calling Ollama adapts the same MCP schema into Ollama's OpenAI-compatible function-calling format. Tool definitions get written once, not three times.
3. **T4 boundaries become structural, not just policy.** `settlement-mcp`'s MMS-resolution tools can't expose a write — the underlying `ShukriaMmsRepo` connection is `read_only: true` at the Ecto adapter level (§1.8, confirmed by reading the Repo definition, not assumed). `switch-mcp` has the same shape: no tool wraps anything that writes to jPOS's tables, because jPOS is a foreign process this umbrella has no write access to at all (§4).

**Implementation decision (spike done, trial pending):** `hermes_mcp` (185K downloads, Phoenix-integrated) is the lead candidate over `conduit_mcp` (2.3K downloads, more recently updated, built-in auth/CORS) — full comparison in [mcp-servers.md](mcp-servers.md). Not yet trial-built against this codebase; that's the remaining Phase 0 step, not a documentation-only decision.

**Scope confirmed:** all five servers are built and owned by this team. Read-only reach into epay-main and jpos-main is sufficient — no other team needs to build an MCP server. `switch-mcp` reads `pos_transaction` straight out of the shared DB; `settlement-mcp`'s merchant tools call through the already-verified `SettlementCore.Mms.*` path (§1.8). The only genuine external dependency is `infra-mcp`'s jPOS/BEAM metrics export (§1.7) — a metrics-export ask, not an MCP-server-build ask. Full per-server tool contracts, with verified field names and risk tiers, are in [mcp-servers.md](mcp-servers.md).

---

## 3. Agent roster

Eight agents, each grounded in modules that already exist.

### A1 — Fleet Health Agent
| | |
|---|---|
| **Observes** | Heartbeat gaps (`tms_terminal_status_logs`), status transitions, `tms_terminals.heart`, online-rate by area/merchant/model |
| **Detects** | Terminal offline; *cohort* offline (an area or merchant dropping together — the signal `OfflineMonitor` cannot see); flapping; clock skew |
| **Acts** | T1: request diagnostics via `RemoteLogService.start_log_session/3`; T1: mark offline; T2: send device command via `MqttCommandBuilder`; T3: escalate to field ops |
| **Replaces** | `OfflineMonitor` fixed 10-min threshold → learned per-model, per-connectivity-profile thresholds |
| **Key value** | Distinguishing *"one terminal died"* from *"an ISP region dropped"* — currently 400 separate alerts instead of one |

### A2 — Config Compliance Agent
| | |
|---|---|
| **Observes** | `parameter_config_version` / `emv_config_version` / `keys_config_version` vs `parameter_templates.version` and `config_file_versions`; `parameter_push_logs` status and aging |
| **Detects** | Version drift; pushes stuck in `pending`/`sent`; repeated push→no-ACK loops; checksum mismatch; a *template* rollout that is failing broadly (not a device fault but a bad template) |
| **Acts** | T1: `AutoPushService.trigger_config_push/4`; T1: regenerate `ParamsZipBuilder` / `L3ConfigZipBuilder` artefacts; T2: adjust per-fingerprint cooldown; T3: quarantine a template version |
| **Replaces** | `VersionComplianceChecker` — same job, but reasons about *why* the previous push failed instead of blindly waiting 30 minutes |
| **Key value** | The current cooldown is a constant. A device that NAKs due to a corrupt zip and a device that is simply offline get identical treatment today. |

### A3 — Rollout Agent (OTA / app packages)
| | |
|---|---|
| **Observes** | `app_upgrade_device_status` (`update_result`, `status`, `pushed_time` → `finish_time`), OTA ACKs on `ota/ack/{sn}`, download failures on `ota/file_download/{sn}` |
| **Detects** | Failure rate crossing threshold within a cohort; stalled downloads; a model-specific regression; slow-burn failures that only appear after 200 devices |
| **Acts** | T1: retry a device; T2: pause rollout; T2: expand canary to next cohort; T3: `AppPackageService.rollback_package/2` |
| **Key value** | Automated canary judgement. Rollout halt currently depends on a human watching `application_upgrade_status_live`. |

### A4 — Settlement Ops Agent
| | |
|---|---|
| **Observes** | `reconciliation_exceptions` (type, aging vs `sla_due_at`), dump-file arrival, `SlaMonitorWorker` checkpoints (06:15 recon / 07:15 MIS generated / 08:15 MIS approved), Oban failures in `settlements`/`reconciliation`/`payouts` |
| **Detects** | SLA at risk **before** breach (the current worker only reports *after*); recurring exception clusters; upstream file late/malformed |
| **Acts** | T1: re-run a failed Oban job; T1: classify + enrich exceptions; T2: re-trigger reconciliation for a date; **T4 (never auto): release a held transaction** |
| **Key value** | Predictive SLA warning + a drafted root-cause packet, instead of an 06:15 breach email with no diagnosis |

### A5 — Risk Triage Agent
| | |
|---|---|
| **Observes** | `risk_rule_hits` in `Hold`, merchant history, `risk_audit_logs`, plus read-only merchant risk/KYC context via `ShukriaMmsRepo` (§1.6) |
| **Detects** | Hit clusters indicating a mis-tuned rule vs genuine fraud; aging holds |
| **Acts** | **T1 only — enrich and recommend.** Assemble evidence, propose a category, surface to the supervisor queue. |
| **Hard constraint** | **Never** auto-releases or auto-holds. Financial disposition stays with a human supervisor, full stop. |

### A6 — Investigation Agent
Read-only, cross-domain, on-demand. Given an incident or a free-text operator question, gathers evidence across all five domains and returns a cited root-cause narrative.

Its flagship scenario is the one mw-core was designed around and could not reach: **merchant transaction disputes** — *"the switch approved it, the merchant says it's not in their settlement report."* Here that is a walk down the §1.5 join path (`switch_dump_records` → `core_transactions` → `settlement_mis_items` → `payout_items`), usually terminating at an existing `reconciliation_exceptions` row that already names the failure. It also serves as the **debugger for the rest of the agent system** — reading `agent_decisions` / `agent_actions` / `agent_outcomes` to explain why another agent did what it did. Detailed in [investigation_agent_design.md](investigation_agent_design.md).

### A7 — Ops Orchestrator
| | |
|---|---|
| **Observes** | All agent findings, including A8's metric/alert-based findings |
| **Does** | Correlates and **deduplicates** (400 offline terminals + 400 failed pushes = one incident, not 800); assigns incident ownership; sequences actions across agents; decides escalation |
| **Note** | This is the "Incident Commander" from the copied design, and it survives the port intact — it was the strongest idea in it. Its value grows once A8 exists: correlating "BEAM memory pressure" (a metric fact) against "reconciliation exceptions spiking" (a DB fact) is exactly the cross-domain judgement a human juggling separate dashboards does badly. |

### A8 — Infrastructure & Switch Health Agent
| | |
|---|---|
| **Observes** | BEAM VM metrics (memory, run-queue length, GC) for `platform_web`/`da_product_app` via Prometheus once exported (§1.7); jPOS Q2/JVM health and `ReversalMetrics` **if and when jpos-main exports them** (cross-team dependency, §1.7); acquirer (YSP/Fiserv) connection/network health, same caveat; AlertManager webhook triggers |
| **Detects** | BEAM memory/run-queue trending toward exhaustion before it becomes an outage; jPOS reversal MANUAL_REVIEW backlog growing; acquirer connectivity degrading before terminals start timing out |
| **Acts** | **T1 only, and only on the Elixir side:** trigger BEAM GC, enable verbose logging, adjust scrape interval. **jPOS itself is untouchable — no tool exists, T4 by absence, stronger than any DB-based boundary in this plan because it is a foreign process this umbrella has no write access to at all.** |
| **Sensors, not DB/MQTT** | This is the one agent where D1's event-driven default doesn't apply — see §1.7. Existing rule-based alerting (AlertManager) stays the threshold engine; the agent's value-add is cross-domain correlation with A1–A5's DB-native findings via A7, not re-deriving thresholds an LLM would do worse than a Prometheus rule. |
| **Dependency** | BEAM export is in-repo work (Phase 3). jPOS/network export needs the Java side to expose something scrapeable — flagged honestly as a cross-team ask, not assumed. See §1.7 and §12. |

---

## 4. Tool catalogue & risk tiers

Every tool is a module implementing `AgentCore.Tool` with a declared contract. The model never sees anything not in this catalogue.

```elixir
defmodule AgentCore.Tools.TriggerConfigPush do
  use AgentCore.Tool

  @impl true
  def spec do
    %{
      name: "trigger_config_push",
      description: "Push a specific config type to one terminal.",
      risk_tier: :t1,
      reversible: true,
      blast_radius: :single_device,
      rate_limit: {5, :per_hour, :per_device},
      requires_healthy_mqtt: true,
      params: %{
        serial_number: %{type: :string, required: true},
        config_type: %{type: :enum, values: ~w(parameter emv_config keys_config application)}
      }
    }
  end

  @impl true
  def execute(%{serial_number: sn, config_type: ct}, _ctx) do
    terminal = TerminalManagement.get_terminal_by_serial!(sn)
    AutoPushService.trigger_config_push(sn, terminal.vendor, terminal.model, ct)
  end
end
```

### Risk tiers

| Tier | Gate | Examples |
|---|---|---|
| **T1 — Safe / reversible** | Auto-execute above confidence floor. Log + notify. | Trigger a config push (single device) · start a remote log session · re-run a failed Oban job · retry one OTA device · enrich a risk hit · trigger BEAM GC |
| **T2 — Bounded impact** | Auto only in `:auto` mode **and** confidence ≥ 85% **and** blast radius ≤ cohort limit. Otherwise queue for approval. | Pause a rollout · expand a canary cohort · re-trigger reconciliation for a date · adjust a learned cooldown · fleet-wide push to ≤ N devices |
| **T3 — High impact** | **Always** requires human approval. | `rollback_package/2` · quarantine a template version · bulk push > N devices · change a group rule |
| **T4 — Prohibited to agents** | Not in the catalogue. No tool exists. | Anything touching keys (`KeysConfigService`, KEK/KCV, RKI) · releasing a held transaction · modifying transaction/settlement amounts · user/role changes · direct SQL · deleting audit records · **anything on jpos-main or epay-main — foreign processes, no tool exists at all** |

**T4 is enforced by absence, not by policy.** There is no tool module, so there is no path from a model output to the operation. This matters more than any prompt instruction.

### Blast-radius ceiling
Independent of tier, a **hard cap** on devices affected per incident and per hour, enforced in the policy engine and not overridable by the model. Suggested initial values: 25 devices/incident, 200 devices/hour, both configurable per environment.

---

## 5. Data model

New migrations in `priv/repo/migrations/` (module prefix `DaProductApp.Repo.Migrations.*`, per repo convention), schemas in `apps/agent_core/lib/agent_core/`.

```
agent_incidents
  id, fingerprint (indexed), domain, severity, status,
  subject_type ('terminal'|'cohort'|'merchant'|'settlement_date'|'rollout'|'infra'),
  subject_ref, opened_at, resolved_at, resolution,
  owning_agent, correlated_incident_id, summary

agent_observations                          -- append-only, pruned
  id, incident_id, source ('mqtt'|'db'|'oban'|'telemetry'|'prometheus'|'alertmanager'),
  observed_at, kind, subject_ref, payload (json)

agent_decisions                             -- full reasoning trace
  id, incident_id, agent, mode, model_id, prompt_version,
  input_digest, reasoning (text), hypothesis, confidence (decimal),
  proposed_actions (json), precedent_incident_ids (json),
  input_tokens, output_tokens, latency_ms, cost_cents, created_at

agent_actions
  id, decision_id, tool_name, params (json), risk_tier,
  status ('proposed'|'awaiting_approval'|'approved'|'rejected'
         |'executing'|'succeeded'|'failed'|'rolled_back'),
  approved_by_id, approved_at, executed_at,
  result (json), error, rollback_action_id, oban_job_id

agent_outcomes                              -- the learning signal
  id, incident_id, action_id, observed_at,
  outcome ('resolved'|'partial'|'no_effect'|'worsened'),
  evidence (json), auto_labelled (bool), operator_feedback

agent_playbooks                             -- semantic memory, human-editable
  id, name, domain, fingerprint_pattern, version, is_active,
  content (markdown), preconditions (json), recommended_tools (json),
  author_id, success_rate, times_applied

agent_learned_params                        -- procedural memory
  id, fingerprint, param_key, param_value (json),
  sample_size, confidence, last_updated_at

agent_policies
  id, agent, environment, mode, min_confidence_t1, min_confidence_t2,
  max_devices_per_incident, max_devices_per_hour,
  enabled, updated_by_id, updated_at
```

`agent_actions` and `agent_decisions` are **append-only and never hard-deleted** — this is the compliance artefact.

---

## 6. Control loop

```
 MQTT heartbeat / OTA ACK / Oban failure / periodic sweep / AlertManager webhook
        │
        ▼
 [1] SENSE      AgentCore.Sensors.* normalise → agent_observations
        │       cheap, deterministic, no LLM. For A8: a Prometheus scrape or an
        │       AlertManager POST, normalised the same way as an MQTT event.
        ▼
 [2] TRIGGER    Deterministic Elixir predicates decide "is this worth reasoning about?"
        │       Debounce + fingerprint. Existing open incident with same
        │       fingerprint → attach observation, do NOT start a new turn.
        │       AlertManager webhooks land here as a trigger source, same as
        │       MQTT/Oban/DB — the alert rule already did the thresholding;
        │       the agent's job starts at "now correlate and explain."
        │       ── This gate is what keeps LLM cost bounded. ──
        ▼
 [3] RECALL     Fetch precedents by fingerprint from agent_incidents,
        │       matching agent_playbooks, agent_learned_params
        ▼
 [4] REASON     Oban job → Claude API with tool-use.
        │       Context: observations + precedents + playbook + tool catalogue
        │       Output: hypothesis, confidence, ordered proposed_actions
        │       → persisted as agent_decisions (ALWAYS, even in shadow mode)
        ▼
 [5] GATE       Policy engine (deterministic):
        │         mode? · risk_tier? · confidence ≥ floor? · blast radius?
        │         rate limit? · circuit breaker? · kill switch?
        │       → :execute | :await_approval | :escalate | :shadow_only
        ▼
 [6] ACT        Tool executes inside an Oban job. Result → agent_actions.
        │       Failure → circuit breaker increments for that tool+fingerprint.
        ▼
 [7] VERIFY     Scheduled outcome check at t+2m / t+15m / t+2h.
        │       AUTO-LABELLED from durable state — no human needed:
        │         push  → parameter_push_logs.status == 'acknowledged'
        │                 AND next heartbeat reports the target version
        │         OTA   → app_upgrade_device_status.update_result
        │         offline → heartbeat resumed
        │         recon → exception closed before sla_due_at
        │         infra (A8) → the underlying metric returned below threshold;
        │                      auto-labelled from the next scrape, same pattern
        │       → agent_outcomes
        ▼
 [8] LEARN      Nightly Oban job aggregates agent_outcomes by fingerprint:
                 · update agent_learned_params (cooldowns, thresholds, retry counts)
                 · update playbook success_rate
                 · flag fingerprints with success_rate < 40% for human review
                 · propose new playbooks from repeated successful novel resolutions
                   → these go to a HUMAN REVIEW QUEUE, never auto-activate
```

**Step 7 is the part the Mercury design could not do well and this one can.** "Latency recovered" is a fuzzy, confounded observation. "The terminal ACKed and its next heartbeat reports version 1.1.0" is a fact. Free, unambiguous, high-volume labels are the whole reason this system can actually learn — and this holds even for A8's metric-based outcomes, since the next scrape is just as unambiguous as the next heartbeat.

---

## 7. Learning: what is and is not claimed

The copied doc promises "confidence 92% → 96%" and month-3 seasonal expertise. Be more careful about the mechanism:

**What genuinely improves, automatically:**
- Retry/cooldown/threshold parameters per fingerprint (`agent_learned_params`) — straightforward statistics over `agent_outcomes`, no model training
- Playbook precedent retrieval — more resolved incidents means better in-context examples
- Confidence *calibration* — comparing stated confidence against actual outcome rate, then applying a correction curve. This is a real, measurable improvement and is more valuable than raw accuracy.

**What does not improve automatically:**
- The base model. There is no fine-tuning here and none is proposed.
- Playbook *content*. New playbooks are drafted by the system but **activated only by a human**.

**Explicit safeguard against learned-in error:** a fingerprint whose success rate drops below 40% over ≥ 10 samples auto-demotes that agent to `:suggest` mode *for that fingerprint* and raises a review task. Learning can lower autonomy, not just raise it.

---

## 8. Safety, compliance & failure modes

Payment terminal estate — the bar is higher than the copied design assumed.

| Control | Implementation |
|---|---|
| **Kill switch** | Global + per-agent, DB-backed, honoured at step [5]. Reachable from the console in one click. |
| **Key material is untouchable** | No tool wraps `KeysConfigService`, RKI, KEK/KCV or slot operations. Enforced by absence. |
| **Financial disposition is human-only** | No tool releases holds or alters amounts. A5 and A4 recommend into existing supervisor/finance queues. |
| **Foreign processes are untouchable** | No tool wraps anything in jpos-main or epay-main. A8 reads their exported metrics (once those exist); it has no write path to either, enforced by absence — same pattern as key material. |
| **Blast-radius cap** | Hard numeric ceilings in the policy engine, not in the prompt. |
| **Circuit breaker** | N consecutive failures for a `{tool, fingerprint}` pair → disable that pair, escalate. Directly addresses the "agent stuck in a loop" risk. |
| **Rollback** | Only for tools declaring `reversible: true` with a paired inverse tool. Irreversible tools are T3 minimum. |
| **Prompt-injection surface** | **Device-supplied data (log payloads, remark fields, status items) is untrusted input.** It is passed to the model as clearly delimited data, never as instructions, and can never itself authorise a tool call — the policy gate is outside the model. This risk did not exist in the Prometheus design and is the single most under-appreciated new attack surface. |
| **Audit** | Every decision persisted with model id, prompt version, input digest, reasoning, confidence, and outcome. Exportable for regulators. |
| **Cost control** | Trigger debounce at step [2]; per-agent daily token budget; budget exhaustion degrades to `:shadow`, never to unbounded spend. |
| **Data minimisation** | No PAN, no cardholder data, no key material in prompts. A redaction pass runs on every context assembly and is unit-tested. |

**Failure mode to test hardest: the agent is confidently wrong about a fleet-wide push.** Mitigated by blast-radius cap + T2 gating + canary-first ordering in the Rollout Agent.

---

## 9. Where Prometheus fits

`promotheus.yml` in this folder scrapes `mercury-switch:/monitoring/metrics` for `iso8583_*`, `network_*`, `beam_*`, `reversal_*`. **None of those metric families exist in this repo today**, and nothing named "mercury-switch" appears anywhere in the codebase — that file is either a different system's config or aspirational, not deployed TMS infrastructure.

**Revised stance, after §1.7.** The earlier draft of this section treated Prometheus as uniformly "optional, secondary, host/infra health only." That was too broad a dismissal. The honest split is:

- **For the TMS-native domains (A1–A5)** — config compliance, fleet health, rollout, settlement, risk — DB state and MQTT events remain the backbone. These domains have real rows and real events; polling a TSDB would be a worse signal than what already exists. §9's original caution stands *here*.
- **For the infra/switch domain (A8, §1.7)** — BEAM health, jPOS process health, acquirer network health — **Prometheus is not secondary, it is the only sensor that exists.** There is no database row for "BEAM memory has been climbing for 45 minutes" or "the YSP SSL channel is flapping." A trend is not observable from a point-in-time query. This is exactly the sensing modality Prometheus + Grafana were built for, and AlertManager is exactly the threshold engine that should keep doing threshold work rather than having an LLM re-derive it.

**What to build, in order:**

1. **BEAM export (Phase 3, in-repo).** Add `prom_ex` (or `telemetry_metrics_prometheus`) to `platform_web` and `da_product_app`, expose `/metrics`, and emit TMS-domain metrics alongside the standard BEAM ones: `tms_terminals_online_total`, `tms_config_push_result_total{config_type,result}`, `tms_ota_rollout_progress`, `tms_recon_exceptions_open`, `tms_agent_decisions_total{agent,outcome}`.
2. **AlertManager as a trigger source (Phase 3).** A webhook receiver in `platform_web` that normalises AlertManager POSTs into `agent_observations` and feeds step [2] of the control loop (§6) — the same trigger path as an MQTT event or a DB write, not a separate system.
3. **jPOS/network export (Phase 3+, cross-team, not blocking).** Requires the Java side to expose something scrapeable — a JMX-to-Prometheus bridge, or extending the existing Q2 `99_sysmon.xml` monitor. Scope and request this early (§12); do not let A8's BEAM-only capability wait on it.
4. **Grafana** as the human-facing dashboard for whatever gets exported — complementary to the Agent Console (§2, L6), not a replacement. The Console shows agent reasoning and decisions; Grafana shows the raw time series underneath them.

Either way, **the rest of the agent system must not depend on Prometheus being deployed.** A8 degrades gracefully to BEAM-only if jPOS/network export never lands; A1–A5 never depend on it at all.

---

## 10. Implementation phases

Deliberately recalibrated from the copied doc's "2.5 FTE × 6 months, $400–600K". That estimate was for a greenfield control plane; here most actuators exist.

Phase-level here; sprint-by-sprint breakdown with UI work mapped in, a critical-path/dependency list, and a risk register live in [implementation-plan.md](implementation-plan.md).

> **Revised after reading the mw-core design.** An earlier draft of this plan put the Config Compliance Agent first and Investigation in Phase 3. mw-core's argument for investigation-first is better: it is read-only, needs no approval gates, has near-zero blast radius, and exercises the same tool-calling runtime every later agent depends on. Its one blocker — the correlation key — doesn't exist here (§1.5). **A6 Investigation now runs first**; A2 Config Compliance follows as the first agent that *acts* and the first that *learns*. This also front-loads a deliverable ops can use on day one (dispute investigation) rather than a shadow-mode agent nobody can see.

### Phase 0 — Foundation (4–5 weeks — widened to cover D6/D8 and the §1.8 spike)
- [ ] Create `apps/agent_core` umbrella app; wire into `mix.exs` releases
- [ ] Migrations for all nine `agent_*` tables + `agent_investigations` + Ecto schemas; `agent_policies` gets a `provider` column (D6)
- [ ] **Entity-resolution data-quality spike** ([entity-resolution.md](entity-resolution.md) §4) — read-only check of whether `tms_terminals.merchant_id`/`serial_number` actually correspond to Cloud Layer/jPOS identity today. Half a day; blocks any tool that assumes the join, not the rest of Phase 0.
- [ ] **MCP server library spike** (D8) — evaluate the Elixir MCP ecosystem against this codebase; not committing to a library in this document.
- [ ] `AgentCore.LLM.Provider` behaviour + Anthropic adapter (primary) — Req-based Messages API tool-use loop, bounded step budget, retry, token accounting, budget guard (D6 has the exact request shape and easy-to-miss constraints). OpenAI and Ollama adapters can land in Phase 3 alongside A8 — not blocking Phase 1/2, since Claude is the default provider either way.
- [ ] `AgentCore.ReadTool` behaviour + **redaction layer at the tool boundary** (PAN → first6+last4, no CVV/track data, no key material) — unit-tested as a pure function
- [ ] `AgentCore.Tool` behaviour + tool registry + `AgentCore.Policy` engine (deterministic, heavily unit-tested)
- [ ] Sensors: MQTT tap on the existing handler, Oban telemetry tap, periodic sweep
- [ ] **Baseline measurement — two weeks of current metric values (see §11)**
- [ ] **Exit criterion:** policy engine has ≥ 90% branch coverage, rejects every T4 attempt in tests, the redaction layer has no path that emits a full PAN, and the entity-resolution spike has a documented match rate (not an assumption)

### Phase 1 — A6 Investigation Agent, read-only (4–6 weeks)
- [ ] Read tools across terminal / config / OTA / settlement / risk, all with hard row caps and mandatory time windows
- [ ] The §1.5 join-path walker: `switch_dump_records` → `core_transactions` → `settlement_mis_items` → `payout_items`, plus `reconciliation_exceptions` lookup — checking the live `pos_transaction` table first (§1.6)
- [ ] Investigation loop with budget enforcement; Investigation Report format; `/agents/investigations` LiveView
- [ ] **Self-inspection tools** (`get_agent_decisions`, `get_outcome_stats`) — needed in Phase 2 to review A2's shadow decisions without a spreadsheet
- [ ] **Exit criterion:** on 20 hand-labelled historical incidents (mixed: disputes, offline cohorts, failed rollouts), A6 identifies the correct primary cause in ≥ 70% **and produces zero confidently-wrong reports** — never `HIGH` confidence on an incorrect finding

### Phase 2 — A2 Config Compliance, shadow → T1 autonomy (4–6 weeks)
- [ ] **A2 Config Compliance Agent** in `:shadow` — cleanest signals, cleanest outcome labels, and an existing rule-based baseline to beat
- [ ] Tools: `trigger_config_push`, `regenerate_config_artifact`, `start_remote_log_session` (all T1)
- [ ] Outcome auto-labeller (step [7]) for push convergence — the first real learning loop
- [ ] Console: incident feed + decision cards + shadow-vs-`VersionComplianceChecker` comparison view
- [ ] Promote to `:auto_low_risk` once the exit criterion holds
- [ ] **Exit criterion:** over ≥ 200 shadow decisions, A2 matches-or-beats `VersionComplianceChecker` on convergence rate, with ≥ 80% of proposals judged correct by ops review

### Phase 3 — Fleet coverage + orchestration + infra sensing (6–8 weeks)
- [ ] **A1 Fleet Health** and **A3 Rollout** in `:shadow` → `:suggest`
- [ ] Approval queue for T2/T3 in the console
- [ ] **A7 Ops Orchestrator** — correlation & dedup (highest immediate operator value: kills alert storms)
- [ ] **A8 Infrastructure & Switch Health** — BEAM export (`prom_ex`/`telemetry_metrics_prometheus`) + AlertManager webhook intake, both in-repo; jPOS/network export requested from the Java side as a parallel, non-blocking track (§1.7, §9, §12)
- [ ] Circuit breakers, rate limits, blast-radius caps live and exercised in staging drills
- [ ] **Exit criterion:** zero unapproved T2+ executions; ≥ 30% of config-compliance incidents auto-resolved; A8 BEAM-only detection catches at least one real memory/run-queue trend in staging before it becomes an incident

### Phase 4 — Settlement, risk, learning (6–8 weeks)
- [ ] **A4 Settlement Ops** — predictive SLA warning ahead of the 06:15/07:15/08:15 checkpoints
- [ ] **A5 Risk Triage** — recommend-only, permanently
- [ ] Nightly learning job; playbook review queue; confidence calibration report
- [ ] jPOS/network export, if the Java-side dependency has landed by now — A8 upgrades from BEAM-only to full infra coverage
- [ ] **Exit criterion:** SLA breaches predicted ≥ 30 min ahead in ≥ 70% of cases

### Phase 5 — Ongoing
Confidence calibration, playbook curation, selective T2 autonomy per fingerprint where success rate justifies it. Cloud Layer (QR/rules engine) and jPOS-adjacent acting agents are legitimate candidates here, once A2–A5 have proven the pattern (§1.6, item 5).

---

## 11. Success metrics

MTTR is the wrong headline metric for TMS. Replace with:

| Metric | Definition | Baseline | Phase 3 | Phase 4 |
|---|---|---|---|---|
| **Config convergence rate** | % of terminals on target version within 24h | measure first | +15 pp | +30 pp |
| **Mean time to convergence** | first drift detection → verified target version | measure first | −40% | −60% |
| **First-push success rate** | pushes ACKed without re-push | measure first | +20 pp | +35 pp |
| **Alert dedup ratio** | raw findings ÷ operator-visible incidents | 1:1 | 4:1 | 8:1 |
| **Rollout failure containment** | rollouts halted before 5% device failure | manual | 70% | 90% |
| **Exception aging** | % of recon exceptions closed before `sla_due_at` | measure first | +15 pp | +30 pp |
| **SLA breach lead time** | warning issued before the 06:15/07:15/08:15 checkpoint | 0 (post-hoc) | — | ≥30 min in 70% |
| **Infra trend detection lead time** | A8 flags a BEAM/jPOS trend before it becomes a user-visible incident | 0 (reactive only) | ≥1 caught in staging | measured in prod |
| **Decision precision** | % of proposals judged correct on review | — | ≥80% | ≥90% |
| **Confidence calibration error** | \|stated confidence − actual success rate\| | — | ≤15 pp | ≤8 pp |
| **Autonomous resolution rate** | incidents closed with no human action | 0% | 30% | 50% |
| **Unauthorised action count** | T2+ executed without a valid gate | — | **0** | **0** |

**Baseline first.** Phase 0 must include a two-week measurement of current values, or none of the deltas above mean anything.

---

## 12. Open decisions

1. **Model choice + data residency.** Recommend Claude (`claude-opus-4-8`) as the default provider for reasoning turns — note that unlike mw-core, nothing is wired here yet, so this is a free choice rather than an inherited one. Confirm whether UAE payment-ops regulation permits sending terminal and settlement telemetry to a US-hosted API. *This is the one question that can invalidate the design and should be answered before Phase 0.* If it fails, D6's Ollama adapter is the designed-in answer, not a fallback improvised later — route the affected agents/environments to it via the `agent_policies.provider` column. Heavier redaction (only non-identifying operational state leaves the estate) is the complementary mitigation regardless of which provider is used.
2. **Cost envelope.** The trigger-gated design should land well under the copied doc's $500–2000/month for Claude/OpenAI usage, but set a hard monthly budget in Phase 0. Ollama has a different cost shape entirely — GPU/CPU infra, not per-token billing — size that separately if it becomes the mandated provider for any environment.
3. **`da_product_app` mirror.** Does `agent_core` need mirroring into the legacy app, or is it tms_core-forward only? Recommend the latter — no mirror.
4. **Console placement.** New top-level nav section, or extend `alerts_live`? Recommend a new `/agents` section with its own RBAC role.
5. **Environment.** Build against UAT with production-shaped data, or a dedicated agent-dev environment?
6. **Which agent goes first** if not A2 — the choice above is argued on signal quality and baseline availability; ops may have a more painful problem.
7. **jPOS/network metrics export — who owns this ask?** §1.7 and §9 both depend on the Java side exposing scrapeable metrics (JMX-to-Prometheus bridge, or extending the Q2 `99_sysmon.xml` monitor). This is a real cross-team dependency and should be scoped and requested in parallel with Phase 0–2, not discovered as a blocker when Phase 3 starts. A8 is designed to degrade gracefully to BEAM-only if this never lands, but the fuller value (jPOS reversal backlog, acquirer connectivity trends) depends on it.
Answer: The JPOS to Promotheus bridge already built and alert manager will push the webhook to this system. We need to provide the endpoint for this.
8. **MCP server library.** D8 restructures the tool layer as one MCP server per domain. Spike done: `hermes_mcp` is the lead candidate over `conduit_mcp` (mcp-servers.md has the comparison) — the remaining step is a build-time trial, not further research. Doesn't block Phase 1's Investigation Agent, which already runs against a plain `AgentCore.ToolRegistry` (built in this pass) and can migrate to MCP once the trial concludes.
9. **Entity-resolution match rate.** §1.8 / [entity-resolution.md](entity-resolution.md) §4 flags that the device→merchant→MID/TID join is unverified in production data. The Phase 0 spike answers this; if the match rate is poor, A1 and A6 need `:unresolved` handling for that hop before it's presented to an operator as a fact.

---

## 13. Summary

**Keep** from the monitoring proposal: the observe→reason→act→learn loop, the risk-tiered gate, the Incident Commander pattern, shadow-mode rollout.

**Keep** from the mw-core investigation design: investigation-first sequencing, the read-only invariant enforced in code rather than prompt, the bounded step budget, and evidence-alongside-narrative.

**Discard:** Prometheus-as-*sole*-backbone, the five infra agents, MTTR as headline metric, the ROI model, the effort estimate — and, from mw-core, the correlation-key work-stream and the three external read-API dependencies, which this repo does not need.

**Add,** because TMS uniquely supports it: durable auto-labelled outcomes, business-domain actuators that already exist, hard blast-radius limits, absence-based enforcement for prohibited operations, and treatment of device-supplied data as an untrusted injection surface.

**Add, after reconciling the infra/switch feedback (§1.7):** a dedicated agent (A8) whose backbone genuinely is Prometheus + AlertManager, because BEAM health, jPOS process health, and acquirer network health have no database row at all — the one place the original monitoring proposal's instincts were right, scoped to the one domain where they actually apply.

**Add, after reconciling entity resolution, skills, MCP, and multi-provider (§1.8, D6–D8):**
- A verified (not assumed) cross-system identity map, with an explicit, honestly-flagged gap where the device→merchant→MID/TID join isn't yet proven in production data — and a half-day spike to resolve it before any tool depends on it.
- "Skills" formalized as `agent_playbooks` per agent domain, deliberately *not* Anthropic's provider-native Skills feature, because that would break multi-provider portability.
- The tool layer restructured as one MCP server per domain (`tms-mcp`, `settlement-mcp`, `switch-mcp`, `risk-mcp`, `infra-mcp`) — reusable outside the agent loop, and the actual mechanism that makes multi-provider tool-calling work without redefining every tool three times.
- `AgentCore.LLM.Provider` with real Anthropic, OpenAI, and Ollama adapters, routed per agent/environment via policy — making the data-residency fallback in §12 a designed capability, not an improvised one if the residency question ever fails.

Four arguments carry the decision to build here:

1. **The correlation already exists — and where it doesn't, that's now known rather than assumed.** mw-core's flagship dispute scenario needed a cross-team correlation-key programme plus three new external read APIs. In `settlement_core` — and even more directly in the live `pos_transaction` table (§1.6) — it is a join path that `ReconciliationEngine` already walks, and when it fails, the system has already written down why. The one place the same discipline turned up a real gap (device↔merchant identity, §1.8) is now a scoped, half-day spike instead of a silent assumption.
2. **Step [7] is free and unambiguous.** TMS can tell you whether the agent was right — the device ACKed and the next heartbeat reports the target version, or the metric crossed back below threshold on the next scrape — thousands of times a day. That is the supervised signal that makes the learning loop real rather than aspirational.
3. **Two sensing modalities, correctly scoped, beat one modality misapplied everywhere.** DB/MQTT events for the business domains that have them; metrics/alerts for the one domain (BEAM/jPOS/network) that has nothing else. A7 Ops Orchestrator is what turns "two separate dashboards" into one correlated incident.
4. **The tool layer and the model layer are decoupled on purpose.** MCP servers per domain plus a provider behaviour with three real backends means neither "which LLM" nor "which system owns this tool" is a one-way door — a real requirement for a system that has to satisfy both data-residency constraints and a payment-terminal blast radius.