# Investigation Agent — Design > **Status:** Proposed (design only, no code yet) > **Scope:** A **read-only** agentic investigation capability for cross-system transaction/incident diagnosis. > **LLM runtime:** OpenAI (the only provider actually wired today — see `MwKernel.LlmConfig`). > **Relationship to the existing "AI-Agentic" work:** This is a **new** capability. The current > `AiAgentic.*` code is a *flow-proposal* pipeline, not an investigation system. See §2. --- ## 1. Problem statement An operator or an alert observes a symptom that **spans multiple systems** and needs the truth correlated across all of them. Canonical example: > "Transaction `X` shows **successful** in the switch, but the **merchant says it is not in their > settlement report**. Is the claim correct? If the auth really succeeded, where did it stop — > settlement batching or payout?" Answering this today is manual: log into the switch, grep logs, query settlement, check payout, mentally correlate. We want an agent that does this **read-only** investigation and returns a narrative + evidence, triggered either by: - a **human chat** question ("investigate txn X"), or - a **Prometheus / Alertmanager** alert firing. Investigation is deliberately chosen as the **first** agentic use-case because it is **read-only**: low blast radius, useful without approval gates, and it exercises the same tool-calling runtime that later remediation agents (see `ARCHITECTURE.md`) will need. --- ## 2. What exists today vs. what this needs | Capability | Exists today | Where | Needed for this? | |---|---|---|---| | Flow-proposal LLM pipeline | ✅ | `InfraRepo.AiFlow.ProposalService` | Reused as a *pattern* only | | Proposal telemetry (ETS counters) | ✅ | `MwKernel.AiAgentic.Telemetry` | Reused as a *pattern* for tool-call spans | | Proposal dashboard (LiveView) | ✅ | `GatewayWebWeb.AiDashboardLive` | UI shell to reuse | | OpenAI text completion | ✅ | `MwKernel.LlmConfig.call_llm/2` | **Must extend** — see below | | Claude / Llama providers | ⚠️ placeholder | `LlmConfig.claude_call/llama_call` | Not used (OpenAI chosen) | | **Multi-step tool-calling loop** | ❌ | — | **NEW — core of this design** | | **MCP / external tool client** | ❌ | — | **NEW** | | **Chat surface** | ❌ | — | **NEW** | | **Alertmanager webhook intake** | ❌ | — | **NEW** | | Local `transactions` table | ✅ | `InfraRepo.Schemas.Transaction` | Read tool source #1 | **Key gap:** `LlmConfig.call_llm/2` is single-shot `text → text`. Investigation requires an `observe → call tool → read result → reason → call next tool → … → conclude` loop using OpenAI's **function/tool calling** API. That loop is the central new component. **Key constraint:** mw-core owns almost none of the truth. Its `transactions` table is a **risk-oriented shadow copy** keyed by `cbs_reference`. Authoritative auth/settlement/payout state lives in the **Switch**, **Settlement**, and **Payout** systems. Therefore most external changes are about **each system exposing a read interface the agent can call** (§6). --- ## 3. Architecture ``` ┌──────────────────────── ENTRY POINTS ────────────────────────┐ │ │ Human chat (LiveView) Alertmanager webhook │ "investigate txn X" POST /alerts/webhook │ │ │ │ └───────────────────────┬────────────────────────┘ │ ▼ │ ┌───────────────────────────────────────────────────────────┐ │ │ InvestigationAgent (mw_kernel, NEW) │ │ │ ─ builds system prompt + tool catalog │ │ │ ─ runs OpenAI tool-calling loop (bounded step budget) │ │ │ ─ read-only guardrail + audit every step │ │ │ ─ emits telemetry spans per tool call │ │ └───────────────┬───────────────────────────────────────────┘ │ │ tool calls (JSON args) │ ┌──────────┬───────┴───────┬───────────────┬───────────────┐ │ ▼ ▼ ▼ ▼ ▼ │ Local Ecto Log search Switch tool Settlement tool Payout tool │ tool tool (MCP/REST) (MCP/REST) (MCP/REST) │ transactions logs get_switch_ get_settlement_ get_payout_ │ / archive auth() status() status() │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ ▼ │ mw-core DB mw-core logs MERCURY SWITCH SETTLEMENT SYS PAYOUT SYS │ (external) (external) (external) │ │ All tools are READ-ONLY. Correlation key threads every hop. ────┘ ``` ### 3.1 Component responsibilities - **InvestigationAgent** (`MwKernel.AiAgentic.InvestigationAgent`, NEW): owns the tool-calling loop, the step budget, the read-only invariant, and audit logging. Provider-thin: it calls into an extended `LlmConfig`. - **LlmConfig (extended)**: add `call_with_tools/3` that hits OpenAI's chat-completions **with a `tools` array** and returns either a final message or a `tool_calls` request. Existing `call_llm/2` stays for the proposal pipeline. - **ToolRegistry** (NEW): describes each available tool to the LLM (name, JSON-schema params, description) — mirroring how `AdapterDiscovery` describes adapters to the proposal LLM. This is the discovery seam. - **Tools**: each is a module implementing a common behaviour (`name/0`, `spec/0`, `invoke/1`). Local tools query Ecto directly; remote tools call an MCP server or authenticated REST endpoint on the owning system. - **Entry points**: a chat LiveView and an Alertmanager webhook controller. Both build an initial prompt and hand off to `InvestigationAgent`. --- ## 4. The tool-calling loop ``` 1. Build messages = [system_prompt, user_or_alert_context] 2. Build tools = ToolRegistry.specs() # OpenAI function schemas 3. LOOP (max N steps, e.g. 8): resp = LlmConfig.call_with_tools(messages, tools) if resp has tool_calls: for each call: assert tool is READ-ONLY (registry flag) # hard guardrail result = Tool.invoke(args) # audited + telemetry span append {role: "tool", ...result} to messages continue else: return final narrative + collected evidence 4. If budget exhausted: return partial findings + "inconclusive, escalate". ``` Design notes: - **Step budget** prevents runaway loops and caps LLM/token cost. - **Read-only invariant** is enforced in `InvestigationAgent`, *not* trusted to the prompt: the registry marks each tool `access: :read`, and any non-read tool is rejected before invocation. (No write tools exist in this design at all — remediation is out of scope.) - **Every tool call is audited** via `mw_audit` and gets a telemetry span (reuse the `Telemetry`/`:telemetry.execute` pattern already in `MwKernel.AiAgentic.Telemetry`). - **Evidence, not just prose**: the agent returns the raw tool results alongside the narrative so the operator can verify — never "trust the summary." --- ## 5. Correlation key — the hard dependency The agent can only *follow* a transaction across four systems if they share a lookup key. Today: - mw-core `transactions.cbs_reference` links to the core-banking reference. - The switch keys by ISO-8583 **RRN / STAN / auth id**. - Settlement keys by **batch reference**. - Payout keys by **disbursement id**. If these do not map cleanly, the agent goes blind between hops. **Resolve this before building the remote tools.** Two acceptable strategies: - **(Preferred) End-to-end trace id**: stamp a single correlation id at ingress and carry it through switch → settlement → payout, stored alongside each system's native key. Every read tool accepts and returns it. - **(Interim) Lineage/mapping service**: a lookup that, given any one key, returns the others. More moving parts, but no changes to each system's internal keying. This is usually the **largest and most cross-team** item — treat it as a prerequisite work-stream, not part of the agent code. --- ## 6. Required changes per system ### mw-core (host — most of the new code) 1. `LlmConfig.call_with_tools/3` — OpenAI tool-calling. 2. `MwKernel.AiAgentic.InvestigationAgent` — the loop, budget, guardrail, audit. 3. `ToolRegistry` + tool behaviour. 4. Local tools: `transactions` / `risk_transaction_archive` Ecto tool; log-search tool. 5. MCP/REST client (reuse `adapter_http` / `adapter_grpc` transports). 6. Chat LiveView + Alertmanager webhook controller (in `gateway_web`). 7. Feature-flag gate (reuse `ai_agentic_enabled` pattern) + audit wiring. ### Mercury Switch (external) - Add a **read** interface: `get_transaction(correlation_id | rrn | stan)` → auth status, MTI, response code, timestamps; plus **log retrieval by reference**. Today it only exposes `/monitoring/metrics`. ### Settlement system (external) - Add read tool `get_settlement(correlation_id | cbs_reference)` → batch id, batch state, cutoff result, expected report date. ### Payout system (external) - Add read tool `get_payout(correlation_id | cbs_reference)` → disbursement status, rail, timestamps, failure reason. ### All systems - Emit / carry the **correlation key** (§5), and expose read access behind auth (service token / mTLS). Every tool call is logged on both sides. --- ## 7. Worked example — the merchant dispute ```mermaid sequenceDiagram participant U as Operator (chat) participant A as InvestigationAgent participant DB as mw-core txn DB participant SW as Switch tool participant SE as Settlement tool participant PO as Payout tool U->>A: "Txn X successful in switch, missing from merchant report. Investigate." A->>DB: get_transaction(cbs_reference=X) DB-->>A: status=settled, risk_decision=approve, correlation_id=C A->>SW: get_switch_auth(C) SW-->>A: auth=APPROVED, rrn, response_code=00, ts Note over A: Claim of "successful" CONFIRMED at switch A->>SE: get_settlement(C) SE-->>A: batch=B123, state=EXCLUDED (missed cutoff) Note over A: Root cause candidate found A->>PO: get_payout(C) PO-->>A: no disbursement (never batched) A-->>U: Narrative: auth succeeded; txn missed settlement cutoff B123, so it never reached payout and is absent from the merchant report. Evidence attached. ``` The agent's value is the **correlation + narrative + evidence**, not any single lookup. --- ## 8. Safety, audit, and cost - **Read-only by construction** — no write tools defined; guardrail enforced in code. - **Full audit trail** — every tool call (inputs, outputs, timing) via `mw_audit`; final narrative persisted like proposals are. - **Telemetry** — per-tool-call spans and a session counter set, mirroring `MwKernel.AiAgentic.Telemetry`; surface on a new tab of the existing dashboard. - **Step + token budget** — bounded loop; inconclusive results escalate to a human rather than guessing. - **Data sensitivity** — tool outputs may contain PII/account data; keep the LLM traffic on the approved provider path and redact where the investigation doesn't need raw PANs/accounts. --- ## 9. Phasing 1. **mw-core-only** — tool-calling loop + local `transactions`/log tools + chat panel. Zero external dependencies; proves the runtime. 2. **+ Switch read tool** — answers "was it really authorized?" 3. **+ Settlement + Payout tools** — completes the merchant-dispute scenario. 4. **+ Alertmanager webhook** — Prometheus-triggered auto-investigation posting a findings card. 5. **(Later, out of scope here)** remediation/action agents from `ARCHITECTURE.md` — these need approval gates and are deliberately *after* a trusted read-only agent. **Hard prerequisite for phases 2–3:** the correlation key (§5).