# AI-Agentic Pipeline — Operations Runbook

**Scope**: Observability, incident response, and day-to-day operations for the
AI-Agentic flow-proposal pipeline introduced in Phases 1–5.

---

## 1. Key Components

| Component | Location | Purpose |
|-----------|----------|---------|
| `MwKernel.AiAgentic.Telemetry` | `apps/mw_kernel` | ETS counter accumulation + telemetry event dispatch |
| `InfraRepo.AiFlow.ProposalService` | `apps/infra_repo` | LLM call + normalization + persistence |
| `InfraRepo.AiFlow.ProposalLifecycle` | `apps/infra_repo` | State machine transitions + audit events |
| `InfraRepo.AiFlow.ProposalNormalizer` | `apps/infra_repo` | Validates & normalises raw LLM graph output |
| `MwKernel.AdapterDiscovery` | `apps/mw_kernel` | Scans adapters for MCP tool definitions |
| `MwKernel.LlmConfig` | `apps/mw_kernel` | LLM provider config + HTTP client |
| `GatewayWebWeb.AiDashboardLive` | `apps/gateway_web` | Ops dashboard — `/admin/ai-dashboard` |
| `MwKernel.FeatureFlags` | `apps/mw_kernel` | Enable/disable flag `"ai_agentic_enabled"` |

---

## 2. Feature Flag — Enable / Disable

### Via UI
1. Navigate to `/admin/ai-dashboard`.
2. Click the **AI Enabled / AI Disabled** pill button in the top-right of the page.
3. Change takes effect immediately (next request / next LiveView mount).

### Via IEx (emergency)
```elixir
# Disable
MwKernel.FeatureFlags.disable("ai_agentic_enabled")

# Re-enable
MwKernel.FeatureFlags.enable("ai_agentic_enabled")

# Check current state
MwKernel.FeatureFlags.enabled?("ai_agentic_enabled")
```

### What gets gated
- `✨ Generate with AI` button on the Flows page (`/admin/flows`)
- AI Proposal REST API (`POST /admin/api/ai/proposals` etc.) — guarded by `AiFeatureGate` plug
- Dashboard reflects current flag state in real time

---

## 3. Alert Threshold — Failure Rate

The failure rate is computed as:

```
failure_rate = (validated_fail + errors) / generated × 100
```

The dashboard shows a **red alert banner** when `failure_rate ≥ 20%`.

To change the threshold, edit `@failure_rate_threshold` in
`apps/mw_kernel/lib/mw_kernel/ai_agentic/telemetry.ex` and redeploy.

> **Note**: Session counters reset on server restart. The alert banner reflects
> only the current session. For lifetime trends use the DB lifetime stats panel.

---

## 4. Telemetry Events Reference

All events are under the `[:mw, :ai_agentic, :proposal, *]` namespace.

| Event | Emitted by | Key measurements |
|-------|-----------|-----------------|
| `:generated` | `ProposalService` after successful LLM call | `duration_ms` |
| `:validated` | `ProposalService` after `ProposalNormalizer` | `duration_ms`, `outcome: :pass/:fail`, `warning_count` |
| `:approved` | `ProposalLifecycle.approve/3` | `count: 1` |
| `:rejected` | `ProposalLifecycle.reject/3` | `count: 1` |
| `:published` | `ProposalLifecycle.publish/3` | `count: 1` |
| `:error` | `ProposalService` on LLM/parse failures | `stage, reason` |

Attach a custom handler (e.g., for Prometheus) with:
```elixir
:telemetry.attach("my-ai-metrics", [:mw, :ai_agentic, :proposal, :generated], &MyHandler.handle/4, nil)
```

---

## 5. Common Error Patterns

### `LLM generation failed: ...`
- **Cause**: `LlmConfig.call_llm/2` returned `{:error, _}`.
- **Check**: LLM provider API key in `config/runtime.exs` (`LLM_API_KEY` env var).
- **Check**: Provider health endpoint (see §8).
- **Action**: If provider down, disable AI flag until resolved.

### `Failed to parse LLM response: ...`
- **Cause**: LLM returned non-JSON or malformed JSON.
- **Check**: Review `raw_graph_json` on the failed proposal in the DB:
  ```sql
  SELECT proposal_id, raw_graph_json FROM ai_flow_proposals
  WHERE validation_status = 'fail'
  ORDER BY inserted_at DESC LIMIT 5;
  ```
- **Action**: Adjust the system prompt in `ProposalService.build_orchestrator_prompt/2` to reinforce JSON-only output.

### `Failed to normalize AI output: ...`
- **Cause**: `ProposalNormalizer` rejected the graph (unknown node type, missing required fields, cycle detected).
- **Check**: `validation_warnings_json` column on the proposal.
- **Action**: Tighten or relax node-type allow-list in `ProposalNormalizer`.

### `Cannot approve proposal with validation status: ...`
- **Cause**: Operator attempted to approve a proposal that failed validation.
- **Action**: Review warnings, edit the proposal canvas manually in Flow Builder, or reject and regenerate.

### `Invalid transition from X to Y`
- **Cause**: State machine guard rejected the transition.
- **Check**: `@valid_transitions` in `ProposalLifecycle`.
- **Valid flow**: `draft_generated → draft_opened → draft_edited → approved → published`

---

## 6. Database Queries

### Proposals by status
```sql
SELECT status, count(*) FROM ai_flow_proposals GROUP BY status;
```

### Proposals stuck in draft (> 48 h old)
```sql
SELECT proposal_id, goal, requested_by, inserted_at
FROM ai_flow_proposals
WHERE status IN ('draft_generated', 'draft_opened', 'draft_edited')
  AND inserted_at < NOW() - INTERVAL 48 HOUR
ORDER BY inserted_at ASC;
```

### Event audit trail for a specific proposal
```sql
SELECT event_type, actor_id, event_payload_json, inserted_at
FROM ai_flow_proposal_events
WHERE proposal_id = 'ap_<id>'
ORDER BY inserted_at ASC;
```

### Archive stale drafts (manual)
```elixir
# In IEx — archive proposals older than 7 days still in draft
import Ecto.Query
alias InfraRepo.{Repo, Schemas.AiFlowProposal, AiFlow.ProposalLifecycle}

cutoff = DateTime.add(DateTime.utc_now(), -7 * 86400, :second)

Repo.all(
  from p in AiFlowProposal,
    where: p.status in ["draft_generated", "draft_opened"],
    where: p.inserted_at < ^cutoff
)
|> Enum.each(fn p ->
  ProposalLifecycle.archive(p, "system_gc", "stale draft auto-archived")
end)
```

---

## 7. Rollback Procedure

If the AI-Agentic pipeline causes critical issues:

### Step 1 — Disable feature flag (instant, no deploy needed)
```elixir
MwKernel.FeatureFlags.disable("ai_agentic_enabled")
```
This immediately hides the AI button from the UI and blocks the API.

### Step 2 — Remove `AiDashboardLive` from supervisor (if crashing)
In `apps/gateway_web/lib/gateway_web/application.ex`, comment out:
```elixir
# MwKernel.AiAgentic.Telemetry,
```
Redeploy. The rest of the app runs normally without AI telemetry.

### Step 3 — Rollback to last stable commit
```bash
git log --oneline -10
git revert <ai-agentic-commit-sha>
```
All AI-Agentic code is isolated in:
- `apps/mw_kernel/lib/mw_kernel/ai_agentic/`
- `apps/mw_kernel/lib/mw_kernel/adapter_discovery.ex`
- `apps/infra_repo/lib/infra_repo/ai_flow/`
- `apps/gateway_web/lib/gateway_web_web/live/ai_dashboard_live.ex`
- `apps/gateway_web/lib/gateway_web_web/live/flows_live.ex` (AI button only)

Core routing and flow execution are **not affected** by AI code removal.

---

## 8. LLM Provider Health Checks

### OpenAI / Azure OpenAI
```bash
# Check API availability
curl -s -o /dev/null -w "%{http_code}" \
  https://api.openai.com/v1/models \
  -H "Authorization: Bearer $LLM_API_KEY"
# Expect: 200
```

### Verify LLM config in runtime
```elixir
MwKernel.LlmConfig.configured?()
# => true / false

Application.get_env(:mw_kernel, :llm_provider)
# => :openai | :azure_openai | :anthropic
```

---

## 9. Monitoring Checklist

For each deployment that includes AI-Agentic, verify:

- [ ] `MwKernel.AiAgentic.Telemetry` GenServer starts without errors in log
- [ ] `/admin/ai-dashboard` loads without crash (DB tables exist)
- [ ] Session counter cards show `0` (not error state) after clean boot
- [ ] Alert banner NOT shown (failure rate 0% at startup)
- [ ] Feature flag status matches expected environment setting
- [ ] At least one LLM test proposal can be generated end-to-end

---

## 10. Related Docs

- [IMPLEMENTATION_STATUS.md](IMPLEMENTATION_STATUS.md) — Phase completion tracking
- [phase1-technical-spec.md](phase1-technical-spec.md) — Persistence schema design
- [phase2-implementation-plan.md](phase2-implementation-plan.md) — Route + auth design
- [ai-agentic-initial-approach.md](ai-agentic-initial-approach.md) — Original vision & trade-offs