# Fraud & AML Platform — Phase 10–18 Implementation Plan **Branch:** `feature/fraud-aml-phase10-18` **Base:** `feature/fraud-aml` (Phases 1–9 complete) **Design principle:** Jube UI content/structure (form fields, labels, navigation hierarchy) rendered with our modern Phoenix LiveView + Tailwind CSS stack. --- ## Reference: Jube Navigation Hierarchy → Our Implementation ``` Jube Menu Our Route (/admin/fraud/...) ─────────────────────────────── ───────────────────────────────────── Cases → Case Search /cases ✅ EXISTS Suppression /suppression ⬜ Phase 12 Lists /lists ⬜ Phase 11 Dictionary /dictionaries ⬜ Phase 11 Models → Synchronisation /models/sync ⬜ Phase 10 Models → Model /models ⬜ Phase 10 Models → References → Request XPath /models/:id/xpaths ⬜ Phase 10 → Inline Functions /models/:id/functions ⬜ Phase 19 (later) → Inline Scripts /models/:id/scripts ⬜ Phase 19 (later) → Gateway Rules /models/:id/gateway ⬜ Phase 10 → Sanctions /sanctions ✅ EXISTS → Tags /tags ⬜ Phase 11 Models → Abstraction → TTL Counters /models/:id/ttl ⬜ Phase 10 → Abstraction Rules /models/:id/abstractions ⬜ Phase 14 → Abstraction Calculation /models/:id/calculations ⬜ Phase 14 Models → Machine Learning → HTTP Adaptation /models/:id/adaptations ⬜ Phase 19 → Exhaustive Adaptation /ml/experiments ⬜ Phase 15 Models → Activation Rules /rules ✅ EXISTS (enhance Ph13) Models → Cases Workflows → Cases Workflows /workflows ⬜ Phase 13 → Cases Workflows Status /workflows/:id/statuses ⬜ Phase 13 → Cases Workflow Form /workflows/:id/forms ⬜ Phase 13 → Cases Workflow Action /workflows/:id/actions ⬜ Phase 13 → Cases Workflow Display /workflows/:id/displays ⬜ Phase 13 → Cases Workflow Macro /workflows/:id/macros ⬜ Phase 13 → Cases Workflow Filter /workflows/:id/filters ⬜ Phase 13 → Cases Workflow XPath /workflows/:id/xpaths ⬜ Phase 13 Models → Reprocessing /reprocessing ✅ EXISTS Visualisation /visualisations ⬜ Phase 17 Sanctions /sanctions ✅ EXISTS Watcher /watcher ⬜ Phase 16 Administration → Visualisations /admin/visualisations ⬜ Phase 17 → Security → Tenants /admin/tenants ⬜ Phase 18 → Security → Roles /admin/roles ⬜ Phase 18 → Security → Permissions /admin/permissions ⬜ Phase 18 → Security → Users /admin/users ⬜ Phase 18 → Performance → HTTP Counters /admin/http-counters ⬜ Phase 18 → Performance → Queue Balances /admin/queues ⬜ Phase 18 → Performance → Model Counters /admin/model-counters ⬜ Phase 18 → Preservation /admin/preservation ⬜ Phase 18 ``` --- ## Legend ``` ⬜ Not Started 🔵 In Progress ✅ Complete 🔴 Blocked ⚠️ At Risk ``` --- ## PHASE 10: Model Configuration Platform **Target:** Weeks 1–3 | **Status:** ✅ Complete ### Goal Give analysts a UI to define, configure, and tune detection models without code changes. Mirrors Jube's `Models → Model` screen including all nested configuration sections. ### Jube Form Fields Being Implemented From `EntityAnalysisModel.cshtml` (full field-by-field parity): **Model Header:** - Name (text), Active (toggle), Locked (toggle) - EntryName (text) — the field name used as the entity identifier - Entry XPath (text) — JSON path to extract entity key from payload - Reference Date Name (text), Reference Date Payload Location (radio: Body/Now), Reference Date XPath (text) **Cache Settings:** - Cache Fetch Limit (number, default 1) - Cache TTL Interval (radio: Seconds/Minutes/Hours/Days) + Value (number, default 1) - Enable Cache (toggle), Enable TTL Counter (toggle), Enable Sanction Cache (toggle) **Response Elevation (nested, shown when enabled):** - Enable Response Elevation Limit (toggle) - Interval (radio: S/M/H/D) + Value (number) + Threshold (number) **Activation Watcher (nested, shown when enabled):** - Enable Activation Watcher (toggle) - Interval (radio: S/M/H/D) + Value (number, default 100) + Threshold (number) - Activation Watcher Sample (number, default 1) **Model Synchronisation UI:** - Node list table: node ID, last heartbeat, sync status, model version - Manual sync trigger button - Sync schedule configuration ### Request XPath UI (`/models/:id/xpaths`) From `EntityAnalysisModelRequestXPath.cshtml`: - Name, Active, Locked - XPath (text) — JSON path expression - Enable Suppression (toggle) - Cache (toggle) - Data Type (select: String/Integer/Float/Date/Boolean/Latitude/Longitude) - Default Value (conditional by data type) - Search Key section (nested): TTL Interval, Fetch Limit, cache sub-settings - Report Table (toggle), Response Payload (toggle) ### TTL Counter UI (`/models/:id/ttl-counters`) From `EntityAnalysisModelTtlCounter.cshtml`: - Name, Active, Locked - Online Aggregation (toggle), Live Forever (toggle) - Data Name (dropdown — links to Request XPath field names) - Sum (toggle, reveals Data Value dropdown) - Interval type (S/M/H/D/Month/Year) + Value (number) - Resolution Interval (Minutes/Hours/Days) - Report Table, Response Payload ### Gateway Rules UI (`/models/:id/gateway-rules`) From `EntityAnalysisModelGatewayRule.cshtml`: - Name, Active, Locked, Priority (number) - Rule (code editor — expression evaluator) - Gateway Sample (number, default 100) - Response Elevation Limit (number) - Counters / Last Counters (read-only) ### New DB Migrations (Phase 10) ``` 20260601000001_add_request_xpaths.exs risk_request_xpaths: id, entity_model_id, name, active, locked, xpath, enable_suppression, cache, data_type, default_value, search_key_enabled, search_key_ttl_interval, search_key_ttl_value, search_key_fetch_limit, report_table, response_payload, inserted_at, updated_at 20260601000002_add_gateway_rules.exs risk_gateway_rules: id, entity_model_id, name, active, locked, priority, rule_expression, gateway_sample, max_response_elevation, counters, inserted_at, updated_at 20260601000003_enhance_entity_models.exs Add to risk_entity_models: entry_name, entry_xpath, reference_date_name, reference_date_location, reference_date_xpath, cache_fetch_limit, cache_ttl_interval, cache_ttl_value, enable_cache, enable_ttl_counter, enable_sanction_cache, enable_response_elevation_limit, elevation_interval, elevation_value, elevation_threshold, enable_activation_watcher, watcher_interval, watcher_value, watcher_threshold, watcher_sample, enable_rdbms_archive, locked, guid 20260601000004_add_configurable_ttl_counters.exs risk_ttl_counters: id, entity_model_id, name, active, locked, online_aggregation, live_forever, data_name, sum_enabled, data_value, ttl_interval, ttl_value, resolution_interval, report_table, response_payload, inserted_at, updated_at ``` ### New LiveViews (Phase 10) ``` gateway_web/live/model_config_live.ex — Model CRUD (index + form) gateway_web/live/model_xpath_live.ex — Request XPath sub-page gateway_web/live/model_ttl_counter_live.ex — TTL Counter sub-page gateway_web/live/model_gateway_rule_live.ex — Gateway Rule sub-page gateway_web/live/model_sync_live.ex — Synchronisation status ``` ### Engine Changes (Phase 10) - `FeatureHydrator`: read `risk_request_xpaths` to extract entity values dynamically - `ScoringPipeline`: add gateway rule evaluation phase before abstraction - `RuleCache`: extend to cache gateway rules + request xpath configs ### Phase 10 Exit Criteria - [ ] Analyst can create/edit/delete a detection model through UI - [ ] Request XPaths can be defined per model via UI - [ ] TTL Counters configurable via UI (not code-only) - [ ] Gateway rules can be created with code expression editor - [ ] FeatureHydrator reads xpath config from DB at runtime (PubSub invalidation) --- ## PHASE 11: Reference Data Platform (Dictionaries, Lists, Tags) **Target:** Weeks 3–5 | **Status:** ✅ Complete ### Goal Allow no-code management of lookup tables, entity lists, and transaction tags. ### Dictionaries (`/dictionaries`) From `EntityAnalysisModelDictionary.cshtml`: - Model selector (top-level dropdown) - Dictionary list panel (left): Name, Active, Locked, Data Name (links to Request XPath fields), Response Payload - Key-Value panel (right): - Add individual KVP: Key (text) + Value (number/text) - Edit/Delete per row - CSV bulk upload (parse key,value columns) - Inline editable list view ### Lists (`/lists`) From `EntityAnalysisModelList.cshtml`: - Model selector - List config: Name, Active, Locked - Values panel: single-column list (add/edit/delete per value) - CSV bulk upload (single-column values) - Used in activation/abstraction rules as `in_list("list_name", value)` ### Tags (`/models/:id/tags`) From `EntityAnalysisModelTag.cshtml` (reference): - Name, Active, Locked - Tag expression (rule-based auto-tagging) - Report Table, Response Payload - Manual tag API: PUT /api/v1/risk/archive/tag ### New DB Migrations (Phase 11) ``` 20260608000001_add_dictionaries.exs risk_dictionaries: id, entity_model_id, name, active, locked, data_name, response_payload, inserted_at, updated_at 20260608000002_add_dictionary_kvps.exs risk_dictionary_kvps: id, dictionary_id, kvp_key, kvp_value, version, inserted_at 20260608000003_add_risk_lists.exs risk_lists: id, entity_model_id, name, active, locked, inserted_at, updated_at 20260608000004_add_risk_list_values.exs risk_list_values: id, list_id, value, version, inserted_at 20260608000005_add_risk_tags.exs risk_tags: id, entity_model_id, name, active, locked, tag_expression, report_table, response_payload, inserted_at, updated_at ``` ### New LiveViews (Phase 11) ``` gateway_web/live/dictionaries_live.ex — Dictionary management with KVP editor gateway_web/live/lists_live.ex — List management with value editor gateway_web/live/model_tags_live.ex — Tag management per model ``` ### Engine Changes (Phase 11) - `AbstractionEngine`: add `dictionary_lookup(name, key)` and `in_list(name, value)` operators - `RuleCache`: cache dictionary + list data (PubSub invalidation on change) - `ScoringPipeline`: pass dictionary/list context into abstraction evaluation ### Phase 11 Exit Criteria - [ ] Dictionaries can be created and populated via UI + CSV upload - [ ] Lists (allowlist/blocklist) can be managed via UI + CSV upload - [ ] `in_list()` and `dictionary_lookup()` functions work in activation rules - [ ] Tags can be defined and manually applied via API --- ## PHASE 12: Suppression Module **Target:** Weeks 5–6 | **Status:** ✅ Complete ### Goal Provide an operational override layer to suppress alerts on known-good entities without touching rules. ### Suppression UI (`/suppression`) From `Suppression.cshtml`: - Suppression Key (dropdown — links to Request XPath field names, e.g., `card`, `account_id`) - Suppression Key Value (text — the specific entity value to suppress) - Fetch button → shows all current suppressions for that key/value in a data grid - Grid columns: Key, Value, Activation Rule Name, Created By, Expires At, Active ### Suppression Management - Add suppression: key + value + optional expiry + reason - Remove suppression - Bulk suppression via CSV upload - Suppression audit log (who suppressed what, when, why) - Auto-expiry: suppression with expiry date auto-lifted ### New DB Migrations (Phase 12) ``` 20260615000001_add_suppressions.exs risk_suppressions: id, tenant_id, entity_model_id, suppression_key, suppression_value, activation_rule_id (nullable — model-level if nil), reason, expires_at, created_by, active, inserted_at, updated_at ``` ### New LiveViews (Phase 12) ``` gateway_web/live/suppression_live.ex — Suppression query + management ``` ### Engine Changes (Phase 12) - `ActivationEngine`: check `SuppressionsCache` before firing an activation rule - New `SuppressionsCache` GenServer: ETS-backed, loaded from DB, PubSub invalidated - `ScoringPipeline`: skip rule evaluation for suppressed entities ### Phase 12 Exit Criteria - [ ] Analyst can suppress a specific card/account without changing rules - [ ] Suppressed entities produce no cases (rules still evaluated but result dropped) - [ ] Suppression with expiry auto-lifts when expired - [ ] Suppression audit log visible in UI --- ## PHASE 13: Case Workflow Engine **Target:** Weeks 6–11 | **Status:** ✅ Complete ### Goal Replace our basic status-change case management with a full configurable workflow platform. ### Case Workflow Templates (`/workflows`) From `CaseWorkflow.cshtml`: - Name, Active, Locked - Enable Visualisation (toggle) → links to a custom dashboard (Phase 17) - Role assignment panel (which roles can access this workflow) ### Workflow Status Configuration (`/workflows/:id/statuses`) From `CaseWorkflowStatus.cshtml`: - Name, Active, Locked - Priority (Ultra High/High/Medium/Low/Ultra Low) - Status Fore Color (color picker), Status Back Color (color picker) - HTTP Endpoint (toggle): POST/GET URL called when status entered - Notification (toggle): Email/SMS with destination, subject, body template - Role assignment (which roles can move to this status) ### Workflow Actions (`/workflows/:id/actions`) From `CaseWorkflowAction.cshtml`: - Name, Active, Locked - HTTP Endpoint: Type (POST/GET) + URL - Notification: Type (Email/SMS) + Destination + Subject + Body - Role assignment (which roles can execute this action) ### Workflow Forms (`/workflows/:id/forms`) From `CaseWorkflowForm.cshtml`: - Name, Active, Locked - Form HTML (code editor with live preview) — custom HTML form stored with case - HTTP Endpoint (POST/GET) — where submitted form data is POSTed - Notification on submit - Role assignment ### Workflow Displays (`/workflows/:id/displays`) - Define which payload fields to display per workflow + role - XPath selectors to extract display values from case payload ### Workflow Filters (`/workflows/:id/filters`) - Named SQL-based search templates for investigator queues - Role-based filter access ### Workflow Macros (`/workflows/:id/macros`) - Named executable actions (e.g., "escalate to AML team") - Expression or HTTP call executed in case context ### Workflow XPaths (`/workflows/:id/xpaths`) - JSON path selectors to extract values from case payload for display ### Case Management Enhancements (extend `CaseManagementLive`) - Workflow-driven status panel (show statuses for assigned workflow) - Case diary: scheduled follow-up entries - Case file attachments: upload/download/version - Dynamic form submission per case - Macro execution buttons - Case action history with actor + timestamp ### New DB Migrations (Phase 13) ``` 20260622000001_add_case_workflows.exs risk_case_workflows: id, tenant_id, name, active, locked, enable_visualisation, visualisation_id (nullable), inserted_at, updated_at 20260622000002_add_case_workflow_statuses.exs risk_case_workflow_statuses: id, workflow_id, name, active, locked, priority, fore_color, back_color, enable_http_endpoint, http_endpoint_type, http_endpoint, enable_notification, notification_type, notification_destination, notification_subject, notification_body, inserted_at, updated_at 20260622000003_add_case_workflow_actions.exs risk_case_workflow_actions: id, workflow_id, name, active, locked, enable_http_endpoint, http_endpoint_type, http_endpoint, enable_notification, notification_type, notification_destination, notification_subject, notification_body, inserted_at, updated_at 20260622000004_add_case_workflow_forms.exs risk_case_workflow_forms: id, workflow_id, name, active, locked, form_html, enable_http_endpoint, http_endpoint_type, http_endpoint, enable_notification, notification_type, notification_destination, notification_subject, notification_body, inserted_at, updated_at 20260622000005_add_case_workflow_filters.exs risk_case_workflow_filters: id, workflow_id, name, active, locked, filter_sql, inserted_at, updated_at 20260622000006_add_case_workflow_macros.exs risk_case_workflow_macros: id, workflow_id, name, active, locked, macro_expression, http_endpoint, inserted_at, updated_at 20260622000007_add_case_workflow_xpaths.exs risk_case_workflow_xpaths: id, workflow_id, name, xpath, active, inserted_at, updated_at 20260622000008_add_case_workflow_roles.exs risk_case_workflow_roles: id, workflow_id, role_name, permission_type (access/action/form/filter/display/macro/xpath), inserted_at 20260622000009_add_case_files.exs risk_case_files: id, case_id, filename, content_type, file_data (binary), version, uploaded_by, inserted_at 20260622000010_add_case_diary.exs risk_case_diary: id, case_id, due_at, subject, body, assigned_to, completed_at, created_by, inserted_at 20260622000011_add_case_form_entries.exs risk_case_form_entries: id, case_id, form_id, submitted_by, form_data (json), submitted_at 20260622000012_enhance_risk_cases.exs Add to risk_cases: workflow_id (FK), workflow_status_id (FK) ``` ### New LiveViews (Phase 13) ``` gateway_web/live/workflow_config_live.ex — Workflow template CRUD gateway_web/live/workflow_status_live.ex — Status configuration gateway_web/live/workflow_action_live.ex — Action configuration gateway_web/live/workflow_form_live.ex — Form builder with HTML preview gateway_web/live/workflow_filter_live.ex — Filter template editor gateway_web/live/workflow_macro_live.ex — Macro editor gateway_web/live/workflow_xpath_live.ex — XPath editor ``` Enhance `CaseManagementLive`: - Workflow status panel with colored status chips - File attachment upload/download - Diary entries panel - Dynamic form rendering + submission - Macro execution buttons - Action execution with confirmation modal ### Phase 13 Exit Criteria - [ ] Analyst can create a workflow with custom statuses (with colors) and transitions - [ ] Status change fires HTTP endpoint and/or email notification - [ ] Case forms can be defined and submitted per case - [ ] File attachments can be uploaded and downloaded from cases - [ ] Workflow-driven queues via filter templates - [ ] Role-based access per workflow enforced --- ## PHASE 14: Abstraction Layer Enhancement **Target:** Weeks 11–13 | **Status:** ✅ Complete ### Goal Separate abstraction rules (derived feature computation) from activation rules (alert triggers). Add the full set of aggregation functions Jube provides. ### Abstraction Rules UI (`/models/:id/abstractions`) From `EntityAnalysisModelAbstractionRule.cshtml`: - Name, Active, Locked - Rule (code editor — expression that computes a derived value) - Search section (toggle): - Search Key (dropdown — links to Request XPath defined keys) - Search Value (text, default "1") - Search Interval (radio: S/M/H/D) + value - Function (dropdown): - Count (1), Distinct Count (2), Sum (3), Average (4), Median (5) - Kurtosis (6), Skew (7), Standard Deviation (8), Mode (11) - Same Count (12), Actual Value (13), Max (14), Min (15), Since (16) - Function Key (conditional dropdown — by data type: string/float/date) - Offset (toggle): Offset Type (First/Last/Skip First/Take Last) + Value - Report Table (toggle), Response Payload (toggle) ### Abstraction Calculation UI (`/models/:id/calculations`) - Calculation Name - Ordered list of abstraction rules to evaluate in sequence - Dependency graph visualization (DAG) - Circular dependency detection ### New DB Migrations (Phase 14) ``` 20260629000001_add_abstraction_rules.exs risk_abstraction_rules: id, entity_model_id, name, active, locked, rule_expression, search_enabled, search_key, search_value, search_interval_type, search_interval_value, function_type, function_key, offset_enabled, offset_type, offset_value, report_table, response_payload, inserted_at, updated_at 20260629000002_add_abstraction_calculations.exs risk_abstraction_calculations: id, entity_model_id, name, active, calculation_order (jsonb array of abstraction_rule_ids), inserted_at, updated_at ``` ### Engine Changes (Phase 14) - `AbstractionEngine`: read abstraction rules from DB, execute in calculation order - Add aggregation functions: Distinct Count (HLL), Sum, Average, Median, Kurtosis, Mode, Skew, Max, Min, Since - `RuleCache`: cache abstraction rules separately from activation rules ### Phase 14 Exit Criteria - [ ] Abstraction rules can be created via UI with all 16 aggregation functions - [ ] Calculation order can be defined (DAG with cycle detection) - [ ] Abstraction results flow into activation rule expressions - [ ] Distinct Count uses HLL from FeatureStore --- ## PHASE 15: ML Experiment Management **Target:** Weeks 13–16 | **Status:** ✅ Complete ### Goal Full ML experiment lifecycle: create, run, compare trials, view metrics, promote to production. ### Experiment List UI (`/ml/experiments`) - Table: Experiment ID, Model, Status, Started At, Best AUC, Trial Count, Action - Create new experiment (model selector + training config) - Click row → experiment detail ### Experiment Detail UI (`/ml/experiments/:id`) From `Jube.Engine/Exhaustive/` patterns: - Summary panel: status badge, total trials, best trial, training data size, elapsed time - Trial comparison table: topology (layers), AUC-ROC, accuracy, precision, recall, F1, training time - Promote button on best trial - Variable importance panel: - Bar chart of feature permutation importance scores - Feature name + importance + direction (positive/negative correlation) - ROC curve chart (plot of TPR vs FPR at various thresholds) - Predicted vs Actual scatter/confusion matrix - Training history panel: loss/accuracy curves by epoch ### Experiment Configuration - Training data source: risk_labels (select label_type, date range) - Feature set selection (from FeatureRegistry) - Topology search: enable/disable exhaustive search vs manual topology - Max trials, max epochs, early stopping threshold - Train/test split ratio ### Model Version Management (extend existing) - Model version history table - Rollback to previous version - A/B testing weight (% of traffic to new model) - Champion/challenger configuration ### New DB Migrations (Phase 15) ``` 20260706000001_add_ml_experiments.exs risk_ml_experiments: id, tenant_id, name, entity_model_id, label_type, date_from, date_to, feature_set (jsonb), max_trials, max_epochs, train_test_split, status, best_trial_id, started_at, completed_at, inserted_at 20260706000002_add_ml_trials.exs risk_ml_trials: id, experiment_id, topology (jsonb), auc_roc, accuracy, precision_score, recall, f1_score, training_time_ms, training_size, is_promoted, promoted_at, inserted_at 20260706000003_add_ml_trial_variables.exs risk_ml_trial_variables: id, trial_id, feature_name, importance_score, direction, inserted_at 20260706000004_add_ml_trial_roc.exs risk_ml_trial_roc: id, trial_id, threshold, tpr, fpr, inserted_at ``` ### New LiveViews (Phase 15) ``` gateway_web/live/ml_experiment_live.ex — Experiment list + create gateway_web/live/ml_experiment_detail_live.ex — Trial comparison, charts, promote ``` ### Phase 15 Exit Criteria - [ ] Analyst can create a training experiment with label/feature config - [ ] Multiple topology trials run and results tracked per experiment - [ ] Variable importance bar chart rendered in UI - [ ] ROC curve rendered for each trial - [ ] Best trial can be promoted to production via UI - [ ] Model version history and rollback available --- ## PHASE 16: Activation Watcher (Real-time Alert Visualization) **Target:** Weeks 16–18 | **Status:** ✅ Complete ### Goal Real-time visualization of where and at what intensity alerts are firing, with replay capability. Jube equivalent: `Watcher → Activation`. ### Watcher UI (`/watcher`) From `Activation.cshtml`: **Controls:** - Realtime toggle (on/off — start/stop live feed) - Map Entity Once toggle (deduplicate by entity on map) - Replay From / Replay To (datetime range pickers) + Replay button **Visualization panels:** - Map panel (geographic): plotted points for each activation with color-coded elevation - Point color = Response Elevation Fore/Back Color from activation rule config - Hover: entity ID, rule name, score, timestamp - Chart panel: time-series bar/line chart of activation count + elevation distribution - Grid panel: recent activations table (Rule, Entity, Score, Elevation, Time, Status) - Status panel: current TPS, active rules, suppressed count, case creation rate **Real-time feed:** - Phoenix PubSub subscription to `"risk:watcher:#{tenant_id}"` topic - Backend: `PersistToActivationWatcher` - publishes activation events after scoring - Sampling: `watcher_sample` from model config (1-in-N activations shown) ### Backend: ActivationWatcher Storage - New `risk_activation_watcher` table for persisted watcher events - `WatcherBroadcaster` module: publishes to PubSub + inserts sampled watcher rows - Replay: query `risk_activation_watcher` by date range and replay via PubSub ### New DB Migrations (Phase 16) ``` 20260713000001_add_activation_watcher.exs risk_activation_watcher: id, tenant_id, entity_model_id, activation_rule_id, entity_value, entity_key, response_elevation, fore_color, back_color, latitude (nullable), longitude (nullable), payload_snapshot (jsonb, sampled), scored_at, inserted_at ``` ### New LiveViews (Phase 16) ``` gateway_web/live/activation_watcher_live.ex — Realtime watcher with map+chart+grid ``` ### Engine Changes (Phase 16) - `ActivationEngine`: after firing a rule, publish watcher event if `enable_activation_watcher` set on model - `EventBroadcaster`: extend to publish `"risk:watcher:#{tenant_id}"` events - Watcher sampler: only publish 1-in-N events per model's `watcher_sample` config ### Phase 16 Exit Criteria - [x] Realtime feed shows activations as they fire (< 1s latency) - [x] Activation points plotted on geo scatter SVG when lat/lon available in payload - [x] Activity bar chart updates live as activations arrive (per-minute buckets, 30-min window) - [x] Replay loads historical activations for a date range via datetime-local inputs - [x] Sampling rate respects model's `watcher_sample` config (WatcherBroadcaster) --- ## PHASE 17: Custom Visualization / Dashboard Builder **Target:** Weeks 18–21 | **Status:** ✅ Complete ### Goal Allow analysts to define custom dashboards from SQL datasources without code deployment. Jube equivalent: `Visualisation` menu + `Administration → Visualisations`. ### Visualisation Registry (`/admin/visualisations`) - Dashboard list: Name, Description, Active, Roles, Data Sources, Created - Create/Edit dashboard: - Name, Description, Active - Role assignment (which roles can view) ### Visualisation Datasources (`/admin/visualisations/:id/datasources`) From `VisualisationRegistryDatasource.cshtml`: - Name, Active - SQL Query (code editor with syntax highlighting) - Column mapping: auto-detected from query - Cache TTL (seconds) - Chart type (table / bar / line / pie / scatter) - X-axis column, Y-axis column(s), Series column ### Visualisation Parameters (`/admin/visualisations/:id/parameters`) From `VisualisationRegistryParameter.cshtml`: - Parameter Name (maps to SQL named param e.g. `@tenant_id`) - Data Type (String/Integer/Date) - Default Value - Required (toggle) - Role restrictions (which roles can set this param) ### Dashboard Viewer (`/visualisations`) From `Visualisation.cshtml`: - Model selector (which dashboard to view) - Dynamic parameter form (renders based on parameter config) - Charts rendered per datasource (table/bar/line/pie) - Refresh button, auto-refresh interval selector ### New DB Migrations (Phase 17) ``` 20260720000001_add_visualisation_registry.exs risk_visualisation_registry: id, tenant_id, name, description, active, inserted_at, updated_at 20260720000002_add_visualisation_datasources.exs risk_visualisation_datasources: id, visualisation_id, name, active, sql_query, chart_type, x_axis_column, y_axis_columns (jsonb), cache_ttl_seconds, inserted_at, updated_at 20260720000003_add_visualisation_parameters.exs risk_visualisation_parameters: id, visualisation_id, name, data_type, default_value, required, inserted_at, updated_at 20260720000004_add_visualisation_roles.exs risk_visualisation_roles: id, visualisation_id, role_name, inserted_at ``` ### New LiveViews (Phase 17) ``` gateway_web/live/visualisation_registry_live.ex — Dashboard CRUD gateway_web/live/visualisation_datasource_live.ex — Datasource + param editor gateway_web/live/visualisation_viewer_live.ex — Dashboard viewer (dynamic render) ``` ### Phase 17 Exit Criteria - [x] Analyst can create a custom dashboard from a SQL query - [x] Charts render from query results (table / bar / bar_horizontal / line / pie / scatter) — server-side SVG, no JS deps - [x] Parameters allow filtering dashboard by date range / entity (`@param_name` → positional binding) - [x] Role-based access: roles assigned per visualisation; `VisualisationRegistryLive` enforces via `set_roles/2` - [x] QueryExecutor: SELECT-only enforcement, 10k row cap, 10s timeout, forbidden keyword rejection - [x] Auto-refresh via configurable `cache_ttl_seconds` with `:timer.send_interval` heartbeat --- ## PHASE 18: Administration Platform **Target:** Weeks 21–24 | **Status:** ✅ Complete ### Goal Full admin back-office: multi-tenancy, role/permission management, user management, platform monitoring. ### Tenant Registry (`/admin/tenants`) - Tenant list: ID, Name, Active, Created, User Count - Create/Edit tenant: Name, Active, Config (JSON blob for per-tenant settings) - Deactivate tenant (soft delete) ### Role Registry (`/admin/roles`) - Role list: Name, Permissions, User Count, Active - Create role: Name, Active - Assign permissions to role (multi-select from permission catalog) ### Permission Specification (`/admin/permissions`) - Permission catalog: Name, Category, Description - Seed: fraud_read, fraud_write, fraud_admin, compliance_read, compliance_export, sanctions_manage, ml_train, ml_promote, admin_full ### User Registry (`/admin/users`) - User list: Email, Name, Tenant(s), Roles, Last Login, Active - Create/Edit user: email, name, tenant assignment(s), role assignment(s) - Reset password, deactivate user - Login audit log: timestamp, IP, success/failure ### Platform Monitoring UI **HTTP Processing Counters (`/admin/http-counters`):** - Table: endpoint, req/s, p50/p95/p99 latency, error rate - Auto-refresh every 10s **Queue Balance (`/admin/queues`):** - Oban queue table: queue name, available/executing/retryable/discarded job counts - Per-queue trend sparkline - Drain button (pause queue) **Model Processing Counters (`/admin/model-counters`):** - Per-model: invocations/s, avg latency, rule hits, ML scores, cases created ### Data Preservation (`/admin/preservation`) - Archive policy config: retain risk_scores for N days, risk_labels for N days - Manual export trigger (extends compliance export API) - Archive size metrics ### New DB Migrations (Phase 18) ``` 20260727000001_add_tenant_registry.exs risk_tenant_registry: id, name, active, config (jsonb), inserted_at, updated_at 20260727000002_add_role_registry.exs risk_role_registry: id, tenant_id, name, active, inserted_at, updated_at 20260727000003_add_permission_spec.exs risk_permission_spec: id, name, category, description, inserted_at 20260727000004_add_role_permissions.exs risk_role_permissions: id, role_id, permission_id, inserted_at 20260727000005_add_user_roles.exs risk_user_roles: id, user_id (FK to users), role_id, tenant_id, inserted_at 20260727000006_add_user_login_audit.exs risk_user_login_audit: id, user_id, ip_address, user_agent, success, failure_reason, logged_at ``` ### New LiveViews (Phase 18) ``` gateway_web/live/admin/tenant_registry_live.ex gateway_web/live/admin/role_registry_live.ex gateway_web/live/admin/permission_spec_live.ex gateway_web/live/admin/user_registry_live.ex gateway_web/live/admin/http_counter_live.ex gateway_web/live/admin/queue_balance_live.ex gateway_web/live/admin/model_counter_live.ex gateway_web/live/admin/preservation_live.ex ``` ### Phase 18 Delivered (vs plan — overlap-adjusted) > Pre-implementation audit found: TenantsLive already fully implemented; no new tenant migration needed. > Hard-coded role list expanded (`AdminUser.@roles`) rather than DB-driven roles table (sufficient for MVP). - [x] `UserRegistryLive` — create/edit/deactivate admin users, temp-password generation, role + status badges - [x] `QueueBalanceLive` — Oban queue depth per state (available/executing/scheduled/retryable/discarded), bar depth indicator, 10s auto-refresh - [x] `ModelCounterLive` — per-model approve/decline/review rates + avg score from `risk_scores`, 30s auto-refresh - [x] `PreservationLive` — per-table retention policy CRUD, row-count estimates, manual purge with confirmation - [x] `LoginAuditLive` — searchable login history with failures-only filter - [x] `SessionController` — records every login attempt (success + failure) with IP + user-agent - [x] `InfraRepo.Admin.Accounts` — full context: user CRUD, temp password gen, login audit, policy upsert/purge - [x] Migrations: `risk_login_audit`, `risk_preservation_policy` - [x] `AdminUser.@roles` expanded to include `fraud_admin`, `fraud_analyst`, `compliance`, `ml_trainer` - [x] Admin sidebar: new "Administration" section with 5 nav items + icons --- ## PHASE 19: HTTP Adaptation + Inline Functions **Target:** Weeks 24–27 | **Status:** 🔵 In Progress ### Goal Extend the scoring pipeline with two no-code extensibility mechanisms: 1. **HTTP Adaptation** — call external APIs during scoring to enrich the feature set (device fingerprint, geo-IP, KYC, velocity from a partner service) 2. **Inline Functions** — define named reusable expressions once and reference them across all activation and abstraction rules 3. **Inline Scripts** — optional sandboxed expression evaluation for complex custom logic that cannot be expressed in the rule DSL Jube equivalents: `Models → References → Inline Functions`, `Models → References → Inline Scripts`, `Models → Machine Learning → HTTP Adaptation`. --- ### 19.1 HTTP Adaptation UI (`/fraud/model-config/:model_id/adaptations`) From `EntityAnalysisModelHttpAdaptation.cshtml`: **Adaptation config fields:** - Name (text), Active (toggle), Locked (toggle) - HTTP Method (radio: GET / POST) - URL Template (text) — supports `@xpath_field_name` token substitution from request XPaths - Request Body Template (textarea, JSON) — same `@param` substitution; shown only for POST - Response Content Type (select: JSON / XML / Plain) - Response Latency Limit ms (number, default 3000) — abort if exceeded - Response Limit (number, default 100KB) — reject oversized responses - Authentication (select: None / Bearer Token / Basic) - Bearer: Token field (text, stored encrypted) - Basic: Username + Password (stored encrypted) - Enable Caching (toggle): - Cache Key Expression (text — which request fields determine cache identity) - Cache TTL Seconds (number, default 300) - Enable Failure Fallback (toggle) — continue scoring with no enrichment on timeout/error **Response Mapping:** - Table of response-field → feature-name mappings - Each row: Response XPath (JSON path into response body), Feature Name (injected into scoring context), Data Type (String/Integer/Float/Boolean) - Add/delete rows inline **Test panel:** - Sample Payload (textarea — paste a JSON transaction payload) - "Test Now" button → live call to the configured URL + shows raw response + mapped values --- ### 19.2 Inline Functions UI (`/fraud/model-config/:model_id/functions`) From `EntityAnalysisModelInlineFunction.cshtml`: **Function config fields:** - Name (text, must match `^[a-zA-Z_][a-zA-Z0-9_]*$`) — referenced as `fn_name()` in rules - Active (toggle), Locked (toggle) - Return Type (select: String / Integer / Float / Boolean / Date) - Function Body (code editor — same expression DSL as activation rules) - Can reference all XPath fields, TTL counters, abstraction results - Cannot call other inline functions (no recursion) - Description (textarea, optional) — displayed in rule editor autocomplete tooltip **Usage tracking:** - Read-only panel showing which activation rules and abstraction rules reference this function - "Where used" list with links to the relevant rule --- ### 19.3 Inline Scripts UI (`/fraud/model-config/:model_id/scripts`) From `EntityAnalysisModelInlineScript.cshtml`: **Script config fields:** - Name (text), Active (toggle), Locked (toggle) - Return Type (select: String / Integer / Float / Boolean) - Script Body (code editor — Elixir expression evaluated in a restricted sandbox) - Allowed: arithmetic, string ops, list ops, `if/case`, pattern matching - Blocked: IO, Process, System, File, :os, :erlang.apply, any atom starting with `Elixir.` - Input Parameter declarations (table): Parameter Name, Data Type, Source XPath - Timeout ms (number, default 50) — abort script if it exceeds wall time - Test panel: inject test values, run script, show output --- ### 19.4 Scoring Pipeline Integration **HTTP Adaptation phase** (new stage between feature hydration and abstraction): ``` FeatureHydrator → [HTTP Adaptation Phase] → AbstractionEngine → ActivationEngine → ... ``` - `MwRisk.HttpAdaptationEngine` — new module - Loads adaptation configs from DB via `AdaptationCache` (ETS, PubSub invalidated) - For each active adaptation: substitute XPath values into URL/body template, call HTTP endpoint - Maps response fields into scoring context under configured feature names - Enforces latency + response size limits; on failure logs and continues (or halts if `enable_failure_fallback = false`) - Caches responses in ETS keyed by `{adaptation_id, cache_key_value}` with TTL **Inline Functions** (evaluated lazily during rule expression parsing): - `MwRisk.FunctionRegistry` — ETS-backed store of compiled function expressions per model - Rule DSL extended: when parser encounters `fn_name(...)`, looks up in `FunctionRegistry` and inlines the expression - PubSub invalidation on function change (same pattern as `RuleCache`) **Inline Scripts** (called explicitly in rule expressions as `script("script_name", arg1, arg2)`): - `MwRisk.ScriptSandbox` — wraps `Code.eval_quoted/3` with AST whitelist validation - Pre-compiles script AST on load; rejects forbidden nodes at compile time - Enforces wall-time timeout via `Task.yield/2` with hard kill on expiry --- ### New DB Migrations (Phase 19) ``` 20260803000001_add_http_adaptations.exs risk_http_adaptations: id, entity_model_id, name, active, locked, http_method, url_template, body_template, response_content_type, response_latency_limit_ms (default 3000), response_limit_bytes (default 102400), auth_type, auth_token_enc, auth_username, auth_password_enc, enable_caching, cache_key_expression, cache_ttl_seconds, enable_failure_fallback, inserted_at, updated_at 20260803000002_add_http_adaptation_mappings.exs risk_http_adaptation_mappings: id, adaptation_id (FK → risk_http_adaptations, delete_all), response_xpath, feature_name, data_type, inserted_at 20260803000003_add_inline_functions.exs risk_inline_functions: id, entity_model_id, name, active, locked, return_type, function_body, description, inserted_at, updated_at 20260803000004_add_inline_scripts.exs risk_inline_scripts: id, entity_model_id, name, active, locked, return_type, script_body, timeout_ms (default 50), inserted_at, updated_at 20260803000005_add_inline_script_params.exs risk_inline_script_params: id, script_id (FK → risk_inline_scripts, delete_all), param_name, data_type, source_xpath, inserted_at ``` --- ### New LiveViews (Phase 19) ``` gateway_web/live/model_adaptation_live.ex — HTTP Adaptation CRUD + test panel gateway_web/live/model_adaptation_live.html.heex gateway_web/live/model_function_live.ex — Inline Function CRUD + usage panel gateway_web/live/model_function_live.html.heex gateway_web/live/model_script_live.ex — Inline Script CRUD + test panel gateway_web/live/model_script_live.html.heex ``` Routes (add to `:fraud` live_session, under model-config group): ```elixir live "/fraud/model-config/:model_id/adaptations", ModelAdaptationLive, :index live "/fraud/model-config/:model_id/functions", ModelFunctionLive, :index live "/fraud/model-config/:model_id/scripts", ModelScriptLive, :index ``` Nav: add three items to the "Model Config" sub-group in the fraud sidebar — "HTTP Adapt", "Functions", "Scripts". --- ### New Engine Modules (Phase 19) ``` apps/mw_risk/lib/mw_risk/http_adaptation_engine.ex — adaptation phase runner apps/mw_risk/lib/mw_risk/adaptation_cache.ex — ETS cache for adaptation configs apps/mw_risk/lib/mw_risk/function_registry.ex — compiled inline function store apps/mw_risk/lib/mw_risk/script_sandbox.ex — sandboxed AST evaluator ``` `ScoringPipeline` changes: - Add `HttpAdaptationEngine.enrich/2` call after `FeatureHydrator.hydrate/2` - Wire `FunctionRegistry` into rule expression evaluator - Expose `ScriptSandbox.call/3` as a callable from rule expressions `InfraRepo.Risk.Adaptations` — new context: - CRUD for adaptations, mappings, functions, scripts, script params - `list_active_adaptations/1`, `list_active_functions/1`, `list_active_scripts/1` --- ### New InfraRepo Schemas (Phase 19) ``` infra_repo/lib/infra_repo/schemas/risk_http_adaptation.ex infra_repo/lib/infra_repo/schemas/risk_http_adaptation_mapping.ex infra_repo/lib/infra_repo/schemas/risk_inline_function.ex infra_repo/lib/infra_repo/schemas/risk_inline_script.ex infra_repo/lib/infra_repo/schemas/risk_inline_script_param.ex ``` --- ### Security Considerations - Auth tokens + passwords stored AES-256-GCM encrypted (extend `FeatureSnapshotCrypto` pattern); decrypted at load time into ETS only - `ScriptSandbox` AST whitelist validated at compile time — any forbidden node raises `{:error, :forbidden_ast}`; no dynamic `Code.eval_string/1` (only `Code.eval_quoted/3` on pre-parsed + validated AST) - `HttpAdaptationEngine` calls made from `Task.async` with `Task.yield/2` — never blocks the scoring GenServer - Response body size enforced before parsing to prevent memory exhaustion --- ### Phase 19 Exit Criteria - [ ] HTTP Adaptation can be configured with URL template, response mapping, and auth - [ ] External enrichment values injected into scoring context and visible in `feature_snapshot` - [ ] Adaptation response cached in ETS with configurable TTL - [ ] Inline Function referenced in activation rule expression evaluates correctly - [ ] Inline Script rejects forbidden AST nodes at compile time (IO, Process, System, File) - [ ] Script timeout enforced — script exceeding `timeout_ms` returns `{:error, :timeout}` - [ ] Test panel in UI shows live adaptation call result + mapped values - [ ] `ScoringPipeline` passes with new enrichment stage: no regression in P95 latency for models without adaptations --- ## Summary Timeline ``` Week 1–3 │ Phase 10: Model Configuration Platform (Model CRUD, XPath, TTL, Gateway) Week 3–5 │ Phase 11: Reference Data (Dictionaries, Lists, Tags) Week 5–6 │ Phase 12: Suppression Module Week 6–11 │ Phase 13: Case Workflow Engine (largest phase) Week 11–13 │ Phase 14: Abstraction Layer Enhancement Week 13–16 │ Phase 15: ML Experiment Management Week 16–18 │ Phase 16: Activation Watcher Week 18–21 │ Phase 17: Custom Visualization Builder Week 21–24 │ Phase 18: Administration Platform Week 24–27 │ Phase 19: HTTP Adaptation + Inline Functions ``` **Total new DB tables:** ~40 migrations **Total new LiveViews:** ~35 pages **Total engine changes:** ~12 modules --- ## UI Design Principles All new pages follow our existing stack — Jube's **content and structure** (form fields, labels, nested sections) with our **modern Tailwind/LiveView UI**: | Jube Pattern | Our Implementation | |---|---| | Form with nested reveal tables | LiveView `phx-change` conditional sections | | Kendo UI Grid | AG Grid (already integrated) or `<.table>` component | | Kendo ListView | Custom LiveView list component with inline edit | | Kendo TreeView (left nav) | Sidebar with `phx-navigate` links | | Color pickers | `` native | | Code editors (rules) | CodeMirror / Monaco editor via JS hook | | Toggle switches | Tailwind custom toggle component | | CSV upload | LiveView `allow_upload` + `consume_uploaded_entries` | | Role allocation panel | Multi-select checkboxes with role badge chips | | Real-time map | Leaflet.js via JS hook (lat/lon from payload) | | Charts (Kendo) | Chart.js via JS hook | --- ## Phase Tracker | Phase | Name | Weeks | Status | Exit Criteria Met | |-------|------|-------|--------|-------------------| | 10 | Model Configuration Platform | 1–3 | ✅ | [x] Model CRUD UI [x] XPath config [x] TTL counter UI [x] Gateway rules | | 11 | Reference Data Platform | 3–5 | ✅ | [x] Dictionaries [x] Lists [x] Tags | | 12 | Suppression Module | 5–6 | ✅ | [x] Suppression UI [x] Engine integration | | 13 | Case Workflow Engine | 6–11 | ✅ | [x] Workflows [x] Statuses [x] Forms [x] Actions [x] Files [x] Diary | | 14 | Abstraction Layer | 11–13 | ✅ | [x] Abstraction rules UI [x] 16 functions [x] Calculations | | 15 | ML Experiment Management | 13–16 | ✅ | [x] Experiments [x] Trials [x] ROC [x] Promote | | 16 | Activation Watcher | 16–18 | ✅ | [x] Realtime feed [x] Geo scatter SVG [x] Activity chart [x] Replay | | 17 | Custom Visualization | 18–21 | ✅ | [x] Dashboard registry [x] SQL datasource [x] All 6 chart types [x] QueryExecutor [x] Role assignment | | 18 | Administration Platform | 21–24 | ✅ | [x] User registry [x] Queue balance [x] Model counters [x] Preservation [x] Login audit | | 19 | HTTP Adaptation + Scripts | 24–27 | 🔵 | [ ] HTTP Adaptation [ ] Inline Functions [ ] Inline Scripts [ ] Pipeline integration |