# ML Pipeline: Training, Inference, and Model Lifecycle

Version: 1.0 | Date: 2026-05-21

---

## 1. ML Approach (mirrors Jube's Exhaustive Adaptation)

Jube uses a combination of:
  - Unsupervised: Isolation Forest (anomaly detection, no labels needed)
  - Supervised:   Neural network (trained on labeled fraud/legit transactions)
  - Hybrid:       Ensemble of both (weighted by performance)

mw-core equivalent:
  - Unsupervised: Scholar.Preprocessing.IsolationForest (Elixir, no Python)
  - Supervised:   Axon (MLP neural network, trained on risk_scores with labels)
  - Hybrid:       Ensemble score = α * ml_score + (1-α) * anomaly_score

---

## 2. Model Training Pipeline

Trigger: Oban job MwRisk.ModelTrainer, scheduled or manual from UI.

Step 1: Feature Extraction
  Query: SELECT feature_snapshot, label FROM risk_scores
         WHERE tenant_id = ? AND labeled_at IS NOT NULL
         AND created_at BETWEEN ? AND ?
  Result: DataFrame (Explorer) → feature matrix + label vector

Step 2: Feature Engineering
  For each feature in feature_set:
    - Clip outliers at 99th percentile
    - Normalize: (value - mean) / std_dev
    - Handle missing: fill with 0.0
  Split: 70% train, 15% validation, 15% test

Step 3: Train Isolation Forest (Scholar)
  model = Scholar.Preprocessing.IsolationForest.fit(x_train,
    n_estimators: 100,
    contamination: 0.05  # expected fraud rate
  )
  anomaly_scores = Scholar.Preprocessing.IsolationForest.predict(model, x_test)

Step 4: Train MLP (Axon, if labeled data available >= 1000 samples)
  model = Axon.input("features", shape: {nil, n_features})
    |> Axon.dense(64, activation: :relu)
    |> Axon.dropout(rate: 0.3)
    |> Axon.dense(32, activation: :relu)
    |> Axon.dense(1, activation: :sigmoid)

  {trained_model, params} = Axon.Loop.trainer(model, :binary_cross_entropy,
    Axon.Optimizers.adam(0.001))
    |> Axon.Loop.metric(:accuracy)
    |> Axon.Loop.run(train_data, epochs: 50, compiler: EXLA)

Step 5: Evaluate & Record Metrics
  AUC-ROC, precision, recall, F1, K-S statistic
  Persist to risk_model_versions with status: "trained"

Step 6: "Exhaustive Adaptation" (Jube concept)
  Try multiple configurations:
    - Different hidden layer sizes [32, 64, 128]
    - Different feature subsets (forward selection)
    - Different contamination rates for Isolation Forest
  Select best by AUC-ROC on validation set.
  Persist winner, retire others.

---

## 3. Model Serving (Nx.Serving)

Nx.Serving batches concurrent scoring requests into single tensor operations.
Started as supervised process in mw_risk.Application.

In application.ex supervision tree:
  {Nx.Serving,
   name: MwRisk.ModelServer,
   serving: loaded_serving,
   batch_size: 32,
   batch_timeout: 5}  # 5ms batching window

Loading a deployed model:
  1. Query risk_model_versions WHERE status = "deployed" AND tenant_id = ?
  2. Load serialized params from artifact_path
  3. Reconstruct Axon model definition from stored hyperparameters
  4. Build Nx.Serving with EXLA backend

Hot-swap on new deployment:
  MwRisk.ModelServer.update_serving(tenant_id, new_serving)
  → no downtime, atomic swap via GenServer state update

---

## 4. Feature Vector Construction

The feature vector must exactly match the feature_set recorded in risk_model_versions.
This is the "dual-write paradox" Jube/Feast documentation warns about.

At scoring time:
  1. features = FeatureHydrator.hydrate(ctx.tenant_id, tx)
  2. feature_vector = build_vector(features, model.feature_set)
     For each feature_key in model.feature_set (in order):
       value = Map.get(features, feature_key, 0.0)
       normalized = (value - model.feature_means[key]) / model.feature_stds[key]
  3. tensor = Nx.tensor([feature_vector])

The feature_set, feature_means, and feature_stds are stored in risk_model_versions.metrics.

---

## 5. Isolation Forest (Unsupervised) Use Cases

Useful before labeled data is available (cold start):
  - New merchant onboarding: no fraud history yet
  - New card token: no velocity history yet
  - Concept drift detection: score distribution shifts

Scholar.Preprocessing.IsolationForest.predict returns anomaly_score ∈ [-1, 1].
Normalize to [0, 1]: (1 - anomaly_score) / 2.

---

## 6. Delayed Label Ingestion (TC40 / Chargeback)

TC40 and chargeback notifications arrive 30–90 days after transaction.
These are the ground truth labels used to retrain models and improve rules.

Oban Worker: MwRisk.LabelIngestionWorker

Step 1: Receive notification (via API or file ingestion):
  POST /api/v1/risk/labels (gateway_api)
  or file upload (adapter_file SFTP polling)
  → creates risk_labels record with status: "unmatched"

Step 2: Match label to original transaction (Oban worker, runs every hour):
  SELECT rs.id FROM risk_scores rs
  WHERE rs.transaction_ref = rl.transaction_ref
    AND rs.tenant_id = rl.tenant_id
  UPDATE risk_scores SET label = ?, labeled_at = NOW() WHERE id = ?
  UPDATE risk_labels SET matched_score_id = ?, matched_at = NOW() WHERE id = ?

Step 3: Trigger model retraining if new label count > threshold:
  Oban.insert(%MwRisk.ModelTrainer{tenant_id: tenant_id, trigger: "new_labels"})

Step 4: Update velocity counter for chargebacks:
  infra_feature_store.TtlCounter.increment(
    tenant_id, :merchant, merchant_id, "1d", :count_chargeback, 1
  )