# Abstraction Rules and Velocity Statistics

Derived/windowed statistics (count, sum, average, distinct count, standard deviation, etc.) used
as inputs to activation rules. mw-core's schema is a near-literal port of Jube's function-type
taxonomy — the same 16 numeric codes, in the same gaps (no 9/10) — confirming this was built
directly against Jube's `AbstractionRule` model. This document focuses mostly on mw-core's
implementation, since that's where this engagement did the deepest work; Jube's pipeline
placement (confirmed directly) and field semantics (confirmed via the TTL-counter research, which
shares the same `Data`/`Abstraction` dictionary-injection model — see
[02](02-RULE-EXPRESSION-ENGINE.md)) are noted where verified.

## Jube (confirmed pipeline placement)

Source: `Jube.Engine/EntityAnalysisModelInvoke/EntityAnalysisModelInvoke.cs:118-187` (the same
trace used in [03](03-GATEWAY-RULES.md)/[04](04-ACTIVATION-RULES.md)).

- Abstraction rules run in **two separate stages**, after TTL counters and before activations:
  `ExecuteAbstractionRulesWithSearchKeysAsync` (those with a grouping/search key — i.e. windowed
  statistics over a value like `AccountId`), then, after a wait-barrier,
  `ExecuteAbstractionRulesWithoutSearchKeys` (presumably simpler payload-derived flags), followed
  by `ExecuteAbstractionCalculations` (a third, distinct stage — matching mw-core's separate
  `AbstractionCalculation` response section, see [01](01-APIS-AND-FLOWS.md)).
- The same `Abstraction("name")` dictionary-injection pattern described in
  [02-RULE-EXPRESSION-ENGINE.md](02-RULE-EXPRESSION-ENGINE.md) applies — a computed abstraction
  value is injected into later-stage rule evaluation via a dictionary lookup by name, not a
  direct function call.
- The function-type taxonomy (Count / Distinct Count / Sum / Average / Median / Kurtosis / Skew /
  Standard Deviation / Mode / Same Count / Actual Value / Max / Min / Since) is mirrored exactly
  in mw-core's schema comment (`# Jube function_type integer codes`,
  `risk_abstraction_rule.ex:5-6`) and its `@function_types [1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14,
  15, 16]` — the gap at 9/10 is preserved from Jube's own numbering, strong evidence this is a
  direct port of Jube's enum rather than an independently-designed list.

## mw-core (verified in depth this engagement)

Source: `apps/mw_risk/lib/mw_risk/abstraction_engine.ex`,
`apps/infra_repo/lib/infra_repo/schemas/risk_abstraction_rule.ex`.

### What was found broken

Every function type that depended on `search_key`/`function_key` was silently non-functional:

- `search_key/1` built a lookup key by **string-concatenating the raw, un-mapped `search_key`
  field** (e.g. `"Payload.AccountId"`) with the interval (`"Payload.AccountId:1D"`), which never
  matched the real hydrated-feature-map keys (`"card:1h:count_tx"`, etc.) — so even the
  function types with *some* implementation (1 Count, 3 Sum, 4 Average, 13 Actual Value) never
  actually found data for any rule using a `search_key`.
- Function types 2 (Distinct Count), 5 (Median), 6 (Kurtosis), 7 (Skew), 8 (Standard Deviation),
  11 (Mode), 12 (Same Count), 14 (Max), 15 (Min) were explicit one-line stubs returning `0.0`
  unconditionally — never implemented at all.
- No mechanism existed to track velocity **by `AccountId`** specifically — the only entities
  `VelocityPipeline` tracked were card/merchant/IP-shaped dimensions, not the account-grouped
  windows several abstraction rules were configured to use.
- `AbstractionEngine.compute/3` only ever received the already-hydrated snake_case feature map,
  never the raw payload — so it had no way to read `AccountId`/`CardPan`/`AmountUSD`/etc. even
  if the lookup keys had been correct.

### What was built (this engagement)

A single generic mechanism replacing the broken/stubbed per-function-type logic:

1. **Write side**: `VelocityPipeline` appends one event per transaction to a small set of
   independent per-field journals (`abs_account`, `abs_card`, `abs_ip` — now model-scoped, see
   [07](07-MULTI-MODEL-SCOPING.md)), using `InfraFeatureStore.Journal` (a Redis sorted set,
   already built but previously unused for this purpose).
2. **Read side**: `AbstractionEngine.evaluate_rule/4` resolves the rule's `search_key` to an
   entity+value+window, fetches the journal events in that window, and computes the requested
   statistic in Elixir — count/distinct-count/sum/average/median/kurtosis/skew/stddev/mode/
   same-count/max/min/since/actual-value **all derive from the same fetched event list**, unifying
   what were 16 separate (mostly stubbed) code paths into one.
3. **In-flight synthetic event**: since `/verify` is a dry run for case/notification purposes (see
   [01](01-APIS-AND-FLOWS.md)), the current transaction's own values are folded into the computed
   window as a synthetic event so a single test call against a cold account shows a sensible
   non-zero result, while the real persisted write (after the decision is known) makes
   *subsequent* calls see real accumulated history.

### `rule_expression` fallback (closes O9/O12 partially) and what's still unconfigured

A second, separate gap: `AbstractionEngine` never read the `rule_expression` column at all — even
though it existed on the schema and was already populated for `CurrencyRiskGate`/`AmlRiskGate`
(which is *why* those two were also stuck at `0.0` despite looking fully configured). Fixed by
wiring `evaluate_rule/4` to fall back to `RuleExpression.evaluate/3` whenever a rule has no usable
`search_key`, mirroring `ActivationEngine`'s existing precedence (`rule_expression`, when present,
governs) — output is `1.0`/`0.0` for true/false, so it composes with downstream Activation rules
exactly like any numeric function_type (`Abstraction.HighValueFlag == 1`).

This single change fixed, with no further code: `CurrencyRiskGate`, `AmlRiskGate`, and (once
configured) `HighValueFlag` (`Payload.AmountUSD > 1000`), `InternationalFlag`
(`Payload.MerchantCountry != Payload.IssuerCountry`), `NightTimeFlag`
(`Payload.LocalHour >= 22 OR Payload.LocalHour <= 5`). `NewMerchantFlag` was configured separately
via the existing `function_type=12` (Same Count, grouped by `AccountId`, counting same-`MerchantId`
events) — verified the polarity is `== 1` (not `0`) for "first transaction with this merchant",
since the synthetic in-flight event used for `/verify`'s dry-run semantics always counts itself.

**O9 fully closed.** `AmountZScore`, `AmountPercentile`, `VelocityScore`, `GeographicAnomalyScore`
needed formulas over *multiple* stats, not just a config value — added four mw-core-specific
`function_type` codes (101-104, intentionally outside Jube's 1-16 range) to `compute_stat/4`,
reusing the same window-events mechanism every other type already uses:

- **101 — Z-Score**: `(current - mean) / stddev` over the window; `0.0` when stddev is `0` (no
  variance yet) rather than dividing by zero.
- **102 — Percentile**: fraction of window values `<=` the current value, as 0-100.
- **103 — Velocity score**: `min(count_in_window / cap, 1.0)`; `cap` configurable via the
  (otherwise-unused-for-this-type) `offset_value` column, default `10`.
- **104 — Geographic anomaly**: `1 - (count matching the current event's value / total count)`.

Confirmed live: building up 3 same-country transactions then a 4th with a new country correctly
produced `0.0` → `0.0` → `0.0` → `0.8` for `GeographicAnomalyScore`. Along the way, found
`MerchantCountry` wasn't in `@event_field_map` or `current_event_data/1` at all (in either
`AbstractionEngine` or `VelocityPipeline`'s journal-write side) — meaning this field could never
have been read from any event, real or synthetic, prior to this fix; `GeographicAnomalyScore` would
have silently returned `1.0` unconditionally (or `0.0` if the window was empty) regardless of
actual history. Fixed on both the read and write side.

All formulas were proposed for review and confirmed before implementation, not guessed at
silently — see O9 in [08-FIXED-GAPS-CHANGELOG.md](08-FIXED-GAPS-CHANGELOG.md).

## Gap table

| Aspect | Jube | mw-core | Tag |
|---|---|---|---|
| Function-type taxonomy | 16 codes (1-8, 11-16) | Same 16 codes, same gaps at 9/10 — confirms direct port | 🟢 Parity (by construction) |
| Pipeline placement | Two-stage (with/without search key) + separate calculations stage | Single `AbstractionEngine.compute/4` call producing both `Abstraction` and `AbstractionCalculation` response sections | 🟡 Intentional simplification — same observable output shape, fewer internal stages |
| Search-key field resolution | Native dictionary lookup (`Data("AccountId")`) | Was broken (raw string concatenation, never matched real keys); fixed via Jube-aware `get_jube/2` field resolution | 🔴 Fixed this engagement |
| Distinct Count / Median / Kurtosis / Skew / StdDev / Mode / Same Count / Max / Min | Implemented (assumed — not independently re-verified against Jube source this round) | Were unconditional `0.0` stubs; now computed from the unified journal-event mechanism | 🔴 Fixed this engagement |
| Account-grouped velocity tracking | Native (same generic `Data`/grouping mechanism as everything else) | Didn't exist; added as a new independent journal namespace (`abs_account`) | 🔴 Fixed this engagement |
| Cross-model isolation | Structural | Was broken (journal entity names had no model component); fixed this engagement | 🔴 Fixed — see [07](07-MULTI-MODEL-SCOPING.md) |
| `rule_expression` fallback for non-windowed flags/gates | N/A — Jube's `Data`/`Abstraction` injection has no equivalent "fallback" concept, it's just code | Was never read by `AbstractionEngine` at all (`CurrencyRiskGate`/`AmlRiskGate` silently stuck at 0.0 despite being configured); fixed by falling back to `RuleExpression.evaluate/3` | 🔴 Fixed this engagement |
| Unconfigured legacy rules (AmountZScore, etc.) | N/A — would need Jube-side config inspection to know if these are configured there | All 8 now configured — 4 via `rule_expression`/existing function_types, 4 via new mw-core-specific function_type codes (101-104) | 🟢 Fixed this engagement |
| `MerchantCountry` missing from the event journal | N/A | Wasn't in `@event_field_map`/`current_event_data/1` (read) or `VelocityPipeline`'s journal-write event map at all — found while verifying `GeographicAnomalyScore` | 🔴 Fixed this engagement |

## Open items

- None remaining specific to this topic — see [08-FIXED-GAPS-CHANGELOG.md](08-FIXED-GAPS-CHANGELOG.md) for the overall open-items table (O2/O4/O10/O11 are out of scope by deliberate decision, not gaps in this topic).
- "Same Count" (function_type 12) compares the current transaction's value for a field against
  past events' values for that same field — implemented, but not independently load-tested at
  volume; flagged as lower-confidence than the more heavily-exercised types (Count/Sum/Average).