# Clearing & Settlement Extension — Requirement & Implementation Plan

**Module:** Clearing Processing (`clearing_core`)
**Part of:** Acquiring Platform Transformation program (Workstream B).
**Depends on:** `platform_core`, `scheme_core` (Phase 5 fee tables).
**Consumed by:** `settlement_core` (depends on `clearing_core`, mirroring its existing
dependency on `dispute_core` per ADR-0002) via `SettlementCore.ClearingBridge`.
**Status:** see [`docs/ACQUIRING_PLATFORM_PROGRESS.md`](../ACQUIRING_PLATFORM_PROGRESS.md).
C1-C6 built (Visa TC-33 + Mastercard IPM inbound/outbound, reconciliation bridge, ops UI).
DCF (MPGS Draft Capture File) is researched but explicitly deferred — see §5.

---

## 1. Purpose

Handle exchange of scheme clearing files — Visa BASE II/TC-33 and Mastercard IPM/T112 —
the actual settlement/interchange record each network sends back for cleared transactions,
plus the presentment/clearing submissions Mercury sends to each network. This closes a
confirmed gap: no Base II/IPM handling existed anywhere in this codebase before this
workstream, because the current external acquirer (YSP) has been absorbing that
responsibility.

## 2. Real formats this is built against

Both formats were grounded in real specification documents and real sample files the user
supplied under `docs/Clearing-System/`, not invented from general knowledge:

### 2a. Visa BASE II / TC-33 — best-specified, built first

Source: `Visa-related/base-ii-clearing-interchange-formats-tc-01-to-tc-49.pdf` (12 pages,
Visa Confidential, 18 Oct 2025 — despite the filename, the actual content is a focused
excerpt covering only **TC33 (Multipurpose Message) TCR0/TCR1**, the "BASE II Clearing and
Settlement Advice"), cross-checked against the real sample
`Visa-related/454183.76.20241022.xml_tc33.txt.ascii`.

- Fixed 168-byte TCR0/TCR1 record pairs. TCR0: PAN, Acquirer Reference Number (ARN, Visa's
  23-digit clearing identifier — **not the same as, and too long for, a standard 12-char
  RRN**, confirmed by a real `Data too long for column 'rrn'` error while building the
  reconciliation bridge), Card Acceptor ID, Terminal ID, Source/Settlement amounts, Usage
  Code. TCR1: Interchange Fee Amount (6 implied decimals, the only field the spec states an
  implied-decimal count for), Fee Program Indicator, Settlement Date, Merchant Name/MCC,
  Purchase Date.
- **Known gap, documented not hidden**: the real `.ascii` sample's Advice Record ID field
  (TCR0 positions 17-19) never reads the spec's constant `"CAS"` — it reads other values
  (`"CP*"`, `"CP1"`, etc.), meaning that sample is a different TC33 sub-message than the one
  this project has field-level documentation for, even though the 168-byte framing and TCR
  sequence numbering (byte 4) both check out against it. The parser/schema are validated
  against hand-built synthetic fixtures matching the spec exactly, not this sample's field
  values — see `ClearingCore.Visa.Tc33Record`'s moduledoc.
- Confirmed two-directional, both using the TC33/TCR record family with different content:
  Acquirer → VisaNet input (presentment batches, submitted transactions) and VisaNet →
  Acquirer output (the daily TC-33A Capture File, settled transactions with fees applied).
  "Members don't send TC-33A back; they reconcile against their own transaction logs."

### 2b. Mastercard IPM / T112 — built from a DE/PDS catalog, no byte-level spec

No official Mastercard IPM Clearing Formats specification or real T112 sample exists in
this project (confirmed with the user). Built anyway, per explicit instruction, from a
**semantic Data Element catalog** — not byte positions:

1. A Visa-TCR ↔ Mastercard-DE/PDS equivalence table the user supplied.
2. Independent corroboration from this project's own Mastercard MPGS Draft Capture File
   research (§5), whose field tables cross-reference the same standard ISO 8583 Data
   Elements (DE3, DE5, DE6, DE7, DE12/13, DE15, DE18, DE25, DE41, DE50/51, DE60, DE90).

The combined ~26-DE catalog lives in `ClearingCore.Mastercard.IpmDeCatalog` — DE number →
field name/type. `ClearingCore.Mastercard.IpmParser`/`IpmBuilder` decode/encode an
*already-field-delimited* `%{de_number => value}` map, not raw file bytes, since the real
byte/VBS framing isn't known. **First thing to try once a real T112 sample exists**:
Mastercard's IPM-family files (including the T067 MPE parameter file already decoded in
`scheme_core` Phase 2a) commonly use a 1014-byte blocked, VBS record framing, and
`cardutil` (the same tool used for T067 — see the `mastercard_mpe_parsing_tools` memory
entry) is a general Mastercard IPM/MDS toolkit, not MPE-specific.

Also confirmed two-directional: Acquirer → Mastercard input (IPM clearing messages) and
Mastercard → Acquirer output (the daily IPM Clearing File, "T112"). "Members don't send
T112 back; they reconcile internally. Adjustments/exceptions are handled via reconciliation
reports (1644, GRN, GDR, GMR) and dispute processes" — those four report types are noted for
future awareness, not built (no spec for them either).

## 3. Architecture (as built)

- `clearing_core` app — depends on `platform_core` + `scheme_core`. Owns:
  - `ClearingBatch` (shared file-tracking across formats, `source`/`direction`/`status`,
    mirrors `SettlementCore.DumpFile`) and `ClearingException` (mirrors
    `SettlementCore.ReconciliationException`'s hold/review/resolve shape, but a distinct
    table — clearing-file-level exceptions are a different reconciliation surface from
    switch-dump-level ones).
  - Per-format raw record schemas: `Visa.Tc33Record`, `Mastercard.IpmRecord` — separate
    schemas per format (matches the `SwitchDumpRecord`/`QrSchemeDumpRecord` precedent),
    each with `match_status`/`core_transaction_id` for the reconciliation bridge.
  - Parsers (`Visa.Tc33Parser`, `Mastercard.IpmParser`) and builders (`Visa.Tc33Builder`,
    `Mastercard.IpmBuilder`) per format, plus a `Mix.Tasks.LoadVisaTc33` ingestion task
    (dry-run/validate/chunked insert, mirrors `settlement_core`'s `load_bin_table.ex`).
- `settlement_core` depends on `clearing_core` and gains `SettlementCore.ClearingBridge` —
  matches parsed records against `core_transactions` and applies a
  `COALESCE(actual, estimate)` `interchange_fee` override, the same pattern
  `ReconciliationEngine` already uses for YSP. **Visa and Mastercard match on different
  columns**: Mastercard's DE37 (RRN) matches `core_transactions.rrn` directly; Visa's ARN
  needed its own column (`core_transactions.acquirer_reference_number`, added via a
  dedicated, additive, nullable-column migration — real bug found and fixed while writing
  the bridge's tests, not assumed away).
- `ysp/`, `ReconciliationEngine`, and `SettlementMisGenerator` are untouched beyond the new
  dependency and bridge module — `SettlementMisGenerator` needed no change since it already
  prefers whatever sits in `core_transactions.interchange_fee` over the `InterchangeRate`
  estimate fallback.
- **Ops UI** (`platform_web`, `/admin/clearing/...`, permission `clearing.view`): a
  dashboard (batch/exception counts by source and status), a batch list with source/
  direction/status filters, a batch detail page showing every decoded record with match
  status, and an exception review queue with a resolve action — so operations can see and
  work everything without database access. New permissions (`clearing.view`,
  `clearing.exceptions.view`, `clearing.exceptions.resolve`) registered under a new
  "Clearing & Settlement" module in `priv/repo/seeds.exs`.

## 4. Real bugs found and fixed while building this

1. **`core_transactions.rrn` is `varchar(12)`**, sized for a standard RRN — cannot hold
   Visa's 23-digit ARN. Caught by a real DB error while writing `ClearingBridge`'s tests,
   not assumed. Fixed with an additive `acquirer_reference_number` column (migration
   `AddAcquirerReferenceNumberToCoreTransactions`), used only by the Visa matching path.
2. **Decimal columns without explicit `precision`/`scale`** (`visa_tc33_records`,
   `mastercard_ipm_records`) silently defaulted to MySQL's `decimal(10,0)` — every
   fractional amount got rounded to a whole number on write (`Decimal.new("1.80")`
   round-tripped as `2`). Caught the same way, fixed with a corrective migration
   (`FixClearingCoreDecimalPrecision`) matching the codebase's established
   `precision: 12, scale: 2` convention for money (`precision: 18, scale: 6` for the one
   field — Interchange Fee Amount — the Visa spec explicitly documents wider decimals for).

## 5. DCF (MPGS Draft Capture File) — researched, explicitly deferred

Mercury's switch (`mercury_device_middlelayer`) already has a live `acquirer/mastercard/`
adapter that **is** the MPGS gateway (per `docs/Switch-System/direct-connect-worklist.md`),
used for Visa/Mastercard/Amex and Mastercard Send (disbursement). The 160-page MPGS DCF spec
(`Master-dcf-Settlement-Disbursement-file/m_mpgs_256dcf_en-us_April2026.pdf`) was read in
full and two real sample files decoded structurally: fixed 256-byte records, `6200`(header)/
`6220`+`6221`(mandatory transaction detail)/`6222`-`6297`(scheme-specific and optional
addenda)/`6240`(trailer), one Header→Details→Trailer block per acquirer ICA.

**Explicitly deferred per the user** ("DCF related implementation can be kept pending as
DCF will can be implemented later"), not built in this pass. The user's described flow for
whenever it's picked up: *"As MasterCard --> Send DCF to our payment platform, we consume
and generate --> IPM 112 format to Master Card Clearing path."* — i.e. DCF consumption
should drive generating the outbound `Mastercard.IpmBuilder` submission (§3), not be a
standalone ingestion feature. Real sample files remain at
`docs/Clearing-System/Master-dcf-Settlement-Disbursement-file/` for whenever this is
resumed.

## 6. Verification

- Visa TC-33 parser: validated against hand-built synthetic TCR0/TCR1 fixtures matching the
  spec exactly (real sample can't validate field values — see §2a's known gap); a smoke
  test confirms the real sample parses without crashing and documents why it decodes to
  zero records (an ASCII-rendering artifact substituting `*` for zero-value bytes).
  `Mix.Tasks.LoadVisaTc33 --dry-run` exercised against the real sample.
- Visa TC-33 builder: round-trips through the parser (build → parse → same field values).
- Mastercard IPM parser/builder: round-trip tests against synthetic DE maps (no real sample
  to validate against).
- `ClearingBridge`: end-to-end tests create a real `CoreTransaction` + a real clearing
  record, run the bridge, and assert the match, the `interchange_fee` override, and the
  no-match exception path — for both Visa (ARN-keyed) and Mastercard (RRN-keyed)
  independently, confirming neither path disturbs the other or the existing YSP/QR paths.
- Full `platform_web` + `clearing_core` suites green (67 + 12 tests respectively), same
  8 pre-existing unrelated failures as documented bug #5, nothing new broken.