Settlement Event Logger
Technical Design Document

ProjectMercury Settlement Platform (UAT)
ModuleSettlement Core
Prepared bySettlement Module Team
DateJune 2026
StatusFinal Design — Ready for Implementation

Contents

  1. Background & Motivation
  2. What Gets Tracked — Event List
  3. Database Design
  4. How the Table Gets Updated
  5. Handling Fallback Scenarios
  6. Bank Confirmation — Cascade Example
  7. The EventLogger Module
  8. Two Implementation Approaches
  9. Design Comparison & Final Schema
  10. Event Logging Principles
  11. UI — Event Log Page
  12. Implementation Steps
  13. Out of Scope

1. Background & Motivation

The Mercury Settlement platform processes the complete lifecycle of merchant payouts — from acquiring raw switch dump files via SFTP, through reconciliation, MIS generation, dual-level Finance approval, payout transmission to banks, and finally merchant notification via Settlement Advice and VAT Invoice.

Each of these steps involves real money movement and regulatory accountability. Currently the outcome of each step is captured in isolated domain tables (sync_log, dump_files, settlement_mis, payout_batches, etc.) with no unified view across the full lifecycle. When an issue arises — a batch mismatch, a delayed payout, a missing VAT invoice — operators must query multiple tables or search through application logs to reconstruct what happened, when, and by whom.

The Settlement Event Logger solves this by maintaining a single, auditable timeline of every significant action across the entire settlement pipeline.

Objectives

2. What Gets Tracked — Event List

The event logger tracks 18 distinct events across 7 pipeline stages. The principle is to log only significant business milestones — not every function call or database row update.

Stage # Event Type Trigger Actor Type
Transaction Sync1core_txn_syncPOS/QR sync API → Oban workerWorker
File Ingestion2sftp_fetchDaily scheduler 03:00 AM or manual triggerScheduler
File Ingestion3dump_file_processingAfter SFTP fetch, per fileWorker
Reconciliation4reconciliation_runAfter dump file processedWorker
Reconciliation5recon_exception_createdDuring recon engine runSystem
Reconciliation6recon_exception_releasedManual — user releases exceptionUser
Adjustments7adjustment_createdManual — user creates adjustmentUser
MIS & Approval8mis_generatedMIS generation worker at 07:00 AMScheduler
MIS & Approval9mis_l1_approvedManual — Finance L1 user actionUser
MIS & Approval10mis_l2_approvedManual — Finance L2 user actionUser
MIS & Approval11mis_rejectedManual — any level rejectionUser
Payout12payout_generatedAfter L2 approval → Oban workerWorker
Payout13payout_transmittedAfter payout generated → bank SFTPWorker
Bank Confirmation14bank_confirmation_uploadedManual — Finance Ops CSV uploadUser
Bank Confirmation15bank_confirmation_approvedManual — Finance Ops approves batchUser
Bank Confirmation16core_transactions_updatedCascade inside bank confirmation approvalSystem
Notification17settlement_advice_sentOban — SettlementAdviceWorkerWorker
Notification18vat_invoice_dispatchedOban — VatInvoiceWorkerWorker

3. Database Design

Final Table: settlement_event_log

CREATE TABLE settlement_event_log ( id BIGSERIAL PRIMARY KEY, event_type VARCHAR NOT NULL, entity_type VARCHAR NULL, entity_id BIGINT NULL, actor_id BIGINT NULL, actor_name VARCHAR NULL, actor_type VARCHAR NOT NULL, -- User | System | Scheduler | Worker | API status VARCHAR NOT NULL, -- success | failed | partial | pending summary TEXT NOT NULL, settlement_date DATE NULL, metadata JSONB NULL, occurred_at TIMESTAMP NOT NULL DEFAULT NOW() );

Field Reference

Field Purpose Example
event_type Machine-readable event identifier mis_l2_approved
entity_type Which domain table this event belongs to settlement_mis, dump_file
entity_id ID of the domain record (soft link — no FK constraint) 42
actor_id User ID if human action; NULL for system events 7 (Finance user), NULL (System)
actor_name Display name — avoids JOIN to users table on every UI query Sunny, Finance L2, Ysp.Scheduler
actor_type Distinguishes manual vs automated origin — enables clean filtering User, System, Scheduler, Worker, API
status Outcome of the event success, failed, partial, pending
summary Human-readable one-line description — displayed directly in UI with no translation MIS for 18-Jun-2026 approved by Finance L2
settlement_date Business date the event relates to — primary filter in UI 2026-06-18
metadata JSONB payload with event-specific details — no schema change needed for new event types {"total_merchants": 47, "net_payable": 58420.50}
occurred_at Exact timestamp when the event happened 2026-06-19 09:15:02

Indexes

-- Primary query: "show me all events for settlement date X" CREATE INDEX idx_event_log_date ON settlement_event_log (settlement_date, occurred_at DESC); -- Filter by event type or failures CREATE INDEX idx_event_log_type_status ON settlement_event_log (event_type, status); -- Filter by actor (who did what) CREATE INDEX idx_event_log_actor ON settlement_event_log (actor_id);

Key Design Decisions

Metadata Examples by Event Type

EventMetadata Payload
mis_generated {"channel": "COMBINED", "total_merchants": 47, "total_transactions": 1320, "net_payable": 58420.50}
mis_l1_approved {"notes": "Verified reconciliation", "previous_status": "pending", "new_status": "l1_approved"}
sftp_fetch {"dump_date": "2026-06-18", "files_found": 2, "files_downloaded": 2, "files_skipped": 0}
reconciliation_run {"dump_file_id": 42, "matched": 1197, "unmatched": 7, "exceptions_created": 7}
payout_transmitted {"batch_id": 28, "merchant_count": 47, "total_amount": 58420.50, "bank": "ENBD", "method": "SFTP"}
bank_confirmation_approved {"batch_id": 11, "filename": "bank_ack_20260619.csv", "successful": 47, "failed": 0}
vat_invoice_dispatched {"invoice_id": 441, "invoice_number": "INV-2026-0441", "merchant_mid": "419926360000000", "amount": 62.00}
Any failure {"reason": "SFTP connection timeout", "retry_attempt": 2, "oban_job_id": 1038}

4. How the Table Gets Updated

There are two types of triggers — system (Oban Workers) and user (Context Functions). In both cases the rule is the same: write to the event log after the operation completes, not before.

4.1 System-Triggered Events (Oban Workers)

These events happen automatically with no user involved — SFTP fetch at 3am, MIS generation at 7am, payout transmission after L2 approval. The event log is updated inside the Oban worker's perform/1 function, after the operation result is known.

Example — MIS Generation Worker

Current code (unchanged):

case SettlementMisGenerator.generate_for_date(date, channel) do {:ok, mis} -> Logger.info("[MisGenerationWorker] MIS generated id=#{mis.id}") :ok {:error, reason} -> Logger.error("[MisGenerationWorker] MIS generation failed") {:error, reason} end

With event logger added (only new lines added after existing logic):

case SettlementMisGenerator.generate_for_date(date, channel) do {:ok, mis} -> Logger.info("[MisGenerationWorker] MIS generated id=#{mis.id}") # unchanged EventLogger.log(%{ # new event_type: "mis_generated", entity_type: "settlement_mis", entity_id: mis.id, actor_id: nil, actor_name: "Ysp.Scheduler", actor_type: "Scheduler", status: "success", summary: "MIS generated for #{mis.mis_date} — #{mis.total_transactions} txns, AED #{mis.total_net_payable}", settlement_date: mis.mis_date, metadata: %{ channel: channel, total_merchants: mis.total_merchants, total_transactions: mis.total_transactions, net_payable: mis.total_net_payable } }) :ok # unchanged {:error, reason} -> Logger.error("[MisGenerationWorker] MIS generation failed") # unchanged EventLogger.log(%{ # new event_type: "mis_generated", actor_type: "Scheduler", actor_name: "Ysp.Scheduler", status: "failed", summary: "MIS generation failed for #{date} — #{inspect(reason)}", settlement_date: date, metadata: %{reason: inspect(reason), channel: channel} }) {:error, reason} # unchanged end
Important: The existing Logger.info, the existing return values (:ok, {:error, reason}), and the existing business logic are all completely untouched. EventLogger.log/1 is added after the result — nothing that currently works can break.

4.2 User-Triggered Events (Context Functions)

These are events where a real user took an action in the UI. The event log is updated inside the context function, after the database update succeeds.

Example — MIS L1 Approval

Current code in context.ex:

def l1_approve_settlement_mis(%SettlementMis{} = mis, user_id, notes \\ nil) do mis |> SettlementMis.l1_approve_changeset(user_id, notes) |> Repo.update() end

With event logger added:

def l1_approve_settlement_mis(%SettlementMis{} = mis, user_id, notes \\ nil) do result = mis |> SettlementMis.l1_approve_changeset(user_id, notes) # unchanged |> Repo.update() # unchanged case result do {:ok, updated_mis} -> EventLogger.log(%{ # new event_type: "mis_l1_approved", entity_type: "settlement_mis", entity_id: updated_mis.id, actor_id: user_id, actor_name: resolve_user_name(user_id), actor_type: "User", status: "success", summary: "MIS for #{updated_mis.mis_date} reviewed and approved at L1", settlement_date: updated_mis.mis_date, metadata: %{notes: notes, previous_status: mis.approval_status} }) {:ok, updated_mis} # unchanged error -> error # unchanged end end
Note: The event is logged only when Repo.update() returns {:ok, ...}. A failed database update never creates a false success event in the log.

5. Handling Fallback Scenarios

Many processes in the pipeline have both an automated path and a manual fallback. The event logger handles this naturally — every attempt, success or failure, auto or manual, gets its own row. Both the failure and the recovery are visible in the same timeline.

Scenario: SFTP Auto-Fetch Fails, Manual Upload Done

Time Event Actor Actor Type Status Summary
03:00:11 sftp_fetch Ysp.Scheduler Scheduler failed SFTP fetch failed for 2026-06-18 — connection timeout
09:14:33 dump_file_upload Finance Ops User success Manual upload: switch_settled_20260618.csv — 1,204 records
09:16:01 reconciliation_run ReconciliationWorker Worker success Recon complete — 1,197 matched, 7 exceptions created

Anyone looking at this date's event log immediately sees: SFTP auto-fetch failed at 3am, a human intervened at 9am, and the pipeline recovered. Without the event log, this 6-hour gap would only be visible by digging through Oban job history or server logs.

Scenario: Reconciliation Fails on First Attempt, Succeeds on Oban Retry

Time Event Actor Status Summary
03:15:22 reconciliation_run ReconciliationWorker failed Recon failed for dump_file #42 — DB timeout
03:16:45 reconciliation_run ReconciliationWorker success Recon complete — 1,197 matched, 7 exceptions

Oban retried automatically after 83 seconds. Both the failure and the success are logged. No manual intervention was needed, and the timeline shows the full picture.

6. Bank Confirmation — Cascade Example

This is the most important scenario because one user action (clicking Approve on a bank confirmation batch) triggers a chain of downstream updates across multiple tables and workers.

What Happens Internally

User clicks "Approve Batch" │ ├── BankConfirmationBatch → status = "approved" ├── For each record in batch: │ ├── PayoutItem → status = "paid", bank_ref set │ ├── PayoutBatch → bank_ref updated, transmitted_at set │ └── core_transactions → settlement_status = "paid", bank_trnx_date set (BULK) │ └── For each confirmed PayoutItem: ├── enqueue SettlementAdviceWorker → sends advice email to merchant ├── enqueue VatInvoiceWorker → generates + emails VAT invoice └── enqueue VatMisDistributionWorker → exports to Finance VAT MIS file

Events Logged for This Cascade

Event When What It Records
bank_confirmation_approved Immediately on approval click batch_id, approved_by, successful_count, failed_count, total_amount
core_transactions_updated Inside mark_transactions_paid per merchant merchant_mid, how many txns marked paid, bank_trnx_date set
settlement_advice_sent When SettlementAdviceWorker completes payout_item_id, merchant, amount, email recipient
vat_invoice_dispatched When VatInvoiceWorker completes invoice_number, merchant, amount

Event Log — One Bank Confirmation Approval (47 merchants)

Time Event Actor Type Status Summary
14:18:44 bank_confirmation_approved Finance Ops User success Batch #11: 47 merchants confirmed, AED 58,420
14:18:44 core_transactions_updated System System success 342 txns marked paid — MID 419926360000000
14:18:44 core_transactions_updated System System success 218 txns marked paid — MID Mercury_CB6F6A5
14:18:45 settlement_advice_sent SettlementAdviceWorker Worker success Advice sent → MID 419926360000000, AED 1,240
14:18:45 settlement_advice_sent SettlementAdviceWorker Worker success Advice sent → MID Mercury_CB6F6A5, AED 890
14:18:46 vat_invoice_dispatched VatInvoiceWorker Worker success INV-2026-0441 → MID 419926360000000, AED 62
14:18:46 vat_invoice_dispatched VatInvoiceWorker Worker success INV-2026-0442 → MID Mercury_CB6F6A5, AED 44.50

This level of granularity means if a settlement advice email failed for one specific merchant, you see exactly which merchant and when — without searching Oban job tables or email logs.

7. The EventLogger Module

The EventLogger.log/1 function is intentionally simple — it does one database insert and never raises an exception. A logging failure never crashes a worker or blocks a user action.

defmodule SettlementCore.EventLogger do require Logger alias SettlementCore.SettlementEventLog alias PlatformCore.Repo def log(attrs) do attrs = Map.put_new(attrs, :occurred_at, NaiveDateTime.utc_now() |> NaiveDateTime.truncate(:second)) %SettlementEventLog{} |> SettlementEventLog.changeset(attrs) |> Repo.insert() |> case do {:ok, _event} -> :ok {:error, cs} -> # Log the failure but never propagate it to the caller. # A logging failure must never disrupt the settlement pipeline. Logger.warning("[EventLogger] Failed to log event: #{inspect(cs.errors)}") :ok end end end
Design guarantee: The settlement pipeline continues regardless of whether the event log insert succeeds. The event logger is a silent observer — it never influences the outcome of any business operation.

8. Two Implementation Approaches

Approach A — Oban Telemetry Hook

Oban emits a telemetry event [:oban, :job, :stop] every time any worker finishes — with the job struct (worker name, args) and outcome (:success or :failure). A single handler module intercepts all of these and maps them to event log entries.

defmodule SettlementCore.ObanEventHandler do def attach do :telemetry.attach( "settlement-oban-logger", [:oban, :job, :stop], &__MODULE__.handle_event/4, nil ) end def handle_event([:oban, :job, :stop], _measurements, metadata, _config) do job = metadata.job state = metadata.state # :success | :failure | :cancelled | :discard case job.worker do "SettlementCore.Workers.MisGenerationWorker" -> EventLogger.log(%{ event_type: "mis_generated", actor_type: "Scheduler", actor_name: "Ysp.Scheduler", status: if(state == :success, do: "success", else: "failed"), summary: "MIS generation #{state} for #{job.args["settlement_date"]}", settlement_date: Date.from_iso8601!(job.args["settlement_date"]), metadata: %{channel: job.args["channel"], oban_job_id: job.id} }) "SettlementCore.Workers.ReconciliationWorker" -> EventLogger.log(%{ event_type: "reconciliation_run", entity_type: "dump_file", entity_id: job.args["dump_file_id"], actor_type: "Worker", actor_name: "ReconciliationWorker", status: if(state == :success, do: "success", else: "failed"), summary: "Reconciliation #{state} for dump_file ##{job.args["dump_file_id"]}", metadata: %{oban_job_id: job.id} }) _ -> :ok end end end

Attached once in application.ex — zero changes to any worker file.

ProsCons
  • Zero changes to existing worker code
  • Automatically captures all retries and failures
  • One place to manage all worker event mappings
  • New workers are logged by adding one case clause here
  • Metadata limited to job args — you get the date and IDs but not counts and amounts from the result unless you query the DB again inside the handler
  • User-triggered events (approvals, uploads) cannot be captured this way — context functions still require changes
  • Business context is thinner — "MIS generated for 2026-06-18" without transaction count or net amount

Approach B — Direct Logging at Point of Outcome

Add EventLogger.log(...) calls directly inside each worker's perform/1 function and each context function, immediately after the operation result is known.

ProsCons
  • Full access to the actual result — logs exact counts, amounts, error messages directly from the result struct
  • Completely explicit — every log point is visible in the code, no magic
  • User-triggered and system-triggered events handled identically
  • Existing logic completely untouched — only new lines added after result handling
  • Touches more files — approximately 18 touch points across workers and context functions
  • Requires discipline to add logging whenever a new worker is created in future
Critically — existing code is not modified, only extended. The before and after for any worker looks like this:
BEFORE: :ok ← existing return, untouched AFTER: EventLogger.log(%{...}) ← new line added above :ok ← existing return, untouched
Nothing that currently works can break.

9. Design Comparison & Final Schema

Two design proposals were reviewed side by side. The table below compares them and gives a verdict on each field.

Field Proposal 1 Proposal 2 Verdict
event_type string VARCHAR Same — adopt
entity_type string VARCHAR Same — adopt
entity_id integer BIGINT BIGINT preferred — safer for large tables
actor_id integer, nullable BIGINT NULL Same — adopt
actor_label Single field combining name + type e.g. "System (Scheduler)" Split into actor_name + actor_type Proposal 2 wins — split is better for filtering
actor_type Not present as separate field User | System | Scheduler | Worker | API Proposal 2 addition — adopt it
status success | failed | partial | pending success | failed | partial Keep pending — needed for bank confirmation upload lifecycle
summary string TEXT — human-readable prose Same principle — store readable prose, not event codes
settlement_date date, nullable DATE NULL Same — adopt
metadata jsonb JSONB Same — adopt
occurred_at naive_datetime TIMESTAMP Same — adopt
Soft entity link Explicit — no FK constraint on entity_id Not mentioned Keep — event log must survive domain record deletion

Final Merged Schema

CREATE TABLE settlement_event_log ( id BIGSERIAL PRIMARY KEY, event_type VARCHAR NOT NULL, entity_type VARCHAR NULL, entity_id BIGINT NULL, -- soft link, no FK constraint actor_id BIGINT NULL, -- NULL for system events actor_name VARCHAR NULL, -- stored at event time, no JOIN needed actor_type VARCHAR NOT NULL, -- User | System | Scheduler | Worker | API status VARCHAR NOT NULL, -- success | failed | partial | pending summary TEXT NOT NULL, -- human-readable prose for UI display settlement_date DATE NULL, -- business date for filtering metadata JSONB NULL, -- event-specific payload occurred_at TIMESTAMP NOT NULL DEFAULT NOW() ); CREATE INDEX idx_event_log_date ON settlement_event_log (settlement_date, occurred_at DESC); CREATE INDEX idx_event_log_type_status ON settlement_event_log (event_type, status); CREATE INDEX idx_event_log_actor ON settlement_event_log (actor_id);

10. Event Logging Principles

Principle 1 — Log Only Significant Business Events

Do not log every function call or database row update. Log only milestones that Finance, Operations, or Compliance would recognise as meaningful steps in the settlement lifecycle. The 18 events defined in Section 2 represent these milestones. This keeps the timeline clean and searchable.

Principle 2 — Every Event Answers Four Questions

Every log entry must answer:

FieldExample 1Example 2
WhatMIS for 18-Jun-2026 approved by Finance L2SFTP fetch failed for 2026-06-18
When09:15:0203:00:08
WhoFinance L2 (User)Ysp.Scheduler (Scheduler)
OutcomeSuccessFailed — SFTP timeout

Principle 3 — Summary Must Be Human-Readable

The summary field stores readable prose, not codes. The UI displays it directly without any translation layer.

WrongCorrect
mis_l2_approvedMIS for 18-Jun-2026 approved by Finance L2
payout_generatedPayout batch generated for 47 merchants, AED 58,420
sftp_fetch failedSFTP fetch failed for 2026-06-18 — connection timeout

Principle 4 — Metadata for Event-Specific Detail

Keep common information in columns (event_type, status, actor_name) and event-specific information in metadata JSONB. No schema change is required when a new event type is introduced. Examples of what goes in metadata: file names, transaction counts, amounts, notes, rejection reasons, exception IDs, bank references, invoice numbers.

Principle 5 — Log Never Blocks the Pipeline

A failure in EventLogger.log/1 is caught silently and written to the application log. It never propagates as an error to the calling worker or context function. The settlement pipeline must continue regardless of whether event logging succeeds.

11. UI — Event Log Page

A new "Event Log" menu item in the Settlement section of the admin panel, built as a Phoenix LiveView page.

Full Daily Timeline — Example for 18-Jun-2026

Time Event Actor Type Status Summary
03:00:11 sftp_fetch Ysp.Scheduler Scheduler success SFTP: 2 files downloaded for 2026-06-18
03:02:45 dump_file_processing SwitchDumpProcessing Worker success switch_settled_20260618.csv — 1,204 records processed
03:04:12 reconciliation_run ReconciliationWorker Worker success Recon complete — 1,197 matched, 7 exceptions created
03:04:13 recon_exception_created System System partial 7 reconciliation exceptions created for dump_file #42
07:00:43 mis_generated Ysp.Scheduler Scheduler success MIS generated for 2026-06-18 — 342 txns, AED 58,420
09:14:22 mis_l1_approved Finance L1 User success MIS for 2026-06-18 reviewed and approved at L1
11:32:05 mis_l2_approved Finance L2 User success MIS for 2026-06-18 approved at L2 — payout initiated
11:32:06 payout_generated PayoutTransmissionWorker Worker success Payout batch #28 generated — 47 merchants, AED 58,420
11:32:09 payout_transmitted PayoutTransmitter Worker success Batch #28 transmitted to bank via SFTP
14:05:31 bank_confirmation_uploaded Finance Ops User pending bank_ack_20260619.csv uploaded — 47 records, awaiting approval
14:18:44 bank_confirmation_approved Finance Ops User success Batch #11: 47 merchants confirmed as paid, AED 58,420
14:18:44 core_transactions_updated System System success 342 core transactions marked paid — MID 419926360000000
14:18:45 settlement_advice_sent SettlementAdviceWorker Worker success Settlement advice sent → MID 419926360000000, AED 1,240
14:18:46 vat_invoice_dispatched VatInvoiceWorker Worker success INV-2026-0441 dispatched → MID 419926360000000, AED 62

UI Features

12. Implementation Steps

# Step Work Impact on Existing Code
1 Database migration Create settlement_event_log table and indexes None
2 Ecto schema Create SettlementCore.SettlementEventLog schema and changeset None — new file
3 EventLogger module Create SettlementCore.EventLogger with log/1 None — new file
4 Wire system events Add EventLogger.log/1 inside 13 Oban worker perform/1 functions Additive only — new lines after existing result handling
5 Wire user events Add EventLogger.log/1 inside 4 context functions (l1_approve, l2_approve, reject, create_adjustment) Additive only
6 Wire cascade events Add EventLogger.log/1 inside mark_transactions_paid in BankConfirmationService Additive only
7 LiveView page Create EventLogLive.Index with filters, timeline, detail expand None — new file
8 Menu item Add "Event Log" entry to MenuProvider Additive only — one new item in list
9 Permissions Add settlement.event_log.view permission to roles seed Additive only
No existing functionality is modified in any step. All changes are additive. The settlement pipeline continues to work exactly as it does today regardless of whether the event logger is fully implemented or not.

13. Recommendation

Use Approach B — Direct Logging at Point of Outcome

The event log is a financial audit trail, not just an engineering debug tool. Finance, Compliance, and Audit teams will use it to verify amounts, counts, and timings for every settlement cycle. An event that says "MIS generated for 2026-06-18" without the transaction count, merchant count, and net payable amount is significantly less useful than one that says "MIS generated for 2026-06-18 — 342 txns across 47 merchants, AED 58,420 net payable."

That richer data only exists in the result struct (mis.total_transactions, mis.total_net_payable) which is available inside the worker after the operation completes — but not available to an external telemetry hook without an additional database query.

Why not Approach A (Oban Telemetry)?

Why Approach B is safe:

14. Out of Scope — Future Enhancements