# Outbox / Inbox Event Lifecycle

**Date:** 2026-04-04  
**Scope:** `wallet_events`, `wallet_gl`

---

## Overview

The system uses a transactional outbox/inbox pattern to guarantee at-least-once delivery of domain events across OTP apps.

```
Producer app
  └─ INSERT outbox record (status: pending)
       └─ Dispatcher broadcasts via PubSub → status: dispatched
            ├─ [Real-time] Consumer receives {:outbox_event, event}
            └─ [Catch-up]  GenServer polls list_dispatched every 5s
                  └─ Consumer claims inbox → processes → marks acknowledged
                          └─ outbox status: acknowledged  ✓ (no longer polled)
```

---

## Outbox Status Lifecycle

| Status | Meaning |
|---|---|
| `pending` | Written by producer, not yet dispatched |
| `dispatching` | Being processed by dispatcher |
| `dispatched` | PubSub broadcast complete; waiting for consumer acknowledgment |
| `acknowledged` | At least one consumer successfully processed the event — **terminal, excluded from catch-up** |
| `failed` | Dispatch or processing failed; eligible for retry |
| `dead_letter` | Exceeded max retry attempts |

---

## Inbox Status Lifecycle

| Status | Meaning |
|---|---|
| `processing` | Consumer claimed the event, INSERT in progress |
| `processed` | Consumer completed successfully |
| `failed` | Consumer processing failed |
| `duplicate` | A second consumer tried to claim the same event |

---

## Deduplication: How It Works

`WalletEvents.Inbox.claim/2` attempts to INSERT an inbox record with `id = "#{consumer_app}:#{event_id}"`.

- **New event:** INSERT succeeds → `{:ok, :new}` → consumer processes
- **Duplicate event:** INSERT fails with a PK constraint error → `{:ok, :duplicate}` → skipped silently

> **Note:** `Ecto.ConstraintError` (PK violation) is caught explicitly and mapped to `:duplicate`. The generic `rescue` clause returns `:new` only for unexpected errors.

---

## The Catch-Up Loop

`WalletGl.TransferEventConsumer` runs a poll every 5 seconds (configurable via `:event_consumer_interval_ms`):

```elixir
WalletEvents.list_dispatched_events(["TransferCompleted.v1"], 200)
```

This queries `WHERE status = 'dispatched'`. Events stay in this query **until acknowledged**.  
When `mark_inbox_processed/2` is called it now also calls `Outbox.mark_acknowledged/1`, which moves the outbox record to `"acknowledged"` — removing it from all future catch-up polls.

---

## Bug Fixed (2026-04-04)

**Symptom:** Logs showed `wallet_gl transfer already has posting; skipping duplicate create` on every catch-up cycle for all historical transfers.

**Cause (1):** `Inbox.claim/2` rescue clause returned `{:ok, :new}` for `Ecto.ConstraintError` (PK duplicate), causing every duplicate claim to re-enter processing.

**Cause (2):** Outbox had no terminal status after `"dispatched"`. Successfully processed events were never removed from the catch-up query, so they fired every 5 seconds indefinitely.

**Fix:**
- `Ecto.ConstraintError` in `claim/2` now returns `{:ok, :duplicate}` 
- Added `"acknowledged"` status to outbox
- `mark_inbox_processed/2` now also calls `Outbox.mark_acknowledged/1`

---

## Cleanup: Stuck Historical Events

If deploying this fix to an environment with existing stuck records, run the one-off cleanup script:

```bash
mix run cleanup_stuck_outbox.exs
```

This finds all `"dispatched"` outbox events whose inbox records are already `"processed"` and bulk-updates them to `"acknowledged"`.

---

## Configuration

| Key | Default | Description |
|---|---|---|
| `:event_consumer_interval_ms` | `5_000` | Catch-up poll interval in ms |
| `@catch_up_limit` | `200` | Max events fetched per catch-up batch |
| `:default_adapter` | `WalletGl.Adapters.TestGlAdapter` | GL posting adapter |
| `:transfer_coa` | `%{debit: "1010", credit: "2010"}` | Chart of accounts for transfer entries |