# Event-Driven Architecture Implementation Guide

## Table of Contents
1. [Overview](#overview)
2. [Architecture Components](#architecture-components)
3. [How It Works](#how-it-works)
4. [Creating New Event Listeners](#creating-new-event-listeners)
5. [Event Lifecycle](#event-lifecycle)
6. [Best Practices](#best-practices)
7. [Management & Monitoring](#management--monitoring)
8. [Troubleshooting](#troubleshooting)

## Overview

This payment processing system implements a **Symfony-style event-driven architecture** using Elixir's native Phoenix.PubSub and GenServer capabilities. The system provides clean separation between business logic (event listeners) and message routing, eliminating duplicate upstream calls and creating a maintainable, extensible architecture.

### Key Benefits
- ✅ **Single Point of Upstream Control**: Eliminates duplicate upstream calls
- ✅ **Clean Separation of Concerns**: Business logic separate from routing
- ✅ **High Extensibility**: Easy to add new acquirer networks
- ✅ **Testable Architecture**: Event listeners can be unit tested independently
- ✅ **Production Ready**: Built on battle-tested Phoenix.PubSub infrastructure

## Architecture Components

### Core Components

```
┌─────────────────────────────────────────────────────────────┐
│                    Event-Driven Architecture                │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  IncomingMessageProcessor ──► EventDispatcher              │
│                                      │                      │
│                                      ▼                      │
│                            ┌─────────────────┐              │
│                            │  Event Listeners │              │
│                            │  • YSP Listener  │              │
│                            │  • VISA Listener │              │
│                            │  • Custom...     │              │
│                            └─────────────────┘              │
│                                      │                      │
│                                      ▼                      │
│                            Single Upstream Router           │
│                                                             │
└─────────────────────────────────────────────────────────────┘
```

### File Structure
```
lib/da_product_app/
├── events/
│   ├── event_dispatcher.ex          # Core event management
│   ├── event_registry.ex            # Listener discovery
│   └── event_listener_behaviour.ex  # Behaviour contract
├── acquirer/
│   └── ysp/
│       └── transaction_event_listener.ex  # YSP business logic
├── startup/
│   ├── event_system_initializer.ex  # Startup logic
│   └── graceful_shutdown.ex         # Shutdown logic
└── switch/
    └── incoming_message_processor.ex # Message routing
```

## How It Works

### 1. Message Flow Overview

```mermaid
graph TD
    A[Device/POS] --> B[IncomingMessageProcessor]
    B --> C[EventDispatcher]
    C --> D[Event: message_received]
    D --> E[Event: before_validation]
    E --> F[Event: after_validation]
    F --> G[Event: before_business_logic]
    G --> H[YSP TransactionEventListener]
    H --> I[Event: after_business_logic]
    I --> J[Event: before_upstream_routing]
    J --> K[Single Upstream Router]
    K --> L[Event: after_upstream_sent]
    L --> M[Event: response_received]
    M --> N[Response to Device]
```

### 2. Event Lifecycle (8 Events)

The system dispatches events at key points in the payment processing pipeline:

1. **`message_received`** - Raw message received from device
2. **`before_validation`** - Before message validation
3. **`after_validation`** - After message validation
4. **`before_business_logic`** - Before acquirer-specific processing
5. **`after_business_logic`** - After acquirer-specific processing
6. **`before_upstream_routing`** - Before routing to upstream
7. **`after_upstream_sent`** - After message sent to upstream
8. **`response_received`** - When response comes back from upstream

### 3. Core Classes

#### EventDispatcher
```elixir
# Core event management using Phoenix.PubSub
GenServer.call(EventDispatcher, {:dispatch, :before_business_logic, payload})
```

#### EventListenerBehaviour  
```elixir
@callback handle_event(event :: atom(), payload :: map()) :: 
  {:ok, map()} | {:error, term()} | :ignore
```

#### Event Listeners
Modules that implement business logic for specific acquirer networks.

## Creating New Event Listeners

### Step 1: Implement the Behaviour

Create a new listener module that implements `EventListenerBehaviour`:

```elixir
# lib/da_product_app/acquirer/visa/transaction_event_listener.ex
defmodule DaProductApp.Acquirer.VISA.TransactionEventListener do
  @moduledoc """
  VISA-specific transaction processing event listener.
  
  Handles VISA network transactions by enriching messages with
  VISA-specific fields and business logic.
  """
  
  @behaviour DaProductApp.Events.EventListenerBehaviour
  
  require Logger
  alias DaProductApp.MercuryISO8583.Packagers.ISOMsg
  alias DaProductApp.Acquirer

  # Configuration
  @visa_acquirer_id 2
  @visa_nii "001"  # VISA Network Identification Number
  
  # Priority (1-100, higher = more priority)  
  @priority 15
  
  @doc """
  Returns the events this listener subscribes to.
  """
  @impl true
  def subscribed_events, do: [:before_business_logic]
  
  @doc """
  Returns the listener priority (higher number = higher priority).
  """
  @impl true  
  def get_priority, do: @priority
  
  @doc """
  Handles business logic events for VISA transactions.
  """
  @impl true
  def handle_event(:before_business_logic, payload) do
    iso_message = Map.get(payload, :iso_message)
    channel_context = Map.get(payload, :channel_context, %{})
    
    # Only process VISA transactions
    if is_visa_transaction?(iso_message) do
      Logger.info("Processing VISA transaction MTI: #{ISOMsg.get_mti(iso_message)}")
      
      case process_visa_business_logic(iso_message, channel_context) do
        {:ok, enriched_message, transaction_data} ->
          updated_payload = payload
          |> Map.put(:iso_message, enriched_message)
          |> Map.put(:visa_transaction_data, transaction_data)
          |> Map.put(:channel_context, Map.put(channel_context, :network_type, :acquirer))
          
          {:ok, updated_payload}
          
        {:error, reason} ->
          Logger.error("VISA transaction processing failed: #{inspect(reason)}")
          {:error, reason}
      end
    else
      # Not a VISA transaction, pass through unchanged
      {:ok, payload}
    end
  end
  
  # Handle other events if needed
  @impl true
  def handle_event(_event, payload), do: {:ok, payload}
  
  # Private functions
  
  defp is_visa_transaction?(iso_message) do
    case ISOMsg.get(iso_message, 24) do  # NII field
      @visa_nii -> true
      _ -> false
    end
  end
  
  defp process_visa_business_logic(iso_message, channel_context) do
    with {:ok, terminal} <- get_terminal_config(iso_message),
         {:ok, enriched_message} <- enrich_visa_message(iso_message, terminal),
         {:ok, transaction_data} <- build_transaction_data(enriched_message, terminal) do
      
      Logger.info("VISA transaction processed successfully")
      {:ok, enriched_message, transaction_data}
    else
      error -> error
    end
  end
  
  defp get_terminal_config(iso_message) do
    terminal_id = ISOMsg.get(iso_message, 41)  # Terminal ID
    
    case Acquirer.get_acquirer_terminal(terminal_id, @visa_acquirer_id) do
      %{} = terminal -> {:ok, terminal}
      nil -> {:error, {:terminal_not_found, terminal_id}}
    end
  end
  
  defp enrich_visa_message(iso_message, terminal) do
    # Add VISA-specific fields
    enriched = iso_message
    |> ISOMsg.set(18, "5999")  # Merchant Category Code
    |> ISOMsg.set(22, "051")   # POS Entry Mode
    |> ISOMsg.set(25, "00")    # POS Condition Code
    |> ISOMsg.set(32, terminal.fiid)  # Acquiring Institution ID
    
    {:ok, enriched}
  end
  
  defp build_transaction_data(iso_message, terminal) do
    transaction_data = %{
      "acquirer_id" => @visa_acquirer_id,
      "network_type" => "VISA", 
      "terminal_id" => terminal.tid,
      "merchant_id" => terminal.mid,
      "processing_date" => Date.utc_today() |> Date.to_string() |> String.replace("-", ""),
      "processing_time" => Time.utc_now() |> Time.to_string() |> String.slice(0..5) |> String.replace(":", ""),
      "currency_code" => terminal.currency_code,
      "mcc_code" => terminal.mcc_code
    }
    
    {:ok, transaction_data}
  end
end
```

### Step 2: Register the Listener

#### Option A: Programmatic Registration
Add to `EventSystemInitializer`:

```elixir
# lib/da_product_app/startup/event_system_initializer.ex
def register_configured_listeners do
  listeners = [
    DaProductApp.Acquirer.YSP.TransactionEventListener,
    DaProductApp.Acquirer.VISA.TransactionEventListener,  # Add new listener
    # Add more listeners here
  ]
  
  results = Enum.map(listeners, fn listener_module ->
    case EventDispatcher.register_listener(listener_module) do
      :ok -> {:ok, listener_module}
      error -> {:error, {listener_module, error}}
    end
  end)
  
  # Process results...
end
```

#### Option B: Configuration-Based Registration
Add to `config/event_listeners.exs`:

```elixir
# config/event_listeners.exs
import Config

config :da_product_app, :event_listeners, [
  %{
    module: DaProductApp.Acquirer.YSP.TransactionEventListener,
    enabled: true,
    priority: 10
  },
  %{
    module: DaProductApp.Acquirer.VISA.TransactionEventListener,
    enabled: true, 
    priority: 15
  }
]
```

### Step 3: Create Tests

```elixir
# test/da_product_app_test/acquirer/visa/transaction_event_listener_test.exs
defmodule DaProductAppTest.Acquirer.VISA.TransactionEventListenerTest do
  use ExUnit.Case
  
  alias DaProductApp.Acquirer.VISA.TransactionEventListener
  alias DaProductApp.MercuryISO8583.Packagers.ISOMsg
  
  describe "VISA TransactionEventListener" do
    test "processes VISA transactions with NII 001" do
      # Create VISA transaction
      iso_message = ISOMsg.new("0200")
      |> ISOMsg.set(24, "001")  # VISA NII
      |> ISOMsg.set(41, "VISA_TERMINAL")
      
      payload = %{
        iso_message: iso_message,
        channel_context: %{}
      }
      
      # Create test terminal
      {:ok, _terminal} = create_test_terminal("VISA_TERMINAL", 2)
      
      # Test event handling
      {:ok, result} = TransactionEventListener.handle_event(:before_business_logic, payload)
      
      assert Map.has_key?(result, :visa_transaction_data)
      assert result.channel_context.network_type == :acquirer
    end
    
    test "ignores non-VISA transactions" do
      # Create non-VISA transaction  
      iso_message = ISOMsg.new("0200")
      |> ISOMsg.set(24, "782")  # YSP NII
      
      payload = %{iso_message: iso_message, channel_context: %{}}
      
      {:ok, result} = TransactionEventListener.handle_event(:before_business_logic, payload)
      
      # Should pass through unchanged
      assert result == payload
      refute Map.has_key?(result, :visa_transaction_data)
    end
  end
  
  defp create_test_terminal(tid, acquirer_id) do
    DaProductApp.Acquirer.upsert_terminal(%{
      tid: tid,
      acquirer_id: acquirer_id,
      status: "ACTIVE",
      # ... other fields
    })
  end
end
```

## Event Lifecycle

### Detailed Event Flow

```elixir
# 1. Message arrives at IncomingMessageProcessor
def process_message(iso_message, channel_context) do
  payload = %{
    iso_message: iso_message,
    channel_context: channel_context,
    timestamp: DateTime.utc_now()
  }
  
  # 2. Dispatch through event pipeline
  with {:ok, payload} <- EventDispatcher.dispatch(:message_received, payload),
       {:ok, payload} <- EventDispatcher.dispatch(:before_validation, payload),
       {:ok, validated_payload} <- validate_message(payload),
       {:ok, payload} <- EventDispatcher.dispatch(:after_validation, validated_payload),
       {:ok, payload} <- EventDispatcher.dispatch(:before_business_logic, payload),
       {:ok, payload} <- EventDispatcher.dispatch(:after_business_logic, payload),
       {:ok, payload} <- EventDispatcher.dispatch(:before_upstream_routing, payload),
       {:ok, response} <- route_to_upstream_single_point(payload),
       {:ok, payload} <- EventDispatcher.dispatch(:after_upstream_sent, Map.put(payload, :response, response)),
       {:ok, final_payload} <- EventDispatcher.dispatch(:response_received, payload) do
    
    {:ok, final_payload}
  else
    error -> handle_error(error, payload)
  end
end
```

### Event Listener Execution Order

1. **Priority-based ordering**: Higher priority listeners execute first
2. **Error handling**: If a listener fails, subsequent listeners still execute
3. **Payload transformation**: Each listener can modify the payload
4. **Short-circuiting**: Listeners can return `:ignore` to skip processing

## Best Practices

### 1. Event Listener Design

#### ✅ Do's
```elixir
# Keep listeners focused on single responsibility
defmodule MyAcquirer.TransactionEventListener do
  @behaviour EventListenerBehaviour
  
  # Clear, descriptive names
  def handle_event(:before_business_logic, payload) do
    # Process only relevant transactions
    if is_my_network?(payload.iso_message) do
      process_transaction(payload)
    else
      {:ok, payload}  # Pass through unchanged
    end
  end
  
  # Always handle errors gracefully
  defp process_transaction(payload) do
    try do
      # Business logic here
      {:ok, enriched_payload}
    rescue
      exception ->
        Logger.error("Transaction processing failed: #{inspect(exception)}")
        {:error, {:processing_failed, exception}}
    end
  end
end
```

#### ❌ Don'ts
```elixir
# Don't process irrelevant events
def handle_event(event, payload) do
  # Process every event (inefficient)
  heavy_processing(payload)
end

# Don't ignore error handling
def handle_event(event, payload) do
  # This can crash the system
  dangerous_operation(payload)
  {:ok, payload}
end

# Don't make listeners dependent on each other
def handle_event(event, payload) do
  # Don't assume other listeners ran first
  other_listener_data = payload.other_network_data  # May not exist!
end
```

### 2. Configuration Management

#### Event Listener Configuration
```elixir
# config/event_listeners.exs
import Config

config :da_product_app, :event_listeners, [
  %{
    module: DaProductApp.Acquirer.YSP.TransactionEventListener,
    enabled: true,
    priority: 10,
    config: %{
      acquirer_id: 1,
      timeout_ms: 5000,
      retry_attempts: 3
    }
  },
  %{
    module: DaProductApp.Acquirer.VISA.TransactionEventListener,
    enabled: System.get_env("VISA_ENABLED", "true") == "true",
    priority: 15,
    config: %{
      acquirer_id: 2,
      timeout_ms: 3000
    }
  }
]
```

#### Environment-specific Configuration
```elixir
# config/dev.exs
config :da_product_app, :event_listeners,
  # All listeners enabled in development
  Enum.map(Application.get_env(:da_product_app, :event_listeners), fn listener ->
    Map.put(listener, :enabled, true)
  end)

# config/prod.exs  
config :da_product_app, :event_listeners,
  # Production configuration with environment variables
  [
    %{
      module: DaProductApp.Acquirer.YSP.TransactionEventListener,
      enabled: System.get_env("YSP_ENABLED") == "true",
      priority: String.to_integer(System.get_env("YSP_PRIORITY", "10"))
    }
    # Other listeners...
  ]
```

### 3. Testing Strategies

#### Unit Testing Event Listeners
```elixir
defmodule MyEventListenerTest do
  use ExUnit.Case
  use DaProductApp.DataCase  # For database access
  
  alias MyApp.MyEventListener
  
  describe "handle_event/2" do
    setup do
      # Create test data
      {:ok, terminal} = create_test_terminal()
      %{terminal: terminal}
    end
    
    test "processes valid transaction", %{terminal: terminal} do
      payload = create_test_payload(terminal.tid)
      
      {:ok, result} = MyEventListener.handle_event(:before_business_logic, payload)
      
      assert_transaction_enriched(result)
    end
    
    test "handles missing terminal gracefully" do
      payload = create_test_payload("INVALID_TERMINAL")
      
      {:error, reason} = MyEventListener.handle_event(:before_business_logic, payload)
      
      assert {:terminal_not_found, "INVALID_TERMINAL"} = reason
    end
  end
end
```

#### Integration Testing Event Flow
```elixir
defmodule EventSystemIntegrationTest do
  use ExUnit.Case
  
  test "complete event flow processes transaction" do
    # Create test message
    iso_message = create_test_transaction()
    
    # Process through complete pipeline
    {:ok, result} = IncomingMessageProcessor.process_message(iso_message, %{})
    
    # Assert all enrichment happened
    assert result.ysp_transaction_data != nil
    assert result.channel_context.network_type == :acquirer
    
    # Verify no duplicate upstream calls
    assert_single_upstream_call()
  end
end
```

### 4. Error Handling Patterns

#### Graceful Degradation
```elixir
def handle_event(:before_business_logic, payload) do
  case process_with_timeout(payload, 5000) do
    {:ok, result} -> 
      {:ok, result}
    {:error, :timeout} ->
      Logger.warning("Processing timeout, using fallback")
      {:ok, add_fallback_data(payload)}
    {:error, reason} ->
      Logger.error("Processing failed: #{inspect(reason)}")
      {:error, reason}
  end
end
```

#### Circuit Breaker Pattern
```elixir
defmodule MyListener do
  @circuit_breaker_key :my_listener_circuit
  
  def handle_event(event, payload) do
    case CircuitBreaker.call(@circuit_breaker_key, fn ->
      process_transaction(payload)
    end) do
      {:ok, result} -> {:ok, result}
      {:error, :circuit_open} ->
        Logger.warning("Circuit breaker open, skipping processing")
        {:ok, payload}  # Graceful degradation
    end
  end
end
```

## Management & Monitoring

### 1. Runtime Management

#### EventDispatcher Health Check
```elixir
# lib/da_product_app/events/event_dispatcher.ex (add to existing module)
def get_health_status do
  %{
    status: :healthy,
    registered_listeners: count_registered_listeners(),
    events_processed: get_event_metrics(),
    uptime: get_uptime(),
    last_error: get_last_error()
  }
end

def get_listener_status do
  get_listeners()
  |> Enum.map(fn {event, listeners} ->
    {event, Enum.map(listeners, fn listener ->
      %{
        module: listener.module,
        priority: listener.priority,
        status: check_listener_health(listener.module)
      }
    end)}
  end)
  |> Enum.into(%{})
end
```

#### Management API
```elixir
# lib/da_product_app/management/event_system_manager.ex
defmodule DaProductApp.Management.EventSystemManager do
  @moduledoc """
  Management API for the event system.
  Provides runtime control and monitoring capabilities.
  """
  
  alias DaProductApp.Events.EventDispatcher
  
  def reload_listeners do
    with :ok <- unregister_all_listeners(),
         :ok <- EventSystemInitializer.register_event_listeners() do
      {:ok, :reloaded}
    end
  end
  
  def disable_listener(listener_module) do
    EventDispatcher.unregister_listener(listener_module)
  end
  
  def enable_listener(listener_module) do
    EventDispatcher.register_listener(listener_module)
  end
  
  def get_system_metrics do
    %{
      event_dispatcher: EventDispatcher.get_health_status(),
      listeners: EventDispatcher.get_listener_status(),
      recent_errors: get_recent_errors()
    }
  end
end
```

### 2. Monitoring & Observability

#### Telemetry Integration
```elixir
# lib/da_product_app/events/event_dispatcher.ex (add telemetry)
defp dispatch_to_listener(listener, event, payload) do
  start_time = System.monotonic_time()
  
  try do
    result = listener.module.handle_event(event, payload)
    
    # Emit success telemetry
    :telemetry.execute(
      [:da_product_app, :event_listener, :success],
      %{duration: System.monotonic_time() - start_time},
      %{listener: listener.module, event: event}
    )
    
    result
  rescue
    exception ->
      # Emit error telemetry
      :telemetry.execute(
        [:da_product_app, :event_listener, :error],
        %{duration: System.monotonic_time() - start_time},
        %{listener: listener.module, event: event, exception: exception}
      )
      
      {:error, {:listener_exception, listener.module, exception}}
  end
end
```

#### Metrics Collection
```elixir
# lib/da_product_app/telemetry.ex (add to existing module)
def handle_event([:da_product_app, :event_listener, :success], measurements, metadata, _config) do
  :prometheus_histogram.observe(
    :event_listener_duration_seconds,
    [listener: metadata.listener, event: metadata.event],
    measurements.duration / 1_000_000  # Convert to seconds
  )
  
  :prometheus_counter.inc(
    :event_listener_processed_total,
    [listener: metadata.listener, event: metadata.event, status: "success"]
  )
end

def handle_event([:da_product_app, :event_listener, :error], measurements, metadata, _config) do
  :prometheus_counter.inc(
    :event_listener_processed_total,
    [listener: metadata.listener, event: metadata.event, status: "error"]
  )
  
  :prometheus_counter.inc(
    :event_listener_errors_total,
    [listener: metadata.listener, event: metadata.event, 
     exception: metadata.exception.__struct__]
  )
end
```

#### Dashboard Queries
```elixir
# Example Prometheus/Grafana queries:

# Event processing rate
rate(event_listener_processed_total[5m])

# Error rate by listener
rate(event_listener_errors_total[5m]) / rate(event_listener_processed_total[5m])

# Processing duration by listener
histogram_quantile(0.95, rate(event_listener_duration_seconds_bucket[5m]))

# Listener availability
up{job="da_product_app"}
```

### 3. Logging & Debugging

#### Structured Logging
```elixir
def handle_event(:before_business_logic, payload) do
  Logger.metadata([
    event: :before_business_logic,
    listener: __MODULE__,
    transaction_id: get_transaction_id(payload),
    mti: get_mti(payload)
  ])
  
  Logger.info("Processing transaction", %{
    terminal_id: get_terminal_id(payload),
    amount: get_amount(payload)
  })
  
  case process_transaction(payload) do
    {:ok, result} ->
      Logger.info("Transaction processed successfully")
      {:ok, result}
    {:error, reason} ->
      Logger.error("Transaction processing failed", %{reason: reason})
      {:error, reason}
  end
end
```

#### Debug Mode
```elixir
# config/dev.exs
config :logger, level: :debug

config :da_product_app, :event_system,
  debug_mode: true,
  trace_events: [:before_business_logic, :after_business_logic]
```

## Troubleshooting

### Common Issues & Solutions

#### 1. Listener Not Receiving Events
```elixir
# Check if listener is registered
EventDispatcher.get_listeners()

# Check if events are being dispatched
Logger.debug("Available listeners for :before_business_logic: #{inspect(EventDispatcher.get_listeners()[:before_business_logic])}")

# Verify listener implements behaviour correctly
Code.ensure_loaded(MyListener)
function_exported?(MyListener, :handle_event, 2)
```

#### 2. Database Connection Errors in Tests
```elixir
# In test setup
setup do
  # Allow EventDispatcher to access database
  event_dispatcher_pid = Process.whereis(EventDispatcher)
  if event_dispatcher_pid do
    Ecto.Adapters.SQL.Sandbox.allow(Repo, self(), event_dispatcher_pid)
  end
  
  :ok
end
```

#### 3. Performance Issues
```elixir
# Profile event processing
:fprof.apply(&EventDispatcher.dispatch/2, [:before_business_logic, payload])

# Use async processing for heavy operations
def handle_event(event, payload) do
  Task.start(fn -> heavy_background_processing(payload) end)
  {:ok, payload}  # Don't block event pipeline
end
```

#### 4. Listener Priority Issues
```elixir
# Verify listener priorities
EventDispatcher.get_listeners()[:before_business_logic]
|> Enum.map(fn listener -> {listener.module, listener.priority} end)
|> Enum.sort_by(fn {_module, priority} -> -priority end)  # Highest first
```

### Debugging Commands

```elixir
# In IEx console:

# Check system health
DaProductApp.Management.EventSystemManager.get_system_metrics()

# Test specific listener
payload = %{iso_message: test_message, channel_context: %{}}
MyListener.handle_event(:before_business_logic, payload)

# Trace event flow
EventDispatcher.dispatch(:before_business_logic, payload, debug: true)

# Check EventDispatcher state
:sys.get_state(EventDispatcher)
```

## Summary

This event-driven architecture provides a robust, scalable foundation for payment processing systems. By following these guidelines and best practices, you can:

- ✅ Create maintainable, testable event listeners
- ✅ Ensure proper separation of concerns
- ✅ Implement effective monitoring and debugging
- ✅ Handle errors gracefully
- ✅ Scale the system as business requirements grow

The system eliminates the duplicate upstream call problem while providing a clean, extensible architecture that can accommodate any number of acquirer networks and business logic requirements.