# Phase 4.1 Monitoring Infrastructure Deployment Guide
# Mercury Device Middleware - Production Monitoring Setup

## Overview

This document provides comprehensive deployment instructions for the Phase 4.1 monitoring infrastructure implemented for the Mercury Device Middleware reversal system.

### Components Implemented

1. **ReversalMetrics** - Real-time telemetry collector
2. **PrometheusExporter** - Metrics endpoint for scraping
3. **Grafana Dashboard** - Visual monitoring interface
4. **Alert Rules** - Proactive issue detection
5. **Application Integration** - Seamless monitoring integration

## Deployment Steps

### 1. Application Integration

The monitoring infrastructure is automatically integrated into the application:

```elixir
# Supervision Tree (lib/da_product_app/application.ex)
children = [
  # ... existing children ...
  {DaProductApp.Telemetry.ReversalMetrics, []}
]

# Router Integration (lib/da_product_app_web/router.ex)
scope "/monitoring", DaProductAppWeb do
  get "/metrics", PrometheusController, :metrics
end
```

### 2. Prometheus Setup

**Installation:**
```bash
# Download and install Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
cd prometheus-*
```

**Configuration:**
```bash
# Copy our configuration
cp config/prometheus/prometheus.yml ./prometheus.yml
cp config/prometheus/reversal_alert_rules.yml ./reversal_alert_rules.yml
```

**Start Prometheus:**
```bash
./prometheus --config.file=prometheus.yml --storage.tsdb.path=./data --web.console.templates=consoles --web.console.libraries=console_libraries
```

### 3. Grafana Setup

**Installation:**
```bash
# Install Grafana
sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt-get update
sudo apt-get install grafana
```

**Configuration:**
```bash
# Start Grafana service
sudo systemctl start grafana-server
sudo systemctl enable grafana-server
```

**Dashboard Import:**
1. Access Grafana at http://localhost:3000 (admin/admin)
2. Add Prometheus data source: http://localhost:9090
3. Import dashboard from: `config/grafana/reversal_dashboard.json`

### 4. Application Deployment

**Development Environment:**
```bash
# Start the application with monitoring
mix phx.server
```

**Production Environment:**
```bash
# Build release
MIX_ENV=prod mix release

# Start with monitoring enabled
_build/prod/rel/da_product_app/bin/da_product_app start
```

## Monitoring Endpoints

### Metrics Endpoint
- **URL:** `http://localhost:4000/monitoring/metrics`
- **Format:** Prometheus exposition format
- **Authentication:** None (configure as needed)

### Health Checks
- **Application:** `http://localhost:4000/health`
- **Prometheus:** `http://localhost:9090/-/healthy`
- **Grafana:** `http://localhost:3000/api/health`

## Key Metrics

### Reversal System Metrics

| Metric | Type | Description |
|--------|------|-------------|
| `reversal_total` | Counter | Total reversal attempts |
| `reversal_duration_seconds` | Histogram | Reversal processing time |
| `reversal_queue_size` | Gauge | Current queue size |
| `reversal_system_health_score` | Gauge | System health (0-1) |

### Network Metrics

| Metric | Type | Description |
|--------|------|-------------|
| `reversal_network_transmission_total` | Counter | Network transmission attempts |
| `reversal_network_duration_seconds` | Histogram | Network response time |
| `reversal_network_errors_total` | Counter | Network transmission errors |

### Packet Generation Metrics

| Metric | Type | Description |
|--------|------|-------------|
| `reversal_packet_generation_total` | Counter | Packet generation attempts |
| `reversal_packet_generation_duration_seconds` | Histogram | Packet generation time |
| `reversal_packet_size_bytes` | Histogram | Generated packet size |

## Alert Configuration

### Critical Alerts
- **ReversalFailureRateHigh:** > 10% failure rate for 2+ minutes
- **NetworkTimeoutRateHigh:** > 5% timeout rate for 2+ minutes  
- **ReversalSystemHealthLow:** Health score < 0.8 for 1+ minute

### Warning Alerts
- **ReversalQueueBacklog:** Queue size > 100 for 5+ minutes
- **ReversalProcessingTimeHigh:** 95th percentile > 30 seconds
- **PacketGenerationFailureRate:** > 2% failure rate

### Info Alerts
- **DuplicateReversalAttempts:** High number of duplicate attempts
- **OrphanedReversalsDetected:** Data cleanup operations

## Dashboard Panels

### System Health Overview
- **Health Score:** Real-time system health indicator
- **Success Rate:** Percentage of successful reversals
- **Active Queues:** Current processing queue status

### Performance Metrics
- **Processing Time:** P50, P95, P99 percentiles
- **Queue Sizes:** Real-time queue monitoring
- **Resource Usage:** Memory and connection pool status

### Error Analysis
- **Failure Rates:** Categorized failure tracking
- **Error Distribution:** Network vs application errors
- **Timeout Analysis:** Network timeout patterns

## Troubleshooting

### Common Issues

**1. Metrics Endpoint Not Accessible**
```bash
# Check if application is running
curl http://localhost:4000/monitoring/metrics

# Verify router configuration
grep -r "monitoring" lib/da_product_app_web/router.ex
```

**2. Prometheus Not Scraping**
```bash
# Check Prometheus targets
curl http://localhost:9090/api/v1/targets

# Verify configuration
./prometheus --config.file=prometheus.yml --web.enable-lifecycle
```

**3. Grafana Dashboard Empty**
```bash
# Test Prometheus data source
curl "http://localhost:9090/api/v1/query?query=up"

# Check Grafana logs
sudo journalctl -u grafana-server -f
```

### Performance Tuning

**Metrics Collection:**
- Adjust collection interval in `ReversalMetrics`
- Configure telemetry attachment scope
- Optimize histogram bucket ranges

**Prometheus Storage:**
- Configure retention period based on needs
- Set up remote storage for long-term retention
- Optimize scrape intervals

**Grafana Performance:**
- Use appropriate query time ranges
- Configure refresh intervals
- Set up alert notification channels

## Security Considerations

### Access Control
- Configure authentication for monitoring endpoints
- Set up Grafana user management
- Implement network-level access controls

### Data Privacy
- Ensure no sensitive data in metrics labels
- Configure data retention policies
- Implement secure transmission (HTTPS)

### Monitoring Infrastructure
- Monitor the monitoring systems themselves
- Set up backup and recovery procedures
- Configure high availability if needed

## Maintenance

### Regular Tasks
- Monitor storage usage and clean up old data
- Update dashboard configurations as needed
- Review and adjust alert thresholds
- Backup Grafana configurations and dashboards

### Capacity Planning
- Monitor metrics collection overhead
- Plan for storage growth
- Scale monitoring infrastructure as needed

## Integration Testing

### Verification Steps
1. **Start all services** (application, Prometheus, Grafana)
2. **Generate test reversals** using development tools
3. **Verify metrics collection** at `/monitoring/metrics`
4. **Confirm Prometheus scraping** in targets page
5. **Validate dashboard display** in Grafana
6. **Test alert firing** by simulating failure conditions

### Test Scenarios
- Normal reversal processing
- High volume processing
- Network timeout simulation
- Error condition handling
- System recovery scenarios

## Phase 4.2 Preparation

This Phase 4.1 infrastructure provides the foundation for:
- **Advanced alerting** with escalation rules
- **Operational runbooks** with automated response
- **Production deployment** with feature flags
- **Comprehensive testing** with monitoring validation

The monitoring infrastructure is now ready for production use and provides comprehensive observability for the Phase 3 reversal system.