# AutoPush Process - Comprehensive Manual Test Cases

## Document Overview
This document provides comprehensive test cases for manual testing of the AutoPush feature in the Terminal Management System (TMS). AutoPush automatically detects missing version configurations (parameters, EMV, keys, application) on terminals and triggers configuration pushes via MQTT.

---

## 1. TEST ENVIRONMENT SETUP

### Prerequisites
- MySQL running with `lic_project_cicd` database
- Phoenix server running on `http://localhost:4000`
- MQTT broker accessible and configured
- Postman or curl for API testing
- Database client for MySQL verification

### Initial Database State Verification
```bash
# Connect to MySQL
mysql -u root -pdataaegis123 -D lic_project_cicd

# Verify tables exist
SHOW TABLES LIKE '%terminal%';
SHOW TABLES LIKE '%push_log%';
SHOW TABLES LIKE '%parameter%';

# Check sample data
SELECT COUNT(*) FROM tms_terminals;
SELECT COUNT(*) FROM parameter_templates;
SELECT COUNT(*) FROM parameter_push_logs;
```

### Test Data Setup
Before running tests, ensure you have:
1. **At least one active terminal** with `serial_number` and `vendor`, `model`
2. **Parameter templates** matching vendor/model combinations
3. **MQTT connection** configured in `config/dev.exs`

---

## 2. SCENARIO 1: MISSING VERSION DETECTION AND AUTO-PUSH

### Test Case 1.1: Detect Missing Version - Parameter Config
**Description**: When a terminal sends status with NULL parameter_config_version, AutoPush should be triggered.

**Setup**:
```sql
-- Create or select a terminal with missing parameter_config_version
UPDATE tms_terminals 
SET parameter_config_version = NULL 
WHERE serial_number = '61250904380091';

-- Verify it's NULL
SELECT serial_number, parameter_config_version FROM tms_terminals 
WHERE serial_number = '61250904380091';
```

**Test Steps**:
1. Terminal publishes MQTT message to `tms/status/61250904380091` with missing `parameter_config_version`:
```json
{
  "oid": "OID123",
  "sn": "61250904380091",
  "vendor": "MoreFun",
  "model": "MF919",
  "uploadTime": "2026-01-15T10:30:00Z",
  "org.device": [
    {"version": null, "type": "parameter_config"}
  ]
}
```

2. Monitor logs for trigger message:
```bash
tail -f logs/app.log | grep -E "Detected missing versions|AutoPushService: Triggering"
```

3. Expected logs:
```
[info] Detected missing versions for 61250904380091, triggering auto-push
[info] AutoPushService: Triggering auto-push for 61250904380091 (MoreFun MF919) - missing version
```

**Verification**:
- [ ] Log shows auto-push was triggered
- [ ] `parameter_push_logs` table has new entry with:
  - `config_type: 'parameter'`
  - `trigger_reason: 'missing_version'`
  - `status: 'pending'`
  - `device_vendor: 'MoreFun'`
  - `device_model: 'MF919'`

---

### Test Case 1.2: Detect Missing Version - EMV Config
**Description**: When a terminal sends status with NULL emv_config_version, AutoPush should be triggered.

**Setup**:
```sql
UPDATE tms_terminals 
SET emv_config_version = NULL 
WHERE serial_number = '61250904380091';
```

**Test Steps**:
Same as Test Case 1.1, but with:
```json
{
  "org.device": [
    {"version": null, "type": "emv_config"}
  ]
}
```

**Expected Results**:
- Auto-push triggered with `config_type: 'emv_config'`
- New push log entry created

---

### Test Case 1.3: Detect Missing Version - Keys Config
**Description**: Keys config auto-push requires RKI endpoint integration.

**Setup**:
```sql
UPDATE tms_terminals 
SET keys_config_version = NULL, kek_kcv = '112233', slot_number = 1
WHERE serial_number = '61250904380091';
```

**Test Steps**:
Same as Test Case 1.1, but with:
```json
{
  "org.device": [
    {"version": null, "type": "keys_config"}
  ]
}
```

**Expected Results**:
- [ ] KeysConfigService calls RKI endpoint
- [ ] `keys.json` file created at `priv/ota/61250904380091/keys.json`
- [ ] Push log entry with:
  - `config_type: 'keys_config'`
  - `file_path: 'priv/ota/61250904380091/keys.json'`
  - `file_size: [actual_size]`
  - `checksum: [SHA256_hash]`

**File Verification**:
```bash
# Check file exists and has valid JSON
ls -lh priv/ota/61250904380091/keys.json
cat priv/ota/61250904380091/keys.json | jq .
```

---

### Test Case 1.4: Detect Missing Version - Application
**Description**: Application config auto-push.

**Setup**:
```sql
UPDATE tms_terminals 
SET app_version = NULL
WHERE serial_number = '61250904380091';
```

**Test Steps**:
Same as Test Case 1.1, with application version field.

**Expected Results**:
- Push log with `config_type: 'application'`

---

### Test Case 1.5: Multiple Missing Versions (All configs)
**Description**: When multiple version fields are missing, AutoPush should push all configs.

**Setup**:
```sql
UPDATE tms_terminals 
SET parameter_config_version = NULL,
    emv_config_version = NULL,
    keys_config_version = NULL,
    app_version = NULL
WHERE serial_number = '61250904380091';
```

**Test Steps**:
1. Send status message with all version fields missing
2. Monitor logs for push operations

**Expected Results**:
- [ ] 4 push logs created (one for each config type)
- [ ] All have `trigger_reason: 'missing_version'`
- [ ] All have correct `config_type` values
- [ ] No errors in logs

**Verification Query**:
```sql
SELECT config_type, status, trigger_reason, created_at 
FROM parameter_push_logs 
WHERE device_vendor = 'MoreFun' AND device_model = 'MF919'
ORDER BY created_at DESC 
LIMIT 4;
```

---

## 3. TEMPLATE MATCHING AND VALIDATION

### Test Case 2.1: Find Best Template - Active Template Preferred
**Description**: When multiple templates exist, active templates should be preferred.

**Setup**:
```sql
-- Create or verify multiple templates for same vendor/model
INSERT INTO parameter_templates (name, vendor, model, is_active, is_default, created_by_id, inserted_at, updated_at)
VALUES 
  ('MF919 Active Template', 'MoreFun', 'MF919', true, false, 1, NOW(), NOW()),
  ('MF919 Default Template', 'MoreFun', 'MF919', true, true, 1, NOW(), NOW());

-- Select active template
SELECT id, name, is_active, is_default FROM parameter_templates 
WHERE vendor = 'MoreFun' AND model = 'MF919';
```

**Test Steps**:
1. Trigger auto-push with missing versions
2. Check logs to see which template was selected

**Expected Results**:
- [ ] Active non-default template is preferred
- [ ] Log shows: `[info] AutoPushService: Created push log X for parameters (template: Y)`

---

### Test Case 2.2: Fallback to Default Template
**Description**: If no active template found, fallback to default template.

**Setup**:
```sql
-- Mark active templates as inactive
UPDATE parameter_templates 
SET is_active = false
WHERE vendor = 'MoreFun' AND model = 'MF919' AND is_default = false;

-- Verify there's a default active template
SELECT id, name, is_active, is_default FROM parameter_templates 
WHERE vendor = 'MoreFun' AND model = 'MF919' AND is_active = true;
```

**Test Steps**:
1. Trigger auto-push with missing versions
2. Monitor logs

**Expected Results**:
- [ ] Default active template used
- [ ] Push log created without errors

---

### Test Case 2.3: No Template Found - Error Handling
**Description**: When no template exists for vendor/model, AutoPush should fail gracefully.

**Setup**:
```sql
-- Delete all templates for a non-existent vendor/model
DELETE FROM parameter_templates 
WHERE vendor = 'UnknownVendor' AND model = 'UnknownModel';

-- Verify deletion
SELECT COUNT(*) FROM parameter_templates 
WHERE vendor = 'UnknownVendor' AND model = 'UnknownModel';
```

**Test Steps**:
1. Trigger auto-push for terminal with vendor='UnknownVendor', model='UnknownModel'
2. Check logs

**Expected Results**:
- [ ] Log shows: `[warning] AutoPushService: No template found for UnknownVendor UnknownModel`
- [ ] No push logs created
- [ ] No errors (graceful failure)

---

## 4. PARAMETER PUSH OPERATIONS

### Test Case 3.1: Push Template Parameters
**Description**: Verify parameter values are correctly pushed from template.

**Setup**:
```sql
-- Create template with parameter values
INSERT INTO parameter_templates (name, vendor, model, is_active, is_default, created_by_id, inserted_at, updated_at)
VALUES ('Test Template', 'MoreFun', 'MF919', true, false, 1, NOW(), NOW())
ON DUPLICATE KEY UPDATE id=id;

-- Get template ID
SET @template_id = (SELECT id FROM parameter_templates WHERE name = 'Test Template' LIMIT 1);

-- Add parameter values
-- First, ensure parameter definitions exist
INSERT INTO parameter_definitions (key, description, data_type, category, display_order, inserted_at, updated_at)
VALUES 
  ('COMM_SERVER_ADDRESS', 'Server Address', 'string', 'COMMUNICATIONS', 1, NOW(), NOW()),
  ('COMM_PORT', 'Server Port', 'integer', 'COMMUNICATIONS', 2, NOW(), NOW()),
  ('TRANS_TIMEOUT', 'Transaction Timeout', 'integer', 'TRANSACTIONS', 1, NOW(), NOW())
ON DUPLICATE KEY UPDATE id=id;

-- Insert template values
INSERT INTO parameter_template_values (template_id, parameter_definition_id, value, inserted_at, updated_at)
SELECT @template_id, id, CASE key 
  WHEN 'COMM_SERVER_ADDRESS' THEN '192.168.1.100'
  WHEN 'COMM_PORT' THEN '8080'
  WHEN 'TRANS_TIMEOUT' THEN '60'
END, NOW(), NOW()
FROM parameter_definitions 
WHERE key IN ('COMM_SERVER_ADDRESS', 'COMM_PORT', 'TRANS_TIMEOUT');
```

**Test Steps**:
1. Create terminal with missing parameter_config_version
2. Trigger auto-push
3. Check parameter_push_logs for parameters_sent field

**Verification**:
```sql
SELECT parameters_sent FROM parameter_push_logs 
WHERE config_type = 'parameter' AND device_model = 'MF919'
ORDER BY created_at DESC LIMIT 1;
```

**Expected Results**:
- [ ] `parameters_sent` contains map with all parameter key-value pairs:
```json
{
  "COMM_SERVER_ADDRESS": "192.168.1.100",
  "COMM_PORT": "8080",
  "TRANS_TIMEOUT": "60"
}
```

---

### Test Case 3.2: MQTT Command Format Validation
**Description**: Verify MQTT payload uses correct format for device.

**Setup**:
- Monitor MQTT broker traffic or logs

**Test Steps**:
1. Trigger auto-push
2. Capture MQTT message sent to device

**MQTT Topic**: `/ota/{product_key}/{serial_number}/update`
Example: `/ota/pFppbioOCKlo5c8E/61250904380091/update`

**Expected Payload Format** (via MQTTCommandBuilder):
```json
{
  "type": "tms_command",
  "command": "UPDATE_PARAMS",
  "requestId": "req-1234567890",
  "parameters": {
    "COMM_SERVER_ADDRESS": "192.168.1.100",
    "COMM_PORT": "8080"
  }
}
```

**Verification**:
- [ ] Log shows: `[info] AutoPushService: Sending parameter push to 61250904380091 on topic: /ota/...`
- [ ] Log shows: `[info] AutoPushService: Parameter push sent successfully to 61250904380091`

---

## 5. KEYS CONFIG PUSH - SPECIAL HANDLING

### Test Case 4.1: RKI Endpoint Integration
**Description**: Keys config push requires calling RKI endpoint.

**Setup**:
- Configure RKI endpoint in `config/dev.exs`:
```elixir
config :da_product_app, :rki_endpoint, "http://path-to-rki:port/getRki"
```

**Test Steps**:
1. Terminal with missing keys_config_version
2. Trigger auto-push
3. Monitor RKI endpoint calls

**Expected Results**:
- [ ] HTTP POST sent to RKI endpoint with parameters:
```json
{
  "serialNumber": "61250904380091",
  "model": "MF919",
  "kekKcv": "112233",
  "slotNumber": 1
}
```
- [ ] Response received and parsed:
```json
{
  "rkiKey": "XA9A9A23CD841F63D20D1B17F24BED2CF",
  "rkiKcv": "CB9DEA",
  "slotNumber": 1
}
```

---

### Test Case 4.2: Keys File Generation and Storage
**Description**: Verify keys.json is created with correct content.

**Test Steps**:
1. Trigger keys config auto-push
2. Check file system

**Verification**:
```bash
# Check file exists
ls -lh priv/ota/61250904380091/keys.json

# View content
cat priv/ota/61250904380091/keys.json | jq .
```

**Expected File Content**:
```json
{
  "rki_key": "XA9A9A23CD841F63D20D1B17F24BED2CF",
  "rki_kcv": "CB9DEA",
  "kek_kcv": "112233",
  "slot_number": 1,
  "serial_number": "61250904380091",
  "model": "MF919",
  "vendor": "MoreFun",
  "created_at": "2026-01-15T10:30:00Z",
  "expires_at": "2027-01-15T10:30:00Z"
}
```

**Required Fields Check**:
- [ ] rki_key is 32-character hex string
- [ ] rki_kcv is valid 6-character hex
- [ ] Timestamps are ISO 8601 format
- [ ] File size > 0
- [ ] JSON is valid

---

### Test Case 4.3: Checksum Calculation and Storage
**Description**: File checksum should be calculated and stored in push log.

**Verification**:
```sql
SELECT file_path, file_size, checksum FROM parameter_push_logs 
WHERE config_type = 'keys_config' AND device_serial_number = '61250904380091'
ORDER BY created_at DESC LIMIT 1;
```

**Expected**:
- [ ] `checksum` is SHA256 hex (64 characters)
- [ ] `file_size` matches actual file size
- [ ] Checksum can be manually verified:
```bash
sha256sum priv/ota/61250904380091/keys.json
```

---

### Test Case 4.4: Keys Config MQTT Payload
**Description**: Verify keys config MQTT payload has download URL.

**Expected Payload** (via MQTTCommandBuilder):
```json
{
  "type": "tms_command",
  "command": "LOAD_KEYS",
  "requestId": "req-1234567890",
  "downloadUrl": "http://demo.ctrmv.com/ota/61250904380091/keys.json"
}
```

**Verification**:
- [ ] Log shows: `[info] AutoPushService: Sending keys config to 61250904380091 on topic: /ota/...`

---

## 6. EMV AND APPLICATION CONFIG PUSH

### Test Case 5.1: EMV Config Push
**Description**: Push EMV configuration when version missing.

**Setup**:
```sql
UPDATE tms_terminals 
SET emv_config_version = NULL
WHERE serial_number = '61250904380091';
```

**Test Steps**:
1. Trigger auto-push
2. Monitor logs

**Expected Payload**:
```json
{
  "type": "tms_command",
  "command": "UPDATE_L3_CONFIG",
  "requestId": "req-1234567890"
}
```

**Verification**:
- [ ] Push log created with `config_type: 'emv_config'`
- [ ] MQTT message sent successfully
- [ ] Log: `[info] AutoPushService: emv_config push sent successfully to 61250904380091`

---

### Test Case 5.2: Application Config Push
**Description**: Push application configuration when version missing.

**Setup**:
```sql
UPDATE tms_terminals 
SET app_version = NULL
WHERE serial_number = '61250904380091';
```

**Test Steps**:
1. Trigger auto-push
2. Monitor logs

**Expected Payload**:
```json
{
  "type": "tms_command",
  "command": "UPDATE_APPLICATION",
  "requestId": "req-1234567890"
}
```

**Verification**:
- [ ] Push log created with `config_type: 'application'`
- [ ] MQTT message sent successfully

---

## 7. ERROR HANDLING AND EDGE CASES

### Test Case 6.1: Invalid MQTT Status Message - Malformed JSON
**Description**: Handler should gracefully handle invalid JSON.

**Test Step**:
1. Send malformed JSON to `tms/status/61250904380091`:
```
Invalid JSON: {incomplete message
```

**Expected Results**:
- [ ] Log shows: `[error] Failed to parse status message`
- [ ] No crashes
- [ ] No push logs created

**Verification**:
```bash
grep -i "error\|failed" logs/app.log | tail -20
```

---

### Test Case 6.2: Terminal Not Found in Database
**Description**: When terminal doesn't exist, it should be created.

**Setup**:
```sql
-- Start with new serial number
DELETE FROM tms_terminals WHERE serial_number = 'NEW_SERIAL_999';
```

**Test Steps**:
1. Send status message with new serial number
2. Check database

**Expected Results**:
- [ ] New terminal created automatically
- [ ] Log shows: `[info] AutoPushService: Triggering auto-push for NEW_SERIAL_999...`
- [ ] Terminal record now in database with correct vendor/model

**Verification**:
```sql
SELECT serial_number, vendor, model FROM tms_terminals 
WHERE serial_number = 'NEW_SERIAL_999';
```

---

### Test Case 6.3: Database Insert Failure - Request ID Type Mismatch
**Description**: Ensure request_id field in push log matches expected type (integer).

**Test Steps**:
1. Monitor logs during auto-push
2. Check for type conversion errors

**Expected Result**:
- [ ] Push log insert succeeds without type errors
- [ ] request_id stored correctly as integer

---

### Test Case 6.4: MQTT Publish Failure
**Description**: When MQTT publish fails, error should be logged.

**Setup**:
- Temporarily stop MQTT broker or disconnect

**Test Steps**:
1. Trigger auto-push while MQTT broker is unavailable
2. Monitor logs

**Expected Results**:
- [ ] Log shows: `[error] AutoPushService: Failed to send [config_type] push to [serial]: ...`
- [ ] Push log created with `status: 'pending'` (not sent)
- [ ] System continues operating (no crash)

---

### Test Case 6.5: RKI Endpoint Timeout
**Description**: Handle timeout when calling RKI endpoint.

**Setup**:
- Configure RKI endpoint to a non-responsive address

**Test Steps**:
1. Trigger keys config auto-push
2. Monitor logs

**Expected Results**:
- [ ] Log shows: `[error] AutoPushService: Failed to generate keys.json...`
- [ ] No crash, system continues
- [ ] Terminal status remains valid

---

### Test Case 6.6: Missing Terminal Fields (vendor/model)
**Description**: Status message without vendor/model should be handled.

**Test Step**:
1. Send MQTT status message missing vendor or model fields
2. Check logs

**Expected Results**:
- [ ] Status message still processed
- [ ] Terminal creation succeeds if fields eventually added
- [ ] Auto-push only triggers when vendor AND model are present

---

## 8. VERSION EXTRACTION AND COMPARISON

### Test Case 7.1: Extract Various Version Formats
**Description**: Test version extraction from different MQTT message formats.

**Test Formats**:

**Format 1: Inline version objects**
```json
{
  "sn": "61250904380091",
  "vendor": "MoreFun",
  "model": "MF919",
  "org.device": [
    {"version": "1.0.0", "type": "parameter_config"},
    {"version": null, "type": "emv_config"},
    {"version": "2.3.1", "type": "keys_config"}
  ]
}
```

**Format 2: Nested version mapping**
```json
{
  "sn": "61250904380091",
  "versions": {
    "parameter_config": "1.0.0",
    "emv_config": null,
    "keys_config": "2.3.1"
  }
}
```

**Expected Results for Both**:
- [ ] Versions correctly extracted
- [ ] NULL versions detected
- [ ] Auto-push triggered only for missing versions

---

### Test Case 7.2: Version Presence Check - All Present
**Description**: When all versions present, no auto-push.

**Setup**:
```sql
UPDATE tms_terminals 
SET parameter_config_version = '1.0.0',
    emv_config_version = '2.0.0',
    keys_config_version = '3.0.0',
    app_version = '4.0.0'
WHERE serial_number = '61250904380091';
```

**Test Steps**:
1. Send MQTT status message with all versions (no NULL)
2. Monitor logs

**Expected Results**:
- [ ] No auto-push triggered
- [ ] Log does NOT show: `Detected missing versions`
- [ ] No new push logs created

---

### Test Case 7.3: Empty String as Missing Version
**Description**: Empty strings should also trigger auto-push (not just NULL).

**Test Steps**:
1. Send MQTT message with empty string `"version": ""`
2. Monitor logs

**Expected Results**:
- [ ] Auto-push triggered
- [ ] Treated same as NULL

---

## 9. DATABASE STATE CONSISTENCY

### Test Case 8.1: Push Log Entries Audit Trail
**Description**: Verify push log captures all necessary audit information.

**Verification Query**:
```sql
SELECT 
  id, terminal_id, config_type, status, trigger_reason,
  device_vendor, device_model, request_id, log_time,
  push_type, version_sent, created_at
FROM parameter_push_logs 
WHERE device_model = 'MF919'
ORDER BY created_at DESC 
LIMIT 5;
```

**Expected Fields**:
- [ ] `config_type`: parameter, emv_config, keys_config, or application
- [ ] `trigger_reason`: missing_version
- [ ] `status`: pending (initially)
- [ ] `request_id`: integer value
- [ ] `log_time`: UTC datetime when push was triggered
- [ ] `device_vendor` and `device_model`: populated from message
- [ ] `push_type`: full (for auto-push)

---

### Test Case 8.2: Terminal Status Update on Auto-Push
**Description**: Terminal record should be updated with latest version info.

**Setup**:
1. Record initial terminal state
2. Trigger auto-push  
3. Check terminal was updated

**Verification Before**:
```sql
SELECT serial_number, parameter_config_version, emv_config_version, 
       keys_config_version, app_version, updated_at
FROM tms_terminals 
WHERE serial_number = '61250904380091';
```

**Expected After**:
- [ ] Received versions updated in terminal record
- [ ] `updated_at` timestamp changed to message upload time
- [ ] Status field set to 'connected'

---

### Test Case 8.3: Template Version Tracking
**Description**: When parameters are pushed, version info should be tracked.

**Test Steps**:
1. Trigger parameter push from template
2. Check push log for version_sent field

**Expected**:
- [ ] `version_sent` field populated with template version (if available)
- [ ] Can be used later to match which template version was used

---

## 10. INTEGRATION TESTS - END-TO-END WORKFLOWS

### Test Case 9.1: Complete Device Onboarding Flow
**Description**: New device connects and receives all configs via auto-push.

**Scenario**:
- New terminal with `serial_number: 'NEWDEV_001'`
- All versions missing
- Active templates available for vendor/model

**Test Steps**:
1. Device connects and sends initial status with all NULL versions
2. Monitor entire push sequence
3. Verify all 4 config types pushed

**Execution**:
```bash
# Monitor logs with timestamps for complete flow
tail -f logs/app.log | grep -E "NEWDEV_001|AutoPushService|MQTT"
```

**Expected Timeline**:
- T+0s: Status received, missing versions detected
- T+1s: Parameter push log created and MQTT sent
- T+2s: EMV config push log created and MQTT sent
- T+3s: Keys config generated and MQTT sent
- T+4s: Application push log created and MQTT sent

**Verification Queries**:
```sql
-- Count pushes per config type
SELECT config_type, COUNT(*) as count 
FROM parameter_push_logs 
WHERE device_vendor = 'MoreFun' AND device_model = 'MF919'
GROUP BY config_type;

-- Check all have pending status initially
SELECT config_type, status FROM parameter_push_logs 
WHERE device_vendor = 'MoreFun' AND device_model = 'MF919'
ORDER BY created_at DESC LIMIT 4;
```

---

### Test Case 9.2: Partial Config Update
**Description**: Update only specific configs and retrigger.

**Setup**:
```sql
-- Only EMV and keys missing, parameters and app present
UPDATE tms_terminals 
SET parameter_config_version = '1.0.0',
    emv_config_version = NULL,
    keys_config_version = NULL,
    app_version = '2.0.0'
WHERE serial_number = '61250904380091';
```

**Test Steps**:
1. Trigger auto-push
2. Verify only 2 pushes (EMV and keys)

**Expected Results**:
- [ ] Only 2 new push logs created
- [ ] Parameter and app configs NOT pushed

---

### Test Case 9.3: Re-push Same Config Multiple Times
**Description**: Retrig push for same config should create new logs.

**Test Steps**:
1. First push for parameter config
2. Clear parameter version again
3. Trigger second push
4. Check logs

**Expected Results**:
- [ ] Two separate push log entries
- [ ] Both have same config_type but different timestamps
- [ ] request_id values are different

---

## 11. PERFORMANCE AND STRESS TESTS

### Test Case 10.1: Bulk Terminal Status Updates
**Description**: Multiple devices sending simultaneous status messages.

**Test Setup**:
- Prepare 10 terminals with missing versions
- Send status messages in quick succession

**Test Script** (Example with curl):
```bash
#!/bin/bash
for i in {1..10}; do
  SERIAL="BULK_DEV_00$i"
  
  # Create/update terminal
  mysql -u root -pdataaegis123 -D lic_project_cicd << EOF
  UPDATE tms_terminals 
  SET parameter_config_version = NULL
  WHERE serial_number = '$SERIAL';
EOF
  
  # Would send MQTT message here
  echo "Triggered for $SERIAL"
  sleep 1
done

# Check push logs created
mysql -u root -pdataaegis123 -D lic_project_cicd << EOF
SELECT COUNT(*) FROM parameter_push_logs 
WHERE created_at > NOW() - INTERVAL 5 MINUTE;
EOF
```

**Expected Results**:
- [ ] All 10 devices processed
- [ ] Auto-push triggered for each
- [ ] Database queries don't timeout
- [ ] No race conditions

---

### Test Case 10.2: Large Parameter Set Push
**Description**: Push large number of parameters (50+).

**Setup**:
```sql
-- Create template with many parameters
INSERT INTO parameter_definitions (key, description, data_type, category, display_order, inserted_at, updated_at)
VALUES 
  ('PARAM_001', 'Parameter 1', 'string', 'OTHER', 1, NOW(), NOW()),
  ('PARAM_002', 'Parameter 2', 'string', 'OTHER', 2, NOW(), NOW()),
  -- ... repeat to 50+ parameters
ON DUPLICATE KEY UPDATE id=id;
```

**Test Steps**:
1. Trigger parameter push with 50+ parameters
2. Monitor MQTT payload size
3. Check push log parameters_sent field

**Expected Results**:
- [ ] All 50+ parameters sent successfully
- [ ] MQTT message formatted correctly
- [ ] Database can handle large map storage
- [ ] No truncation or data loss

---

## 12. MANUAL TESTING PROCEDURES - API TESTING

### Prerequisites for API Testing
```bash
# 1. Start server
mix phx.server

# 2. Set API base URL
BASE_URL="http://localhost:4000"

# 3. Get valid authentication token (if required)
# Adjust based on your authentication setup
TOKEN="your_auth_token"
```

### Test Case 11.1: Check Terminal Status
**API Endpoint**: GET `/api/v1/terminals/{serial_number}`

**Test Request**:
```bash
curl -X GET "$BASE_URL/api/v1/terminals/61250904380091" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json"
```

**Expected Response**:
```json
{
  "id": 7,
  "serial_number": "61250904380091",
  "vendor": "MoreFun",
  "model": "MF919",
  "status": "connected",
  "parameter_config_version": "1.0.0",
  "emv_config_version": "2.0.0",
  "keys_config_version": null,
  "app_version": "1.5.0",
  "kek_kcv": "112233",
  "slot_number": 1,
  "last_keys_update": null,
  "updated_at": "2026-01-15T10:30:00Z"
}
```

**Verification**:
- [ ] Status is 'connected'
- [ ] Versions populated correctly
- [ ] Keys config fields present

---

### Test Case 11.2: Get Push Logs for Terminal
**API Endpoint**: GET `/api/v1/terminals/{terminal_id}/push-logs`

**Test Request**:
```bash
curl -X GET "$BASE_URL/api/v1/terminals/7/push-logs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json"
```

**Expected Response**:
```json
{
  "total": 5,
  "page": 1,
  "page_size": 10,
  "push_logs": [
    {
      "id": 42,
      "config_type": "keys_config",
      "status": "pending",
      "trigger_reason": "missing_version",
      "device_vendor": "MoreFun",
      "device_model": "MF919",
      "request_id": 1234567890,
      "log_time": "2026-01-15T10:30:00Z",
      "file_path": "priv/ota/61250904380091/keys.json",
      "file_size": 512,
      "checksum": "abc123...",
      "created_at": "2026-01-15T10:30:05Z"
    },
    {
      "id": 41,
      "config_type": "parameter",
      "status": "pending",
      "trigger_reason": "missing_version",
      "device_vendor": "MoreFun",
      "device_model": "MF919",
      "request_id": 1234567889,
      "log_time": "2026-01-15T10:30:00Z",
      "parameters_sent": {
        "COMM_SERVER_ADDRESS": "192.168.1.100",
        "COMM_PORT": "8080"
      },
      "created_at": "2026-01-15T10:30:04Z"
    }
  ]
}
```

**Verification**:
- [ ] All recent pushes returned
- [ ] Correct config types
- [ ] Parameters visible in response
- [ ] File info present for keys config

---

### Test Case 11.3: Get Parameter Template Details
**API Endpoint**: GET `/api/v1/parameter-templates/{template_id}`

**Test Request**:
```bash
curl -X GET "$BASE_URL/api/v1/parameter-templates/5" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json"
```

**Expected Response**:
```json
{
  "id": 5,
  "name": "MF919 Default Parameters",
  "vendor": "MoreFun",
  "model": "MF919",
  "is_active": true,
  "is_default": true,
  "template_values": [
    {
      "parameter_definition": {
        "key": "COMM_SERVER_ADDRESS",
        "description": "Server IP Address",
        "data_type": "string",
        "category": "COMMUNICATIONS"
      },
      "value": "192.168.1.100"
    },
    {
      "parameter_definition": {
        "key": "COMM_PORT",
        "description": "Server Port",
        "data_type": "integer",
        "category": "COMMUNICATIONS"
      },
      "value": "8080"
    }
  ],
  "created_at": "2025-12-01T00:00:00Z"
}
```

**Verification**:
- [ ] All template parameters returned
- [ ] Correct values shown
- [ ] Parameter definitions included

---

### Test Case 11.4: List All Active Templates
**API Endpoint**: GET `/api/v1/parameter-templates?is_active=true`

**Test Request**:
```bash
curl -X GET "$BASE_URL/api/v1/parameter-templates?is_active=true&vendor=MoreFun" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json"
```

**Expected Response**:
```json
{
  "total": 3,
  "page": 1,
  "page_size": 10,
  "templates": [
    {
      "id": 5,
      "name": "MF919 Default Parameters",
      "vendor": "MoreFun",
      "model": "MF919",
      "is_active": true,
      "is_default": true,
      "created_at": "2025-12-01T00:00:00Z"
    },
    {
      "id": 6,
      "name": "MF919 Production Parameters",
      "vendor": "MoreFun",
      "model": "MF919",
      "is_active": true,
      "is_default": false,
      "created_at": "2026-01-01T00:00:00Z"
    }
  ]
}
```

**Verification**:
- [ ] Only active templates returned
- [ ] Correct vendor/model filter applied
- [ ] Pagination working

---

### Test Case 11.5: Filter Push Logs by Status
**API Endpoint**: GET `/api/v1/parameter-push-logs?status=pending`

**Test Request**:
```bash
curl -X GET "$BASE_URL/api/v1/parameter-push-logs?status=pending&limit=20" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json"
```

**Expected Response**:
```json
{
  "total": 12,
  "page": 1,
  "page_size": 20,
  "push_logs": [
    {
      "id": 45,
      "terminal_id": 7,
      "config_type": "keys_config",
      "status": "pending",
      "trigger_reason": "missing_version",
      "request_id": 1234567892,
      "log_time": "2026-01-15T10:35:00Z",
      "created_at": "2026-01-15T10:35:05Z"
    }
  ]
}
```

**Verification**:
- [ ] Only pending logs returned
- [ ] Date filters work if provided
- [ ] Config type filters work

---

### Test Case 11.6: Query Push Logs by Device
**API Endpoint**: GET `/api/v1/parameter-push-logs?device_vendor=MoreFun&device_model=MF919`

**Test Request**:
```bash
curl -X GET "$BASE_URL/api/v1/parameter-push-logs?device_vendor=MoreFun&device_model=MF919&days=7" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json"
```

**Expected Results**:
- [ ] Returns all pushes for MoreFun MF919 in last 7 days
- [ ] Grouped by config_type if requested
- [ ] Includes summary statistics

---

## 13. LOG ANALYSIS AND MONITORING

### Test Case 12.1: Real-time Log Monitoring
**Description**: Active monitoring of auto-push operations as they happen.

**Monitoring Command**:
```bash
# Watch for AutoPush specific messages with timestamps
tail -f logs/app.log | grep -E "AutoPush|KeysConfig|has_missing_versions" | \
  cut -d'[' -f1-3,5- | awk '{$1=$2=$3=""; print "["$0"]"}'
```

**Expected Log Stream**:
```
[2026-01-15 10:30:00.123] [info] Detected missing versions for 61250904380091
[2026-01-15 10:30:01.456] [info] AutoPushService: Triggering auto-push
[2026-01-15 10:30:02.789] [info] AutoPushService: Created push log 42
[2026-01-15 10:30:03.012] [info] AutoPushService: Sending parameter push
[2026-01-15 10:30:04.345] [info] Parameter push sent successfully
[2026-01-15 10:30:05.678] [info] AutoPushService: Sending keys config push
[2026-01-15 10:30:06.901] [info] Keys config sent successfully
```

**Verification Checklist**:
- [ ] Timestamps are sequential
- [ ] All push types appear in logs
- [ ] No error messages in sequence
- [ ] Total elapsed time < 10 seconds

---

### Test Case 12.2: Error Log Analysis
**Description**: Search logs for any error conditions during auto-push.

**Search Command**:
```bash
# Find all errors in last 100 lines
tail -100 logs/app.log | grep -i error

# Search for specific error patterns
grep -E "Failed|Exception|Error" logs/app.log | tail -20

# Check for AutoPush specific errors
grep "AutoPushService" logs/app.log | grep -i error
```

**Expected Result**:
- [ ] No errors related to auto-push if all tests passed
- [ ] Only expected errors from negative test cases

---

### Test Case 12.3: Database Query Performance
**Description**: Check query performance during auto-push operations.

**Monitor with MySQL**:
```bash
# Connect to MySQL and enable query log
mysql -u root -pdataaegis123 -D lic_project_cicd

# Check slow query log
SHOW VARIABLES LIKE 'slow_query_log%';
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 0.5;

# Trigger auto-push and check slow queries
SELECT * FROM mysql.slow_log ORDER BY start_time DESC LIMIT 10;
```

**Expected Results**:
- [ ] No queries exceeding 0.5 seconds
- [ ] All template lookups use indexes
- [ ] Insert/update operations complete quickly

---

## 14. REGRESSION AND SMOKE TESTS

### Test Case 13.1: Smoke Test - Basic Happy Path
**Quick 5-minute validation that core functionality works**

**Procedure**:
```bash
# 1. Check server is running
curl http://localhost:4000 | head -20

# 2. Check database connectivity
mysql -u root -pdataaegis123 -D lic_project_cicd -e "SELECT COUNT(*) AS terminal_count FROM tms_terminals;"

# 3. Create/update test terminal with missing versions
mysql -u root -pdataaegis123 -D lic_project_cicd << EOF
UPDATE tms_terminals 
SET parameter_config_version = NULL,
    emv_config_version = NULL
WHERE serial_number = 'SMOKE_TEST_001'
LIMIT 1;
EOF

# 4. Simulate MQTT status message (would need MQTT client)
# Send status via MQTT broker

# 5. Wait 5 seconds for processing
sleep 5

# 6. Check push logs created
mysql -u root -pdataaegis123 -D lic_project_cicd << EOF
SELECT COUNT(*) AS push_count, config_type 
FROM parameter_push_logs 
WHERE created_at > NOW() - INTERVAL 1 MINUTE
GROUP BY config_type;
EOF

# Expected: 2 rows (parameter and emv_config)
```

---

### Test Case 13.2: Regression - No Unintended Side Effects
**Description**: Ensure auto-push doesn't affect other system functions.

**Checks**:
```bash
# 1. Other services still accessible
curl -s http://localhost:4000/api/v1/merchants | jq . | head -20

# 2. User authentication working
curl -s -X POST http://localhost:4000/api/v1/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@example.com","password":"test"}' | jq .

# 3. Settlement APIs still working
curl -s http://localhost:4000/api/v1/merchant/settlements | jq . | head -20

# 4. Database still responsive
mysql -u root -pdataaegis123 -D lic_project_cicd -e "SELECT NOW() AS current_time;"

# 5. MQTT messages for other types still processed
# (Check if non-status MQTT messages still work)
```

**Expected Results**:
- [ ] All other API endpoints respond normally
- [ ] Database not locked or slow
- [ ] No increase in error logs
- [ ] Authentication unaffected

---

## 15. VALIDATION CHECKLIST AND SIGN-OFF

### Pre-Deployment Validation Checklist

- [ ] **Database Integrity**
  - [ ] All migration scripts executed successfully
  - [ ] `tms_terminals` table has version columns
  - [ ] `parameter_push_logs` table has all required fields
  - [ ] No constraint violations

- [ ] **AutoPush Core Logic**
  - [ ] Missing version detection works for all 4 config types
  - [ ] Template matching finds correct template (active > default)
  - [ ] Template fallback works when no active template
  - [ ] Graceful error when no template found

- [ ] **Parameter Push**
  - [ ] Template parameter values extracted correctly
  - [ ] MQTT payload formatted correctly per device
  - [ ] Request ID generated and stored properly
  - [ ] Parameters_sent map contains all key-value pairs

- [ ] **Keys Config Push**
  - [ ] RKI endpoint integration works
  - [ ] keys.json file created with valid content
  - [ ] File checksum calculated and verified
  - [ ] Download URL included in MQTT payload

- [ ] **EMV/Application Push**
  - [ ] EMV config pushed with correct command type
  - [ ] Application config pushed with correct command type
  - [ ] Both follow same MQTT formatting as parameters

- [ ] **Error Handling**
  - [ ] Gracefully handles missing templates
  - [ ] Gracefully handles RKI endpoint failures
  - [ ] Gracefully handles MQTT publish failures
  - [ ] No system crashes from bad MQTT messages

- [ ] **Database Consistency**
  - [ ] Push logs capture all audit fields
  - [ ] Terminal records updated with received versions
  - [ ] No orphaned push logs
  - [ ] Transaction integrity maintained

- [ ] **API Testing**
  - [ ] Terminal status endpoint returns correct data
  - [ ] Push logs queryable by device/config type/status
  - [ ] Template details include all parameters
  - [ ] Pagination working on all endpoints

- [ ] **Performance**
  - [ ] Auto-push completes for all 4 configs in < 10 seconds
  - [ ] Bulk operations handle 10+ devices without issues
  - [ ] No database query timeouts
  - [ ] No memory leaks in logs

- [ ] **Monitoring & Logs**
  - [ ] All major steps logged with timestamps
  - [ ] Error messages are clear and actionable
  - [ ] Log levels appropriate (info, warning, error)
  - [ ] No sensitive data in logs

---

## 16. KNOWN ISSUES AND TROUBLESHOOTING

### Issue: request_id Type Mismatch
**Symptom**: `[error] AutoPushService: Failed to create [config_type] push log: ... errors: [request_id: {"is invalid", [type: :integer, validation: :cast]}]`

**Root Cause**: `MQTTCommandBuilder.generate_request_id()` may return string format, but `parameter_push_logs.request_id` is integer field.

**Resolution**:
```elixir
# In AutoPushService, ensure request_id is converted to integer
request_id = System.unique_integer([:positive])  # Use this instead
```

---

### Issue: Keys File Not Found After Generation
**Symptom**: Push log created but file doesn't exist at expected path.

**Troubleshooting**:
```bash
# Check if directory exists
ls -la priv/ota/

# Check Phoenix upload directory
ls -la priv/

# Verify file permissions
ls -la priv/ota/61250904380091/
```

**Resolution**: Ensure `priv/ota/` directory exists and is writable by Phoenix process.

---

### Issue: Template Not Applied Correctly
**Symptom**: Auto-push triggered but wrong template used.

**Troubleshooting**:
```sql
-- Check templates for vendor/model
SELECT id, name, is_active, is_default FROM parameter_templates 
WHERE vendor = 'MoreFun' AND model = 'MF919'
ORDER BY is_active DESC, is_default DESC;

-- Check which template was used in push log
SELECT template_id FROM parameter_push_logs 
WHERE config_type = 'parameter' AND device_model = 'MF919'
ORDER BY created_at DESC LIMIT 1;

-- Compare template details
SELECT id, name, is_active, is_default FROM parameter_templates 
WHERE id = [template_id_from_above];
```

**Resolution**: Update template is_active or is_default flags as needed.

---

### Issue: MQTT Message Not Triggering Auto-Push
**Symptom**: Status message received but no auto-push triggered.

**Troubleshooting**:
```bash
# Check if handler received message
grep "Received status update for terminal" logs/app.log

# Check MQTT decode success
grep "Decoded JSON\|Failed to parse" logs/app.log

# Check version detection
grep "Extracted versions\|has_missing_versions" logs/app.log

# Look for vendor/model in message
grep "vendor\|model" logs/app.log | tail -10
```

**Resolution**: Ensure MQTT message includes vendor, model, and version fields.

---

## APPENDIX A: SQL QUICK REFERENCE

### Create Test Terminal
```sql
INSERT INTO tms_terminals (
  serial_number, oid, status, vendor, model, 
  parameter_config_version, emv_config_version, 
  keys_config_version, app_version,
  inserted_at, updated_at
) VALUES (
  'TEST_DEVICE_001', 'OID123', 'offline', 
  'MoreFun', 'MF919',
  NULL, NULL, NULL, NULL,
  NOW(), NOW()
);
```

### Create Parameter Template
```sql
INSERT INTO parameter_templates (
  name, vendor, model, is_active, is_default, 
  created_by_id, inserted_at, updated_at
) VALUES (
  'MF919 Test Template', 'MoreFun', 'MF919',
  true, true, 1, NOW(), NOW()
);
```

### Reset Auto-Push Testing
```sql
-- Clear all push logs for testing
DELETE FROM parameter_push_logs 
WHERE created_at > DATE_SUB(NOW(), INTERVAL 1 DAY)
AND device_model = 'MF919';

-- Reset terminal versions to NULL
UPDATE tms_terminals 
SET parameter_config_version = NULL,
    emv_config_version = NULL,
    keys_config_version = NULL
WHERE model = 'MF919'
AND serial_number LIKE 'TEST_%';
```

---

## APPENDIX B: CURL REQUESTS FOR API TESTING

### Sample Auth Header (if JWT required)
```bash
AUTH_HEADER="Authorization: Bearer eyJhbGc..."
CONTENT_TYPE="Content-Type: application/json"
BASE_URL="http://localhost:4000"
```

### Get All Terminals
```bash
curl -s "$BASE_URL/api/v1/terminals" \
  -H "$AUTH_HEADER" \
  -H "$CONTENT_TYPE" | jq '.data[] | {id, serial_number, vendor, model, status}'
```

### Get Push Logs for Last 24 Hours
```bash
curl -s "$BASE_URL/api/v1/parameter-push-logs?days=1" \
  -H "$AUTH_HEADER" \
  -H "$CONTENT_TYPE" | jq '.push_logs[] | {id, config_type, status, device_model, created_at}'
```

### Filter Pending Pushes
```bash
curl -s "$BASE_URL/api/v1/parameter-push-logs?status=pending" \
  -H "$AUTH_HEADER" \
  -H "$CONTENT_TYPE" | jq '.push_logs | length'
```

---

## APPENDIX C: MQTT MESSAGE EXAMPLES

### Status Message with Missing Parameter Version
```json
{
  "oid": "OID123",
  "sn": "61250904380091",
  "uploadTime": "2026-01-15T10:30:00Z",
  "vendor": "MoreFun",
  "model": "MF919",
  "org.device": [
    {
      "version": null,
      "type": "parameter_config"
    }
  ]
}
```

### Status Message with All Versions
```json
{
  "oid": "OID123",
  "sn": "61250904380091",
  "uploadTime": "2026-01-15T10:30:00Z",
  "vendor": "MoreFun",
  "model": "MF919",
  "org.device": [
    {
      "version": "1.0.0",
      "type": "parameter_config"
    },
    {
      "version": "2.0.0",
      "type": "emv_config"
    },
    {
      "version": "3.0.0",
      "type": "keys_config"
    },
    {
      "version": "4.0.0",
      "type": "application"
    }
  ]
}
```

---

## APPENDIX D: EXPECTED DATABASE SCHEMA VALIDATION

### Verify Schema for TMS Terminals
```sql
DESCRIBE tms_terminals;

-- Expected columns:
-- - serial_number (VARCHAR, NOT NULL, UNIQUE)
-- - status (VARCHAR)
-- - vendor (VARCHAR)
-- - model (VARCHAR)
-- - parameter_config_version (VARCHAR)
-- - emv_config_version (VARCHAR)
-- - keys_config_version (VARCHAR)
-- - kek_kcv (VARCHAR)
-- - slot_number (INTEGER)
-- - last_keys_update (DATETIME)
```

### Verify Schema for Parameter Push Logs
```sql
DESCRIBE parameter_push_logs;

-- Expected columns:
-- - id (BIGINT, PRIMARY KEY)
-- - terminal_id (BIGINT, FOREIGN KEY)
-- - template_id (BIGINT, FOREIGN KEY)
-- - request_id (BIGINT)
-- - config_type (VARCHAR)
-- - status (VARCHAR)
-- - trigger_reason (VARCHAR)
-- - device_vendor (VARCHAR)
-- - device_model (VARCHAR)
-- - file_path (VARCHAR)
-- - file_size (BIGINT)
-- - checksum (VARCHAR)
-- - parameters_sent (JSON)
-- - version_sent (VARCHAR)
-- - log_time (DATETIME)
-- - created_at (DATETIME)
-- - updated_at (DATETIME)
```

---

## APPENDIX E: PERFORMANCE BENCHMARKS

### Expected Performance Numbers
| Operation | Expected Time | Max Acceptable | Test Serial |
|-----------|---|---|---|
| Trigger auto-push | 50-200ms | 500ms | See Test 9.1 |
| Generate keys.json with RKI | 1-2s | 5s | See Test 4.1 |
| Send all 4 MQTT messages | 100-500ms | 2s | See Test 9.1 |
| Create push log entry | 10-50ms | 100ms | All tests |
| Template lookup | 5-20ms | 100ms | All tests |
| Complete device onboarding | 4-6s | 15s | See Test 9.1 |

---

## Testing Sign-Off

**Test Completed By**: _______________
**Date**: _______________
**Environment**: Development / Staging / Production
**Total Tests Run**: _____
**Tests Passed**: _____
**Tests Failed**: _____
**Known Issues**: 
```
[Use Appendix C for documentation]
```

**Recommendation**: 
- [ ] APPROVED - Ready for deployment
- [ ] CONDITIONAL - Pass with known issues documented
- [ ] REJECTED - Resolve critical issues before deployment

**Comments**:
___________________________________________________________
___________________________________________________________

---

**Document Version**: 1.0
**Last Updated**: January 15, 2026
**Next Review**: Post-Production Deployment Validation