# MF919 Auto-Push Implementation Plan (Updated)

## Purpose

Define the complete end-to-end delivery model for MF919 parameter and EMV configuration push including ZIP packaging, dynamic merchant data injection, and MQTT delivery.

## Source Artifacts

| Artifact | Delivery ZIP | Push Command | Scope |
| --- | --- | --- | --- |
| `default_params.properties` | `params.zip` | `UPDATE_PARAMS` | Per-terminal (dynamic values injected) |
| `default_merchants.properties` | `params.zip` | `UPDATE_PARAMS` | Per-terminal (fully dynamic from DB) |
| `YSDK_L3_configuration.xml` | `l3config.zip` | `UPDATE_L3_CONFIG` | Shared across all MF919 terminals |
| `keys.json` | direct file | `LOAD_KEYS` | Per-terminal (already implemented) |

## Delivery Model

MF919 terminals receive configuration via three MQTT file-download commands, each referencing a remote `downloadUrl`.  All commands follow the structure:

```json
{
  "type": "tms_command",
  "command": "UPDATE_PARAMS",
  "requestId": "req-...",
  "downloadUrl": "https://..."
}
```

The three payloads are:

1. **`params.zip`** → `UPDATE_PARAMS`
   - Contains `default_params.properties` and `default_merchants.properties`
   - Built dynamically per terminal at push time
   - Static keys come from the MF919 parameter template stored in the database
   - Dynamic keys (`MERCHANT_ID`, `TERMINAL_ID`, `BASE_MERCHANT_NAME`) are fetched from the external transaction database

2. **`l3config.zip`** → `UPDATE_L3_CONFIG`
   - Contains a single `YSDK_L3_configuration.xml`
   - Shared across the entire MF919 fleet (no per-terminal variation)
   - Stored as a managed file artifact in the TMS; new versions are uploaded by operations staff
   - Terminal version tracked via `emv_config_version` in the terminal record

3. **`keys.json`** → `LOAD_KEYS`
   - Already implemented via `KeysConfigService`
   - No changes required

---

## What Is Already Done

The cloud agent has seeded the parameter foundation (PR #18):

- `mf919_parameter_categories.exs` — MF919 category tree under `device_setup`
- `mf919_parameter_definitions.exs` — All parameter keys (BASE_, COMM_, TRANS_, PRINT_, PINPAD_, etc.)
- `mf919_default_template.exs` — `MoreFun / MF919 Default Parameters` template + template values
- `mf919_merchant_overlay.exs` — Demonstration of MERCHANT_ID/TERMINAL_ID/BATCH_NO as override records

The above is the **source of truth** for static parameter values and gives the UI a way to manage them.

---

## params.zip: Dynamic Build Architecture

### What goes into params.zip

Two `.properties` files are packaged together:

| File | Keys | Source |
| --- | --- | --- |
| `default_params.properties` | All BASE_, COMM_, TRANS_, PRINT_, PINPAD_, ELECSIGN_, SCAN_, PASSWORD_, TOMS_, NFC_, OTHER_, EXTERNAL_ keys | MF919 ParameterTemplate values (manageable via UI) |
| `default_params.properties` | `BASE_MERCHANT_NAME` (override) | External DB: `users.meta->>'business_name'` |
| `default_merchants.properties` | `MERCHANT_ID` | External DB: `pos_merchant.merchantid` |
| `default_merchants.properties` | `TERMINAL_ID` | External DB: `pos_terminal.terminalid` |
| `default_merchants.properties` | `BATCH_NO` | Template default value `000001` (or terminal record) |

### External DB Queries Required at Push Time

All three queries are keyed by terminal `serial_number`:

```sql
-- Merchant ID (for default_merchants.properties)
SELECT merchantid
FROM pos_merchant
WHERE id = (
  SELECT pos_merchant_id FROM pos_terminal
  WHERE serial_number = :serial_number LIMIT 1
) LIMIT 1;

-- Terminal ID (for default_merchants.properties)
SELECT terminalid
FROM pos_terminal
WHERE serial_number = :serial_number LIMIT 1;

-- Merchant business name (for BASE_MERCHANT_NAME in default_params.properties)
SELECT JSON_UNQUOTE(JSON_EXTRACT(u.meta, '$.business_name'))
FROM users u
WHERE u.id = (
  SELECT um.user_id FROM user_metadata um
  WHERE um.merchant_refrence_number = (
    SELECT merchantid FROM pos_merchant
    WHERE id = (
      SELECT pos_merchant_id FROM pos_terminal
      WHERE serial_number = :serial_number LIMIT 1
    ) LIMIT 1
  ) LIMIT 1
) LIMIT 1;
```

> These run against the external MySQL schemas (`shukria_transactions.pos_terminal`, `shukria_transactions.pos_merchant`, `shukria_mms_new_local.user_metadata`, `shukria_mms_new_local.users`). The existing `Repo` in `tms_core` or a new `ExternalRepo` must be configured to access these.

### params.zip Build Flow

```
auto_push triggers on heartbeat (params_version IS NULL)
         |
         v
1. Load ParameterTemplate (vendor=MoreFun, model=MF919)
         |
         v
2. Load all ParameterTemplateValues → key/value map
         |
         v
3. Query external DB for terminal:
     - merchantid  (MERCHANT_ID)
     - terminalid  (TERMINAL_ID)
     - business_name (BASE_MERCHANT_NAME override)
         |
         v
4. Build default_params.properties:
     - All keys from template EXCEPT MERCHANT_ID/TERMINAL_ID/BATCH_NO
     - Override BASE_MERCHANT_NAME with DB value
         |
         v
5. Build default_merchants.properties:
     [DEFAULT]
     MERCHANT_ID={merchantid from DB}
     TERMINAL_ID={terminalid from DB}
     BATCH_NO={from template, default 000001}
         |
         v
6. Create in-memory ZIP containing both files as params.zip
         |
         v
7. Upload params.zip to file store → get download_url
         |
         v
8. Log push (ParameterPushLog, request_id as integer)
         |
         v
9. MQTT publish:
     topic: /ota/{product_key}/{serial_number}/update
     payload: {"type":"tms_command","command":"UPDATE_PARAMS",
               "requestId":123456,"downloadUrl":"https://..."}
```

### New Module: `ParamsZipBuilder`

Location: `apps/tms_core/lib/tms_core/terminal_management/params_zip_builder.ex`

Responsibilities:
- Accept `serial_number` and `template_values` map
- Query external DB for `merchantid`, `terminalid`, `business_name`
- Build the two `.properties` file contents as binaries
- Create an in-memory ZIP using `:zip.create/2` (Erlang stdlib)
- Return `{:ok, zip_binary}` or `{:error, reason}`

---

## l3config.zip: Shared EMV File Architecture

### Design Decision

`YSDK_L3_configuration.xml` is the same file for all MF919 terminals. It does **not** contain per-terminal data. A single version is pushed to every terminal.

### Storage

Store as a versioned file artifact rather than in `parameter_template_values`:

- Store `YSDK_L3_configuration.xml` at `priv/static/mf919/emv/YSDK_L3_configuration.xml` (or in an object store)
- Track the current version string in app config (e.g. `"1.0.0"`)
- Track deployed version per terminal via `emv_config_version` on the terminal record

When operations staff needs to update the EMV config:
1. Replace the XML file
2. Bump the version string in config
3. Clear `emv_config_version` on affected terminals → auto-push will re-send

### l3config.zip Build Flow

```
auto_push triggers on heartbeat (emv_config_version IS NULL)
         |
         v
1. Load YSDK_L3_configuration.xml from configured path
         |
         v
2. Create in-memory ZIP: l3config.zip containing the XML
         |
         v
3. Upload l3config.zip to file store → get download_url
         |
         v
4. Log push (ParameterPushLog, request_id as integer,
             config_type="emv_config")
         |
         v
5. MQTT publish:
     topic: /ota/{product_key}/{serial_number}/update
     payload: {"type":"tms_command","command":"UPDATE_L3_CONFIG",
               "requestId":123456,"downloadUrl":"https://..."}
```

### New Module: `L3ConfigZipBuilder`

Location: `apps/tms_core/lib/tms_core/terminal_management/l3_config_zip_builder.ex`

Responsibilities:
- Load `YSDK_L3_configuration.xml` from configured path
- Create in-memory ZIP containing the file with filename `YSDK_L3_configuration.xml`
- Return `{:ok, zip_binary}` or `{:error, reason}`

---

## Bugs to Fix in Existing Code

### Bug 1: request_id type mismatch

**Location**: `auto_push_service.ex` → `push_regular_config/5` and `push_keys_config/4`

**Problem**: `MQTTCommandBuilder.generate_request_id()` returns `"req-{ts}-{rand}"` (string), but `ParameterPushLog.request_id` is `integer` → Ecto insert fails → push never completes.

**Fix**: Replace `MQTTCommandBuilder.generate_request_id()` with `System.unique_integer([:positive])` in both functions.

### Bug 2: MF919 push_template_parameters sends inline map instead of ZIP

**Location**: `auto_push_service.ex` → `push_template_parameters/5` + `send_mqtt_parameter_push/5`

**Problem**: Current flow builds a `parameters_map` and passes it inline in the MQTT payload. `build_mf919_command/1` ignores the `"parameters"` key entirely — MF919 only understands `downloadUrl`.

**Fix**: Replace `push_template_parameters` flow for MF919 with:
1. Call `ParamsZipBuilder.build/2`
2. Upload ZIP → get `download_url`
3. Pass `download_url` to MQTT command builder

### Bug 3: send_mqtt_config_push for emv_config has no downloadUrl

**Location**: `auto_push_service.ex` → `send_mqtt_config_push/5`

**Problem**: Sends `UPDATE_L3_CONFIG` with no `downloadUrl`. Terminal expects a file to download.

**Fix**: Before calling `send_mqtt_config_push` for `emv_config`, call `L3ConfigZipBuilder.build/0`, upload, pass `download_url`.

---

## Parameter Template Management (UI)

The MF919 parameter template remains **manageable via the TMS UI**:

- Operations staff can edit `COMM_SERVER_ADDRESS`, `COMM_PORT`, `TRANS_SALE`, etc. via the ParameterTemplate admin
- Changes are picked up on next auto-push (clear `params_version` on terminal to force re-push)
- `MERCHANT_ID`, `TERMINAL_ID`, and `BASE_MERCHANT_NAME` are **never stored in the template** — they are always injected dynamically from the DB at build time

### Key Split at Build Time

At `ParamsZipBuilder` time, keys are split like this:

| File | Keys Included |
| --- | --- |
| `default_merchants.properties` | MERCHANT_ID, TERMINAL_ID, BATCH_NO |
| `default_params.properties` | All other keys from template + BASE_MERCHANT_NAME (dynamic) |

---

## Parameter Inventory (Reference)

### default_merchants.properties (fully dynamic)

| Key | Source | Note |
| --- | --- | --- |
| `MERCHANT_ID` | `pos_merchant.merchantid` via serial_number | Never in template |
| `TERMINAL_ID` | `pos_terminal.terminalid` via serial_number | Never in template |
| `BATCH_NO` | Template default `000001` | Start value; terminal increments it |

### default_params.properties (mostly from template)

| Group | Keys | Dynamic? |
| --- | --- | --- |
| Base | `BASE_TRACE_NO`, `BASE_MAX_REFUND_AMOUNT`, `BASE_CURRENCY_CODE`, `BASE_MAX_TRANS_COUNT` | No |
| Base | `BASE_MERCHANT_NAME` | **Yes** — from `users.meta->>'business_name'` |
| PINPAD | `PINPAD_MASTER_KEY_INDEX`, `PINPAD_ALGORITHM_TYPE`, `PINPAD_TIMEOUT`, `EXTERNAL_PINPAD`, `EXTERNAL_PINPAD_YSDK`, `EXTERNAL_PINPAD_CONNECT_MODE` | No |
| PRINT | `PRINT_COUNT`, `PRINT_EXTERNAL`, `PRINT_REMARKS`, `PRINT_EXTERNAL_CONNECT_MODE`, `PRINT_EXTERNAL_SERIAL_BAUDRATE` | No |
| TRANS | `TRANS_SALE`, `TRANS_VOID`, `TRANS_REFUND`, `TRANS_BALANCE`, `TRANS_PREAUTH`, `TRANS_MOBILE_PAY`, `TRANS_INSTALLMENT` | No |
| COMM | `COMM_USE_SSL`, `COMM_TIMEOUT`, `COMM_TPDU`, `COMM_SERVER_ADDRESS`, `COMM_PORT`, `COMM_NII` | No |
| ELECSIGN / SCAN | `ELECSIGN_IS_SUPPORT`, `SCAN_PRIORITY_SCANNER`, `SCAN_EXTERN_CONNECT_MODE`, `SCAN_EXTERN_USB_WAIT_TIME`, `SCAN_EXTERN_SERIAL_BAUDRATE` | No |
| PASSWORD | `PASSWORD_ADMIN`, `PASSWORD_SYSTEM_ADMIN`, `PASSWORD_SECURITY` | No |
| OTHER / TOMS / NFC | `TOMS_FLY_PARAMETERS`, `TOMS_FLY_RECEIPT`, `NFC_RECEIPT`, `OTHER_TIP_INPUT`, `OTHER_THRID_BILL_SHOW`, `OTHER_VOID_CARD`, `OTHER_VOID_PIN` | No |

---

## Category Layout (Already Seeded)

```
device_setup
  └── mf919_device_setup
        ├── mf919_base
        ├── mf919_merchant      ← MERCHANT_ID, TERMINAL_ID, BATCH_NO (reference only, not in template values)
        ├── mf919_comm
        ├── mf919_transactions
        ├── mf919_printing
        ├── mf919_pinpad
        ├── mf919_scanner
        └── mf919_security
```

---

## Implementation Sequence

### Phase 1: Seed Data ✅ Done (PR #18)

- Categories, definitions, default template, merchant overlay seed scripts created.

### Phase 2: Bug Fixes in AutoPushService

1. Fix `request_id` integer/string mismatch in `push_regular_config/5` and `push_keys_config/4`
2. Fix `push_template_parameters/5` for MF919: switch from inline map to ZIP + upload path

### Phase 3: ParamsZipBuilder Service

1. Create `TmsCore.TerminalManagement.ParamsZipBuilder`
2. Implement external DB queries (serial_number → merchantid, terminalid, business_name)
3. Build `.properties` file contents
4. ZIP packaging using Erlang `:zip` module
5. Upload ZIP and return `download_url`

### Phase 4: L3ConfigZipBuilder Service

1. Create `TmsCore.TerminalManagement.L3ConfigZipBuilder`
2. Load `YSDK_L3_configuration.xml` from configured path
3. ZIP packaging and upload
4. Return `download_url`

### Phase 5: Wire up AutoPushService

1. For MF919 `params` push: call `ParamsZipBuilder`, then MQTT with `downloadUrl`
2. For MF919 `emv_config` push: call `L3ConfigZipBuilder`, then MQTT with `downloadUrl`
3. Store `YSDK_L3_configuration.xml` in `priv/static/mf919/emv/`

### Phase 6: Push Log and UI

1. Push logs should record `config_type`, `file_path`, `checksum` for both ZIP types
2. Logs UI should show file metadata for MF919 ZIP-based pushes

### Phase 7: Dynamic BATCH_NO and STAN from pos_terminal_data ✅ Done

1. Created `DaProductApp.PosTransactions.PosTerminalData` Ecto schema
   - Maps `pos_terminal_data` table (fields: `pos_terminalId`, `stan`, `batch_number`, `transaction_date`, `transaction_time`, `transaction_id`)
2. Added `get_terminal_data/1` to `ParamsZipBuilder`
   - Queries latest row from `pos_terminal_data` by `terminalid` (ORDER BY id DESC LIMIT 1)
   - Falls back to `%{batch_number: "000001", stan: "000001"}` if no row found
3. `default_merchants.properties`: `BATCH_NO` now sourced from `pos_terminal_data.batch_number`
4. `default_params.properties`: `BASE_TRACE_NO` now sourced from `pos_terminal_data.stan`

### Phase 8: Save version fields from heartbeat ✅ Done

1. MQTT handler `handle_message/3` for `["tms", "status", ...]` now includes `app_version: versions.application` in `terminal_update_attrs`
2. All four version fields saved on every heartbeat: `app_version`, `parameter_config_version`, `emv_config_version`, `keys_config_version`

---

## File Upload Integration

Both ZIP builders need a common file upload step. This should either:
- Reuse the same upload mechanism already used by `KeysConfigService`
- Or write to a shared HTTP file server path

The upload should return an absolute `downloadUrl` that the MF919 terminal can reach over the network.