# Bulk merchant load testing — runbook

How to test CloudLayer's MPGS **Direct** flow with the 1000 bulk test merchants
(`BULKTEST0001` … `BULKTEST1000`) against this simulator, without touching the
client's merchants or the client's CloudLayer instance.

Every command below lists **what it does**, **what you should see**, and **what
to do if you don't**. Run them on the server (`78.47.213.209`).

---

## 1. How the pieces fit

```
 scripts/bulk_load_test.py        (or the browser test page, section 4)
        │  Direct API: POST /pgpayments/api/direct/pay
        │  Authorization: Bearer <merchant's own API key>
        ▼
 CloudLayer dev instance — port 4056 (MPGS-Cloudlayer)
        │  public URL: https://mercurypay.aritic.app/pn/...  (Apache /pn → 4056)
        │  MySQL: shukria_transactions + shukria_mms_demo_local
        │  calls "MPGS" at the merchant's custom_gateway_host
        ▼
 MastercardSimulator — port 4444  (this repo, in-memory)
```

| Instance | Port | Used by | MPGS target |
|---|---|---|---|
| CloudLayer (client) | 4065 | client testing (`shukriapg.aritic.app`) | real MPGS test facility |
| CloudLayer (dev) | 4056 | **bulk load testing** (`mercurypay.aritic.app/pn`) | simulator for BULKTEST merchants |

Both instances share one database and one `MPGS_CREDENTIAL_KEY`. Separation is by
**configuration**: bulk merchants point at `http://localhost:4444`, client merchants
point at `https://mtf.gateway.mastercard.com`, and load tests call 4056 only.

### One Direct payment, step by step

1. Load test / page → CloudLayer `POST /pgpayments/api/direct/pay` with card + amount.
2. CloudLayer checks the API key, loads + decrypts the merchant's MPGS credentials,
   allocates an order number (`BT0001-n`), saves a `PENDING` row in `pg_transactions`.
3. CloudLayer → simulator, as `merchant.BULKTEST0001` / `bulk_password_123`:
   - `POST /api/rest/version/100/merchant/BULKTEST0001/session` — create session
   - `PUT  …/session/:id` — attach the card
   - `PUT  …/order/BT0001-n/transaction/1` with `apiOperation=PAY`, only `session.id`
4. Simulator reads the card from the session and decides (see test cards below).
5. CloudLayer saves the result (`CLOSED` / `APPROVED`) and returns JSON to the caller.

### Test cards (simulator)

| Card | Result |
|---|---|
| `5123450000000008` | APPROVED (`00`), scheme MASTERCARD |
| `5999990000000000` | DECLINED (`05` Do Not Honour) |
| `5777770000000000` | DECLINED (`51` Insufficient Funds) |
| any card, amount > 99 999 | DECLINED (`61`) |

Expiry `12/27`, CVV `123` for all.

---

## 2. One-time setup (already done — verify only)

| Item | Where |
|---|---|
| 1000 MMS users `bulktest-b0930-0001…1000@example.com` (user_id 1704–2703) | `shukria_mms_demo_local.users` |
| Credentials: merchant_id `BULKTEST####`, region `custom`, host `http://localhost:4444`, method `direct`, `direct_approved=1`, 3DS off, order prefix `BT####-` | `shukria_transactions.mpgs_merchant_credentials` |
| One Direct API key per merchant (hash only in DB) | `shukria_transactions.mpgs_direct_api_keys` |
| Plaintext keys | `/home/priyanka/bulk_keys.csv` (mode 600 — **never commit, never share**) |
| Simulator branch `feature/bulk-merchant-direct-flow` | this repo |

Onboarding was done with CloudLayer's own task (only touches the bulk users):

```bash
# from MPGS-Cloudlayer/CloudLayer — DO NOT re-run unless re-onboarding
INCLUDE_PAYMENT_GATEWAY=true MPGS_CREDENTIAL_KEY=<shared key> \
  mix mpgs.bulk_onboard --api-password 'bulk_password_123' \
    --keys-out /home/priyanka/bulk_keys.csv --dry-run
```

---

## 3. Pre-flight checks (run before every test session)

Every command is self-contained: copy it as-is into any terminal. The simulator
login `merchant.BULKTEST0001:bulk_password_123` works for all `/admin/*` routes.

### 3.1 Simulator is up

```bash
curl -s http://localhost:4444/health
```
- **Expect:** `{"status":"UP",...}`
- **If not:** start it from this repo: `mix run --no-halt` (port 4444 by default).

### 3.2 CloudLayer 4056 is up

```bash
curl -s -o /dev/null -w "%{http_code}\n" -X POST \
  http://localhost:4056/pgpayments/api/direct/pay -H 'Content-Type: application/json' -d '{}'
```
- **Expect:** `401` (no key sent — proves the Direct API is listening).
- **If `000`:** 4056 is down. Start it from `MPGS-Cloudlayer/CloudLayer`.
- **Never** test this endpoint with a *valid* key and an empty body — CloudLayer
  accepts it and creates a real 0.00 transaction.

### 3.3 Clear the simulator's transactions

```bash
curl -s -u "merchant.BULKTEST0001:bulk_password_123" -X DELETE http://localhost:4444/admin/transactions
```
- **What:** wipes the simulator's in-memory transactions so stats show only this session.
- **Expect:** `{"message":"All transactions cleared","result":"OK"}`
- Does **not** touch CloudLayer's database.

### 3.4 Check the simulator counts

```bash
curl -s -u "merchant.BULKTEST0001:bulk_password_123" http://localhost:4444/admin/stats | python3 -m json.tool
```
- **What:** totals, counts by status, and counts per merchant.
- **Expect after 3.3:** `"total": 0`.

### 3.5 Database connection pool on 4056

```bash
grep -n pool_size /var/www/internaltesting/priyanka/prverification/MPGS-Cloudlayer/CloudLayer/config/dev.exs
```
- **What:** how many MySQL connections 4056 may use per repo.
- **Expect:** `30` on both lines (see section 8 for why). After changing it, **restart 4056**.

---

## 4. Single payment checks

### 4.1 Simulator only — Direct flow with curl (no CloudLayer)

Proves the simulator handles the three Direct calls correctly. Run the three
commands **in the same terminal**: the first saves the session id in `$SID` and
the other two use it (`echo $SID` must print `SESSION_...`).

```bash
SID=$(curl -s -u "merchant.BULKTEST0001:bulk_password_123" -X POST -H 'Content-Type: application/json' -d '{}' \
  http://localhost:4444/api/rest/version/100/merchant/BULKTEST0001/session \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["session"]["id"])')
echo $SID
```
- **What:** creates a session. **Expect:** `SESSION_...`

```bash
curl -s -u "merchant.BULKTEST0001:bulk_password_123" -X PUT -H 'Content-Type: application/json' \
  -d '{"sourceOfFunds":{"provided":{"card":{"number":"5123450000000008","expiry":{"month":"12","year":"27"},"securityCode":"123","nameOnCard":"Test"}}}}' \
  "http://localhost:4444/api/rest/version/100/merchant/BULKTEST0001/session/$SID"
```
- **What:** attaches the card to the session. **Expect:** `{"result":"SUCCESS",...}`
- Simulator log shows the number masked (`************0008`), no `securityCode`.

```bash
curl -s -u "merchant.BULKTEST0001:bulk_password_123" -X PUT -H 'Content-Type: application/json' \
  -d "{\"apiOperation\":\"PAY\",\"order\":{\"amount\":\"10.00\",\"currency\":\"AED\"},\"session\":{\"id\":\"$SID\"}}" \
  "http://localhost:4444/api/rest/version/100/merchant/BULKTEST0001/order/MANUAL-1/transaction/1" | python3 -m json.tool | grep -E '"result"|gatewayCode|scheme'
```
- **What:** PAY with only the session id. **Expect:** `SUCCESS`, `APPROVED`, `MASTERCARD`.
- **If `scheme` is `UNKNOWN`:** the session card was not picked up.
- **If the shell shows `>`:** a quote is unbalanced — press Ctrl+C and re-paste.

### 4.2 End-to-end in the browser (CloudLayer 4056 → simulator)

Page: `https://internal.aritic.app/priyanka/mgps_direct_bulk/index.php`
(a copy of `/var/www/mercury/mgps_direct`; its `config.php` points at
`https://mercurypay.aritic.app/pn/pgpayments/api/direct` with BULKTEST0001's key).

1. Open the page, pay with `5123450000000008`, `12/27`, `123` → approved.
2. Pay again with `5999990000000000` → declined.
3. Check `curl -s -u "merchant.BULKTEST0001:bulk_password_123" http://localhost:4444/admin/stats` → BULKTEST0001 shows 1 APPROVED, 1 DECLINED.

To test another merchant, change `DIRECT_API_KEY` in that copy's `config.php` to the
merchant's key from `bulk_keys.csv`:

```bash
awk -F, '$3=="BULKTEST0002"{print $4}' /home/priyanka/bulk_keys.csv
```

The original `/var/www/mercury/mgps_direct` page is the **client's** (→ 4065). Do not edit it.

---

## 5. Load test — `scripts/bulk_load_test.py`

### 5.1 What it does

For each stage N: sends one Direct payment for each of the first N merchants in
`bulk_keys.csv`, **all released at the same instant**, each with its own API key.
Before/after each stage it reads the simulator's `/admin/stats` to confirm every
merchant's payment reached the gateway. Writes every request to
`load_results_<timestamp>.csv` (no keys). Asks for confirmation unless `--yes`.

### 5.2 Commands

**Always bypass Apache for load tests** (`--url http://localhost:4056/...`).
Apache allows only 150 concurrent requests for *every* site on the server,
including the client's 4065 traffic.

Run from the repo root:

```bash
cd /var/www/internaltesting/priyanka/prverification/MastercardSimulator
```

| Purpose | Command |
|---|---|
| Baseline, one payment | `python3 scripts/bulk_load_test.py --stages 1 --yes --url http://localhost:4056/pgpayments/api/direct/pay` |
| Small burst | `python3 scripts/bulk_load_test.py --stages 10 --yes --url http://localhost:4056/pgpayments/api/direct/pay` |
| Ramp, stop at first failure | `python3 scripts/bulk_load_test.py --stages 10,20,50,100 --stop-on-fail --yes --url http://localhost:4056/pgpayments/api/direct/pay` |
| All declines | add `--card 5999990000000000` |
| Longer pause between stages | add `--pause 30` |
| More patience per request | add `--timeout 120` |

Options: `--keys` (CSV path), `--url`, `--stages`, `--pause`, `--timeout`,
`--card`, `--amount`, `--currency`, `--sim`, `--stop-on-fail`, `--yes`, `--out`.

### 5.3 Reading the output

```
── Stage 100: 100 merchants ──
  CloudLayer   success 100  |  not success 0
  latency ms   p50 10429  p95 10957  max 11044  mean 8953
  wall time    11.2s  ->  9.0 payments/s
  simulator    +100 transactions {'APPROVED': 100}
  ✓ every merchant in this stage reached the simulator
```

| Line | Meaning |
|---|---|
| `success` / `not success` | CloudLayer's answer per request (`status == "success"`) |
| `latency ms` | per-request time; watch p95 as stages grow |
| `wall time -> payments/s` | stage throughput = N ÷ wall time |
| `failures:` | grouped by HTTP code + message, with example merchants |
| `simulator +N` | transactions that actually reached the gateway |
| `⚠ … no new simulator transaction` | CloudLayer failed **before** calling MPGS |

**Failed in CloudLayer but present in the simulator** = the gateway approved but
CloudLayer reported an error. This is the most serious outcome (see section 8).

Approved-card runs: any simulator transaction with `"scheme": "UNKNOWN"` means the
card never reached the gateway:

```bash
curl -s -u "merchant.BULKTEST0001:bulk_password_123" http://localhost:4444/admin/transactions \
  | python3 -c 'import sys,json; t=json.load(sys.stdin)["transactions"]; print("total",len(t),"| no card:",sum(x["scheme"]=="UNKNOWN" for x in t))'
```
(The decline card `5999…` also reports `UNKNOWN`; only use this check with `5123…`.)

---

## 6. Isolation tests — simulator or CloudLayer?

### 6.1 Simulator alone (no CloudLayer, no DB, no Apache)

100 merchants run session → card → PAY directly against port 4444, all at once.

```bash
python3 - <<'EOF'
import json, time, base64, threading, urllib.request
from concurrent.futures import ThreadPoolExecutor
B = "http://localhost:4444/api/rest/version/100/merchant/"
N = 100
bar = threading.Barrier(N)
def call(m, method, path, body):
    req = urllib.request.Request(B + m + path, json.dumps(body).encode(), method=method)
    req.add_header("Content-Type", "application/json")
    req.add_header("Authorization", "Basic " + base64.b64encode(f"merchant.{m}:bulk_password_123".encode()).decode())
    return json.load(urllib.request.urlopen(req, timeout=30))
def one(i):
    m = f"BULKTEST{i:04d}"; bar.wait(); t = time.perf_counter()
    sid = call(m, "POST", "/session", {})["session"]["id"]
    call(m, "PUT", f"/session/{sid}", {"sourceOfFunds": {"provided": {"card": {"number": "5123450000000008", "expiry": {"month": "12", "year": "27"}}}}})
    r = call(m, "PUT", f"/order/SIMONLY-{i}/transaction/1", {"apiOperation": "PAY", "order": {"amount": "10.00", "currency": "AED"}, "session": {"id": sid}})
    return r["result"], (time.perf_counter() - t) * 1000
t0 = time.perf_counter()
res = list(ThreadPoolExecutor(N).map(one, range(1, N + 1)))
lat = sorted(l for _, l in res)
print(f"{sum(r == 'SUCCESS' for r, _ in res)}/{N} ok | p50 {lat[N//2]:.0f}ms max {lat[-1]:.0f}ms | wall {time.perf_counter()-t0:.2f}s")
EOF
```
- **Expect:** `100/100 ok`, well under 1s.
- Leaves 100 `SIMONLY-*` transactions in the simulator — clear with 3.3 afterwards.

### 6.2 CloudLayer intake alone (no simulator, no transaction created)

100 requests at once with an invalid key: CloudLayer does the key lookup, rejects, stops.

```bash
seq 100 | xargs -P100 -I{} curl -s -o /dev/null -w "%{time_total}\n" \
  -X POST http://localhost:4056/pgpayments/api/direct/pay \
  -H "Authorization: Bearer shkdirect_invalid" -H "Content-Type: application/json" -d '{}' \
  | sort -n | awk '{a[NR]=$1} END {print "p50", a[int(NR/2)], "s  max", a[NR], "s"}'
```

### 6.3 Interpreting 6.1 + 6.2

| 6.1 simulator | 6.2 CloudLayer intake | Conclusion |
|---|---|---|
| fast | slow | CloudLayer queues requests before any payment work |
| fast | fast | slowdown is in CloudLayer's later steps (save, order number, MPGS calls) |
| slow | — | simulator problem |

---

## 7. Database checks (read-only)

Connect with the credentials from `MPGS-Cloudlayer/CloudLayer/config/dev.exs`
(`mysql -u root -p`). Only run `SELECT` / `SHOW` here.

**Bulk merchant configuration — expect one row, n = 1000:**
```sql
SELECT region, custom_gateway_host, method, direct_approved, threedsecure, active,
       COUNT(*) n, MIN(merchant_id), MAX(merchant_id)
FROM shukria_transactions.mpgs_merchant_credentials
WHERE user_id BETWEEN 1704 AND 2703
GROUP BY 1,2,3,4,5,6;
```

**Nobody outside the bulk users uses a BULKTEST id or the simulator host:**
```sql
SELECT user_id, merchant_id, custom_gateway_host
FROM shukria_transactions.mpgs_merchant_credentials
WHERE NOT (user_id BETWEEN 1704 AND 2703)
  AND (merchant_id LIKE 'BULKTEST%' OR custom_gateway_host LIKE '%4444%');
```
(User 1629 `TEST_MERCHANT_2` predates this work and points at the simulator; it is not a client merchant.)

**Outcome of one load-test run** — replace the run id (`LT<run_id>` from the order ids in the results CSV):
```sql
SELECT closure_status, response_code, (gateway_session_id IS NOT NULL) has_session, COUNT(*) n
FROM shukria_transactions.pg_transactions
WHERE order_number LIKE 'LT1007104256-%'
GROUP BY 1,2,3;
```
| Row | Meaning |
|---|---|
| `CLOSED` / `APPROVED` | success |
| `PENDING`, has_session = 1 | reached the gateway, CloudLayer failed afterwards |
| `PENDING`, has_session = 0 | CloudLayer failed before contacting the gateway |
| fewer rows than requests | CloudLayer failed before saving anything |

**When did rows get saved (finds where time is spent):**
```sql
SELECT created_dateTime, COUNT(*) n
FROM shukria_transactions.pg_transactions
WHERE order_number LIKE 'LT1007104256-%'
GROUP BY 1 ORDER BY 1;
```

**MySQL headroom (connections are shared with every app on the server):**
```sql
SHOW VARIABLES LIKE 'max_connections';
SHOW STATUS LIKE 'Threads_connected';
```

---

## 8. Findings so far (2026-10-07)

| Load (burst) | Path | Pool | Success | p50 | Notes |
|---|---|---|---|---|---|
| 1 | Apache | 3 | 100% | 0.8s | baseline |
| 10 | Apache | 3 | 100% | 2.5s | queued on 3 DB connections |
| 50 | Apache | 10 | 88% | 5.4s | 6 × HTTP 500, 4 of them approved at gateway |
| 100 | Apache | 10 | 77% | 5.2s | 23 × HTTP 500 at exactly ~5.2s, 4 approved at gateway |
| 100 | Apache | 30 | 100% | 10.4s | no failures, ~9 payments/s |

1. **5-second cut-off.** With too few DB connections, requests waiting ~5s are
   dropped with HTTP 500. Raising `pool_size` to 30 removed the failures.
2. **Approved payments end up `EXPIRED` (CloudLayer bug).** When CloudLayer fails
   *after* the gateway approves, the row stays `PENDING` and the expiry job later
   marks it `EXPIRED` without asking the gateway. Real-world effect: card charged,
   merchant told it failed, no record. Evidence: `pg_transactions` 1466, 1473, 1475, 1482.
3. **Time is spent before the row is saved.** In the pool-30 run, rows were saved
   2–10s after the burst started; each PAY reached the simulator in the same second
   its row was saved. The simulator answers in < 1ms.
4. **Apache caps concurrency at 150 for all sites.** Load through Apache measures
   Apache and degrades other sites — use the direct `localhost:4056` URL.
5. **The server is shared and overloaded:** 4 CPU cores, load average 11–13 at rest,
   ~346 of 600 MySQL connections in use. Throughput numbers here are a lower bound.
   Raise `pool_size` only while measured throughput keeps rising.

"1000 merchants" ≠ 1000 requests in the same millisecond. Agree a target **rate**
(e.g. 1000 merchants within 1 minute ≈ 17 payments/s) and test sustained load at that rate.

---

## 9. Cleanup

**Simulator** (in-memory; also cleared by restarting it):
```bash
curl -s -u "merchant.BULKTEST0001:bulk_password_123" -X DELETE http://localhost:4444/admin/transactions
```

**Results files** (local, not committed):
```bash
ls load_results_*.csv
```

**Deactivate the bulk merchants** when testing is finished (only touches users
matching the bulk email pattern; dry-run first):
```bash
# from MPGS-Cloudlayer/CloudLayer
INCLUDE_PAYMENT_GATEWAY=true MPGS_CREDENTIAL_KEY=<shared key> \
  mix mpgs.bulk_onboard --deactivate --dry-run
```

**Keys file:** `/home/priyanka/bulk_keys.csv` holds 1000 live keys. Delete it once
the merchants are deactivated.