# Runbook: Service Controls and Circuit Breakers

**Service:** WalletWeb Admin Console
**Owner:** SRE Team
**Escalation:** Platform Team Lead

---

## Overview
Procedures for managing service controls, feature toggles, and circuit breakers during incidents or planned maintenance.

## Prerequisites
- Admin console access with `sre` or `admin` role
- Change control ticket (for production changes)
- Incident ticket (for emergency controls)

## Procedure

### Accessing Service Controls

1. Log in to `/admin/login` with SRE or admin credentials
2. Navigate to `/admin/services`
3. View current control states

### Control Types

#### 1. Feature Toggles
Enable/disable features without code deployment.

**Available Toggles:**
- `enable_transfers`: Master toggle for all transfer operations
- `enable_external_transfers`: External bank transfers only
- `enable_bill_payments`: Bill payment functionality
- `enable_loan_applications`: Loan application submissions
- `enable_insurance_enrollment`: Insurance product enrollment
- `enable_card_operations`: Card freeze/unfreeze/limits

**To Toggle a Feature:**
1. Locate feature in Service Controls list
2. Click toggle switch
3. Enter reason in modal (required)
4. Enter change control/incident ticket ID
5. Confirm with MFA
6. System will emit `ServiceControlChanged` event

**Effect:** Disabled features return user-friendly messages to customers. No data loss.

#### 2. Circuit Breakers
Automatically trip when dependency failure thresholds are exceeded.

**Monitored Dependencies:**
- Payment gateway (external API)
- SMS provider (OTP delivery)
- Email service (notifications)
- KYC provider API
- Settlement rails

**Circuit States:**
- 🟢 **Closed**: Normal operation, requests routed to dependency
- 🟡 **Open**: Dependency unavailable, requests fail-fast
- 🔵 **Half-Open**: Testing recovery, limited requests allowed

**Manual Circuit Control:**
1. Locate circuit breaker in list
2. Click "Force Open" to manually trip (emergency)
3. Click "Reset" to attempt recovery (after fix confirmed)

**Auto-Recovery:** Circuits automatically transition to half-open after cooldown period (5 minutes), then closed if dependency responds successfully.

#### 3. Rate Limit Overrides
Temporarily adjust rate limits for specific operations.

**Use Cases:**
- Increase limits during high-volume periods (e.g., salary day)
- Decrease limits during suspected attack (e.g., credential stuffing)

**To Override:**
1. Navigate to "Rate Limits" tab
2. Select operation type (e.g., `transfer_create`, `login_attempt`)
3. Enter new limit value
4. Set expiry time (auto-reverts after expiry)
5. Enter justification and ticket ID
6. Confirm

---

## Common Scenarios

### Scenario 1: External Payment Gateway Outage

**Symptoms:** Failed external transfers, error logs showing timeout/503

**Actions:**
1. Open circuit breaker for payment gateway
   - This fails new external transfers fast (no hanging requests)
2. Disable `enable_external_transfers` toggle
   - Prevents customers from initiating new external transfers
3. Post status update to customer-facing status page
4. Monitor gateway vendor status page for recovery
5. When gateway recovers:
   - Reset circuit breaker
   - Re-enable `enable_external_transfers` toggle
   - Verify with test transaction

### Scenario 2: SMS OTP Delivery Delays

**Symptoms:** Customers reporting OTP not received, SMS queue backlog

**Actions:**
1. Check SMS circuit breaker status (may be half-open due to delays)
2. If SMS provider is degraded:
   - Force circuit open to prevent further queue buildup
   - Enable fallback: email OTP (if configured)
3. Reduce rate limit for `send_otp` operation (prevent abuse)
4. Coordinate with SMS vendor
5. After resolution:
   - Reset circuit breaker
   - Restore normal rate limits

### Scenario 3: Planned Maintenance (e.g., Settlement System Upgrade)

**Pre-Maintenance (1 hour before):**
1. Disable `enable_external_transfers` toggle
2. Post maintenance notice to customers
3. Coordinate with stakeholders

**During Maintenance:**
- Monitor service controls dashboard
- Internal transfers continue normally (not affected)

**Post-Maintenance:**
1. Verify settlement system health
2. Run smoke test: initiate test external transfer
3. Re-enable toggle
4. Monitor for anomalies (15-30 minutes)

---

## Troubleshooting

### Toggle Change Fails
- **Error**: "Policy denied: insufficient role"
  - **Fix**: Verify you have `sre` or `admin` role
- **Error**: "MFA verification failed"
  - **Fix**: Re-authenticate

### Circuit Breaker Won't Reset
- **Check**: Is dependency actually healthy? Test manually via curl/postman
- **Check**: Check recent error logs (last 5 minutes)
- **Action**: If dependency is confirmed healthy, force reset with override flag

### Feature Still Active After Disable
- **Check**: Cache TTL (feature flags cached for 60 seconds)
- **Wait**: Allow 60 seconds for cache expiry
- **Verify**: Test from incognito/private browser window

---

## Audit and Compliance
- All service control changes emit privileged audit events
- Changes logged with: admin_id, control_name, old_value, new_value, reason, ticket_id
- Weekly review by Platform Team for unauthorized changes
- Emergency overrides (no ticket ID) flagged for post-incident review

---

## Rollback
All service controls are **instantly reversible**:
- Toggle a feature back on
- Close/reset a circuit breaker
- Remove rate limit override

No code deployment required. Effect is immediate after cache TTL (60s).

---

## Related Runbooks
- [Health Check and SLO Monitoring](./health-check-slo.md)
- [Incident Response](./incident-response.md)
- DR Rehearsal (SRE internal runbook)