Navid
67 lines
4.3 KiB
Markdown
67 lines
4.3 KiB
Markdown
# GL/Kuknos settlement operations runbook
|
|
|
|
## Scope and invariants
|
|
|
|
GL is the financial source of truth and admission boundary. Kuknos is required eventual settlement. Never edit or delete GL journals, transaction events, settlement rows, or operator-audit rows. Never retry a CONFIRMED settlement. Never mark an application transaction successful from an operator console without matching GL and Kuknos evidence.
|
|
|
|
The `kuknos-mock` service in `DevOps/local/compose.yml` exists only for deterministic local orchestration tests. Never deploy it or point staging/production settlement configuration at it; release validation requires an authorized transaction against the real Kuknos network.
|
|
|
|
## Normal lifecycle
|
|
|
|
1. The application records/adopts the value change through GL.
|
|
2. GL stores one signed XDR under a stable idempotency key.
|
|
3. The worker claims PENDING or RETRYABLE work with a bounded lease.
|
|
4. Kuknos submission returns a transaction hash and GL records CONFIRMED.
|
|
5. Wallet reconciliation stores the hash and final application status.
|
|
6. GL reconciliation compares the confirmed hash with lifecycle evidence.
|
|
|
|
## Health and alert interpretation
|
|
|
|
- `database_ready=false`: stop financial ingress immediately. Read-only pages may remain available.
|
|
- `settlement_ready=false`: stop financial ingress. Check GL database and worker logs.
|
|
- Increasing `settlement_pending`: check worker activity and Horizon latency.
|
|
- Increasing `settlement_retryable`: check Horizon response codes, TLS, DNS, passphrase, and XDR validity.
|
|
- Any `settlement_manual_review`: operator investigation is required.
|
|
- Any open reconciliation issue: compare the settlement, latest GL transaction event, Wallet transaction, and Kuknos transaction before taking action.
|
|
- `MISSING_LEDGER_CONFIRMATION`: Kuknos confirmed but matching lifecycle evidence did not arrive within ten minutes.
|
|
- `TRANSACTION_HASH_MISMATCH`: GL lifecycle evidence and Kuknos settlement reference different hashes. Treat this as a high-severity integrity incident.
|
|
|
|
## Manual-review procedure
|
|
|
|
1. Record the incident/ticket identifier.
|
|
2. Locate the settlement by idempotency key in AdminPanel.
|
|
3. Verify source service, source transaction ID, network, attempts, signed XDR, expected hash, and last error.
|
|
4. Query Kuknos independently using the known transaction hash and source account sequence. A timeout is not proof that submission failed.
|
|
5. Compare the Wallet transaction and GL transaction-event history.
|
|
6. If Kuknos already accepted the transaction, do not retry; allow reconciliation to repair local state or escalate a mismatch.
|
|
7. If Kuknos proves the transaction was not accepted and the signed XDR remains valid, select the settlement in AdminPanel, enter a specific reason including the incident ID, and invoke retry.
|
|
8. Confirm that GL creates an immutable `RETRY` audit row containing actor, reason, prior status, and resulting status.
|
|
9. Watch the settlement until CONFIRMED or MANUAL_REVIEW. Do not repeatedly force retries.
|
|
|
|
## Outage recovery
|
|
|
|
1. Keep financial writes disabled while GL, PostgreSQL, or Kuknos readiness is uncertain.
|
|
2. Restore PostgreSQL first, then GL, then the settlement worker, then Wallet/API financial ingress.
|
|
3. Do not purge queues or pending rows. Stale worker locks release automatically after their lease.
|
|
4. Confirm GL health and backlog age are stable.
|
|
5. Run/observe reconciliation and resolve every mismatch before reopening financial writes.
|
|
6. Re-enable traffic gradually and monitor retry rate, oldest pending age, database errors, and transaction conservation.
|
|
|
|
## Rollback
|
|
|
|
1. Disable financial ingress.
|
|
2. Retain database migrations and all settlement/audit records.
|
|
3. Roll application images back to the previous known-good versions.
|
|
4. Start GL before Wallet financial services.
|
|
5. Resume the worker with the same database; stable idempotency keys make pending submissions restart-safe.
|
|
6. Reconcile all work created across the deployment boundary before reopening traffic.
|
|
|
|
## Required production alerts
|
|
|
|
- GL database or settlement readiness false for two consecutive probes.
|
|
- Oldest pending settlement above five minutes.
|
|
- Retryable count increasing for five minutes.
|
|
- Any transition to MANUAL_REVIEW.
|
|
- Any new reconciliation issue, especially hash mismatch.
|
|
- Reconciliation job failure or absence of successful runs for five minutes.
|