dev-procfile/report/08-settlement-operations-runbook.md

GL/Kuknos settlement operations runbook

Scope and invariants

GL is the financial source of truth and admission boundary. Kuknos is required eventual settlement. Never edit or delete GL journals, transaction events, settlement rows, or operator-audit rows. Never retry a CONFIRMED settlement. Never mark an application transaction successful from an operator console without matching GL and Kuknos evidence.

The kuknos-mock service in DevOps/local/compose.yml exists only for deterministic local orchestration tests. Never deploy it or point staging/production settlement configuration at it; release validation requires an authorized transaction against the real Kuknos network.

Normal lifecycle

1. The application records/adopts the value change through GL.
2. GL stores one signed XDR under a stable idempotency key.
3. The worker claims PENDING or RETRYABLE work with a bounded lease.
4. Kuknos submission returns a transaction hash and GL records CONFIRMED.
5. Wallet reconciliation stores the hash and final application status.
6. GL reconciliation compares the confirmed hash with lifecycle evidence.

Health and alert interpretation

• database_ready=false: stop financial ingress immediately. Read-only pages may remain available.
• settlement_ready=false: stop financial ingress. Check GL database and worker logs.
• Increasing settlement_pending: check worker activity and Horizon latency.
• Increasing settlement_retryable: check Horizon response codes, TLS, DNS, passphrase, and XDR validity.
• Any settlement_manual_review: operator investigation is required.
• Any open reconciliation issue: compare the settlement, latest GL transaction event, Wallet transaction, and Kuknos transaction before taking action.
• MISSING_LEDGER_CONFIRMATION: Kuknos confirmed but matching lifecycle evidence did not arrive within ten minutes.
• TRANSACTION_HASH_MISMATCH: GL lifecycle evidence and Kuknos settlement reference different hashes. Treat this as a high-severity integrity incident.

Manual-review procedure

1. Record the incident/ticket identifier.
2. Locate the settlement by idempotency key in AdminPanel.
3. Verify source service, source transaction ID, network, attempts, signed XDR, expected hash, and last error.
4. Query Kuknos independently using the known transaction hash and source account sequence. A timeout is not proof that submission failed.
5. Compare the Wallet transaction and GL transaction-event history.
6. If Kuknos already accepted the transaction, do not retry; allow reconciliation to repair local state or escalate a mismatch.
7. If Kuknos proves the transaction was not accepted and the signed XDR remains valid, select the settlement in AdminPanel, enter a specific reason including the incident ID, and invoke retry.
8. Confirm that GL creates an immutable RETRY audit row containing actor, reason, prior status, and resulting status.
9. Watch the settlement until CONFIRMED or MANUAL_REVIEW. Do not repeatedly force retries.

Outage recovery

1. Keep financial writes disabled while GL, PostgreSQL, or Kuknos readiness is uncertain.
2. Restore PostgreSQL first, then GL, then the settlement worker, then Wallet/API financial ingress.
3. Do not purge queues or pending rows. Stale worker locks release automatically after their lease.
4. Confirm GL health and backlog age are stable.
5. Run/observe reconciliation and resolve every mismatch before reopening financial writes.
6. Re-enable traffic gradually and monitor retry rate, oldest pending age, database errors, and transaction conservation.

Rollback

1. Disable financial ingress.
2. Retain database migrations and all settlement/audit records.
3. Roll application images back to the previous known-good versions.
4. Start GL before Wallet financial services.
5. Resume the worker with the same database; stable idempotency keys make pending submissions restart-safe.
6. Reconcile all work created across the deployment boundary before reopening traffic.

Required production alerts

• GL database or settlement readiness false for two consecutive probes.
• Oldest pending settlement above five minutes.
• Retryable count increasing for five minutes.
• Any transition to MANUAL_REVIEW.
• Any new reconciliation issue, especially hash mismatch.
• Reconciliation job failure or absence of successful runs for five minutes.