diff --git a/Procfile b/Procfile index 181b809..df92ded 100644 --- a/Procfile +++ b/Procfile @@ -1,21 +1,22 @@ # Go services — air builds on first run and rebuilds+restarts on any .go change -api: cd api && sleep 2 && air -auth: cd auth && air -gl: cd GL && air +api: cd api && sleep 2 && air -log.silent=true 2>&1 | cor +auth: cd auth && air -log.silent=true 2>&1 | cor +gl: cd GL && air -log.silent=true -c .gl.air.toml 2>&1 | cor +gl-dash: cd GL && air -log.silent=true -c .dashboard.air.toml 2>&1 | cor # Wallet binary hosts multiple sub-services; each gets its own air config + cfg file. # All five run `make build` independently — Go's build cache serialises concurrent builds safely. -wallet: cd wallet && air -c .wallet.air.toml -wallet-market: cd wallet && air -c .market.air.toml -wallet-alert: cd wallet && air -c .alert.air.toml -wallet-internal: cd wallet && air -c .internal.air.toml -wallet-stream: cd wallet && air -c .stream.air.toml +wallet: cd wallet && air -log.silent=true -c .wallet.air.toml 2>&1 | cor +wallet-market: cd wallet && air -log.silent=true -c .market.air.toml 2>&1 | cor +wallet-notif: cd wallet && air -log.silent=true -c .notification.air.toml 2>&1 | cor +wallet-internal: cd wallet && air -log.silent=true -c .internal.air.toml 2>&1 | cor +wallet-stream: cd wallet && air -log.silent=true -c .stream.air.toml 2>&1 | cor # Django — built-in autoreloader watches .py / .html / .po files and restarts on change -admin: cd AdminPanel && .venv/bin/python src/manage.py runserver 0.0.0.0:8080 --traceback +# admin: cd AdminPanel && .venv/bin/python src/manage.py runserver 0.0.0.0:8080 --traceback # Next.js — yarn dev includes HMR out of the box; buf generate runs automatically before dev server ui: cd ui && pnpm dev -H 0.0.0.0 -p 3000 # Mkdocs - Darano and Dezone documnetaitons -docs: cd docs && make dev +# docs: cd docs && make dev diff --git a/report/07-production-readiness-status.md b/report/07-production-readiness-status.md new file mode 100644 index 0000000..f99e9c5 --- /dev/null +++ b/report/07-production-readiness-status.md @@ -0,0 +1,78 @@ +# Production-readiness status — 2026-09-12 + +## Verdict + +The complete application runs locally from `DevOps/local/compose.yml`. All 16 declared containers start; declared health checks pass; the UI, API, AdminPanel, and GL explorer answer smoke checks. + +The code is substantially production-hardened, but production approval remains blocked on authenticated end-to-end acceptance in staging and operational validation against the production Kuknos network. Those checks require staging credentials and deployment access that were not available during this work. + +The only configured SSH target, `ParsVDS-N1`, was inspected read-only and is an infrastructure host; it does not run the Darano staging services or contain a Darano deployment directory. The actual staging host remains unidentified. + +## Completed implementation + +- GL is the admission boundary and durable Kuknos settlement coordinator. +- Settlement states are persisted as PENDING, SUBMITTED, CONFIRMED, RETRYABLE, and MANUAL_REVIEW, with stable idempotency keys, bounded backoff, restart-safe claims, and stale-lock recovery. +- Wallet persists the settlement key and signed XDR, re-enqueues missing GL work after restart, and reconciles GL state back to application transactions. +- Transfers, IPG settlement, IRT withdrawals, commissions, referral claims, wallet activation, trustlines/gas funding, and marketplace settlement now build signed XDR and enqueue GL instead of submitting from production request handlers. +- Marketplace exchange and commission legs are one atomic, multi-operation, multi-signature transaction. +- GL/database/Kuknos unavailability makes financial writes fail closed while read-only services remain available. +- Production configuration rejects the direct-submit IRT development mode. +- Vandar tokens are protected with versioned AES-256-GCM encryption and plaintext rows are migrated before the watcher starts. +- AdminPanel exposes the GL settlement queue read-only. Authorized operators can retry a settlement only through an authenticated GL command, with a required actor and reason. GL stores the operation in an immutable audit table. +- GL periodically compares confirmed settlements with lifecycle evidence, persists missing/hash-mismatch findings in `kuknos_reconciliation_issues`, emits structured mismatch counts, and exposes the read-only issue registry in AdminPanel. +- AdminPanel's obsolete source-only Kavenegar and HTML conversion dependencies were replaced with internal adapters. `UV_DEFAULT_INDEX` was removed from every project Dockerfile to avoid uv cache/index corruption; AdminPanel pins a fixed setuptools wheel for deterministic source-package builds. +- The redemption repository's transactional insert is implemented and tested; it no longer contains a reachable not-implemented panic. Alert retry cancellation is deterministic even when cancellation occurs on the final attempt. +- The unsupported authenticator-app setup HTTP endpoint was removed from routing and Swagger. Its protobuf RPC is explicitly deprecated; the implemented SMS/OTP `ProcessTFAReq` and `CheckTFACode` flow remains the supported 2FA contract. + +## Verified locally + +- `go test ./...`, `go test -race ./...`, `go vet ./...`, `go build ./...`, and formatting checks pass for API, Auth, Wallet, and GL. +- Focused AdminPanel tests and Django system checks pass. +- Current Docker images build for GL, API, Wallet, and AdminPanel. +- GL migrations 1–5 are applied in the live local PostgreSQL instance, including the durable `kuknos_reconciliation_issues` registry. +- The AdminPanel client authenticates to the GL retry command; a missing-ID smoke call reaches GL and returns FAILED_PRECONDITION rather than an auth or transport failure. +- Local development uses an explicitly local-only Horizon-compatible mock because the external test Horizon returned HTTP 403 from the containers. Production and staging configuration continue to require the real Kuknos endpoint. +- The actual GL submission worker submitted a persisted local smoke transaction to that endpoint and recorded `CONFIRMED`, one attempt, and a 64-character transaction hash. +- Local smoke responses: UI 200, API 200, AdminPanel 302 login redirect, GL explorer 200. +- API readiness now exposes Wallet readiness plus GL database readiness, settlement enablement/readiness, pending/retryable/manual-review counts, and oldest-pending age. +- API readiness independently probes RabbitMQ with a bounded connection, exposes `queue.reachable`, and returns HTTP 503 when the queue is unreachable. +- Live failure injection verified that stopping GL changes API readiness to HTTP 503 and restarting it restores HTTP 200. Stopping RabbitMQ likewise returns HTTP 503 with `queue.reachable=false`; recovery restores HTTP 200. The streamer and cron use `unless-stopped` restart policies so a transient container failure cannot leave either background process silently stopped. +- Local startup ordering now waits for Wallet's RPC dependencies, the Alert service uses its current notification command, and GL local configuration uses the current database and graceful-shutdown keys. +- The staging Compose topology now defines GL, its health check, durable settlement worker, and GL explorer; API and Wallet processes declare GL startup dependency, and AdminPanel is wired to the GL database and authenticated operator endpoint. The manifest validates with required secret placeholders but has not been deployed. +- GL accepts its database password, settlement admin token, and Horizon endpoint from runtime environment variables, keeping those values out of its staging TOML templates. +- Wallet and Auth accept all database, queue, provider, encryption, wallet-signing, SMTP, and blockchain credentials through explicit runtime overrides. The staging Compose file requires these values, and current staging TOML templates no longer contain non-empty credential material. Previously committed values still require rotation and Git-history incident handling. +- Local infrastructure uses non-conflicting host ports: PostgreSQL 5433, Redis 6380, AMQP 5673, RabbitMQ UI 15673. Container-to-container ports are unchanged. + +## Remaining release gates + +1. Identify and grant deployment access to the actual host serving `stage.darano.ir`; the available `ParsVDS-N1` target does not host it. +2. Publish the new GL image and deploy the updated `DevOps/services/srv-stage` production-readiness topology; create `gl_db` before first startup and verify migrations 1–5 on the staging database. +3. Provision replacement staging credentials from a host-only secret source and rotate every value previously exposed in Git history. Current templates and Compose wiring no longer embed the credentials. +4. Run authenticated staging scenarios for login/2FA, wallet creation, deposits/IPG, withdrawals and approval, transfers, marketplace/ICO, redemption, referrals/commissions, balances, and history. +5. Exercise failure injection in a deployed environment: GL unavailable before acceptance, Kuknos timeout/retry, restart with pending work, duplicate submission, reconciliation mismatch, and concurrent balance mutations. +6. Confirm TLS, backups and restore, alert routing, queue limits, rollback procedure, and Kuknos credentials/network passphrase. +7. Run a real low-value Kuknos settlement and reconcile its transaction hash. Local tests prove orchestration and retry behavior but intentionally do not spend or submit a real financial transaction. + +## Rollback and operations + +- Stop financial ingress first; keep read-only UI/API and GL explorer running. +- Do not delete settlement rows or rewrite GL journals during rollback. +- Roll application images back while retaining database migrations and pending settlement records; the worker can safely resume because submissions are keyed idempotently. +- Investigate MANUAL_REVIEW records in AdminPanel. Retry only after checking the Kuknos transaction/hash and recording a reason. Confirmed records cannot be retried. +- If GL, its database, or Kuknos readiness is uncertain, leave financial writes disabled until reconciliation is clean. + +## Local commands + +```bash +docker compose -f DevOps/local/compose.yml build +docker compose -f DevOps/local/compose.yml up -d +docker compose -f DevOps/local/compose.yml ps +``` + +Local URLs: UI `http://localhost:3001`, API `http://localhost:3000`, AdminPanel `http://localhost:8080`, GL explorer `http://localhost:8601`. + +The complete stack and both GL and RabbitMQ outage/recovery paths were verified live after Docker Desktop was restarted. + +Public staging was rechecked on 2026-09-12: `stage.darano.ir` responds, while `stage.api.darano.ir` still returns the legacy `{"data":"Server is up and running"}` payload. The detailed GL readiness build in this report has therefore not been deployed to staging. + +See `08-settlement-operations-runbook.md` for incident handling, recovery, rollback, and alert requirements. diff --git a/report/08-settlement-operations-runbook.md b/report/08-settlement-operations-runbook.md new file mode 100644 index 0000000..8416249 --- /dev/null +++ b/report/08-settlement-operations-runbook.md @@ -0,0 +1,66 @@ +# GL/Kuknos settlement operations runbook + +## Scope and invariants + +GL is the financial source of truth and admission boundary. Kuknos is required eventual settlement. Never edit or delete GL journals, transaction events, settlement rows, or operator-audit rows. Never retry a CONFIRMED settlement. Never mark an application transaction successful from an operator console without matching GL and Kuknos evidence. + +The `kuknos-mock` service in `DevOps/local/compose.yml` exists only for deterministic local orchestration tests. Never deploy it or point staging/production settlement configuration at it; release validation requires an authorized transaction against the real Kuknos network. + +## Normal lifecycle + +1. The application records/adopts the value change through GL. +2. GL stores one signed XDR under a stable idempotency key. +3. The worker claims PENDING or RETRYABLE work with a bounded lease. +4. Kuknos submission returns a transaction hash and GL records CONFIRMED. +5. Wallet reconciliation stores the hash and final application status. +6. GL reconciliation compares the confirmed hash with lifecycle evidence. + +## Health and alert interpretation + +- `database_ready=false`: stop financial ingress immediately. Read-only pages may remain available. +- `settlement_ready=false`: stop financial ingress. Check GL database and worker logs. +- Increasing `settlement_pending`: check worker activity and Horizon latency. +- Increasing `settlement_retryable`: check Horizon response codes, TLS, DNS, passphrase, and XDR validity. +- Any `settlement_manual_review`: operator investigation is required. +- Any open reconciliation issue: compare the settlement, latest GL transaction event, Wallet transaction, and Kuknos transaction before taking action. +- `MISSING_LEDGER_CONFIRMATION`: Kuknos confirmed but matching lifecycle evidence did not arrive within ten minutes. +- `TRANSACTION_HASH_MISMATCH`: GL lifecycle evidence and Kuknos settlement reference different hashes. Treat this as a high-severity integrity incident. + +## Manual-review procedure + +1. Record the incident/ticket identifier. +2. Locate the settlement by idempotency key in AdminPanel. +3. Verify source service, source transaction ID, network, attempts, signed XDR, expected hash, and last error. +4. Query Kuknos independently using the known transaction hash and source account sequence. A timeout is not proof that submission failed. +5. Compare the Wallet transaction and GL transaction-event history. +6. If Kuknos already accepted the transaction, do not retry; allow reconciliation to repair local state or escalate a mismatch. +7. If Kuknos proves the transaction was not accepted and the signed XDR remains valid, select the settlement in AdminPanel, enter a specific reason including the incident ID, and invoke retry. +8. Confirm that GL creates an immutable `RETRY` audit row containing actor, reason, prior status, and resulting status. +9. Watch the settlement until CONFIRMED or MANUAL_REVIEW. Do not repeatedly force retries. + +## Outage recovery + +1. Keep financial writes disabled while GL, PostgreSQL, or Kuknos readiness is uncertain. +2. Restore PostgreSQL first, then GL, then the settlement worker, then Wallet/API financial ingress. +3. Do not purge queues or pending rows. Stale worker locks release automatically after their lease. +4. Confirm GL health and backlog age are stable. +5. Run/observe reconciliation and resolve every mismatch before reopening financial writes. +6. Re-enable traffic gradually and monitor retry rate, oldest pending age, database errors, and transaction conservation. + +## Rollback + +1. Disable financial ingress. +2. Retain database migrations and all settlement/audit records. +3. Roll application images back to the previous known-good versions. +4. Start GL before Wallet financial services. +5. Resume the worker with the same database; stable idempotency keys make pending submissions restart-safe. +6. Reconcile all work created across the deployment boundary before reopening traffic. + +## Required production alerts + +- GL database or settlement readiness false for two consecutive probes. +- Oldest pending settlement above five minutes. +- Retryable count increasing for five minutes. +- Any transition to MANUAL_REVIEW. +- Any new reconciliation issue, especially hash mismatch. +- Reconciliation job failure or absence of successful runs for five minutes.