Production-readiness status — 2026-09-12

Verdict

The complete application runs locally from DevOps/local/compose.yml. All 16 declared containers start; declared health checks pass; the UI, API, AdminPanel, and GL explorer answer smoke checks.

The code is substantially production-hardened, but production approval remains blocked on authenticated end-to-end acceptance in staging and operational validation against the production Kuknos network. Those checks require staging credentials and deployment access that were not available during this work.

The only configured SSH target, ParsVDS-N1, was inspected read-only and is an infrastructure host; it does not run the Darano staging services or contain a Darano deployment directory. The actual staging host remains unidentified.

Completed implementation

GL is the admission boundary and durable Kuknos settlement coordinator.
Settlement states are persisted as PENDING, SUBMITTED, CONFIRMED, RETRYABLE, and MANUAL_REVIEW, with stable idempotency keys, bounded backoff, restart-safe claims, and stale-lock recovery.
Wallet persists the settlement key and signed XDR, re-enqueues missing GL work after restart, and reconciles GL state back to application transactions.
Transfers, IPG settlement, IRT withdrawals, commissions, referral claims, wallet activation, trustlines/gas funding, and marketplace settlement now build signed XDR and enqueue GL instead of submitting from production request handlers.
Marketplace exchange and commission legs are one atomic, multi-operation, multi-signature transaction.
GL/database/Kuknos unavailability makes financial writes fail closed while read-only services remain available.
Production configuration rejects the direct-submit IRT development mode.
Vandar tokens are protected with versioned AES-256-GCM encryption and plaintext rows are migrated before the watcher starts.
AdminPanel exposes the GL settlement queue read-only. Authorized operators can retry a settlement only through an authenticated GL command, with a required actor and reason. GL stores the operation in an immutable audit table.
GL periodically compares confirmed settlements with lifecycle evidence, persists missing/hash-mismatch findings in kuknos_reconciliation_issues, emits structured mismatch counts, and exposes the read-only issue registry in AdminPanel.
AdminPanel's obsolete source-only Kavenegar and HTML conversion dependencies were replaced with internal adapters. UV_DEFAULT_INDEX was removed from every project Dockerfile to avoid uv cache/index corruption; AdminPanel pins a fixed setuptools wheel for deterministic source-package builds.
The redemption repository's transactional insert is implemented and tested; it no longer contains a reachable not-implemented panic. Alert retry cancellation is deterministic even when cancellation occurs on the final attempt.
The unsupported authenticator-app setup HTTP endpoint was removed from routing and Swagger. Its protobuf RPC is explicitly deprecated; the implemented SMS/OTP ProcessTFAReq and CheckTFACode flow remains the supported 2FA contract.

Verified locally

go test ./..., go test -race ./..., go vet ./..., go build ./..., and formatting checks pass for API, Auth, Wallet, and GL.
Focused AdminPanel tests and Django system checks pass.
Current Docker images build for GL, API, Wallet, and AdminPanel.
GL migrations 1–5 are applied in the live local PostgreSQL instance, including the durable kuknos_reconciliation_issues registry.
The AdminPanel client authenticates to the GL retry command; a missing-ID smoke call reaches GL and returns FAILED_PRECONDITION rather than an auth or transport failure.
Local development uses an explicitly local-only Horizon-compatible mock because the external test Horizon returned HTTP 403 from the containers. Production and staging configuration continue to require the real Kuknos endpoint.
The actual GL submission worker submitted a persisted local smoke transaction to that endpoint and recorded CONFIRMED, one attempt, and a 64-character transaction hash.
Local smoke responses: UI 200, API 200, AdminPanel 302 login redirect, GL explorer 200.
API readiness now exposes Wallet readiness plus GL database readiness, settlement enablement/readiness, pending/retryable/manual-review counts, and oldest-pending age.
API readiness independently probes RabbitMQ with a bounded connection, exposes queue.reachable, and returns HTTP 503 when the queue is unreachable.
Live failure injection verified that stopping GL changes API readiness to HTTP 503 and restarting it restores HTTP 200. Stopping RabbitMQ likewise returns HTTP 503 with queue.reachable=false; recovery restores HTTP 200. The streamer and cron use unless-stopped restart policies so a transient container failure cannot leave either background process silently stopped.
Local startup ordering now waits for Wallet's RPC dependencies, the Alert service uses its current notification command, and GL local configuration uses the current database and graceful-shutdown keys.
The staging Compose topology now defines GL, its health check, durable settlement worker, and GL explorer; API and Wallet processes declare GL startup dependency, and AdminPanel is wired to the GL database and authenticated operator endpoint. The manifest validates with required secret placeholders but has not been deployed.
GL accepts its database password, settlement admin token, and Horizon endpoint from runtime environment variables, keeping those values out of its staging TOML templates.
Wallet and Auth accept all database, queue, provider, encryption, wallet-signing, SMTP, and blockchain credentials through explicit runtime overrides. The staging Compose file requires these values, and current staging TOML templates no longer contain non-empty credential material. Previously committed values still require rotation and Git-history incident handling.
Local infrastructure uses non-conflicting host ports: PostgreSQL 5433, Redis 6380, AMQP 5673, RabbitMQ UI 15673. Container-to-container ports are unchanged.

Remaining release gates

Identify and grant deployment access to the actual host serving stage.darano.ir; the available ParsVDS-N1 target does not host it.
Publish the new GL image and deploy the updated DevOps/services/srv-stage production-readiness topology; create gl_db before first startup and verify migrations 1–5 on the staging database.
Provision replacement staging credentials from a host-only secret source and rotate every value previously exposed in Git history. Current templates and Compose wiring no longer embed the credentials.
Run authenticated staging scenarios for login/2FA, wallet creation, deposits/IPG, withdrawals and approval, transfers, marketplace/ICO, redemption, referrals/commissions, balances, and history.
Exercise failure injection in a deployed environment: GL unavailable before acceptance, Kuknos timeout/retry, restart with pending work, duplicate submission, reconciliation mismatch, and concurrent balance mutations.
Confirm TLS, backups and restore, alert routing, queue limits, rollback procedure, and Kuknos credentials/network passphrase.
Run a real low-value Kuknos settlement and reconcile its transaction hash. Local tests prove orchestration and retry behavior but intentionally do not spend or submit a real financial transaction.

Rollback and operations

Stop financial ingress first; keep read-only UI/API and GL explorer running.
Do not delete settlement rows or rewrite GL journals during rollback.
Roll application images back while retaining database migrations and pending settlement records; the worker can safely resume because submissions are keyed idempotently.
Investigate MANUAL_REVIEW records in AdminPanel. Retry only after checking the Kuknos transaction/hash and recording a reason. Confirmed records cannot be retried.
If GL, its database, or Kuknos readiness is uncertain, leave financial writes disabled until reconciliation is clean.

Local commands

docker compose -f DevOps/local/compose.yml build
docker compose -f DevOps/local/compose.yml up -d
docker compose -f DevOps/local/compose.yml ps

Local URLs: UI http://localhost:3001, API http://localhost:3000, AdminPanel http://localhost:8080, GL explorer http://localhost:8601.

The complete stack and both GL and RabbitMQ outage/recovery paths were verified live after Docker Desktop was restarted.

Public staging was rechecked on 2026-09-12: stage.darano.ir responds, while stage.api.darano.ir still returns the legacy {"data":"Server is up and running"} payload. The detailed GL readiness build in this report has therefore not been deployed to staging.

See 08-settlement-operations-runbook.md for incident handling, recovery, rollback, and alert requirements.

8.7 KiB Raw Blame History Unescape Escape