Change & Release Management Procedure

The real deploy mechanism: the box pulls, nothing pushes to it

Production deploy is pull-based, via a systemd timer, not push-based CI/CD. This is a deliberate architectural choice, not a stopgap: .github/workflows/deploy.yml "is gone. It never once succeeded, and it never could have without a Tailscale credential" — the host's ufw policy allows port 22 only on the tailscale0 interface and from Docker's internal network, so a GitHub Actions runner's public IP is dropped by firewall policy, not by a fail2ban ban (docs/runbooks/deploy.md:15-19, confirmed fail2ban-client status sshd reports 0 bans ever — it is firewall policy). The upside stated in the runbook: CI holds no SSH key and no root credential for production. This procedure documents that mechanism as it actually runs, not an idealized push-deploy pipeline.

The actual change path

1. PR opened, CI gate runs. See PRO-04 — dos-ci.yml's verify check must pass before merge.
2. PR merged (squash) to `main`. This is the only human-initiated step in the deploy path;

everything after this is automatic.

1. Next timer tick pulls and redeploys. stratify-deploy.timer runs

/usr/local/bin/stratify-deploy-agent every 2 minutes. When it detects origin/main has moved, it fetches, rebuilds the stratify-web Docker image from its own git checkout at /data/stratify/src, restarts the container, and health-checks the result (docs/runbooks/deploy.md:20-24).

1. Successful deploys are silent by design — no notification is sent on success; the way to

confirm what's live is:

   ssh [email protected] 'cat /var/lib/stratify-deploy/deployed_sha'

1. A failed health check is recorded, not retried automatically. The failing SHA is written

to /var/lib/stratify-deploy/failed_sha and the timer will not attempt that SHA again — this prevents a broken build from rebuilding itself every 2 minutes, but it also means a failed deploy requires a human to notice and act (see "Post-deploy verification" below); it is not self-healing.

# watch a deploy happen
ssh [email protected] 'journalctl -u stratify-deploy -f'

# what is live right now
ssh [email protected] 'cat /var/lib/stratify-deploy/deployed_sha'

# force a redeploy of current main
ssh [email protected] 'rm -f /var/lib/stratify-deploy/deployed_sha && systemctl start stratify-deploy'

Post-deploy verification

docs/runbooks/deploy.md documents exactly one smoke check, tied to the manual deploy path (infra/hetzner/deploy-web-hetzner.sh), not the automatic timer-driven path:

curl -s -o /dev/null -w '%{http_code}\n' https://app.stratifyinvest.com/api/health

Gap, stated plainly: there is no documented post-deploy verification step tied to the automatic timer-driven path specifically — the timer's own health check gates whether the deploy "succeeds" from the agent's point of view (it decides whether to write deployed_sha or failed_sha), but there is no separate, human-facing smoke-check procedure that runs after every automatic deploy to confirm the application is actually behaving correctly beyond that internal health check. Until one exists, the practical mitigation is running the curl smoke test above manually after a change you have reason to watch closely, and checking deployed_sha / journalctl -u stratify-deploy when in doubt. This gap should be closed by defining a lightweight smoke-check script the deploy agent (or a human, post-merge) runs against the health endpoint and one or two critical paths (e.g. /, /api/v1/mandates with a sandbox key, per the smoke-test list already used in the DB-recovery runbook, docs/runbooks/db-recovery.md:53-54).

Rollback — what's actually possible today, not an idealized version

There is no automated rollback. The runbook is explicit about why: "The containers are left running rather than rolled back — compose build has already replaced the previous image, so there is nothing to roll back to; the failure is surfaced loudly instead of guessed at" (docs/runbooks/deploy.md:41-44).

What's actually available:

1. Revert commit + wait for the next timer tick. The standard path: open a revert PR, get it

through the CI gate (PRO-04), merge to main, and the timer picks it up within 2 minutes like any other change. This is the same mechanism as a forward deploy — there is no separate, faster "rollback" code path, only a normal deploy of reverted code.

1. Manual intervention on the host, for anything more urgent than waiting for a revert PR to

clear CI: e.g. docker stop stratify-web to take the app down entirely (matches the DB-recovery runbook's "stop all writes" step, docs/runbooks/db-recovery.md:40-41), or manually checking out a known-good commit at /data/stratify/src and forcing a rebuild — this bypasses the CI gate entirely and should be treated as an emergency measure, logged after the fact, not a routine option.

1. `STRATIFY_READ_ONLY=1` as a soft-degradation option — server actions fail gracefully rather

than the app being fully down (docs/runbooks/db-recovery.md:88), useful when the underlying issue is data-related rather than a bad deploy specifically.

There is no idealized one-command rollback to describe honestly beyond the above — this section states what's actually possible today.

Database migrations — deliberately excluded from the automatic path

Unlike the web app, DB migrations are not applied by the deploy timer. Applying DDL unattended on every merge was a deliberate design decision against (docs/runbooks/deploy.md:61-62, "Deliberately not automated"). Migrations are run explicitly:

HETZNER_STRATIFY_HOST=100.72.131.123 HETZNER_STRATIFY_SSH_USER=root \
  ./scripts/with-stratify-hetzner-supabase-db.sh ./scripts/hetzner-supabase-db-push.sh --dry-run

--dry-run first, then dropped to apply. This means a merged PR that includes both application code and a migration does not atomically deploy both — the code goes live automatically within 2 minutes, the migration only when someone runs the command above. Any change that depends on a migration having landed first needs to sequence these two steps manually and deliberately, not assume the deploy timer handles it.

Secrets — also deliberately excluded

The deploy agent never writes to /data/stratify/web/.env. An agent that could rewrite production secrets would have to hold them, which defeats the point of the pull-based design removing CI's credentials (docs/runbooks/deploy.md:56-60). Secret rotation goes through the manual deploy path, infra/hetzner/deploy-web-hetzner.sh, run from a trusted machine — see PRO-01's leaver checklist and docs/secrets.md's rotation table for when this is triggered.

Evidence produced

• journalctl -u stratify-deploy on the host — the deploy log, timestamped, per attempt.
• /var/lib/stratify-deploy/deployed_sha and /var/lib/stratify-deploy/failed_sha — the current

and last-failed state, respectively.

• GitHub's merge history on main, cross-referenced against deployed_sha to confirm what's

actually live matches what was merged.

• Manual-deploy runs (secrets rotation, migrations) are ad hoc today — not centrally logged beyond

shell history and whatever the operator notes at the time. This is a gap: neither deploy-web-hetzner.sh nor hetzner-supabase-db-push.sh writes a persistent, reviewable record of when they were run and by whom, beyond the git commit each deploy corresponds to.

Review

Reviewed whenever the deploy mechanism itself changes (e.g. if a post-deploy smoke-check script is added, closing the gap noted above), and otherwise annually.