Stratify
Engineering

Deploy

docs/runbooks/deploy.md

Source updated 03. Aug. 2026

Runbook: deploying to production

Prod is a single Hetzner box (coolify-main) reachable only over Tailscale — the public IP is fail2ban-banned for this network, all ports.

tailnet IP   100.72.131.123   (ssh root@ works)
public IP    128.140.8.187    (DROPped — do not probe)
app          https://app.stratifyinvest.com
supabase     https://app.stratifyinvest.com/supabase

How deploys happen: the box pulls

.github/workflows/deploy.yml is gone. It never once succeeded, and it never could have without a Tailscale credential — ufw on this host allows port 22 only on tailscale0 and from Docker's 10.0.0.0/8, so a GitHub runner's public IP is dropped by policy. (There is no fail2ban "ban" to lift: fail2ban-client status sshd reports 0 bans, ever. It is firewall policy, not a jail.)

Instead the box polls. stratify-deploy.timer runs /usr/local/bin/stratify-deploy-agent every 2 minutes; when origin/main moves it fetches, rebuilds the web image from its own git checkout at /data/stratify/src, restarts stratify-web, and health-checks. Nothing needs to reach into the box, so CI holds no SSH key and no root credential.

# watch a deploy happen
ssh [email protected] 'journalctl -u stratify-deploy -f'

# what is live right now
ssh [email protected] 'cat /var/lib/stratify-deploy/deployed_sha'

# force a redeploy of current main
ssh [email protected] 'rm -f /var/lib/stratify-deploy/deployed_sha && systemctl start stratify-deploy'

# timer health
ssh [email protected] 'systemctl list-timers stratify-deploy.timer --no-pager'

A commit whose health check fails is recorded in /var/lib/stratify-deploy/failed_sha and not retried, so a broken build does not rebuild itself every 2 minutes. Delete that file to retry. The containers are left running rather than rolled back — compose build has already replaced the previous image, so there is nothing to roll back to; the failure is surfaced loudly instead of guessed at.

Install or refresh the agent (idempotent, run from a machine on the tailnet with gh authenticated):

./infra/hetzner/install-deploy-agent.sh

It generates a read-only deploy key on the box, registers it on the repo, clones main to /data/stratify/src, and installs the systemd unit + timer.

Deliberately not automated

  • Secrets (/data/stratify/web/.env) — the agent never writes them. An agent that

could rewrite secrets would have to hold them, which defeats the point of removing CI's credentials. Rotate with deploy-web-hetzner.sh from a trusted machine.

  • Database migrations — applying DDL unattended on every merge is a bad trade. Run

scripts/hetzner-supabase-db-push.sh deliberately (see below).

Repo secrets

None are required any more. dos-ci.yml uses none, and deploy.yml is gone. If any TS_OAUTH_*, HETZNER_ROOT_SSH_KEY, SUPABASE_*, CRON_SECRET or OPENROUTER_API_KEY secrets are still set on the repo, they are leftovers and can be deleted.

Manual deploy (fallback, and how secrets get rotated)

Still supported and still the way to change /data/stratify/web/.env. Run from the repo root, on main, with a clean tree:

HETZNER_STRATIFY_HOST=100.72.131.123 HETZNER_STRATIFY_SSH_USER=root \
  ./infra/hetzner/deploy-web-hetzner.sh

This rsyncs the working tree (not a git export), builds the Docker image on the box, and recreates stratify-web. Because it syncs the working tree, .env.local and friends are explicitly excluded — Next.js gives .env.local priority at build time, so a leftover dev file would otherwise bake localhost URLs into the prod bundle. Check git status before deploying.

Smoke test:

curl -s -o /dev/null -w '%{http_code}\n' https://app.stratifyinvest.com/api/health

GoTrue mail templates

Templates are fetched by GoTrue over HTTP from the web app, so web must be deployed before auth is recreated or GoTrue 404s and silently falls back to its bare built-in templates.

./scripts/hetzner-supabase-mail-templates.sh

The script refuses to proceed unless all five templates are reachable _and_ contain the expected Go variables, diffs the remote compose override before overwriting it, and then recreates stratify-auth. Recreating auth interrupts authentication for a few seconds.

Verify afterwards:

# template + OTP env actually applied
ssh [email protected] 'docker inspect stratify-auth --format "{{range .Config.Env}}{{println .}}{{end}}" \
  | grep -E "MAILER_TEMPLATES|OTP_EXP|OTP_LENGTH|RATE_LIMIT_VERIFY"'

# password login still works after the recreate
curl -s -X POST "https://app.stratifyinvest.com/supabase/auth/v1/token?grant_type=password" \
  -H "apikey: $ANON" -H 'Content-Type: application/json' \
  -d '{"email":"…","password":"…"}' | jq -r 'has("access_token")'

Database migrations

HETZNER_STRATIFY_HOST=100.72.131.123 HETZNER_STRATIFY_SSH_USER=root \
  ./scripts/with-stratify-hetzner-supabase-db.sh ./scripts/hetzner-supabase-db-push.sh --dry-run

Drop --dry-run to apply. Postgres listens only on the box's 127.0.0.1:5433, so this tunnels over SSH.

Getting a real OTP without sending mail

Useful for verifying the auth flow end to end:

SR=$(security find-generic-password -s SUPABASE_SERVICE_ROLE_KEY -a stratify-hetzner -w)
curl -s -X POST "https://app.stratifyinvest.com/supabase/auth/v1/admin/generate_link" \
  -H "apikey: $SR" -H "Authorization: Bearer $SR" -H 'Content-Type: application/json' \
  -d '{"type":"magiclink","email":"…"}' | jq -r '.properties.email_otp // .email_otp'

Gotchas

  • docker logs --since 30d silently returns nothing — Go durations have no d

unit. Use --since 144h or an explicit --since 2026-07-01T00:00:00, or you will read an empty log as evidence of absence.

  • GoTrue's own /auth/v1 links do not work at all; see

ADR-0002. Sign-in, password reset and invites are all 6-digit codes. Links to the web app are fine — that is what the recovery and invite mails point at (/reset-password). Only email-change still carries a GoTrue link, and it is unreachable from either client.

  • Mobile ships separately — deploying web does not update the app. updates.enabled

is false, so there is no OTA path; a change to the app needs a TestFlight build.