Business Continuity & Disaster Recovery Test Procedure

Defines the recurring exercise that validates stratify can actually do what its continuity commitments claim — restore data, verify the audit chain survives, and keep the product usable during a host outage. This procedure operationalises the commitments the (not yet drafted) Business Continuity & ICT Resilience policy will state, and cross-references docs/runbooks/db-recovery.md, which already defines the mechanics this procedure tests.

Stated plainly up front: neither the restore drill nor the failover toggle has ever been executed. db-recovery.md's own setup checklist has the restore-drill line unchecked ("einmal manuell getestet"), and its decision log's most recent row reads "TBD | Erster vollständiger Hetzner-Restore-Drill abgeschlossen." This procedure exists so the first real run has a defined shape, not because a track record already exists.

1. Scope of the test

Two independent components, both already described operationally in db-recovery.md but never exercised:

1. Restore drill — cross-references the backup/restore procedure (PRO-07, once written) and

db-recovery.md §1–§4. Confirms the nightly pg_dump backups (scripts/hetzner-backup-postgres.sh, 14-dump retention, 05:30 UTC) are actually restorable, not just produced.

1. Failover toggle test — exercises the manual read-only-mode toggle for a host outage

(db-recovery.md §3.4: STRATIFY_READ_ONLY=1), confirming the product degrades gracefully rather than failing hard.

2. Restore drill — steps

1. Select the most recent nightly dump from /data/stratify/backups/stratify-<stamp>.sql.gz on the

Hetzner host.

1. Restore it into a throwaway database on the same host (not against stratify-db directly):

   gunzip -c /data/stratify/backups/stratify-<stamp>.sql.gz \
     | docker exec -i stratify-db psql -U postgres -d postgres_restore

1. Confirm the restore succeeded: row counts on a sample of tables (subscribers, audit_log,

signal_deliveries) are non-zero and roughly consistent with what's expected for that date.

1. Run `verifyChain()` against the restored database. This is the concrete pass/fail criterion

for audit-chain integrity — not "the restore completed" but verifyChain() returning { ok: true, rows: N } with no firstBadId divergence (apps/web/lib/audit/chain.ts:101-137).

1. Record actual RTO (wall-clock time from "decision to restore" to "restore verified") against the

stated SLO of ≤60 minutes, and actual RPO (age of the dump used) against the stated SLO of ≤24 hours (db-recovery.md line 57).

1. Tear down the throwaway database.

3. Failover toggle test — steps

1. In a controlled window (not during a real incident), set STRATIFY_READ_ONLY=1 on the Hetzner

host and redeploy, or stop the write path per db-recovery.md §3.4.

1. Confirm server actions fail soft with the expected "wir sind kurz nicht erreichbar" message

rather than a hard error.

1. Smoke-test the read paths that should remain usable in read-only mode: /,

/walkthrough, /feed, /api/v1/mandates (with a sandbox key) — the same smoke-test list db-recovery.md §3.1 already specifies for the DROP-TABLE scenario, reused here since the product-usability bar is the same.

1. Confirm the product remains genuinely usable in read-only mode from a user's perspective, not

just that it doesn't 500.

1. Revert the toggle, redeploy, and confirm writes resume (e.g. a test signal delivery or admin

action completes normally).

4. What "test passed" means, concretely

• Restore drill passed: dump restored into a throwaway DB, sample row counts sane, and

verifyChain() returns { ok: true } with no divergence. A restore that completes but where verifyChain() reports a firstBadId is a failed test, and should itself trigger the audit-chain-corruption runbook (db-recovery.md §3.3) as if it were a live incident, not just be logged and moved past.

• Failover toggle passed: the toggle was actually flipped (not just read about), server actions

degraded as expected, the four listed routes remained usable, and writes resumed cleanly after revert.

• A test that skips either the verifyChain() step or the live toggle flip — e.g. "we read through

the runbook and it looks right" — does not count as a passed test under this procedure.

5. Cadence and ownership

db-recovery.md §4 already defines a quarterly rotation across four distinct sub-tests:

This procedure's annual scope is the union of all four — one full pass per year satisfies both the restore-drill and failover-toggle components described above, run across the year rather than as a single annual event. Antonios's Q3 role is deliberate: verifying audit-chain integrity is a compliance judgment as much as a technical one, consistent with his RACI accountability for breach-notification decisions (governance/GOV-03-roles-and-raci.md).

6. Where the result is logged

• docs/runbooks/db-recovery.md §5 Decision Log — append a dated row per test run, replacing the

standing "TBD" placeholder once the first restore drill actually completes.

• A corresponding dated entry under docs/compliance/isms/evidence/, recording actual RTO/RPO

against the stated SLOs and the verifyChain() result.

• Per db-recovery.md §4's own instruction, each test should also write an ops.recovery_drill

event to audit_log (actorRole: 'system') — this has not yet happened for any quarter, since no drill has run.

7. Review

Reviewed annually. The first execution of this procedure — in full, both components — should be scheduled as part of closing the Phase 5 gap list in PLAN.md §5 ("one restore drill logged").