Backups

What's in Postgres

Everything that's not a secret:

• Tasks + events (volume depends on traffic).
• Agents (names, token hashes, capabilities).
• Users, memberships, sessions (sessions are ephemeral).
• Audit log (pruned to Z4J_AUDIT_RETENTION_DAYS, default 90 - most important to back up, and the window is why: rows past it are deleted, not archived).
• Schedules.

What's in env / secrets

• Z4J_SECRET, Z4J_SESSION_SECRET.
• Z4J_AUDIT_CHAIN_SECRET, the dedicated audit-chain signing key. Back this up with the same care as the database and keep it somewhere the database operator cannot read. A database backup restored without this key gives you the audit rows but no way to prove they are the rows you wrote.
• Agent plaintext tokens (only the operator who minted them holds these - brain stores hashes).

Backup strategy

1. Postgres - z4j backup, a raw pg_dump, or a managed-service snapshot, at least daily. Point-in-time recovery for RPO < 1 hour. If you use a raw pg_dump with shell redirection on POSIX, set umask 077 first: without it the dump lands world-readable and exposes users, API key material, sessions and the audit log. On POSIX, z4j backup creates mode 0600 itself. On Windows, its output inherits the destination directory's DACL, so use an ACL-restricted backup directory.
2. Secrets - version-controlled in your secret manager (Vault, AWS Secrets Manager, GCP Secret Manager).
3. Agent tokens - re-mintable. If lost, revoke the old agent and mint a new token. No need to back up plaintext.

Restore drill

The keys come first, and "first" means before you start the command. z4j restore builds its settings once at startup and never re-reads the environment, so a key exported partway through is invisible to it.

It uses that one frozen keyring at three separate points: staging and inspecting the archive, then authenticating the existing target, then authenticating the restored contents after the load. A key that is wrong or missing therefore fails at whichever of those it reaches first, and the last of them happens after PostgreSQL has already been populated, leaving the target full and fenced rather than empty. See backup and restore for what each failure point leaves behind and how to recover from it.

1. Put the source's keys in the drill environment, before anything else:
   • Z4J_AUDIT_CHAIN_SECRET (+ Z4J_AUDIT_CHAIN_PREVIOUS_SECRETS), or the restored chain cannot be verified at all.
   • Z4J_SECRET (+ Z4J_PREVIOUS_SECRETS), covering every value that was ever in force when a live agent token was minted. Tokens are stored as hashes keyed by the master secret.
   • Z4J_SESSION_SECRET if you want existing browser sessions to survive.
2. Provision the drill target as a real z4j installation and migrate it to the current head. It cannot be an empty database. See backup and restore for the full ceremony, the client-binary requirements, and what restore checks.
3. Stop the brain. Nothing checks this for you: z4j restore requires --force, and --force only records your assertion that it is stopped.
4. Restore: z4j restore <archive> --force.
5. Start the brain and verify: z4j audit verify, then z4j migrate current --check-heads, then z4j status.
6. Spot-check: log in, list tasks, check agent status.

On agents: the drill proves the token hashes survived, and not much more. They deliver data only if the drill environment's Z4J_SECRET is the CURRENT one their Z4J_HMAC_SECRET was derived from, because the per-project frame key comes from the current master with no fallback. A drill host with its own secret shows agents authenticating, registering online, and then flapping on a fast reconnect loop with no data arriving. That is expected in a drill and is not a backup defect.

Run this drill at least quarterly in staging. There is no dry-run mode, so restoring is the only way to learn whether an archive restores, which is exactly why the drill needs somewhere it is safe to actually do it.

Step 3 needs one qualification, and a restore is exactly the case where it bites. A plain z4j audit verify checks the log against chain state held in the same database, so a backup that is internally consistent verifies clean whether or not it is the newest history you had. If you keep an exported head (see HMAC audit chain), pass it, and the restore will tell you which history you are looking at rather than only that it hangs together:

z4j audit verify --known-head "$(cat /secure/last-known-head)"

z4j restore accepts the same --known-head envelope and prints the assessment on its rollback: line.

Audit log export

For compliance, export the audit log per project. There is no separate /export route; the format query param on the list endpoint switches it to a file download, and the caller must be an admin on that project:

curl -b "$COOKIE_JAR" \
  "https://z4j.example.com/api/v1/projects/billing/audit?format=csv&since=2026-01-01T00:00:00Z" \
  > audit-2026-q1.csv

format is csv, json, or xlsx, and one export is capped at 50,000 rows; narrow with since, action_prefix, outcome, or user_id rather than dumping the whole log.

These exports carry the audit fields (id, timestamp, action, target, result, outcome, user, source IP, user agent, metadata). They do not carry row_hmac or prev_row_hmac, so they are a record to hand to a SIEM or an auditor, not something a downstream system can re-verify the chain from. Chain verification is z4j audit verify against the database with the audit-chain key. That key being separate from Z4J_SECRET is what lets you give an auditor database access plus the chain key so they can run the verification themselves, without handing over the master key that protects agent tokens, frame keys, and stored TOTP secrets. Sessions use the independent Z4J_SESSION_SECRET.

Retention

• Events - tune Z4J_EVENT_RETENTION_DAYS (default 30). Older partitions are dropped by the retention worker.
• Audit log - tune Z4J_AUDIT_RETENTION_DAYS (default 90). Older rows are pruned by the retention worker. Because pruning the oldest rows eventually removes the genesis row, the sweeper advances an authenticated prune watermark on the chain state, and the verifier accepts a first surviving row whose prev_row_hmac matches it. A deleted middle row or an altered row_hmac still fails, so retention does not quietly turn into a clean-looking truncation.

Secrets rotation vs. restore

Two independent keys rotate independently, and mixing them up is the usual reason a restored database will not verify.

The audit chain is signed with Z4J_AUDIT_CHAIN_SECRET, not Z4J_SECRET. When you rotate it, rows written from that point on use the new key. Pre-rotation rows stay verifiable as long as the prior value is listed in Z4J_AUDIT_CHAIN_PREVIOUS_SECRETS (comma-separated), which the verifier accepts while new writes use the current key. If you restore Postgres from before a rotation, the matching Z4J_AUDIT_CHAIN_PREVIOUS_SECRETS entry must still be present, or those rows are unverifiable and no longer recoverable.

Rotating Z4J_SECRET is a separate operation with a separate carry-over list, Z4J_PREVIOUS_SECRETS. Two things about it are commonly stated wrongly, including by earlier versions of these pages. It does not log anyone out: sessions are signed with Z4J_SESSION_SECRET, which rotates independently and has its own carry-over list. And carrying the previous value does not keep agents working: it covers bearer-token verification only, while frames are signed with a key derived from the current master alone, so every agent has to be re-credentialed. See incident response for the full sequence. It does not re-sign, break, or repair a single audit row.