Backup and restore

z4j ships two CLI commands for database snapshots:

z4j backup --output PATH - point-in-time snapshot to a single file
z4j restore PATH --force - restore from a backup file

Both auto-detect the backend from Z4J_DATABASE_URL; you do not choose it. That is a detection step, not a conversion step. An archive is only restorable into the backend it came from: a SQLite snapshot is a SQLite database file, a PostgreSQL archive is a pg_dump custom-format file, and pointing z4j restore at the other backend fails on an unreadable archive rather than migrating anything. There is no supported SQLite-to-PostgreSQL move, in either direction. Changing backend means starting a new installation: keep the old volume and export the records you must retain through the API or the audit export.

Backend	Backup mechanism	Restore mechanism
SQLite	`VACUUM INTO` (online; brain keeps serving)	Authenticated replace (brain must be stopped)
PostgreSQL	`pg_dump -Fc -Z6` (online)	Authenticated clear, fence, then a single-transaction load

These commands handle the database only. For a full disaster-recovery plan you also need your keys and ~/.z4j/allowed-hosts (operator-managed), and the agent-side buffers. See What else to back up.

Restore is a ceremony, not a file copy

This is the part most worth reading before you need it. z4j restore is not a thin wrapper around pg_restore, and treating it as one leads to procedures that fail in an incident.

The target must already be a z4j database. Restore stages and inspects the archive first, then authenticates the existing target against it. That target has to be a database at the current release migration head, with activated schedule-control authority, and with an audit chain that verifies under the keyring in the local environment. An empty database satisfies none of those and is refused. There is no "restore into a fresh database" path: create the database, bring it to head the ordinary way, and only then restore into it.

On PostgreSQL the clear is committed BEFORE the load, and only the load is all-or-nothing. Restore proves the target's relation set matches the release contract, dumps the current target to a recovery archive, then drops exactly that relation set and advances a durable fence in one committed transaction, and only then loads your archive with --single-transaction --exit-on-error.

That ordering decides what you are looking at when something fails, and there are four outcomes, not two:

Refused during preflight: the target is untouched. The archive is staged and inspected first, then the target is authenticated, so most bad inputs are caught before anything is written.
The load failed: the target is EMPTY and fenced, not untouched. The clear was already committed and nothing rolls it back for you.
The load succeeded but the restored contents failed authentication: the target is POPULATED and fenced. This is the one that misleads, because the database looks restored. It is not finalised, and the data in it did not match what the archive's preflight promised.
Everything succeeded: fully restored and the fence is cleared.

The recovery action is the same for both fenced outcomes: z4j restore --force --rollback-operation <uuid>, which replays the recovery archive taken before the clear. Do not judge by whether the database has rows in it. Two earlier versions of this page got this wrong in opposite directions, one claiming a failure leaves the target untouched and one claiming it always leaves it empty; the fence is the thing to read, not the row count.

Do not substitute a bare pg_restore --clean. It is not the same operation and it is not idempotent against a partially populated target. It takes no recovery archive, so a failure partway through leaves a half-cleared database that looks restored and has nothing to replay. If a managed restore is fenced and incomplete, the way out is z4j restore --force --rollback-operation <uuid>, not a manual pg_restore.

--force is an acknowledgement, not a check. It means "I, the operator, assert the brain is stopped". Nothing verifies it. There is no liveness detection anywhere in the restore path, so stopping the brain first is your responsibility and the tool cannot catch the mistake for you.

Restore is forward-only. An archive from the previous release is migrated up to the current head as part of the restore and finalised there. You get your old data on the new schema, running the new code. Restore rolls back data; it never rolls back code or schema.

The window is narrow: the current head, the previous release's head, and one older head. An archive from before that is refused, and the refusal lists the migration heads this release can read and tells you to install the release matching your archive's head, restore there, and upgrade from it. It names heads rather than version numbers, so map it with z4j migrate history.

Client requirements on PostgreSQL

These apply to z4j backup and z4j restore, and they bite on Debian and Ubuntu. Your own pg_dump invocations have no such requirements; the managed ceremony does, because it pins the executable by device, inode, size and digest so a binary swapped mid-run is detected.

The client must be a real file, not a symlink. Debian and Ubuntu ship /usr/bin/pg_dump, /usr/bin/pg_restore and /usr/bin/psql as symlinks to pg_wrapper, a dispatcher that picks a versioned binary at run time, after the pin has been taken. Those are refused. Put the versioned directory first on PATH instead:

export PATH=/usr/lib/postgresql/18/bin:$PATH

apt install postgresql-client on its own is not enough, because it installs exactly the wrapper symlinks that are refused. Install the versioned package (postgresql-client-18 from apt.postgresql.org for PostgreSQL 18) and point PATH at its bin directory. A brew linked libpq on macOS has the same problem for the same reason.

The client major must match the server major. pg_dump 16 cannot dump an 18 server.

The path to the binary has to be trustworthy, all the way up. On POSIX the client itself, and every ancestor directory up to /, must be owned by root or by you and must not be group or world writable, and no ancestor may be a symlink. This is what makes the digest pin meaningful: a directory someone else can write to is a directory where the binary can be replaced between the pin and the run. If you keep a private PostgreSQL client somewhere like /opt, check the whole chain, because the refusal names only the offending directory (PostgreSQL client ancestor has an untrusted owner: /opt) and the reason is not obvious from the message alone.

None of this applies when you run the tools inside the database container: the binaries there are real files already matched to the server.

SQLite backup

z4j backup --output /var/backups/z4j-$(date +%Y-%m-%d).db

z4j keeps serving requests during the backup; VACUUM INTO produces a consistent snapshot via SQLite's online backup API. The output is a self-contained SQLite file. On POSIX, z4j backup creates it mode 0600 before writing data. On Windows, it inherits the destination directory's DACL; use a directory restricted to the backup identity and intended administrators.

z4j: backup complete
  backend:    sqlite
  output:     /var/backups/z4j-2026-04-24.db
  size:       12.34 MiB
z4j: move this file off-host (scp, rclone, S3, ...) for true disaster recovery.

A backup left on the same host as the original is half a backup. Push it off-host immediately:

rclone copy /var/backups/z4j-2026-04-24.db s3:my-backups/z4j/
scp /var/backups/z4j-2026-04-24.db backup-host:/srv/z4j-backups/

PostgreSQL backup

z4j backup --output /var/backups/z4j-$(date +%Y-%m-%d).dump

Uses pg_dump -Fc -Z6 --no-owner --no-acl: custom format, compressed, and portable across environments. z4j keeps serving requests.

If you prefer your own tooling, a plain pg_dump works too, but set the umask yourself:

umask 077
destination="z4j-$(date +%F).dump"
temporary="$(mktemp "${destination}.part.XXXXXX")"
pg_dump -Fc -Z 6 --no-owner --no-acl -U z4j -d z4j > "$temporary" &&
  mv -f "$temporary" "$destination" ||
  { status=$?; rm -f "$temporary"; exit "$status"; }

The umask 077 is not decoration. Shell redirection creates the temporary file at the process umask, which is 022 on an ordinary login. Without it the dump lands 0644 and every account on the box can read your users, API key material, sessions and the whole audit log. The same-directory temporary file is renamed only after pg_dump succeeds, so a failed retry cannot replace a good archive with a partial one. On POSIX, z4j backup needs no umask; it creates its output mode 0600 itself. On Windows, the output inherits the destination directory's DACL, which must already restrict access to the backup identity and intended administrators.

Restoring

Stop the brain first. Nothing checks this for you. The commands below assume the unit runs as User=z4j, uses /srv/venv/bin/z4j, and has Z4J_HOME=/srv/z4j/.z4j. Substitute the service's actual identity, executable, and home as one set. Running the CLI as the responder's login user can select a different database and secret store. The example also assumes the remaining effective settings are available through that home. sudo -u does not import a unit's Environment= or EnvironmentFile= values; if the service keeps its database credentials there, invoke the CLI through the same environment wrapper or load that exact source before running the command.

sudo systemctl stop z4j
sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j \
  /srv/venv/bin/z4j restore /var/backups/z4j-2026-04-24.dump --force
sudo systemctl start z4j

On SQLite there is no <dbpath>.pre-restore-bak. Earlier releases worked that way and this page had not caught up; the current mechanism keeps everything in an operation-local directory beside the database, <dbdir>/.z4j-restore/<operation-id>/. It holds a copy of the target taken before anything is touched, and, once installation begins, the displaced live files. That directory is removed once the restore succeeds, so nothing survives a successful restore for you to fall back on, and looking for a sibling .bak file will find nothing at any point.

Recover with z4j restore --force --rollback-operation <uuid> while the operation is still pending, rather than moving files by hand. If you want a copy you can keep, take a z4j backup before restoring.

If the restore stages instead of completing

A restore that finds processes it cannot confirm are stopped does not proceed. It stages the operation and reports a challenge, and the run is completed by re-invoking with the operation ID rather than the path:

sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j \
  /srv/venv/bin/z4j restore --force --operation <uuid> \
    --attest-stopped-executors <sha256-from-the-challenge>

That is deliberate: restoring while a scheduler or agent is still writing would fork the audit chain. Stop the named executors, then resume.

flag	when you need it
`--force`	always; acknowledges that you have stopped the brain
`--expected-sha256`	pin the source bytes, so a swapped file is refused
`--operation UUID`	resume one exact staged restore after a challenge
`--attest-stopped-executors SHA256`	the staged operation named executors that must be stopped first
`--known-head JSON`	let z4j judge whether the restored history is current or an ancestor
`--rollback-operation UUID`	undo a staged restore

Verifying after a restore

sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j /srv/venv/bin/z4j check
sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j \
  /srv/venv/bin/z4j migrate current --check-heads
sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j /srv/venv/bin/z4j status
sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j /srv/venv/bin/z4j audit verify

z4j check proves less than it looks like it proves. It confirms the configuration loads and the database answers a trivial query, then prints whatever migration revision it finds. It does not compare that revision against the head your code expects, and a database with no alembic_version table at all still exits 0 and reports green. Use z4j migrate current --check-heads when the distinction matters, which it does after any restore: that one exits non-zero unless every head is applied. z4j doctor is not a substitute, because it runs check and returns its exit code.

If audit verify fails after a restore, check the audit-chain key first. The chain is signed with Z4J_AUDIT_CHAIN_SECRET, a dedicated key, not with Z4J_SECRET. Restoring rows from one install into an environment holding a different Z4J_AUDIT_CHAIN_SECRET fails verification for every row written under the old value. The fix is to make the signing key available: put the backup's key back as the current key before restoring, or perform an explicit rotation after restoring. For the second path, keep every brain stopped, set the new key as Z4J_AUDIT_CHAIN_SECRET, list the backup's old key in Z4J_AUDIT_CHAIN_PREVIOUS_SECRETS, and run z4j audit rotate-chain-key under the same service identity and configuration. Start the brains only after that command commits the authenticated state transition, then run z4j audit verify. Merely listing the old key in the previous-key window makes old rows verifiable; it does not authorize new writes under a different current key.

The current Z4J_SECRET alone derives frame-signing keys. Its current and previous window is used for agent bearer tokens, stored TOTP secrets, and legacy audit compatibility, but neither setting determines whether the active dedicated audit chain verifies. Sessions use a third, separate key again (Z4J_SESSION_SECRET).

A verification that passes is worth reading precisely. It says the restored log is internally consistent with the chain state restored alongside it. It does not say the backup is the newest history you had, because an older backup is internally consistent too. If you export chain heads off-box (see HMAC audit chain), pass the most recent one and the answer distinguishes the two:

sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j \
  /srv/venv/bin/z4j restore /var/backups/z4j-2026-04-24.dump --force \
    --known-head "$(cat /secure/last-known-head)"

z4j restore prints the assessment on its rollback: line, and reports ROLLBACK_NOT_ASSESSED when you do not pass an envelope. z4j audit verify --known-head takes the same envelope and can be re-run at any time.

Scheduling backups

The CLI is designed for cron and systemd timer usage. Sample systemd timer:

[Unit]
Description=z4j daily backup
After=z4j.service

[Service]
Type=oneshot
User=z4j
Environment=Z4J_HOME=/srv/z4j/.z4j
Environment=Z4J_DATABASE_URL=sqlite+aiosqlite:////srv/z4j/.z4j/z4j.db
# Do NOT write to /var/backups from an unprivileged unit. On stock Ubuntu it
# is root:root 0755, so a User=z4j service fails there with EACCES on the
# first run. StateDirectory makes systemd create /var/lib/z4j-backup owned by
# this unit's user before ExecStart, which needs no chown step an operator can
# forget, and exports the path as $STATE_DIRECTORY.
StateDirectory=z4j-backup
StateDirectoryMode=0750
UMask=0077
# The shell wrapper is required, and the doubled %% with it. In a unit file
# `%` introduces a systemd SPECIFIER, not a strftime code: written bare,
# `%Y-%m-%d` expands to the unit-file directory, your machine ID and the
# credentials directory, which contain slashes, so the backup fails every run
# against a path that is not even in the target directory. Doubling to `%%`
# without a shell is no better: systemd then passes the literal text
# `%Y-%m-%d` through, every run targets the same filename, and `z4j backup`
# refuses an existing destination from the second run onward. Only running
# `date` in a shell, with `%%` so systemd hands the shell a real `%`,
# produces a dated file.
ExecStart=/bin/sh -c 'exec /srv/venv/bin/z4j backup --output "$STATE_DIRECTORY/z4j-$(date +%%Y-%%m-%%d).db"'
ExecStartPost=/bin/sh -c 'find "$STATE_DIRECTORY" -name "z4j-*.db" -mtime +14 -delete'

If you would rather keep backups in /var/backups, create a subdirectory the service owns first (install -d -o z4j -g z4j -m 0700 /var/backups/z4j) and point both lines at it. What does not work is writing to /var/backups directly as an unprivileged user.

[Unit]
Description=Run z4j backup daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

On Docker, write inside the one mounted path

/data is the only path the shipped Compose files persist. Both mount a named volume there (z4j_data for SQLite, z4j_brain_state for PostgreSQL) and mount nothing else. An earlier version of this page scheduled a backup into /backups, which is mounted nowhere: the container runs as an unprivileged user that cannot create a directory at the container root, so the job fails every night, and the retention find after it ran on the host against a directory that does not exist there either.

The two stacks need different commands, and the wrong one fails every run.

SQLite stack

The brain image can back itself up, because SQLite needs no external tool. Write into /data, or add an explicit mount to the service in the Compose file you deploy and make it writable by the image's user:

0 3 * * * docker exec z4j z4j backup --output /data/backups/z4j-$(date +\%Y-\%m-\%d).db

PostgreSQL stack

Do not run docker exec z4j z4j backup here. The brain image ships libpq5, the client library, and no PostgreSQL client binaries, while z4j backup shells out to pg_dump on this backend. Every run fails. That is the same constraint the rollback page states for restore, and it applies identically to backup.

Dump from the database container instead. It has the real binaries, they already match the server, and it is on the private network:

0 3 * * * umask 077 && cd /srv/z4j && d=/var/backups/z4j/z4j-$(date +\%Y-\%m-\%d).dump && t=$(mktemp "$d.part.XXXXXX") && { docker compose -f docker-compose.postgres.yml exec -T z4j-postgres pg_dump -Fc -Z 6 --no-owner --no-acl -U z4j -d z4j > "$t" && mv "$t" "$d"; } || { rm -f "$t"; exit 1; }

That is one physical line on purpose. cron has no line continuation: a backslash at the end of a crontab line does not join it to the next one, and crontab -n rejects the wrapped form outright with bad minute. An earlier version of this page wrapped it across three lines for readability, which made it unusable.

It dumps to a temporary file and renames only on success. That is the part worth keeping if you rewrite this. Redirecting straight into the final dated name, as an earlier version did, makes the shell create and truncate that file before pg_dump has produced a byte: a failed run leaves a zero-length or partial archive sitting where a good one should be, and a same-day retry destroys the previous successful archive on the way to failing again. That directly contradicts the rule further down this page that a failed run must not count toward retention. The mktemp plus mv makes the final name appear only when the dump actually succeeded, and the rm -f on the failure branch stops part-files accumulating.

Two more things it gets right that are easy to lose. The cd is required because cron runs with your home directory as the working directory, so a relative -f docker-compose.postgres.yml resolves to nothing. And /var/backups/z4j is a subdirectory you create and own (install -d -o z4j -g z4j -m 0700 /var/backups/z4j), because /var/backups itself is root:root 0755 on stock Ubuntu.

The umask 077 belongs on the host side of that pipeline, where it is written above, not inside the container. The redirect is performed by the host's shell, so a umask set within the container protects nothing: the file still lands 0644 with your users, API key material, sessions and audit log in it. This is the one place the two halves of the command run in different places, and it is easy to get backwards.

If your database credentials are not the defaults, read them from the environment inside the container rather than hard-coding them, by replacing the pg_dump ... portion with sh -ceu 'exec pg_dump -Fc -Z 6 --no-owner --no-acl -U "$POSTGRES_USER" -d "$POSTGRES_DB"'.

An archive produced this way is a plain pg_dump custom-format file, which z4j restore accepts.

Hold any schedule you build to these:

The job must reach the database. The shipped PostgreSQL service publishes no host port and joins a private Compose network, so anything dumping it has to be on that network too.
The archive must be created private. On POSIX, z4j backup creates mode 0600 itself. On Windows, it inherits the destination directory's DACL, so that directory must already restrict access to the backup identity and intended administrators. A raw pg_dump redirected with > does not create a private destination automatically.
A failed run must be loud, and must not count toward retention. Deleting the oldest backup because a broken run created a zero-byte newest one is how a backup set evaporates quietly.
Verify by restoring, not by listing files.

Testing your restores

An untested backup is a guess, and the failure modes are dull: the dump was empty, the cron job stopped silently months ago, the file was never readable off-host, or the client major stopped matching the server after an OS upgrade.

The rehearsal target has to be a real z4j installation at the current head, not a blank database, for the reasons above. Give it the same Z4J_AUDIT_CHAIN_SECRET as the source environment so the archive's chain verifies, restore into it, and compare z4j status counts against what the source reported. There is no dry-run mode: restoring is the only way to learn whether an archive restores.

What else to back up

Path	What it carries	How often it changes
DB (SQLite file or Postgres)	All operational state: users, agents, tasks, audit chain, schedules	Continuous
`Z4J_AUDIT_CHAIN_SECRET` (+ `Z4J_AUDIT_CHAIN_PREVIOUS_SECRETS`)	The audit-chain signing key. Required outside development, no fallback. Lose it and the restored audit log is unverifiable	Only when you rotate
`Z4J_SECRET` + `Z4J_SESSION_SECRET` (+ their `Z4J_PREVIOUS_*` forms)	Agent-token and session keys. `Z4J_PREVIOUS_SECRETS` also decrypts stored MFA secrets written under an older master key	Only when you rotate
Agent buffer directories	Undelivered frames and the brain-issued schedule epoch. Preserve the complete directory including `-wal` and `-shm` sidecars	Continuous
`~/.z4j/secret.env` (SQLite/pip)	Wherever any of the above are not set via env, this file is the only copy	When a key is minted or rotated
`~/.z4j/allowed-hosts`	Operator-managed Host allow-list	When you add/remove hosts
Agent tokens (in your apps)	Bearer tokens minted from the dashboard	When you mint/rotate
`.env` / Compose file	Whatever you set `Z4J_*` env vars to	When you change config

Rotation is exactly when a recovery inventory goes stale, so re-check it after every rotation rather than after every release.

Keep Z4J_AUDIT_CHAIN_SECRET somewhere the database operator cannot read: a backup and its signing key sitting in the same trust boundary is a chain that proves less than it appears to.