Incident response

A control plane needs a clear "what do I do" for the outages you can't prevent. This page covers the four most likely incidents operators will face.

Brain is down

Symptoms: dashboard unreachable, agents reporting disconnected, z4j check fails.

Step 1: is the process running?

# systemd
sudo systemctl status z4j
sudo journalctl -u z4j -n 100 --no-pager

# Docker
docker compose ps
docker compose logs --tail=100 z4j

Process crashed → look for a Python traceback in the last 100 log lines. Common causes:

DB unreachable (Postgres) - network / credential / PG-down. Fix Postgres first, then systemctl restart z4j.
Migration failure - alembic error on boot. Z4J_AUTO_MIGRATE=false z4j serve to start without migrating, then inspect + run z4j migrate history / z4j migrate current.
Disk full (SQLite) - df -h shows 100% on the z4j data volume. Free space or rotate backups. SQLite has a minimum free-space requirement for WAL commits.
Port conflict - something else grabbed 7700. ss -tlnp | grep 7700 to identify.

Step 2: is it accessible?

Brain process is running but dashboard returns 400 / 502 / connection-refused.

400 invalid_host - see allowed hosts. On pip/SQLite with Z4J_ALLOWED_HOSTS unset, run z4j allowed-hosts add <local-name> and restart. In Docker, PostgreSQL, or any deployment with the env var set, update its JSON Z4J_ALLOWED_HOSTS value in the service environment and restart; the persistent file is ignored on those paths.
502 bad gateway - reverse proxy can't reach z4j. Check reverse proxy logs; verify z4j bound to the port the proxy expects (Z4J_BIND_HOST / Z4J_BIND_PORT).
connection-refused - firewall / security group blocking inbound. Verify with telnet <brain-host> 7700 from the client side.

Step 3: nothing obvious, restore

If z4j is broken beyond systemctl restart, restore from the most recent backup. This systemd example assumes User=z4j, /srv/venv/bin/z4j, and Z4J_HOME=/srv/z4j/.z4j; substitute the unit's actual values and load the same EnvironmentFile= if it carries database credentials:

# Locate the latest
ls -lt /var/backups/ | head -5

# Restore. Stop the brain first: --force only records that you say it is
# stopped, and nothing verifies it.
sudo systemctl stop z4j
sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j \
  /srv/venv/bin/z4j restore /var/backups/z4j-2026-04-24.db --force
sudo systemctl start z4j
sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j \
  /srv/venv/bin/z4j migrate current --check-heads
sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j /srv/venv/bin/z4j status
sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j /srv/venv/bin/z4j audit verify

See backup and restore.

Audit chain tampered

The audit log is HMAC-chained (row_hmac + prev_row_hmac) and signed with a dedicated key, Z4J_AUDIT_CHAIN_SECRET, which is separate from Z4J_SECRET. z4j audit verify walks the chain and reports every row that fails.

What that catches is real, and it is most of what actually goes wrong: a row edited or removed from the middle of the log, a row written by a code path that went around the audit service, an older adapter, a retention job that deleted more than it should, and a DELETE run by hand.

What the walk alone does not catch: a database role that can write both audit_log and audit_chain_state can delete a suffix of the log and put back a copy of the state row the brain itself signed while that suffix did not yet exist. Head, counts, and links then all agree, and verification reports the shortened history as clean. Catching that needs a head exported earlier to somewhere the database role cannot write, compared against the live chain with z4j audit verify --known-head. See HMAC audit chain for how to set that up. Until you have it, read a clean verification as "nothing outside z4j's own write path touched this log", not as "nothing touched this log".

Detection

Run periodically (daily cron, or after any suspicious event), and against a retained head if you keep one:

z4j audit verify
z4j audit verify --known-head "$(cat /secure/last-known-head)"

Output on a healthy chain:

verified active: 5421
verified frozen: 0
known-head: CURRENT_MATCH

Output on a tampered chain:

verified active: 5380
verified frozen: 0
known-head: UNPROVABLE
MISMATCHES (2):
  4f2b8c31-9d0e-4a71-8c55-1e6a2f0b7d34 active row HMAC mismatch
  active row count 5380 != authenticated 5421

The known-head line appears only when you pass --known-head. UNPROVABLE there means the head you exported earlier is no longer anywhere in this chain, which is the rollback case above: it is a finding even when every retained row verifies and no MISMATCHES block is printed. Exit status is 0 for a clean chain, 1 for any finding, and 2 if the command could not load settings or reach the database, so a nightly cron can page on non-zero.

Response playbook

Preserve evidence, and preserve the right key. Snapshot the DB, and capture the audit-chain key that was in force when the break happened: Z4J_AUDIT_CHAIN_SECRET, plus every value in Z4J_AUDIT_CHAIN_PREVIOUS_SECRETS if a rotation was in flight. That is the key the log is signed with and the only one that can verify it later. Capture Z4J_SECRET, Z4J_SESSION_SECRET, and the service's actual $Z4J_HOME/secret.env as well: the master secret is evidence for agent tokens, frame keys, and encrypted TOTP values, while the independent session secret verifies session cookies. None of them verifies the audit chain.

For SQLite, first resolve the database path from the service's effective configuration. The responder's ~/.z4j is not the service's $Z4J_HOME. A filesystem or volume snapshot that captures the database and all of its sidecars at one instant is the best byte-level evidence. If you must make a file-level capture, stop the brain and copy the main file plus every sidecar that exists as one evidence set. This example assumes the service database really is /srv/z4j/.z4j/z4j.db; replace it with the resolved path before running it:
Terminal window
```
sudo systemctl stop z4j
db=/srv/z4j/.z4j/z4j.db
evidence="$(sudo mktemp -d /var/tmp/z4j-evidence.XXXXXXXX)"
sudo chmod 0700 "$evidence"
sudo cp --preserve=all -- "$db" "$evidence/"
for sidecar in "$db-wal" "$db-shm" "$db-journal"; do
  if sudo test -e "$sidecar"; then
    sudo cp --preserve=all -- "$sidecar" "$evidence/"
  fi
done
```
Do not use z4j backup or VACUUM INTO for evidence. They produce a logically equivalent database, not the same bytes: pages are rewritten and compacted, which discards free-page contents and other artefacts that may be exactly what you are looking for. This is the one place where the backup command is the wrong tool.

Do not call a sequence of live cp operations an atomic snapshot. Copying a database that is still being written can mix the main file and WAL from different instants. If containment or volatile-evidence concerns prevent a stop, use a storage-level snapshot when available. If you still make a live file copy, capture every sidecar, record the exact procedure and timestamps, and label the result potentially inconsistent.
Check server access logs (journalctl -u z4j, reverse-proxy logs, cloud provider flow logs). Correlate the timestamps of the rows named in MISMATCHES with access patterns.
Check for unauthorised DB writes, and treat this as the primary line of enquiry rather than a later step. A role with write access to audit_log and audit_chain_state can produce a log that verifies clean, so the database is where the answer is even when verification passed. For Postgres: pg_waldump plus a role audit. For SQLite: look for the file being modified outside z4j.
Rotate the audit-chain key if you believe Z4J_AUDIT_CHAIN_SECRET was exposed. Stop every brain replica before changing the key. A brain reads its audit-key window once at process startup; if the CLI authenticates the chain state with a new key while a replica is still running, that replica keeps the old keyring and its next audited write fails.

Bare z4j audit rotate-chain-key is a no-op on a managed install: with no rotation already pending it prints "no pending managed rotation; pass --begin-managed to mint a new key" and exits 0, so it reads as success and rotates nothing. For a systemd managed install, use the complete stop, rotate, start, verify ceremony:
Terminal window
```
sudo systemctl stop z4j
sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j \
  /srv/venv/bin/z4j audit rotate-chain-key --begin-managed
sudo systemctl start z4j
sudo -u z4j env Z4J_HOME=/srv/z4j/.z4j /srv/venv/bin/z4j audit verify
```
With multiple replicas, scale every brain replica to zero, run the rotation once from an administrative environment that shares the managed secret store, make the updated key window available to every replica, start them, and then run z4j audit verify. If you supply keys explicitly instead of letting z4j manage them, use the same stop, rotate, start, verify order. The new value must already be current and the old one already listed in Z4J_AUDIT_CHAIN_PREVIOUS_SECRETS in the CLI and every replica's configuration before you run the command; otherwise it returns without rotating. Either way, keep the old value in that list so the verifier still accepts rows signed under it. Rotation never re-signs or re-anchors those rows: retire the old key with z4j audit retire-chain-key --key-id <id> only after retention has removed every row signed by it and the command confirms the authenticated live count is zero. Rotating this key does not touch sessions or agent tokens.
Rotate Z4J_SECRET separately if you believe the application secret was exposed. Plan on re-enrolling every agent: Z4J_PREVIOUS_SECRETS does not keep them working. This has two layers and only the first one honours the carry-over list, which is why it is easy to get wrong.

The bearer token is a hash keyed by the master secret, and verification walks the whole keyring, so an agent holding an old token still completes the handshake while the outgoing value is carried. But every frame after the handshake is signed with a per-project key derived from the CURRENT master alone, with no previous-secret fallback, on both the WebSocket and long-poll transports. An agent that authenticated with the old derivation therefore sends its first data frame, fails envelope verification, and the brain closes the connection logging frame verification failed, closing.

What you observe is agents online with nothing arriving, and it looks different on each transport. Either way the agent registers as online before its first frame is verified, so the dashboard shows a healthy-looking fleet delivering no data, and the agent's own logs stay nearly silent. Nothing in that picture points at key rotation, which is the trap.

On WebSocket, the connection is closed after the handshake, and because the close lands mid-session the agent treats it as an ordinary drop rather than an auth failure. It reconnects on the fast schedule, roughly every 1 to 30 seconds, indefinitely.

On long-poll, there is no close at all: the brain answers HTTP 200 and reports the frames as rejected. The client treats that as retryable and burns roughly twenty zero-progress attempts, on the order of a minute and a half, before it even reconnects. So the same fault presents as a much slower, quieter stall, and looking only for a reconnect loop will miss it.

And the token is not what broke. The old bearer token still verifies; what is stale is each agent's Z4J_HMAC_SECRET, the derived value it signs frames with. There is no route that reveals or rotates that value for an existing agent, because it is returned once at mint time, so creating a replacement agent is the only supported way to obtain the new one. That is a product gap, not a cryptographic necessity, and it is why the remedy is re-credentialing rather than a config change.

Carrying the outgoing value does not help agents recover. All it does is keep their bearer token resolving, which is what produces the flapping above rather than a clean rejection. An earlier version of this page said it lets them reach the API to be re-credentialed; no such path exists. Minting a replacement needs a browser session with a CSRF token and project-admin authority on that project, plus a fresh MFA step-up if the acting admin has MFA enrolled. Re-credentialing is an operator task start to finish.

Inventory API-key consumers before the restart. API-key hashes are verified with the current Z4J_SECRET only; Z4J_PREVIOUS_SECRETS is not consulted. Every existing API key therefore stops authenticating when the brain restarts with the new master. Make sure an administrator can sign in without an API key, record every automation that needs a replacement, then mint new API keys after restart and update those consumers. There is no grace window, so API-key clients are unavailable until that is complete.

Outstanding invitation and password-reset links are also hashed with the current master only. Rotation invalidates them, and they must be reissued after the restart. Existing browser sessions are separate and remain governed by Z4J_SESSION_SECRET.

Z4J_PREVIOUS_SECRETS also has legacy audit-compatibility uses. Legacy and pre-activation audit-row verification, prune-watermark verification, and the audit activation and reseal tools consult the same key window. If any such transition work remains, complete it and verify the resulting audit authority before retiring the old master. An installation with dedicated authenticated audit authority uses the separate audit-chain key for its active chain.

For user credentials, MFA sets the long-running retirement clock. Stored TOTP secrets are encrypted under a key derived from the master, and they are decrypted by walking the same keyring. A stored secret is re-encrypted under the new master only when that specific user submits a valid TOTP code through the TOTP verification path. A successful login through a trusted device or recovery code does not decrypt or re-wrap it. There is no bulk re-wrap and no management command that performs one.

So the two clocks are unrelated: agents are done when you have re-credentialed them; an MFA enrolment is done only after a successful TOTP verification since the rotation, an MFA reset, or re-enrolment. Merely logging in is not enough. Drop Z4J_PREVIOUS_SECRETS on the MFA clock, not the agent one.

Dropping it while dormant accounts remain orphans their TOTP enrolment, and the symptom is unhelpful: MFA verification fails inside the brain and returns an opaque HTTP 500 rather than anything an operator can read. Three ways out, in order of preference:
- Put the old value back in Z4J_PREVIOUS_SECRETS and restart every brain replica. Settings are loaded at process startup, so changing the environment alone does not update a running brain. Decryption resumes after restart. Keep the retired secret in your secret store rather than destroying it the moment you drop it from the environment.
- The user signs in with a recovery code and re-enrols. Recovery codes are argon2id hashes with no master involvement, so they are unaffected, but recovery-code login alone does not re-wrap the old TOTP secret. This costs them a code and fails for anyone who has none left.
- z4j reset-mfa <email> clears that user's enrolment, recovery codes and trusted devices so they can start over. Shell-only, and it writes an audit row attributed to the OS user who ran it.
What is unrecoverable is that specific TOTP enrolment, not the account.

Two things this does not do, despite an earlier version of this page saying otherwise. It does not log anyone out: session cookies are signed with the independent Z4J_SESSION_SECRET, which has its own Z4J_PREVIOUS_SESSION_SECRETS carry-over list, and rotating one key does not touch the other. And it does not re-sign or invalidate a single audit row, because those are signed with the audit-chain key, so it neither repairs a broken chain nor breaks a good one.
Restore from a backup taken before the tamper timestamp if the current DB is compromised.
File an incident report - what was touched, over what period, and what was read or exfiltrated. The audit log records successful control-plane actions taken through z4j. It mostly does not record refusals: only write requests (POST, PUT, PATCH, DELETE) against a project's schedule endpoints leave a denial row, and those are enqueued best-effort and dropped under backpressure. Everywhere else an authorization denial raises without writing anything, so a compromised account probing what it can reach leaves no trace in the log. Read the log as "what succeeded", pair it with reverse-proxy or application access logs for the attempts, and use your database-side logs for anything done directly against the tables.

Why HMAC chaining matters, and where it stops

The audit log is z4j's tamper-evident record for everything that goes through z4j. An application bug, a downgraded adapter, a compromised operator account working through the product, or an over-eager retention job cannot quietly alter it: the chain fails verification and names the rows, so you learn both that it happened and where. That is worth running on a schedule, because without periodic audit verify a break can sit undetected for months.

It stops at the database. Someone who can write the audit tables directly does not have to break the chain, they can roll it back to a state that really was authentic, and verification will pass. So the chain is evidence against everything that goes through z4j, and it is not evidence against whoever holds the database credentials. If you need the second property, you need the off-box head export described in HMAC audit chain, and database credentials guarded as tightly as the audit key itself.

Leaked agent token

An agent bearer token (the plaintext value returned by the mint dialog once) was pasted into a public Slack / committed to a public git repo / exposed in a build log.

Immediate response

Revoke the token in the dashboard: /projects/<slug>/agents → click the agent → revoke.
Mint a replacement for the legitimate process. There is no rotate-in-place operation. Minting creates a new agent row, ID, and bearer token; the returned hmac_secret is the project's current signing secret.
Update the legitimate agent's config with the returned credentials. Restart the agent process.

Revocation first commits a durable soft tombstone: it sets revoked_at, replaces the old token hash with a value that cannot authenticate, and leaves the old row in place. If the replacement reuses the same name, the mint operation moves the tombstone into z4j's reserved-name namespace before inserting the new row. Only after the revoke commits does the brain try to kick active WebSockets locally and across replicas, and those kicks are best-effort. Independently, every authenticated inbound frame rechecks the durable marker before work is accepted. Do not use a notification or heartbeat latency estimate as the security boundary.

Damage assessment

The current audit schema attributes user control-plane actions with user_id or api_key_id. It has no actor_type or actor_id column and does not record agent transport frames as agent actors. You can use it to find control-plane actions that targeted the agent, including mint and revoke:

# SQL query - against z4j's Postgres / SQLite
SELECT occurred_at, user_id, api_key_id, action, target_type, target_id, source_ip
FROM audit_log
WHERE target_type = 'agent'
  AND target_id = '<revoked-agent-uuid>'
  AND occurred_at > '2026-04-23 12:00:00'
ORDER BY occurred_at;

For activity reported through the agent transport, inspect the raw events and your brain or reverse-proxy connection logs. Raw events retain the reporting agent ID:

SELECT occurred_at, kind, engine, task_id
FROM events
WHERE agent_id = '<revoked-agent-uuid>'
  AND occurred_at > '2026-04-23 12:00:00'
ORDER BY occurred_at;

events.occurred_at is supplied by the agent, so widen the time window and corroborate it with server-side logs when the agent itself may be compromised.

An agent bearer token is accepted by the agent transport: the /ws/agent WebSocket and its /api/v1/agent/* long-poll fallback. It is not a user API key. The bearer alone can authenticate as that agent, open or replace its connection, and receive commands sent to it. Valid post-handshake frames also need the project's hmac_secret. If both credentials leaked, the attacker can report lifecycle events, heartbeats, status, registry data, and command acknowledgements or results as that agent.

An agent token does NOT grant:

User REST API access, including task or schedule operations
Read access to project task history or engine state through the user API
Agent minting or revocation
User, membership, or project administration
Audit-log access

The blast radius is the authority of that agent transport identity, not anything a user can do in the project.

Lost last admin

The first-boot setup token is single-use. The CLI can reset the password of an existing user, but its admin-creation commands cannot add a user after the installation has left first boot.

Password forgotten

sudo -u z4j /srv/venv/bin/z4j changepassword [email protected] --password-stdin

Type or pipe the new password. This works without an admin login - it's an operator-side recovery tool. The command:

Hashes the new password with Argon2id
Bumps password_changed_at, invalidating every existing session for that user

No active admin remains

z4j bootstrap-admin, z4j createsuperuser, and the Z4J_BOOTSTRAP_ADMIN_* variables all require an empty users table. They cannot recover a populated installation whose admins were removed or deactivated. Restore a known-good backup or obtain supported recovery help rather than editing user rows or password hashes by hand.

Lost email access

If another active admin remains, use the authenticated admin surface to create the replacement identity and retire the old one. There is no email-rename or post-first-boot user-creation command in the CLI. If no active admin remains, use the backup or supported-recovery path above.

What to practice

Once a quarter: do a restore drill. The drill target has to be a real z4j installation at the current migration head, not a blank database, and it needs the same Z4J_AUDIT_CHAIN_SECRET as the source or the archive's chain will not verify. Restore into it, then check z4j migrate current, z4j status and z4j audit verify. Do not treat z4j check alone as the verdict: it exits 0 without comparing the revision against your code. This catches backup-process regressions before you need them.
Monthly: run z4j audit verify on the live install. On-boot is too late - the break could have happened weeks ago.
Before any upgrade: take a backup. Every time. See upgrade and rollback.