Audit log

What gets logged

Every side-effect z4j performs:

• Auth - login, logout, password change, failed login, invitation accepted, password reset requested/completed.
• Users / memberships - invite, accept, role change, remove.
• Agents - token mint and token revoke. Revocation replaces the token hash and sets revoked_at; the agent row remains as a tombstone so its historical events keep a valid, non-null agent_id reference.
• Actions - task retry, cancel, bulk retry, queue purge.
• Schedules - create, update, pause, resume, delete.
• Project - create, settings change, delete.

Events flowing through the queue are not in the audit log - those are in events. Audit is "who pressed what button," not "what did the workers do."

Refusals are mostly not here. A request rejected for insufficient role writes no audit row on most API families. The exception is a mutating schedule route, where authentication, authorization, not-found, and validation failures are offered to a bounded best-effort queue so IDOR probing leaves a breadcrumb, and that queue can drop under pressure. See error format.

Chain structure

Each row contains:

id, occurred_at, user_id, api_key_id, project_id, action, target_type,
target_id, result, outcome, event_id, metadata, source_ip, user_agent,
row_hmac, prev_row_hmac, hmac_version, hmac_key_id, chain_generation

row_hmac = HMAC-SHA256(Z4J_AUDIT_CHAIN_SECRET, domain_prefix || canonical(row_fields || prev_row_hmac))

• canonical(...) is a deterministic JSON serialization (sorted keys, no whitespace, UTF-8).
• prev_row_hmac refers to the previous row in (occurred_at, id) order. id is a random UUID and is not an ordering sequence; the writer keeps that pair strictly monotonic.
• The first row of a generation has prev_row_hmac = null.
• See HMAC audit chain for the exact preimage.

The chain is signed with Z4J_AUDIT_CHAIN_SECRET, a dedicated key independent of Z4J_SECRET. Outside development it is required and has no fallback: the brain refuses to start rather than reuse the master application key that protects agent tokens, frame keys, and stored TOTP secrets. Sessions use the separate Z4J_SESSION_SECRET. Keep the audit-chain key somewhere the database operator cannot read.

Deleting or rewriting a row breaks the chain at that point, and every subsequent row fails to verify. That is what catches the realistic failures: a bug that writes outside the audit service, an older adapter, a retention job that removed more than it should, a DELETE run by hand.

It is not unconditional, and the limit is worth knowing before you rely on it. A role that can write both audit_log and the authenticated chain state can delete a suffix of the log and put back a copy of the state row from before those rows existed, which the brain itself signed when it was current. Links, head, and counts then agree and verification passes. Writing new history that verifies still requires the audit key; rolling back to history that really happened does not. See HMAC audit chain for the full threat model and for the off-box head export that closes this.

Verification

z4j audit verify walks the entire log in chain order and checks every row's HMAC and prev_row_hmac link against an authenticated chain-state row that carries the expected head and row counts. It reports every finding it saw, not just the first, and exits 0 for a clean chain and 1 for any finding. Exit 2 is reserved for a settings-load failure or an out-of-range --limit; a database connection failure or a verification exception also exits 1, so route infrastructure failures by matching the Boundary-F integrity verification refused message rather than by exit code alone.

Pass --known-head with a head envelope you exported earlier to also assess whether the current chain still contains it. The result is one of CURRENT_MATCH, PRUNE_MATCH, CURRENT_PRUNE_MATCH, VERIFIED_ANCESTOR, INVALID, or UNPROVABLE; a log rolled back past that head reports UNPROVABLE rather than verifying clean.

For a scheduled check, either wrap the CLI invocation in cron or a Kubernetes CronJob, or enable the brain's own verifier worker with Z4J_AUDIT_CHAIN_VERIFY_ENABLED=true. That worker is off by default, runs daily by default, is leader-gated, and records a failed verification without stopping the brain. Run it one way or the other: a chain nobody walks is not evidence of anything, and an on-demand run only ever proves the chain was intact at the moment somebody asked.

The audit_chain probe on the deep-health endpoint is a narrower check: it authenticates the chain-state row and confirms the brain would boot, and it deliberately does not read audit_log. A tampered audit row passes there. The scheduled verifier above is what covers the rows.

Retention

Audit rows are pruned on a schedule, oldest first. Z4J_AUDIT_RETENTION_DAYS sets the window (default 90) and a background sweeper does the work in bounded batches, so a large log is trimmed over several passes rather than in one long transaction.

Pruning the oldest rows removes the genesis row, so the sweeper records the prune boundary as an authenticated watermark. The verifier accepts a first surviving row whose prev_row_hmac matches that watermark, while a deleted middle row or an altered row_hmac still fails. Retention therefore does not quietly turn into a clean-looking truncation.

Set Z4J_AUDIT_RETENTION_DAYS to your real obligation before the first sweep runs. Rows past the window are deleted, not archived, so if you need history beyond it, ship the log somewhere else as it is written.

Export

Export from the audit API for a project, /api/v1/projects/<slug>/audit, with format=csv, format=json, or format=xlsx alongside the usual action / outcome / user / time-window filters. One export is capped at 50,000 rows; narrow the filter rather than dumping the whole log.

These exports carry the audit fields (id, timestamp, action, target, result, outcome, user, source IP, user agent, metadata) and not row_hmac or prev_row_hmac, so they are a record to hand to a SIEM or an auditor, not something a downstream system can re-verify the chain from. Chain verification is z4j audit verify against the database, with the audit-chain key.

What it does not give you

• Not a SIEM - use it as a source for a SIEM (export + ship to Datadog / Splunk / Loki).
• Not a compliance certificate - tamper-evidence is a primitive; SOC 2 auditors want policies and training, not just tech.
• Not a replacement for database-side auditing - what the chain signs is what z4j wrote. Someone acting directly on the database is outside it, and can leave a log that still verifies. Treat the database credentials as equivalent to control of this record, and add Postgres-side auditing if that matters to you.

See security § HMAC audit chain for the threat model.