HMAC audit chain

Chain construction

Every audit row stores these columns (some legacy-classification fields are NULL on active rows):

id, occurred_at, user_id, api_key_id, project_id, action, target_type,
target_id, result, outcome, event_id, metadata, source_ip, user_agent,
row_hmac, prev_row_hmac, hmac_version, hmac_key_id, chain_generation,
legacy_frozen, legacy_integrity_class, legacy_origin

Where:

row_hmac = HMAC-SHA256(
  Z4J_AUDIT_CHAIN_SECRET,
  b"z4j/audit-chain/row/v2\x00" +
  canonical_json({
    version, id, occurred_at, user_id, api_key_id, project_id,
    action, target_type, target_id, result, outcome, event_id,
    metadata, source_ip, user_agent, prev_row_hmac,
    hmac_key_id, chain_generation
  })
)

The chain is signed with a dedicated key, Z4J_AUDIT_CHAIN_SECRET, which is independent of Z4J_SECRET. Outside development it is required and has no fallback: the brain refuses to start rather than quietly signing the audit log with the same key it uses for everything else.

The separation is the point. If the audit key were derived from application configuration that lives beside the database, anyone who could reach the database could also reach the key, and the chain would only prove that the rows had not been edited by someone with less access than the person most likely to edit them. Keep this key somewhere the database operator cannot read.

canonical_json is sorted-keys, no whitespace, UTF-8. UUIDs use canonical lowercase text, timestamps use UTC ISO-8601 to microseconds, and IP addresses use their compressed form. prev_row_hmac is the row_hmac of the previous row in (occurred_at, id) order. The writer keeps that pair strictly monotonic; id itself is a random UUID and is not an ordering sequence. The first row of a generation has prev_row_hmac = null.

Properties

• Tamper-evident - change any field → row_hmac no longer verifies.
• Insertion-evident - inserting a row in the middle breaks the next row's prev_row_hmac link.
• Deletion-evident - deleting a row breaks the chain at the gap, and leaves the retained rows disagreeing with the authenticated head and row count held in audit_chain_state.

Each of those is checked on every verification, and each of them catches the realistic failures: a bug that writes outside the audit service, an older adapter, a retention job that removes more than it should, an operator running a DELETE by hand.

What this does not protect against

A role that can write audit_log and audit_chain_state. The authenticated state the verifier compares the log against is a single row in the same database. That role does not have to forge it. It can put back a copy of that row from before the rows it wants removed -- a copy the brain itself signed when it was current, so it still authenticates -- delete the rows written since, and leave a log whose head, counts and links all agree. The verifier reports it clean.

Key separation still buys something real, and it is worth being exact about what. Writing new audit history that verifies requires the audit key. Rolling the log back to history that really happened requires only write access to two tables, and a copy of a one-row table from any earlier point, which any backup contains.

So the honest summary is: the chain is evidence against everything that goes through z4j, and it is not evidence against whoever holds the database.

Making it hold against the database

Verification accepts a head you recorded earlier and reports whether the current chain still contains it. Record the current row_hmac on a schedule, somewhere append-only that the database role cannot reach (an object-store bucket with a retention lock, a second database under different credentials, a log service that will not accept edits), and verify against it:

# Record the head to the external sink, on whatever schedule you keep.
z4j audit export-head > /secure/last-known-head

# Later, check the log against the head you recorded.
z4j audit verify --known-head "$(cat /secure/last-known-head)"

export-head authenticates the chain state against the configured keys before printing, and refuses rather than exporting a head it cannot prove. It writes the envelope to stdout and everything else to stderr, so the redirect above produces a complete file. Add --verify to walk the whole active generation first and refuse to export a head from a chain that did not verify clean, which is what an unattended job should do: anchoring a compromised chain records it as the trusted state.

Write the file exactly as the command emits it. The parser is strict, and its key set is closed: uppercase hex, or any key beyond the six the envelope carries, reports INVALID and exits nonzero. Appending a note or a timestamp to the file makes it unreadable by the command that consumes it.

The result is one of CURRENT_MATCH, PRUNE_MATCH, CURRENT_PRUNE_MATCH, VERIFIED_ANCESTOR, INVALID or UNPROVABLE. A log that was rolled back past the head you recorded reports UNPROVABLE instead of verifying clean. The same result occurs on an untampered log after retention has pruned past that head. Record heads more often than your audit retention window, and interpret UNPROVABLE with no other mismatch as "the anchor is gone; determine why", not as proof of an attack by itself.

This is the piece that turns the chain into something a hostile database role cannot quietly defeat, and it only works if the export lands somewhere durable and gap-detecting. The built-in audit webhook is explicitly best-effort and drops rows under backpressure, so it is not that place.

Verification

z4j audit verify

Walks the log and prints the rows verified in the active and frozen generations. It prints at most 100 finding strings, followed by the number it did not print; the MISMATCHES (n) header carries the true total. Exit status is 0 for a clean chain and 2 only for a settings-load failure or an invalid --limit. Findings, database connection failures, and verification exceptions all exit 1, so route infrastructure failures by matching the Boundary-F integrity verification refused message rather than by exit code alone.

Two ways to run it periodically, and you want one of them, because a chain nobody walks is not evidence of anything:

• Wrap the CLI in cron or a Kubernetes CronJob.
• Enable the built-in verifier worker: Z4J_AUDIT_CHAIN_VERIFY_ENABLED=true. It is off by default, runs daily by default, and accepts Z4J_AUDIT_CHAIN_VERIFY_INTERVAL_SECONDS values from 900 seconds through 604800 seconds. An out-of-range value is rejected at startup rather than clamped. The worker is leader-gated so N replicas do not each walk the same table. A failed verification is logged at error level and counted; it does not stop the brain that is already running, preserving your ability to investigate. Do not restart until it is repaired: startup runs the full verification and refuses to serve an unclean chain. See monitoring for the metrics.

Note what the periodic walk covers. The audit_chain probe on the deep-health endpoint is a much narrower check: it authenticates the chain-state row and reports scope: "state-only", deliberately never reading audit_log. A tampered audit row passes there. The verification above is what covers rows.

Activation

A chain has to start somewhere, and the honest question on an existing deployment is what to do with history written before there was a chain.

The upgrade classifies the existing audit history in place. When every row classifies unambiguously, the chain activates during the migration and those rows are frozen as authenticated legacy history, so the ordinary upgrade is an ordinary upgrade.

When the history cannot be classified beyond doubt, the migration stops and returns nonzero, naming the rows and the reason. That covers a forked or truncated chain, rows signed with a key you no longer hold, and an audit table that exists but is empty, because a row count alone cannot distinguish a fresh install from one that was pruned or restored. In that case nothing is half-applied: the preparation revision stands and chain state is not created. The brain will not serve at that revision, and downgrade is refused while the preparation is pending. The only supported path is forward: complete the explicit operator attestation and re-run. Plan the upgrade window accordingly.

Refusing here is deliberate. Adopting history that cannot be proven would hand anyone who emptied the audit log a clean genesis and a chain that verifies, which is the outcome activation exists to withhold. It is worth being exact about the scope of that: refusing here closes the activation path to it. It does not close the database path described above, and nothing in the current design does.

Key rotation

Changing the environment variables alone does not rotate the authenticated state. It lets verification load both keys, but every audited action is refused until the explicit rotation ceremony commits. Stop the brain replicas and treat the configuration change and command as one maintenance step.

For a packaged SQLite install whose key lives in z4j's managed secret store:

z4j audit rotate-chain-key --begin-managed

For explicitly supplied keys, put the new key in Z4J_AUDIT_CHAIN_SECRET, keep the old value and any earlier retained values in comma-separated Z4J_AUDIT_CHAIN_PREVIOUS_SECRETS, make that same window available to every replica, and then run:

z4j audit rotate-chain-key

The command writes an audit.chain_key_rotated row under the new key and re-signs authenticated chain state. Rotation does not re-sign old rows. Keep the old key configured until retention has removed every row signed under it, then use the key's 64-character lowercase ID to prove it is safe to retire:

z4j audit retire-chain-key --key-id <id>

That command refuses while a live row still needs the key. Removing the key early makes startup refuse because the configured key window no longer covers the authenticated row counts.

There is no standalone "begin a fresh chain" operation; only the full, destructive z4j reset ceremony begins a replacement generation. The ordinary project audit export omits both chain-HMAC columns and is not offline evidence. Preserve an authorized database backup and the corresponding old audit key if retired rows must remain verifiable.

Handing the log to an auditor

The project audit export (?format=csv, json, or xlsx) carries the audit columns and not row_hmac or prev_row_hmac. It is a record to ship to a SIEM or hand to an auditor, not something a downstream system can re-verify the chain from. Do not build a compliance workflow on the assumption that an exported file proves itself.

Verification runs against the database with the audit-chain key, but read-only database access is not sufficient. On PostgreSQL the verifier takes LOCK TABLE audit_log IN SHARE MODE and selects audit_chain_state with FOR UPDATE; the role therefore needs write privilege on both tables. It also loads the full application settings, including required Z4J_SECRET and Z4J_SESSION_SECRET values. Giving those privileges and production secrets to an external auditor defeats the separation this page is trying to preserve. Run verification yourself and hand over its output, or give the auditor an isolated restored copy of the database and a purpose-built settings environment instead of access to the live brain.

Performance

Verification is a linear full-log walk. Runtime depends on row count, page size, database, and host; measure your own deployment before choosing a maintenance window. PostgreSQL verification holds a SHARE lock on audit_log for the whole walk, so audited writes wait, and startup performs the same walk before the brain accepts traffic.