Production hardening

z4j ships with backwards-compatible defaults that work out of the box but lean permissive. For production deployments, opt into the fail-closed mode of each subsystem below. None of these are required for z4j to function -- they're defense-in-depth for operators who want hardened defaults instead of trust-the-CA / trust-the-operator-config.

Scheduler gRPC -- require explicit CN allow-list

When Z4J_SCHEDULER_GRPC_ENABLED=true, brain accepts mTLS-authenticated gRPC connections from any client cert that the configured CA bundle validates -- the "trust the CA" deployment model. For production, populate Z4J_SCHEDULER_GRPC_ALLOWED_CNS with the explicit list of CNs you've minted via z4j mint-scheduler-cert, AND set the require flag so a misconfigured boot fails closed instead of falling back to trust-the-CA:

Z4J_SCHEDULER_GRPC_ENABLED=true
Z4J_SCHEDULER_GRPC_ALLOWED_CNS='["scheduler-prod-1","scheduler-prod-2"]'
Z4J_SCHEDULER_GRPC_REQUIRE_ALLOWLIST=true

For the brain to push schedule triggers to the scheduler, set the outbound gRPC client variables on the brain. Without Z4J_SCHEDULER_TRIGGER_URL the brain uses its in-process scheduler path and the TLS variables are ignored:

Z4J_SCHEDULER_TRIGGER_URL=scheduler.example.com:7802
Z4J_SCHEDULER_TRIGGER_TLS_CERT=/etc/z4j/pki/brain.crt
Z4J_SCHEDULER_TRIGGER_TLS_KEY=/etc/z4j/pki/brain.key
Z4J_SCHEDULER_TRIGGER_TLS_CA=/etc/z4j/pki/scheduler-ca.crt

The allow-list for the brain's CN lives on the scheduler side (SCHEDULER_GRPC_ALLOWED_CNS in the scheduler's own env), not on the brain.

CN project bindings (multi-project deployments)

If you run schedulers per-project (one scheduler instance per tenant, each with its own CN), bind each CN to its project list so a leaked cert can only act on the projects it was minted for:

Z4J_SCHEDULER_GRPC_CN_PROJECT_BINDINGS='{"scheduler-acme":["acme"],"scheduler-globex":["globex"]}'

Without bindings (the default), every allow-listed CN can drive RPCs for any project. With bindings, requests outside the bound project list return PERMISSION_DENIED.

Notification webhooks -- HTTPS-only by default

z4j defaults to HTTPS-only for generic webhook channels. Operator-configured http:// URLs are rejected at config-time and at dispatch-time to prevent payload + custom-header leakage in transit.

If you have a legitimate internal-network http endpoint (intranet receiver, dev rig), opt back in:

Z4J_NOTIFICATIONS_WEBHOOK_ALLOW_HTTP=true

Slack / PagerDuty / Discord / Telegram channels always use the provider's HTTPS endpoint and are unaffected by this setting.

Audit log retention

The audit log is HMAC-chained, and z4j's own write path only ever appends to it. It is not retained forever: a retention sweeper prunes it, and Z4J_AUDIT_RETENTION_DAYS defaults to 90. Set this to your real obligation before the first sweep runs, because rows past the window are deleted rather than archived:

Z4J_AUDIT_RETENTION_DAYS=365   # rolls daily; trims rows older than 365d

Pruning the oldest rows eventually removes the genesis row, so the sweeper advances an authenticated prune watermark on the chain state. The verifier accepts a first surviving row whose prev_row_hmac matches that watermark, while a deleted middle row or an altered row_hmac still fails, so retention cannot quietly become a clean-looking truncation. Details in hmac-audit-chain.

Verifying the audit chain

Schedule verification. A tamper-evident log is only tamper-evident if somebody looks, and an on-demand run only ever proves the chain was intact at the moment someone asked. Either wrap z4j audit verify in cron or a Kubernetes CronJob, or turn on the built-in worker, which is off by default:

Z4J_AUDIT_CHAIN_VERIFY_ENABLED=true
Z4J_AUDIT_CHAIN_VERIFY_INTERVAL_SECONDS=86400   # floor 900, ceiling 604800

For anything stronger than "nothing outside z4j's own write path touched this log", export the chain head on a schedule to a sink the database role cannot rewrite and verify against it with z4j audit verify --known-head. That is the one check that survives a hostile database role. The built-in audit webhook is not that sink: it buffers in memory and drops rows under backpressure, so a gap in its copy cannot tell an outage from a deletion.

Production environment flag

Z4J_ENVIRONMENT alone selects environment-sensitive behavior. The exact value dev enables development relaxations; every other value uses the non-development path. Z4J_ALLOWED_HOSTS is a separate host-header allow-list that is required outside development, but it does not participate in choosing the environment. See allowed-hosts for the four-layer host header allow-list.

Recommended summary

For a production deployment, set:

# Environment
Z4J_ENVIRONMENT=production
Z4J_ALLOWED_HOSTS='["z4j.example.com"]'

# Scheduler gRPC server (brain accepts inbound from z4j-scheduler)
Z4J_SCHEDULER_GRPC_ENABLED=true
Z4J_SCHEDULER_GRPC_REQUIRE_ALLOWLIST=true
Z4J_SCHEDULER_GRPC_ALLOWED_CNS='["..."]'

# Scheduler trigger client (brain dials z4j-scheduler outbound)
Z4J_SCHEDULER_TRIGGER_URL=scheduler.example.com:7802
Z4J_SCHEDULER_TRIGGER_TLS_CERT=/etc/z4j/pki/brain.crt
Z4J_SCHEDULER_TRIGGER_TLS_KEY=/etc/z4j/pki/brain.key
Z4J_SCHEDULER_TRIGGER_TLS_CA=/etc/z4j/pki/scheduler-ca.crt

# Webhooks: HTTPS-only is the default; only set this if you need
# plaintext for an internal endpoint.
# Z4J_NOTIFICATIONS_WEBHOOK_ALLOW_HTTP=true

# Audit retention
Z4J_AUDIT_RETENTION_DAYS=365

Each setting is documented individually in the env vars reference.