Monitoring

Health endpoints

GET /api/v1/health          # liveness, public
GET /api/v1/health/ready    # readiness, public
GET /api/v1/health/system   # build and platform detail, authenticated
GET /api/v1/health/deep     # per-subsystem report, authenticated

Liveness

/api/v1/health does no I/O and returns exactly:

{ "status": "ok" }

Nothing else. The endpoint is publicly reachable so container runtimes can poll it cheaply, and it deliberately carries no version field: publishing the brain version on an unauthenticated endpoint lets an attacker pin known CVEs to your install. Version detail lives on /health/system, behind authentication.

Readiness

/api/v1/health/ready returns 200 once lifespan startup has finished and a SELECT 1 round-trips within two seconds. Otherwise it returns 503 with {"status": "unready", "reason": "starting"} or {"status": "unready", "reason": "database"}. That is the whole check: it does not compare the schema against the migration head, and it does not look at agent connectivity. Pair it with a Kubernetes readinessProbe.

Per-subsystem detail

/api/v1/health/deep (authenticated) reports one entry per subsystem and returns 503 when any of them failed. Every response carries a coverage object saying that its verdict is not the one the boot path would give the same database, in both directions: it checks less than startup does, and it can also fail on a deadline startup does not have. A check that could not complete is reported as failed rather than omitted, because an omission reads as healthy.

The audit_chain check authenticates the chain-state row and reports scope: "state-only" on every answer, clean or not. It does not read audit_log, so a tampered audit row passes it. Do not wire an audit-integrity alert to this endpoint; that is the scheduled verifier's job, below.

/api/v1/health/system (authenticated) is build and platform detail: version, Python, OS, database version and size, package versions. It is not a subsystem report.

Audit-chain verification

Verification walks every retained row, so it is not something a health probe can do on each poll. It runs in two places instead:

• On demand, via z4j audit verify (CLI, dashboard button).
• On a schedule, via a built-in leader-gated worker. It is off by default. Set Z4J_AUDIT_CHAIN_VERIFY_ENABLED=true to turn it on, and Z4J_AUDIT_CHAIN_VERIFY_INTERVAL_SECONDS to change the cadence (default 86400, floor 900, ceiling 604800).

A failed verification is logged at error level and counted. It does not stop the brain, because refusing to serve would destroy your ability to investigate the thing that just tripped.

With the worker enabled, two metrics carry the result:

• z4j_audit_chain_verifications_total{outcome} (counter): runs by outcome. clean, failed (the chain did not verify), or error (the run could not complete, which is a different problem and must not be read as evidence of tampering).
• z4j_audit_chain_rows_verified (gauge): rows walked by the most recent run.

Neither metric is emitted while the worker is disabled, so an alert on them is also an alert on having switched it off.

Remember what a clean result covers. It says nothing outside z4j's own write path touched the log. A role that can write audit_log and audit_chain_state directly can roll the log back to an earlier state the brain itself signed, and this verification will report it clean. To detect that you need a head exported somewhere that role cannot write, checked with z4j audit verify --known-head. See HMAC audit chain.

Metrics

Scrape /metrics (Prometheus format, token-gated). Full metric list: metrics API.

Alerts to set

Alert	Trigger
Brain down	up{job="z4j"} == 0 for 2m
Audit chain broken	increase(z4j_audit_chain_verifications_total{outcome="failed"}[1d]) > 0
Audit chain unwatched	increase(z4j_audit_chain_verifications_total[2d]) == 0
Agents dropping	sum(z4j_agents_online) falls by > 20% in 5m
Task failure rate	rate(z4j_tasks_total{state="task.failed"}[5m]) > 0.1
Task backlog growing	engine-specific (Celery/RQ/...)
Background task failing	max(z4j_background_task_error_active) == 1 for 15m
High HTTP 5xx	Not available from /metrics. The brain exports no HTTP request counter; alert on 5xx at your reverse proxy.

Logs

JSON to stdout. Fields:

• ts, level, logger, msg
• request_id (per HTTP request)
• user_id (when authenticated)
• project_id
• agent_id (when relevant)

Ship with Fluent Bit / Vector / Loki / Datadog.

Error tracking / APM

Two optional integrations ship with the brain. Both are off until you configure them.

• Sentry. Install z4j[sentry] and set Z4J_SENTRY_DSN. Unhandled exceptions in HTTP handlers, background workers and domain code are captured, and every event passes through a redaction pass before the SDK ships it. See Sentry.
• OpenTelemetry. Install z4j[otel] and set Z4J_OTEL_EXPORTER_OTLP_ENDPOINT. FastAPI requests, SQLAlchemy queries and outbound httpx calls are traced and exported over OTLP. Traces only; metrics stay on /metrics. See OpenTelemetry.

Neither is required. Application logs go to stdout as JSON and can be shipped by your log pipeline (Fluent Bit / Vector / Loki / Datadog) instead. There is no bundled APM agent beyond the OTLP exporter above.