Metrics API

Endpoint

GET /metrics
Authorization: Bearer <Z4J_METRICS_AUTH_TOKEN>

Not a REST-versioned endpoint (Prometheus convention), and closed by default. Normal z4j serve bootstrap auto-mints and persists a Z4J_METRICS_AUTH_TOKEN; scrapes must present it. If neither a token nor Z4J_METRICS_PUBLIC=1 is configured, the endpoint returns 401. Setting Z4J_METRICS_PUBLIC=1 deliberately opens it and emits a startup warning.

Catalog

Ingest and lifecycle

• z4j_events_ingested_total{project,engine,kind} (counter): wire-level event ingest count by project, engine adapter, and event kind.
• z4j_tasks_total{project,task_name,state} (counter): observed tasks by final state. High-cardinality task names are bounded by the server's overflow policy.
• z4j_task_duration_seconds{project,task_name} (histogram): task wall-clock duration buckets.
• z4j_commands_total{project,action,status} (counter): commands dispatched to agents.
• z4j_command_late_results_total{status} (counter): command results received after the row was already terminal.

Agents, workers, queues

• z4j_agents_online{project} (gauge): currently connected agent count.
• z4j_workers_online{project} (gauge): currently online worker count.
• z4j_queue_depth{project,queue,engine} (gauge): pending messages in a queue.
• z4j_ws_connections (gauge): live WebSocket connections held by this worker.

Database

• z4j_db_pool_size (gauge): configured SQLAlchemy pool size. Populated at scrape time. Rises only when pool_size setting changes; useful baseline.
• z4j_db_pool_checked_out (gauge): pool connections currently checked out (in active use). Steady-state under burst load is a key contention indicator; should be much less than pool_size.
• z4j_brain_rss_bytes (gauge): brain process RSS in bytes, sampled at scrape time from /proc/self/status. 0 on non-Linux. The slope under sustained load is the headline leak signal; flat or slow growth is healthy, rapid growth indicates either an unbounded cache (tune Z4J_DATABASE_STATEMENT_CACHE_SIZE) or a new retention path. See Brain memory tuning for the playbook.
• z4j_postgres_deadlocks_total (counter): total Postgres DeadlockDetectedError instances observed via the asyncpg/SQLAlchemy handle_error event listener. Should hover at or near zero in steady state; sustained non-zero rates on INSERT INTO workers / UPDATE agents / UPDATE queues point at lock-order contention.

Notifications

• z4j_notifications_sent_total{project,channel_type,status} (counter): notification deliveries attempted.
• z4j_notifications_cooldown_skipped_total{project,trigger} (counter): dispatches skipped because the cooldown window had not elapsed.

In-memory state

• z4j_inmemory_state_items{subsystem} (gauge): registered process-local state, currently longpoll_sessions, activity_rate_limit_users, otel_dynamic_sensitive_hosts, metric_task_name_projects, and, when enabled, audit_forwarder_queue. It is refreshed by the worker handling the scrape; multiprocess aggregation sums live workers, whose other samples may be one scrape behind.

Reliability and self-watch

• z4j_swallowed_exceptions_total{module,site} (counter): intentional exception swallows at I/O boundaries (metric updates, WebSocket close during shutdown, asyncpg teardown). A sustained non-zero rate signals a subsystem in trouble even when no error-level log fires.
• z4j_background_task_error_active{task} (gauge): covers audit_retention and wal_checkpoint; 1 means that worker's most recent pass failed, and 0 means a later pass succeeded. In multiprocess mode the maximum is retained, including a failed worker that later died, until the serve run restarts.

Audit retention and WAL

• z4j_audit_retention_pruned_total (gauge despite the _total suffix): cumulative audit rows pruned during the current serve run; it resets on restart. Do not apply counter-only functions such as rate() or increase().
• z4j_audit_retention_last_run_timestamp (gauge): Unix timestamp of the most recent retention pass.
• z4j_audit_retention_last_deleted (gauge): rows deleted by the worker whose pass was most recently published. In a multiworker deployment this is not a brain-wide deletion total and can be zero even when another worker pruned rows; use z4j_audit_retention_pruned_total to confirm cumulative work for the current serve run.
• z4j_audit_chain_verifications_total{outcome} (counter): scheduled audit-chain verification runs by outcome. clean, failed (the chain did not verify and needs investigation), or error (the run could not complete, which is a different problem and must not be read as evidence of tampering). Only emitted while Z4J_AUDIT_CHAIN_VERIFY_ENABLED is on.
• z4j_audit_chain_rows_verified (gauge): rows walked by the most recent successful or integrity-failed verification that walked at least one row. An error or zero-row run leaves the prior value in place, so pair it with z4j_audit_chain_verifications_total{outcome=...}.
• z4j_wal_checkpoint_last_run_timestamp (gauge): Unix timestamp of the most recent SQLite WAL checkpoint (0 on Postgres).
• z4j_wal_checkpoint_pages_last (gauge): pages checkpointed in the most recent successful SQLite WAL pass; -1 means no useful result yet and is also the permanent value on PostgreSQL.

Grafana dashboard

Shareable Grafana dashboards live in the repository checkout's deploy/grafana/ directory (overview, tasks, agents, scheduler, notifications). They are not included in the installed wheel or the z4j package sdist. Import in Grafana via Dashboards, New, Import, then upload the JSON. The overview dashboard covers:

1. Brain process RSS plus 5-min slope (the headline leak signal).
2. Postgres deadlock rate over 5-min windows.
3. DB pool: configured size vs current checked-out count.
4. Event ingest rate and total agents online (for correlation against the leak signal).

Configure your Prometheus datasource to scrape the brain's /metrics with the Z4J_METRICS_AUTH_TOKEN bearer:

prometheus.yml

scrape_configs:
  - job_name: z4j
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials: $Z4J_METRICS_AUTH_TOKEN
    static_configs:
      - targets: ["z4j.internal:7700"]