Skip to content
z4j docs

Agent offline alerts

An agent cannot report its own death, so offline detection lives brain-side: a periodic health sweep watches every agent’s heartbeat and confirms outages. Detection is the emit site for the worker.offline automation trigger and for agent.offline notification subscriptions.

Two thresholds, two meanings

Section titled “Two thresholds, two meanings”

The sweep runs every Z4J_AGENT_HEALTH_SWEEP_SECONDS (default 10) and applies two distinct thresholds:

  1. State flip. An agent whose last heartbeat is older than Z4J_AGENT_OFFLINE_TIMEOUT_SECONDS (default 30) is marked offline on the Agents page. This is the dashboard’s honest, fast view. (An agent that disconnects cleanly is flipped immediately by the gateway; the sweep covers zombie sockets, network partitions, and a brain worker that died before running its close handler.)
  2. Confirmed episode. An agent still silent past Z4J_AGENT_OFFLINE_TIMEOUT_SECONDS + Z4J_AGENT_OFFLINE_ALERT_GRACE_SECONDS (default 30 + 60) is a confirmed-down episode, and that is what alerts.

The extra grace exists to absorb deploy restarts and reconnect blips: an agent back online within timeout + grace flips state on the dashboard but never pages anyone. The grace only delays alerting, never the state flip.

What a confirmed episode emits

Section titled “What a confirmed episode emits”

Each newly confirmed episode produces, once:

  • An agent.offline_detected row in the HMAC-chained audit log, with the agent’s name, adapters, last heartbeat, how long it has been silent, and the timeout + grace in force at detection time.
  • Any worker.offline automation rule on the project, through the same governed executor the task-event path uses (per-project kill switch, circuit breaker, dry-run, and per-rule transactions all apply). An offline agent is the absence of a heartbeat, so there is no task context: retry / cancel actions on a worker.offline rule resolve to no target; use notify.
  • Any agent.offline notification subscription on the project, across the delivery channels and the in-app bell. This path is independent of automation, so offline alerting works on a brain with no rules armed.
  • A bump on the z4j_agents_offline_detected_total{project} Prometheus counter, so a dying fleet is visible on a metrics dashboard without scraping audit rows.

Automation firing and the subscription fanout are independent, best-effort side effects of the committed detection: a failure in one is logged and never blocks the other, and never causes the episode to be recorded twice.

One alert per episode, fleet-wide

Section titled “One alert per episode, fleet-wide”

An episode is keyed by (agent_id, last heartbeat). The heartbeat anchor is frozen while the agent is down and advances only when it reconnects, so:

  • An agent that stays down for a week alerts once, however many sweeps span the outage.
  • An agent that recovers and later dies again is a fresh episode and alerts again.

The dedup claim is durable and shared across brain replicas (the agent_offline_alerts table), so in an HA deployment exactly one replica alerts per episode, across sweeps, restarts, and failovers. The claim is also conditional on the agent still being offline on the same anchor at claim time: an agent that reconnects between candidate selection and the claim yields no claim and no false alert.

Failure handling errs toward visibility: if writing the alert fails after the claim is won, the claim is released so the next sweep retries. Claims of ended episodes age out after 30 days to keep the ledger bounded; the claim of an unchanged, ongoing outage is kept regardless of age, so a long outage is never re-alerted on claim expiry.

Burst behavior

Section titled “Burst behavior”

A network partition can take a whole fleet down at once. At most 100 new episodes alert per sweep; the overflow is not lost, it is picked up by subsequent sweeps (its claims are still unrecorded), spreading a fleet-wide event over a few ticks instead of writing hundreds of audit rows and rule firings in one.

Settings

Section titled “Settings”
Variable Default Description
Z4J_AGENT_OFFLINE_TIMEOUT_SECONDS 30 Heartbeat age at which an agent is marked offline on the dashboard.
Z4J_AGENT_OFFLINE_ALERT_GRACE_SECONDS 60 Extra silence past the timeout before the episode is confirmed and alerted. 0 alerts at the plain timeout.
Z4J_AGENT_HEALTH_SWEEP_SECONDS 10 Sweep cadence.

Tune the grace to your deploy style: rolling restarts that take two minutes per agent want a grace above 120; a fleet that must page fast on real outages wants it low.

Common patterns

Section titled “Common patterns”

Page on confirmed outage

Section titled “Page on confirmed outage”

A PagerDuty channel plus an agent.offline project subscription pages the on-call the moment an episode is confirmed. Severity for agent.offline defaults to critical. See Notifications.

Alert-and-annotate with automation

Section titled “Alert-and-annotate with automation”

A worker.offline rule with a notify action raises an in-app notification for every project member and leaves an audit trail per firing. Combine with a metrics alert on z4j_agents_offline_detected_total for the belt-and-braces version.

See also

Section titled “See also”