Schedule fire history

Every fire dispatched through z4j-scheduler writes one row of history. The schedule row itself carries last_run_at and the run counters ("what is going on right now"); the fire history carries the per-fire detail operators need to answer "did the 3am cron actually fire, and what happened to it?"

Fire history exists for schedules fired by z4j-scheduler (it requires the scheduler integration, Z4J_SCHEDULER_GRPC_ENABLED=true). Schedules surfaced by the per-engine adapters (celery-beat, rq-scheduler, ...) fire inside their native scheduler and do not produce fire rows.

Lifecycle of a fire

A fire row progresses through these states:

Status	Meaning
pending	Row created; the fire arrived from the scheduler but the command has not been issued yet.
accepted	The brain accepted the fire and is dispatching the command.
delivered	The brain dispatched the schedule.fire command to an online agent.
buffered	No matching agent was online; the fire landed in the pending_fires buffer for replay once an agent for the right engine reconnects. buffer_stale and buffer_expired mark a buffered fire whose definition changed since, or whose buffer time ran out.
acked_success	The agent ran the task and reported success.
acked_failed	The agent reported a failure result. error_code / error_message carry the detail.
failed	Brain-side or transport error; the fire never reached an agent.
operator_skipped	An operator skipped the fire during recovery.
terminal_completed, terminal_failed, terminal_cancelled, terminal_timeout	Outcomes set by the fire-recovery routes or by the ack timeout, for fires that never received an ordinary ack.

One row per fire: the scheduler's idempotency key (fire_id, derived from the schedule and its tick boundary) is unique, so HA retries and network duplicates collapse into a single row instead of double-counting.

When the ack arrives, the row also records acked_at and latency_ms (ack minus dispatch), so per-schedule delivery latency is readable without a join.

Consecutive failed / acked_failed fires feed the schedule circuit breaker: after Z4J_SCHEDULE_CIRCUIT_BREAKER_THRESHOLD consecutive failures the schedule is auto-disabled and a schedule.circuit_breaker.tripped audit row is written.

Reading the history

GET /api/v1/projects/{slug}/schedules/{schedule_id}/fires

Role: viewer. Returns the most recent fires, newest first. limit defaults to 100, capped at 1000.

[
  {
    "id": "...",
    "fire_id": "...",
    "schedule_id": "...",
    "command_id": "...",
    "status": "acked_success",
    "scheduled_for": "2026-03-14T03:00:00Z",
    "fired_at": "2026-03-14T03:00:00.412Z",
    "acked_at": "2026-03-14T03:00:01.006Z",
    "latency_ms": 594,
    "error_code": null,
    "error_message": null,
    "triggered_by_user_id": null
  }
]

scheduled_for is the tick boundary the scheduler computed (when the fire was supposed to happen); fired_at is when the brain recorded the fire. A growing gap between the two is a scheduler falling behind.

The dashboard renders the same data as the Last 50 fires panel on each schedule's detail page: status pill, scheduled-for, fired-at, latency, and the error detail inline. It is the first place to look when a schedule "ran but nothing happened".

Seeing a failing schedule before the breaker acts

The circuit breaker disables a schedule once its most recent fires form an unbroken run of failures that reaches Z4J_SCHEDULE_CIRCUIT_BREAKER_THRESHOLD. The brain counts that run for every enabled schedule on each breaker tick, and the count is visible before it reaches the threshold.

The schedules list and the single-schedule read carry consecutive_failures: how many of the schedule's newest fires failed in a row, newest first, stopping at the first fire that is not a failure. 0 means the newest fire is not a failure, which includes a fire that is still pending, buffered or in flight. Responses to mutations (create, update, enable, pause, trigger) do not recount and carry null. Only failed and acked_failed count, because those are what the breaker counts. The schedules list also carries circuit_breaker_threshold so a client can show the run as a proportion. The dashboard renders it as a Health badge on the schedules table, for example 3 of 5 failing, and stays blank while a schedule is healthy. A threshold of 0 means the breaker is switched off; the count is still reported, over the newest twenty fires, and the badge says the breaker is off.

The picture behind the count is the run strip: the last twenty fires (fifty on the schedule detail page) as a row of cells, oldest on the left, coloured by outcome. Green is a success, red is any outcome that ended badly (including fires an operator marked failed during recovery and fires that timed out), grey is a fire still in flight or buffered, and a hollow cell is a fire that was skipped or cancelled. The strip therefore answers "what happened", while the Health badge beside it follows the breaker's narrower count. It appears on each row of the schedules table, above the fire-history table on the schedule detail page, and on the project overview in a "Schedules needing attention" panel that lists schedules with a non-zero run, worst first, and renders nothing when every schedule is healthy.

The strips read one endpoint:

GET /api/v1/projects/{slug}/schedules/runs?id=<uuid>&id=<uuid>&limit=20

id repeats, at most 500 per request (the dashboard sends at most a hundred per request), and limit is 1 to 50. The response is one row per requested schedule with its newest limit fires reduced to fire_id, status, scheduled_for, fired_at and latency_ms, plus the circuit_breaker_threshold. Ids that belong to another project are dropped from the response rather than refused, so a stale id on a cached page does not turn a refresh into an error. The full fire record, including error details, stays on the per-schedule /fires route.

The query behind both the breaker and the strips is a per-schedule top-N read (a LATERAL join on Postgres), bounded by the retention cutoff on fired_at. That is the column the prune worker deletes by, so the bound excludes nothing that still exists: a fire caught up for an old slot, or replayed from the buffer, has a recent fired_at and stays visible, and a schedule that fires monthly keeps its full history.

Manual fires and attribution

The Run now action (dashboard, or POST .../schedules/{schedule_id}/trigger, operator role) fires the underlying task once without touching the schedule's cadence. When the trigger routes through z4j-scheduler, the resulting fire row carries triggered_by_user_id, the operator who pressed the button. Scheduler-driven cadence fires carry null.

Attribution is validated at fire time: if the named user is not a member of the project when the fire lands, the fire is recorded unattributed rather than rejected. Deleting a user later nulls the reference but preserves the fire row, so history survives offboarding.

Retention

Fire history is bounded by Z4J_SCHEDULE_FIRES_RETENTION_DAYS (default 30). The mechanics differ by database.

Postgres: partition drops

On Postgres the schedule_fires table is RANGE-partitioned by scheduled_for into daily partitions. An hourly, leader-elected partition manager:

• Pre-creates a daily partition for today plus the next 14 days, so an insert never fails for lack of a partition and a worker outage shorter than the lookahead cannot exhaust the window.
• Drops whole daily partitions older than the retention window. A partition drop is a fast, lock-cheap DDL, so retention never turns into a long row-by-row DELETE on a hot table. Before dropping, the worker probes the partition's actual MAX(scheduled_for) and refuses the drop if live rows would be destroyed (name-vs-content drift or clock skew fails safe).

Rows that fall outside every daily partition land in the schedule_fires_default DEFAULT partition (data migrated from a pre-partitioned table, or fires with far-off scheduled_for values). A separate hourly sweep DELETEs expired rows from the DEFAULT partition only.

A non-empty DEFAULT partition blocks partition management

Postgres refuses to create a daily partition whose date range overlaps rows sitting in the DEFAULT partition. When that happens the brain logs a warning and bumps z4j_schedule_fires_partition_failures_total{reason="default_blocked"}. The DELETE sweep clears the DEFAULT partition over time; a sustained non-zero default_blocked rate is worth an alert, because those days cannot be reclaimed by partition drop until DEFAULT is empty.

Per-partition DDL runs in its own transaction, so one contended drop or one blocked create is logged and skipped without poisoning the rest of the tick.

SQLite: DELETE-based prune

On SQLite there is no partitioning: schedule_fires is a plain table and the hourly prune worker DELETEs rows older than the retention window. Fine at homelab fire rates; operators with high-frequency schedules should be on Postgres anyway.

Settings

Variable	Default	Description
Z4J_SCHEDULE_FIRES_RETENTION_DAYS	30	Days fire rows live. Postgres reclaims by daily partition drop; SQLite by DELETE. Range 1..3650.
Z4J_SCHEDULE_CIRCUIT_BREAKER_THRESHOLD	5	Consecutive failed fires before the schedule is auto-disabled. 0 disables the breaker.

See Environment variables for the full scheduler knob list.

See also

• z4j-scheduler for the scheduler itself and its misfire detection.
• Schedules API for the surrounding CRUD + trigger endpoints.
• Automation rules for reacting to schedule.misfired events.