Schedule fire history

Every fire dispatched through z4j-scheduler writes one row of history. The schedule row itself carries last_run_at and the run counters (“what is going on right now”); the fire history carries the per-fire detail operators need to answer “did the 3am cron actually fire, and what happened to it?”

Fire history exists for schedules fired by z4j-scheduler (it requires the scheduler integration, Z4J_SCHEDULER_GRPC_ENABLED=true). Schedules surfaced by the per-engine adapters (celery-beat, rq-scheduler, …) fire inside their native scheduler and do not produce fire rows.

Lifecycle of a fire

A fire row progresses through these states:

Status	Meaning
pending	Row created; the fire arrived from the scheduler but the command has not been issued yet.
delivered	The brain dispatched the schedule.fire command to an online agent.
buffered	No matching agent was online; the fire landed in the pending_fires buffer for replay once an agent for the right engine reconnects.
acked_success	The agent ran the task and reported success.
acked_failed	The agent reported a failure result. error_code / error_message carry the detail.
failed	Brain-side or transport error; the fire never reached an agent.

One row per fire: the scheduler’s idempotency key (fire_id, derived from the schedule and its tick boundary) is unique, so HA retries and network duplicates collapse into a single row instead of double-counting.

When the ack arrives, the row also records acked_at and latency_ms (ack minus dispatch), so per-schedule delivery latency is readable without a join.

Consecutive failed / acked_failed fires feed the schedule circuit breaker: after Z4J_SCHEDULE_CIRCUIT_BREAKER_THRESHOLD consecutive failures the schedule is auto-disabled and a schedule.circuit_breaker.tripped audit row is written.

Reading the history

GET /api/v1/projects/{slug}/schedules/{schedule_id}/fires

Role: viewer. Returns the most recent fires, newest first. limit defaults to 100, capped at 1000.

[
  {
    "id": "...",
    "fire_id": "...",
    "schedule_id": "...",
    "command_id": "...",
    "status": "acked_success",
    "scheduled_for": "2026-03-14T03:00:00Z",
    "fired_at": "2026-03-14T03:00:00.412Z",
    "acked_at": "2026-03-14T03:00:01.006Z",
    "latency_ms": 594,
    "error_code": null,
    "error_message": null,
    "triggered_by_user_id": null
  }
]

scheduled_for is the tick boundary the scheduler computed (when the fire was supposed to happen); fired_at is when the brain recorded the fire. A growing gap between the two is a scheduler falling behind.

The dashboard renders the same data as the Last 50 fires panel on each schedule’s detail page: status pill, scheduled-for, fired-at, latency, and the error detail inline. It is the first place to look when a schedule “ran but nothing happened”.

Manual fires and attribution

The Run now action (dashboard, or POST .../schedules/{schedule_id}/trigger, operator role) fires the underlying task once without touching the schedule’s cadence. When the trigger routes through z4j-scheduler, the resulting fire row carries triggered_by_user_id, the operator who pressed the button. Scheduler-driven cadence fires carry null.

Attribution is validated at fire time: if the named user is not a member of the project when the fire lands, the fire is recorded unattributed rather than rejected. Deleting a user later nulls the reference but preserves the fire row, so history survives offboarding.

Retention

Fire history is bounded by Z4J_SCHEDULE_FIRES_RETENTION_DAYS (default 30). The mechanics differ by database.

Postgres: partition drops

On Postgres the schedule_fires table is RANGE-partitioned by scheduled_for into daily partitions. An hourly, leader-elected partition manager:

• Pre-creates a daily partition for today plus the next 14 days, so an insert never fails for lack of a partition and a worker outage shorter than the lookahead cannot exhaust the window.
• Drops whole daily partitions older than the retention window. A partition drop is a fast, lock-cheap DDL, so retention never turns into a long row-by-row DELETE on a hot table. Before dropping, the worker probes the partition’s actual MAX(scheduled_for) and refuses the drop if live rows would be destroyed (name-vs-content drift or clock skew fails safe).

Rows that fall outside every daily partition land in the schedule_fires_default DEFAULT partition (data migrated from a pre-partitioned table, or fires with far-off scheduled_for values). A separate hourly sweep DELETEs expired rows from the DEFAULT partition only.

A non-empty DEFAULT partition blocks partition management

Postgres refuses to create a daily partition whose date range overlaps rows sitting in the DEFAULT partition. When that happens the brain logs a warning and bumps z4j_schedule_fires_partition_failures_total{reason="default_blocked"}. The DELETE sweep clears the DEFAULT partition over time; a sustained non-zero default_blocked rate is worth an alert, because those days cannot be reclaimed by partition drop until DEFAULT is empty.

Per-partition DDL runs in its own transaction, so one contended drop or one blocked create is logged and skipped without poisoning the rest of the tick.

SQLite: DELETE-based prune

On SQLite there is no partitioning: schedule_fires is a plain table and the hourly prune worker DELETEs rows older than the retention window. Fine at homelab fire rates; operators with high-frequency schedules should be on Postgres anyway.

Settings

Variable	Default	Description
Z4J_SCHEDULE_FIRES_RETENTION_DAYS	30	Days fire rows live. Postgres reclaims by daily partition drop; SQLite by DELETE. Range 1..3650.
Z4J_SCHEDULE_CIRCUIT_BREAKER_THRESHOLD	5	Consecutive failed fires before the schedule is auto-disabled. 0 disables the breaker.

See Environment variables for the full scheduler knob list.

See also

• z4j-scheduler for the scheduler itself and its misfire detection.
• Schedules API for the surrounding CRUD + trigger endpoints.
• Automation rules for reacting to schedule.misfired events.