Automation rules

An automation rule watches for an event on a project and, when the event matches, runs an action on your behalf. Rules turn a pattern you would otherwise watch for by hand (“this task keeps failing”, “this schedule stopped firing”) into a governed, fully audited reaction.

Automation is off until you create a rule, and every rule is scoped to a single project.

In the dashboard

Each project has an Automation area in its sidebar. It lists the project’s rules with their trigger, action, on-or-off state, and recent firing activity, and it holds the per-project kill switch that pauses every rule at once. Create or edit a rule with the built-in builder: pick a trigger, add optional conditions, choose one or more actions, and toggle dry-run to validate against live traffic before you arm it. Destructive actions in the builder require an admin who has passed a fresh second-factor check.

Anatomy of a rule

• Trigger: the event class the rule reacts to: task.failed, task.succeeded, task.retried, task.orphaned, worker.offline, or schedule.misfired.
• Conditions: optional filters on the event (engine, queue, task name, exception, priority, runtime, or a failure fingerprint). A rule with no conditions matches every event of its trigger.
• Actions: what to do when a match fires. Each rule can carry more than one action.
• Dry-run: a rule marked dry-run evaluates and records what it would have done, in the audit log, without taking the action. Use it to validate a rule against live traffic before arming it.

Actions

• notify: raise an in-app notification for every active member of the project. This is the safe default and is never blocked by the circuit breaker’s failsafe (see below).
• retry: re-enqueue the failed task through the reporting agent, using only the task name the brain already knows (never a broker re-read and never redacted arguments), so the pickle-safe retry path is preserved.
• cancel: revoke the task through the reporting agent.

retry and cancel are destructive actions. Other action types (webhook and the queue-purge / schedule-pause families) are recognised by the grammar but not yet executed; a rule that requests one is rejected when you create it rather than silently accepted.

Governance

Automation acts on your infrastructure, so it is gated at several layers.

• RBAC: creating or re-arming a rule that carries a destructive action requires the project ADMIN role and a fresh MFA check. A notify-only rule requires OPERATOR.
• Fire-time re-check: a destructive action re-verifies at fire time that the rule’s creator is still an admin of the project. If the creator was removed or demoted, the action is skipped and audited rather than issued.
• Deprovisioning: revoking a member disables every rule that member created, so a departed operator’s automation stops firing entirely.
• Kill switch: each project has a single automation on/off switch. With it off, no rule on the project loads, so nothing fires. It is the one control that stops all automation instantly.

Circuit breaker

Every rule carries a rolling-window circuit breaker: at most a configured number of firings per window. A rule that exceeds its budget (a runaway loop, or an event storm) trips into failsafe mode, where destructive actions are suppressed but notifications still go out, so you keep getting alerted while the blast radius is contained. The breaker resets when the window rolls.

Each rule also runs in its own committed transaction, so a database error on one rule cannot roll back or block the others matching the same event.

Reacting to a dead scheduler

The schedule.misfired trigger is raised by the brain’s misfire detector: a periodic sweep that flags any enabled interval or cron schedule whose expected fire is past a grace window, which usually means its scheduler is down or partitioned. A rule on schedule.misfired lets you page an operator or take action the moment a schedule goes quiet, rather than discovering it later. Recent misfires for a schedule are also readable through the schedule’s misfires endpoint, and each detection is exported as a Prometheus counter you can alert on. See misfire detection for the grace and sweep knobs.

Reacting to a dead agent

The worker.offline trigger is raised by the brain’s agent-health worker when an agent’s heartbeat has been silent past the offline timeout plus an alert grace window. The grace absorbs deploy restarts and reconnect blips, and each outage alerts exactly once, however many sweeps it spans; an agent that recovers and later dies again is a fresh episode. Detection also writes an audit row, fans out to agent.offline notification subscriptions, and bumps a Prometheus counter. See Agent offline alerts for the timeout and grace knobs and the exactly-once mechanics.

Reacting to an orphaned task

The task.orphaned trigger is raised by the reconciliation loop when it confirms a stuck task really did start and never report back: the periodic sweep probes tasks stuck in a non-terminal state, and when the engine’s result backend answers with a terminal state the brain corrects its snapshot and fires the trigger, exactly once per correction. Use it to be told when your event pipeline lost a terminal event, or to chain a retry on tasks that turned out to have failed silently.

Audit

Every firing writes an HMAC-chained audit row: which rule, which trigger, which action, the outcome, and whether it was a dry run. A tripped breaker, a skipped destructive action, and an auto-disabled rule all leave their own audit trail, so the full history of what automation did (and chose not to do) is tamper-evident. See Audit log.