Reconciliation

The stuck-task problem

If a worker crashes mid-task, the engine may emit task_started but never task_success or task_failure. Without reconciliation, the dashboard shows "running" forever.

The reconciliation worker

Runs in z4j every Z4J_RECONCILIATION_SWEEP_SECONDS (default 5 minutes). For each agent:

1. Find all tasks in state started older than Z4J_RECONCILIATION_STALE_THRESHOLD_SECONDS (default 15 minutes).
2. Issue a reconcile_task command to the agent that owns the task. It is an ordinary command row, audited under the action reconcile_task, carrying an idempotency key so two sweeps inside one window collapse onto a single probe.
3. The agent asks its engine: Celery AsyncResult.status, RQ job.get_status(), and so on, and answers with one canonical state: pending, started, success, failure, or unknown.
4. unknown is a no-op. It means the adapter has no result backend to consult, which is not evidence that the task is gone, so the brain changes nothing and the task is probed again on a later sweep.
5. A terminal answer (success or failure) is applied, and the brain records the outcome the missing event would have carried. Applying it fires the task.orphaned automation trigger once for that correction.
6. A non-terminal answer is applied only when the row has not been written since the probe was issued. If it has, the brain has fresher information than the probe saw and the response is dropped. A task already in a terminal state is never moved back out of it.

Why not rely on the engine alone

Each engine reports differently:

• Celery - AsyncResult is Redis-TTL-bound; after expiry it returns PENDING regardless.
• RQ - cleanly reports lost, but requires periodic cleanup for stale jobs.
• arq - no introspection API for historical jobs; relies on application-level result store.

The reconciliation worker owns the "was it really lost?" question so no adapter has to.

Tunables

Setting	Default	Meaning
Z4J_RECONCILIATION_SWEEP_SECONDS	300	Seconds between reconciliation passes
Z4J_RECONCILIATION_STALE_THRESHOLD_SECONDS	900	How old a "started" task must be before the worker queries the agent

Audit impact

Each probe writes an audit log entry under the action reconcile_task, naming the task it asked about. The correction itself is visible as the task's state change and, where an automation rule is armed, as a task.orphaned firing.

Limits

• Reconciliation does not retry anything. It corrects the brain's snapshot to whatever the engine says is true. Retrying is a separate operator decision, or an automation rule armed on task.orphaned.
• If the agent is offline, reconciliation skips that agent and retries next pass.
• No retroactive reconciliation across deploys - only tasks still present in tasks + events are examined.