Skip to content

Bulk retry

Retrying one failed task is simple. Retrying nine thousand of them is a different problem, because the set you are retrying keeps moving while you retry it, and because a retry that runs twice is a retry that ran once too often.

A bulk retry in z4j is therefore a durable request you submit and then watch, not a button that fires and forgets.

Submitting a bulk retry commits four things in a single transaction: the request identity, its audit row, a reservation on your idempotency key, and the complete per-task plan.

The plan matters most. The work that will run is decided up front, at submit time, rather than re-derived later from the filter. That is what makes a repeat safe: submitting the same request again returns the original outcome instead of re-expanding the filter against a fleet that has moved on.

Without that, a request which matched nothing, repeated after a task had failed into range, would run a retry that the first response had already reported as “nothing matched”. You would have been told no work happened, and work would have happened.

Each task in the plan is claimed exactly once, at the moment it is sent, under a parent-wide in-flight bound. Two consequences follow:

  • A large retry paces itself instead of stampeding every worker at once.
  • An ambiguous claim is never returned to the pending set. If z4j cannot tell whether a task was sent, it will not send it again on a guess.

That second point is the whole design. The alternative, retrying anything uncertain, converts an unclear situation into a duplicate execution, and for a destructive or non-idempotent task that is worse than doing nothing.

The state of a request is derived from its per-task counters. It is never stored separately, so it cannot drift from the tasks it describes.

State Meaning
no_match The filter matched nothing. No work was planned and none will run.
in_progress Tasks remain pending or unobserved, and the request is running.
paused Work remains, and an operator paused it. Resume when ready.
blocked Work remains, but the request’s attempt budget expired. Resume explicitly.
succeeded Every task in the plan succeeded.
failed Every task in the plan failed.
partial A mix of outcomes, or some tasks resolved and others stayed unknown.
indeterminate Nothing resolved either way. Every outcome is unknown.

blocked is not an error. It means the request stopped on its own budget rather than running indefinitely, and it needs a person to decide whether the remaining work should still happen. Resuming is deliberate for that reason.

A task can finish in an unknown state, meaning z4j sent it and did not receive a trustworthy answer. Unknown is reported rather than guessed.

Late results can refine it. An authenticated result arriving after the fact moves a task from unknown to succeeded or failed, so a request that looked indeterminate can settle into partial or succeeded without you resubmitting anything.

The distinction worth holding onto: failed means z4j knows the retry did not work. unknown means z4j does not know, and is saying so instead of picking the more comfortable answer.

Pause a running request to stop it claiming further tasks. Work already in flight is not recalled, because it has already been handed to a worker.

Resume returns a paused or blocked request to the pending set. Tasks that already resolved are not re-run: the plan is fixed, and each entry is claimed at most once for the life of the request.

The dashboard shows request state, per-task progress, and the outcome breakdown. Every submission and every refusal writes to the audit chain, including a request that was refused before any work was planned, so a bulk retry that did nothing still leaves a record that it was asked for.