Upgrades

Semver promise

• Major -- breaking wire-protocol changes. Plan a coordinated upgrade.
• Minor -- additive features, fully backward-compatible within the major.
• Patch -- bug fixes, no behavior changes.

See versioning for the full policy.

Upgrade order

1. Upgrade z4j (the brain) first. Newer brains accept older agents within the same major.
2. Upgrade agents on a schedule. Rolling restarts are fine; agents auto-reconnect.
3. Watch the dashboard for the per-agent "outdated" badge -- the brain renders it when an agent's last advertised protocol_version is older than the brain's current protocol.

Database migrations

Alembic migrations run automatically on brain start when Z4J_AUTO_MIGRATE=true (the default).

On a multi-replica deployment there is one supported order, and it is not a rolling one. Stop every replica of the old release, run the migration as a one-shot job, then start replicas of the new release. Stop, not drain: draining removes a replica from HTTP load balancing while its background schedule writers keep running, and it is those writers, not request traffic, that produce the incompatible envelopes. Leaving old replicas running across a migration that adds a column the schedule-control path reads costs executions, not just latency; the reasons are below under Brain replicas have to cross a schema change together.

Set Z4J_AUTO_MIGRATE=false on the runtime replicas so only the one-shot job migrates:

The image's entrypoint is already the z4j binary, so whatever you pass after the image tag starts at the SUBCOMMAND. Repeating the binary name there makes the container invoke it twice and fail; begin with migrate. The job also needs the same /data volume and Z4J_SECRET the running brain uses, or management bootstrap refuses to start.

Run it through Compose, not a bare docker run. The shipped files put the brain's state on a named volume (z4j_data for SQLite, z4j_brain_state for PostgreSQL) and the database on a private network (z4j_net) that is not published, and the brain needs Z4J_SESSION_SECRET as well as Z4J_SECRET. compose run inherits the volume, the network and the whole environment; a bare docker run gets a fresh empty volume and cannot resolve the database host. Add -f docker-compose.postgres.yml if that is the file you deploy.

# Point at the NEW image and pull it FIRST. `compose run` uses whatever image
# the file resolves to, which is still the version you are upgrading FROM
# until you change it -- so without this the one-shot job runs the OLD brain,
# reports "upgrade head" success at the old head, and applies nothing. The
# migration then happens during new-replica startup, or the new brain refuses
# because you just turned auto-migration off.
#
# The compose file and the image variable are a PAIR: each file reads only its
# own variable. Setting one while running the other is a silent no-op, because
# compose keeps the tag already in the file and the migration then reports
# success at the old head. Run the block for the stack you deploy, whole.

# --- SQLite stack: docker-compose.yml, which reads Z4J_IMAGE ---
echo 'Z4J_IMAGE=z4jdev/z4j:<new-version>' >> .env
docker compose pull z4j
docker compose run --rm -e Z4J_AUTO_MIGRATE=false z4j migrate upgrade head
docker compose run --rm z4j migrate current   # confirm it moved

# --- PostgreSQL stack: docker-compose.postgres.yml, which reads Z4J_BRAIN_IMAGE ---
echo 'Z4J_BRAIN_IMAGE=z4jdev/z4j:<new-version>' >> .env
docker compose -f docker-compose.postgres.yml pull z4j
docker compose -f docker-compose.postgres.yml run --rm \
  -e Z4J_AUTO_MIGRATE=false z4j migrate upgrade head
docker compose -f docker-compose.postgres.yml run --rm z4j migrate current

grep image: <your-compose-file> confirms which variable your file reads if you have customised it.

Then start the brain replicas on the new image.

Brain replicas have to cross a schema change together

A multi-replica brain runs mixed versions for as long as the rollout takes, and for most migrations that is fine. It is not fine for a migration that adds a column the schedule-control path reads, because the two versions then disagree about what a schedule row means:

• An older replica does not know the column exists, so it writes change-log envelopes without it. A newer brain refuses to project those envelopes and ends the scheduler's watch stream rather than guess. The scheduler recovers on its own, by reconnecting and re-syncing from live rows, but it refuses to fire while it does, so the window shows up as schedules running late.
• Worse in the other direction: an older replica cannot honour a hold it has no column for. A schedule paused through a newer replica keeps firing through an older one.

This is not merely slower scheduling. Each refused envelope ends the watch, the scheduler will not dispatch while it is reconnecting, a fire more than the grace window late is classified as missed, and the default catch-up policy is skip, which records the slot and moves past it. A schedule that came due inside one of those windows does not run late. It does not run.

So this is a requirement, not a recommendation:

1. Stop every brain replica of the older release. Not "most", not "roll them quickly": a single old writer is enough to produce the envelopes.
2. Run the migration.
3. Start replicas of the new release only.

Schedulers can stay up throughout. They fail closed, reconnect and re-sync on their own once every brain is on one side of the change.

Scaling to a single replica first does NOT avoid the stop. It narrows the window to one process, which is worth doing, but that replica is still an old writer until it exits, so its own restart has to be a full stop and start rather than a rolling replace. In Kubernetes terms: strategy: Recreate, or maxSurge: 0 with maxUnavailable: 100%. There is no configuration that keeps a brain of the previous release serving while the migration runs.

This upgrade takes brief downtime on a multi-replica deployment. Plan for it rather than looking for a way around it.

The release notes say when a given upgrade adds such a column.

None of this applies when only ONE brain process is alive across the migration. That is a statement about overlapping processes, not about replicas: 1: Kubernetes defaults to RollingUpdate and rounds maxSurge UP, so a single-replica Deployment starts the new pod before stopping the old one and produces the window anyway. Set strategy: { type: Recreate }. The same caution applies to any supervisor that starts a replacement before the previous process exits.

Rolling back

An installation that upgraded and then merely ran can be brought back down. Every schedule owned by the z4j scheduler carries a fingerprint of the cadence runtime that wrote it, and nothing on the running path rewrites it: not the fire path, not the cursor path. The only write sites are creating a schedule and cutting one over to a new owner. So on an installation that upgraded and then ran, the rows still carry the identity the older release wrote, and handing them back restores exactly the state that release last saw. Nothing is rewritten, so no image, registry or signature is involved.

The downgrade makes you declare what the release you are returning to computes. Measure it in the target environment, under that release:

python -c 'from z4j_brain.domain.schedule_cadence import cadence_runtime_fingerprint as f; print(f())'

Then stop everything that writes, run z4j migrate downgrade with Z4J_ROLLBACK_TARGET_FINGERPRINT set to that digest, and install the older release afterwards. The schema has to come down first, because the older binary refuses to start against the newer one. Upgrade and rollback carries the sequence step by step.

It refuses in three cases, and every guard is evaluated over the whole downgrade plan before its first step runs, so nothing is dropped and the database is left as it was. It refuses when a schedule was created or re-saved since the upgrade, because those rows carry an identity the target did not write. It refuses when a schedule is paused, and resuming paused schedules does not clear that, it just starts them firing again, which is what the guard is there to prevent, so do not release incident holds trying to get past it. And it refuses when an agent is revoked.

Declaring a fingerprint proves less than it looks like. The value has to equal what the rows already carry, so it cannot make a bad downgrade succeed, but nothing ties it to a running target: paste it out of your own database and the check passes without telling you anything. If the release you install computes something different from what the rows carry, it disables every schedule it cannot agree with, durably, until an operator re-enables each one by hand. Measure the target.

When it refuses, restore your pre-upgrade backup under the release you are returning to. z4j restore is forward-only, so it runs under the older binary against a target that release provisioned and brought to its own head. See upgrade and rollback for the full sequence and backup and restore for what the target must satisfy.

Everything written since the upgrade is lost on that path. Take the pre-upgrade backup every time, because whether the downgrade is available to you depends on what happens after the upgrade, not on anything you can check before it.

Below the boundary activations there is no downgrade at any point in the release history either; those revisions refuse unconditionally. See database migrations for which revisions refuse and why.

Version skew tolerance

Brain	Agent	Behavior
same major, same minor	same major, same minor	Full feature parity.
same major, newer	same major, one minor older	Works. New brain-side features may be unavailable on older agents; the dashboard shows the "outdated" badge.
same major, newer	same major, more than one minor older	Works but flagged loudly. Plan the agent rollout.
any	major mismatch	Rejected at the handshake with WebSocket close code 4426 (Upgrade Required).

Rolling-upgrade caveats

The primary WebSocket transport is protocol-compatible across a same-major minor gap, so the steady-state event, heartbeat, and command data plane keeps flowing in both directions while a fleet is mid-upgrade. Two paths need operator awareness during the mixed-version window:

• Long-poll fallback (corporate-proxy path only). The long-poll HMAC binds the agent and project identity the brain advertises in its response headers, so long-poll delivery only works when the brain and agent are on the same release. If you rely on the HTTPS long-poll fallback rather than WebSocket, upgrade the brain and agents together, or set Z4J_AGENT_ID and Z4J_PROJECT_ID to the real UUIDs shown on the brain's agents page (the agent logs this instruction). WebSocket deployments are unaffected.
• Queue purges. The purge_queue confirm-token is a keyed HMAC. A purge issued across a version boundary is refused (it fails safe -- it never purges the wrong queue). Do not issue queue purges until the whole fleet is on the new release. If you must purge against a not-yet-upgraded brain, set Z4J_ACCEPT_LEGACY_PURGE_TOKEN=1 on the agents for the grace window only.

Reading release notes

Every release ships a CHANGELOG entry at /reference/changelog. Breaking changes are flagged explicitly.

Upgrading Postgres

z4j supports Postgres 17 and 18. Use pg_upgrade or logical replication; z4j is agnostic to which migration path you choose.