Upgrade and rollback

z4j ships patch releases on roughly weekly cadence (faster when fixing security or operator UX). Patch upgrades are designed to be in-place and reversible.

Pre-flight

Before any upgrade:

z4j doctor      # config valid, DB at head, no warnings you don't expect
z4j status      # snapshot row counts so you can verify after restart
z4j backup --output /var/backups/z4j-pre-upgrade-$(date +%Y-%m-%d-%H%M).db

The backup is the most important step. Without it, an upgrade that introduces an unexpected migration is unrecoverable.

Pip upgrade

# 1. Stop the brain
sudo systemctl stop z4j

# 2. Upgrade the wheel (no-cache so we always get the published version)
sudo -u z4j /srv/venv/bin/pip install --no-cache-dir --upgrade z4j

# 3. Confirm version + that migrations are still in place
sudo -u z4j /srv/venv/bin/z4j version
sudo -u z4j /srv/venv/bin/z4j migrate current     # pre-restart sanity

# 4. Restart - alembic upgrades to head automatically on serve
sudo systemctl start z4j

# 5. Verify
journalctl -u z4j -n 30 --no-pager     # look for the boot banner
sudo -u z4j /srv/venv/bin/z4j check
sudo -u z4j /srv/venv/bin/z4j status   # row counts should match step-1 snapshot

z4j auto-runs alembic upgrade head on every serve start unless you set Z4J_AUTO_MIGRATE=false. Patch versions inside the same minor line are migration-compatible. From 1.4.0 onwards, schema is bidirectional inside 1.4.x - every migration ships a working downgrade() and a CI round-trip test enforces it. Data is preserved via backup-restore, not via downgrade. See database migrations for the full policy.

Troubleshooting: `Unable to locate executable '/srv/venv/bin/z4j'`

If systemd reports this after an upgrade:

z4j.service: Unable to locate executable '/srv/venv/bin/z4j': No such file or directory
z4j.service: Failed at step EXEC spawning /srv/venv/bin/z4j: No such file or directory
z4j.service: Main process exited, code=exited, status=203/EXEC

…the z4j console script is missing from the venv even though pip list shows z4j is installed at the right version. The cause is a venv where the z4j dist-info metadata exists but the wheel content was never actually unpacked, so pip install --upgrade z4j short-circuits with “Requirement already satisfied” and never drops the binary into bin/. Common triggers: the venv was built across a package-rename cut, manual cleanup deleted files but left dist-info, or a previous install was interrupted.

The fix is a force-reinstall that ignores the metadata check:

sudo -u z4j /srv/venv/bin/pip install --force-reinstall --no-deps z4j
ls -la /srv/venv/bin/z4j   # should now exist, mode 0755
sudo systemctl restart z4j

--no-deps keeps it fast since dependencies aren’t the issue; only z4j’s own wheel needs to be replanted. If z4j status, z4j check, or z4j doctor themselves work but only the systemd boot fails, this is the diagnosis.

Docker upgrade

# 1. Pull the new tag (specific version recommended over :latest in production)
docker compose pull z4j

# 2. Restart - migrations run on container start
docker compose up -d z4j

# 3. Verify
docker compose logs --tail=50 z4j
docker compose exec z4j z4j check
docker compose exec z4j z4j status

For Postgres deployments, z4j image and the Postgres image are independent - upgrading z4j does not touch Postgres data.

Pinning vs `:latest`

Tag	When to use
`z4jdev/z4j:1.4.0`	Production - reproducible deploys, controlled upgrade cadence
`z4jdev/z4j:latest`	Homelab / small teams - track current stable, don’t pin
`z4jdev/z4j:1.4`	Float on the latest `1.4.x` patch but stay off `1.5.x`

The 1.4 minor line guarantees bidirectional schema migrations: pip install <newer> and pip install <older> both reach a clean schema state inside 1.4.x. Row data is preserved via backup-restore, not via downgrade. Across major versions (1.x → 2.x), expect a documented manual step.

Rollback - pip

If z4j check or z4j status shows something wrong after upgrade:

# 1. Stop the brain
sudo systemctl stop z4j

# 2. Reinstall the previous version (replace 1.4.0 with whatever you had before)
sudo -u z4j /srv/venv/bin/pip install --no-cache-dir --force-reinstall z4j==1.4.0

# 3. If migrations changed schema, restore from the pre-upgrade backup
sudo -u z4j /srv/venv/bin/z4j restore /var/backups/z4j-pre-upgrade-2026-05-03-0300.db --force

# 4. Restart
sudo systemctl start z4j

# 5. Verify
sudo -u z4j /srv/venv/bin/z4j check
sudo -u z4j /srv/venv/bin/z4j status

If you skipped the backup and the migration is irreversible, you have to either:

Roll forward (fix the bug in the new version, ship a patch)
Restore from the most recent off-host backup (snapshot from before the upgrade)

There’s no “undo a migration” magic - alembic migrations are forward-only by design unless explicitly written with a downgrade path.

Rollback - Docker

# 1. Pin to the previous tag in compose.yml
sed -i 's|z4jdev/z4j:1.4.1|z4jdev/z4j:1.4.0|' docker-compose.yml

# 2. Re-create the container with the older image
docker compose up -d --force-recreate z4j

# 3. If schema changed, restore
docker compose exec z4j z4j restore /backups/z4j-pre-upgrade-2026-05-03.dump --force

# 4. Verify
docker compose logs --tail=50 z4j

For Docker Postgres deployments, restore to the live Postgres instance via pg_restore running inside z4j container - the connection string is already set.

Major-version upgrades

When 1.x → 2.0 ships, the release notes will document the manual steps. Plan for:

A required Z4J_* env var change (something deprecated in 1.x is removed)
A breaking wire-protocol bump (agents need matching minor version)
A schema change that’s not backward-compatible with 1.x readers

We won’t ship 2.0 without a z4j upgrade-to-2 migration helper. For now, this section is a placeholder.

Upgrading agents

Agent packages (z4j-django, z4j-celery, etc.) version independently of z4j. The compatibility rule:

Any v1.4.x agent talks to any v1.4.x brain.

Patch a brain or an agent on its own; you don’t need to coordinate the two.

To bulk-upgrade every agent in your app’s venv:

pip install --upgrade --no-cache-dir z4j-django z4j-celery z4j-celerybeat

Restart your web process + worker after the upgrade. The agents reconnect to z4j automatically; no token rotation needed.