Troubleshooting

Run the doctor first

Before anything else, run the framework-side doctor as the same user the service runs under. It probes everything below in order and reports the first specific failure.

Stack	Command
Django	python manage.py z4j_doctor
Flask	python -m z4j_flask doctor
FastAPI	python -m z4j_fastapi doctor
Bare Python	python -m z4j_bare doctor

Add --no-websocket to skip the WS round-trip when the brain is intentionally offline. Add --json for scripts. Exits 0 on all-green, 1 on any failure.

Agent shows offline

1. Run the doctor - it surfaces the most common failures with a specific reason. The list below covers the rest.
2. Check the agent’s app logs for [z4j] lines at boot. Look for handshake errors.
3. Verify Z4J_BRAIN_URL uses wss:// in production, ws:// in local dev.
4. Verify the token is the one shown at mint time (tokens are not recoverable - re-mint if lost).
5. Egress firewall: the agent’s host must reach the brain on TCP 443 (or wherever your proxy is).
6. Proxy WebSocket passthrough: nginx-ingress needs proxy_set_header Upgrade and proxy_set_header Connection.

Agent silently fails to start under gunicorn / uvicorn

PermissionError: ... /var/www/.z4j (or any path that’s not the running user’s writable home) in the service log. The service user has an unwritable $HOME. Upgrade to z4j-bare 1.0.6+ which auto-relocates the buffer to $TMPDIR/z4j-{uid}/buffer-{pid}.sqlite and logs a single WARNING instead of crashing. See service-user deployments.

Events don’t appear

1. Agent is online (Agents page shows online)?
2. Engine is auto-detected (agent drawer → engines list)?
3. For Django: INSTALLED_APPS includes z4j_django after any Celery apps?
4. For Flask: z4j.init_app(app, ...) was called on the app factory?
5. For FastAPI: agent is inside the lifespan context manager?
6. Task names in registry (agent drawer → registry)?

”Agent version mismatch” banner

You’ve upgraded the brain to a newer major than the agent. Re-deploy agents with pip install -U z4j-*. Agents up to one major version behind still work but may lack new features.

Audit chain verify fails

Someone modified the audit_log table directly, or a backup restore is incomplete.

1. Identify first_broken_id.
2. If known-intentional (e.g., planned DB surgery), document the break externally. The chain from there is not recoverable.
3. If unexpected, treat as a compromise event - preserve the DB, alert security, investigate.

409 conflict_duplicate_name on mint-token

Another agent already has that (project_id, name). Pick a different name or delete the old agent first.

Schedules showing read_only

The scheduler backend doesn’t support writes (e.g. celery-beat with PersistentScheduler, rq-scheduler in v1.0). See schedulers overview.

Password reset email not arriving

1. SMTP configured? z4j-brain smtp-test --to [email protected].
2. From: domain has SPF / DKIM / DMARC?
3. Sender reputation good? Gmail drops many SMTP senders silently.

Very high CPU

1. Check rate(z4j_events_persisted_total[1m]) - is one agent emitting millions of events?
2. Redaction patterns not looping on giant payloads? - the redactor has a 2 MiB payload cap.
3. Hot endpoint - check z4j_http_request_duration_seconds.

503 agent_offline on a retry

The target agent dropped between UI showing the action and the brain dispatching. Wait for reconnect or pick a different agent handling the same engine.

When stuck

Include the brain logs (with X-Request-Id), agent logs, and a description of the sequence - file at github.com/z4jdev/z4j/issues.