Docs / Self-hosting
Run the backend on Proxmox
Run the backend, which tracks runs, claims, messages and the dashboard across your machines, in an unprivileged Debian container on Proxmox VE behind Tailscale Serve HTTPS.
This guide builds the backend as an unprivileged Debian LXC on Proxmox VE 9.x. The backend tracks runs, claims and messages across your machines and serves the dashboard, and the commands call it Fleet Hub, so this page says “the hub” from here on. The hub is a Brigade service listening on loopback only, and Tailscale Serve is the only way in. It keeps its live view in one SQLite database. Local journals stay authoritative, so an outage delays the aggregate view and makes clients spool events until it returns. For what the hub is and when to use it, read Run the backend first. The ExecStart lines here run the default engine, which has no shared memory. Shared memory needs extra flags: see Turn on shared memory.
Set up with your agent
Run tailscale serve status to print the hub’s HTTPS tailnet name. This guide calls it <HUB_TAILNET_FQDN>. Replace every other angle-bracket value before you run a command.
Target layout
- Proxmox VE 9.x host.
- A Debian 12 or Debian 13 CT template.
- 1 vCPU, 512 MB RAM, 4 GB root disk.
- An unprivileged CT with nesting disabled.
- Tailscale inside the CT, hostname
brigade-hub. - The hub bound to
127.0.0.1:3774, behind Tailscale Serve HTTPS. - The SQLite database at
/var/lib/brigade/fleet-hub.db. - A root-owned, mode
0600systemd environment file at/etc/brigade/fleet-hub.env.
The hub exposes these endpoints:
| Endpoint | Auth | Purpose |
|---|---|---|
GET /health | none | Liveness check |
POST /events | node token | Append an event. The admin token works only with --allow-admin-writes. |
GET /status | admin or node token | Latest non-terminal state per node and run. Add ?all=1 to include finished runs. |
POST /claims, GET /claims | node token to post, admin or node token to read | Repository claims |
GET /nodes, POST /nodes | admin token only | Node enrollment |
GET /, GET /view/machines, GET /view/repos | admin token or dashboard cookie | The Fleet dashboard |
GET /deck/roster, POST /deck/roster | admin Authorization header to save | The roster page |
The hub drops a repeated event, matched on (node_id, run_id, sequence, digest), so reposting after a lost response is safe.
Two bearer credentials exist. The admin token in /etc/brigade/fleet-hub.env is the control plane: it enrolls nodes, reads status and claims, and mints dashboard enrollment codes. Each machine’s node token, created by brigadestack fleet nodes add, is that machine’s identity. The hub derives node_id from it and refuses an event or claim for any other node with HTTP 403. The hub stores only a SHA-256 of each node token.
1. Create the CT on the Proxmox host
Download a Debian 12 or 13 standard CT template into the template storage first, and confirm the exact filename and storage names.
Run as root on the Proxmox host:
pct create <CTID> <TEMPLATE_STORAGE>:vztmpl/<DEBIAN_TEMPLATE>.tar.zst \
--hostname brigade-hub \
--ostype debian \
--cores 1 \
--memory 512 \
--swap 512 \
--rootfs <ROOT_STORAGE>:4 \
--unprivileged 1 \
--features nesting=0 \
--net0 name=eth0,bridge=<PROXMOX_BRIDGE>,ip=<CT_IPV4/CIDR>,gw=<GATEWAY_IPV4>,type=veth \
--onboot 1 \
--start 0 \
--ssh-public-keys <PUBLIC_KEY_FILE>
pct start <CTID>
pct enter <CTID>
The CT needs outbound access to the Tailscale package repository and the tailnet. Do not enable nesting for this service.
Tailscale normally needs /dev/net/tun. Check from inside the CT:
test -c /dev/net/tun && echo "TUN device present" || echo "TUN device missing"
If the device is missing, leave nesting disabled and add only the TUN device permission on the Proxmox host:
printf '%s\n' \
'lxc.cgroup2.devices.allow: c 10:200 rwm' \
'lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file' \
>> /etc/pve/lxc/<CTID>.conf
pct restart <CTID>
pct enter <CTID>
2. Install Tailscale in the CT
Run as root inside the CT. The repository path comes from Debian’s codename, so the same commands work on Debian 12 and 13.
apt-get update
apt-get install -y ca-certificates curl gnupg
. /etc/os-release
curl -fsSL "https://pkgs.tailscale.com/stable/debian/${VERSION_CODENAME}.noarmor.gpg" \
-o /usr/share/keyrings/tailscale-archive-keyring.gpg
curl -fsSL "https://pkgs.tailscale.com/stable/debian/${VERSION_CODENAME}.tailscale-keyring.list" \
-o /etc/apt/sources.list.d/tailscale.list
apt-get update
apt-get install -y tailscale
systemctl enable --now tailscaled
tailscale up --authkey=tskey-auth-<PREAUTH_KEY> --hostname=brigade-hub --accept-dns=true
tailscale status
tailscale ip -4
The pre-auth key must be reusable, or have enough uses for the rebuilds you expect. Type it in the CT’s own shell. Do not put it in a repository, a chat, or a systemd unit. If you would rather not use a key, leave out --authkey=...: tailscale up then prints a login URL to approve in a browser.
The IPv4 address only shows that Tailscale is connected. The hub itself binds to loopback.
Confirm the MagicDNS name resolves from the CT and from one client machine:
getent hosts brigade-hub
tailscale ping brigade-hub
3. Install Python and Brigade
Debian 12 ships Python 3.11 and Debian 13 ships a newer one. Confirm the interpreter before you install the CLI.
python3 --version
apt-get install -y python3 python3-venv pipx sqlite3 curl
install -d -o root -g root -m 0755 /opt/pipx /usr/local/bin
python3 -c 'import sys; print(sys.version); raise SystemExit("Python 3.11+ required") if sys.version_info < (3, 11) else None'
PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin \
pipx install --force \
"git+https://github.com/escoffier-labs/brigade.git@<BRIGADE_GIT_REF>"
/usr/local/bin/brigade --version
Set <BRIGADE_GIT_REF> to a reviewed tag or commit, and use the same ref on the hub and on every client while you roll out a new build. This install creates the brigade entry point. It is the same CLI as brigadestack, and the service below runs /usr/local/bin/brigade.
4. Create the admin token file
The service reads the admin token from BRIGADE_FLEET_TOKEN in /etc/brigade/fleet-hub.env. Run this in the CT as root. It generates the token on the hub host and writes it straight to the file without printing it:
install -d -o root -g root -m 0750 /etc/brigade
install -d -o root -g root -m 0700 /var/lib/brigade
umask 077
generated_token="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')" && \
[ -n "$generated_token" ] && \
printf 'BRIGADE_FLEET_TOKEN=%s\n' "$generated_token" > /etc/brigade/fleet-hub.env
unset generated_token
chown root:root /etc/brigade/fleet-hub.env
chmod 0600 /etc/brigade/fleet-hub.env
If you keep the token in 1Password, read it into the file instead. This expects op to be signed in for the vault, with the item field named password:
install -d -o root -g root -m 0750 /etc/brigade
install -d -o root -g root -m 0700 /var/lib/brigade
umask 077
fleet_token="$(op read 'op://<VAULT>/brigade-fleet-hub/password')" && \
printf 'BRIGADE_FLEET_TOKEN=%s\n' "$fleet_token" > /etc/brigade/fleet-hub.env
unset fleet_token
chown root:root /etc/brigade/fleet-hub.env
chmod 0600 /etc/brigade/fleet-hub.env
Check the file metadata, not its contents:
stat -c '%U:%G %a %n' /etc/brigade/fleet-hub.env
The output should include root:root 600. The hub does not persist the admin token in SQLite. It keeps node tokens only as SHA-256 digests.
The operator machine that enrolls nodes needs the same admin token in ~/.brigade/fleet.token, mode 0600. Move it with your usual secret channel and never through chat. One option, run on the operator machine, streams it over SSH without printing it:
umask 077
mkdir -p ~/.brigade
ssh root@<HUB_SSH_HOST> "sed -n 's/^BRIGADE_FLEET_TOKEN=//p' /etc/brigade/fleet-hub.env" > ~/.brigade/fleet.token
5. Install and start the service
Write /etc/systemd/system/brigade-fleet.service:
[Unit]
Description=Brigade fleet hub
After=network-online.target tailscaled.service
Wants=network-online.target
Requires=tailscaled.service
[Service]
Type=simple
User=root
WorkingDirectory=/var/lib/brigade
EnvironmentFile=/etc/brigade/fleet-hub.env
ExecStart=/usr/local/bin/brigade fleet serve --host 127.0.0.1 --port 3774 --db /var/lib/brigade/fleet-hub.db --trust-forwarded-proto
Restart=on-failure
RestartSec=5
PrivateTmp=true
NoNewPrivileges=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/brigade
[Install]
WantedBy=multi-user.target
--host is required. Never bind to all interfaces or a routable address: the hub must listen on loopback so Tailscale Serve is its only network frontend. The explicit --db path keeps the database under /var/lib/brigade, away from any user’s home directory. That directory must be owned by root with mode 0700.
A new hub requires a node token for every write. Add --allow-admin-writes only to keep clients that still share one token working during a migration.
Start it and put Tailscale Serve in front:
systemctl daemon-reload
systemctl enable --now brigade-fleet.service
systemctl status --no-pager brigade-fleet.service
journalctl -u brigade-fleet.service -n 50 --no-pager
tailscale serve --bg 3774
tailscale serve status
Tailscale Serve provides the HTTPS MagicDNS origin https://<HUB_TAILNET_FQDN> and proxies it to the local service. The startup log should name 127.0.0.1:3774 and /var/lib/brigade/fleet-hub.db. Keep the Serve proxy on this CT. Do not point it at the Tailscale interface or another host.
6. Verify health and authenticated status
Health needs no token:
curl --fail --silent --show-error https://<HUB_TAILNET_FQDN>/health
Expected response:
{"ok": true, "service": "brigade-fleet-hub"}
From the operator machine, check the protected status endpoint with the admin token:
curl --fail --silent --show-error \
-H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
https://<HUB_TAILNET_FQDN>/status
An empty result is valid before any client reports. HTTP 401 means the request reached the hub but the token did not match.
For a direct endpoint test, enroll a throwaway node and post one synthetic event with its token. Use a node and run id nobody could confuse with a real run, and revoke the node afterward:
smoke_token="$(curl --fail --silent --show-error \
-X POST \
-H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
-H 'Content-Type: application/json' \
--data '{"action":"add","node_id":"smoke-<NODE_ID>","label":"smoke test"}' \
https://<HUB_TAILNET_FQDN>/nodes | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')"
curl --fail --silent --show-error \
-X POST \
-H "Authorization: Bearer $smoke_token" \
-H 'Content-Type: application/json' \
--data '{"node_id":"smoke-<NODE_ID>","run_id":"smoke-<RUN_ID>","repo":"<REPO_NAME>","seat":"<SEAT>","harness":"<HARNESS>","state":"run.started","ts":"<ISO8601_UTC>","sequence":1,"digest":"<EVENT_DIGEST>"}' \
https://<HUB_TAILNET_FQDN>/events
curl --fail --silent --show-error \
-H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
https://<HUB_TAILNET_FQDN>/status
curl --fail --silent --show-error \
-X POST \
-H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
-H 'Content-Type: application/json' \
--data '{"action":"revoke","node_id":"smoke-<NODE_ID>"}' \
https://<HUB_TAILNET_FQDN>/nodes
unset smoke_token
The first post to /events reports accepted: 1. Posting the same body again reports a duplicate, which shows the idempotency key works. Posting the same event with the admin token instead answers HTTP 403 on a hub without --allow-admin-writes, which shows node tokens are required.
7. Enroll and configure each client
Every machine needs its own node token. On the machine, print its identity. brigadestack node --machine creates it once under BRIGADE_HOME (~/.brigade/node.toml by default). Every event and claim is stamped with this identity, never a workspace-local one.
brigadestack node --machine --json | python3 -c 'import json,sys; print(json.load(sys.stdin)["node_id"])'
From the operator machine that holds the admin token ([fleet] token_file in its own ~/.brigade/fleet.toml, or BRIGADE_FLEET_TOKEN), enroll that identity. The token prints once and never again, because the hub keeps only its hash:
brigadestack fleet nodes add <NODE_ID> --label <HOSTNAME>
brigadestack fleet nodes list
Deliver the printed token to the machine through your normal secret-distribution process and install it as a file only that user can read, mode 0600. The admin token stays on the operator machine. A node does not need it.
mkdir -p ~/.brigade
install -m 0600 <NODE_TOKEN_FILE> ~/.brigade/fleet-node.token
chmod 0700 ~/.brigade
cat > ~/.brigade/fleet.toml <<'EOF'
[fleet]
hub_url = "https://<HUB_TAILNET_FQDN>"
node_token_file = "~/.brigade/fleet-node.token"
EOF
chmod 0600 ~/.brigade/fleet.toml
The config keys are [fleet] hub_url, [fleet] node_token_file (this machine’s identity), and optionally [fleet] token_file (the admin token, only where brigadestack fleet nodes runs). BRIGADE_FLEET_HUB_URL overrides hub_url, BRIGADE_FLEET_NODE_TOKEN overrides the node token file, and BRIGADE_FLEET_TOKEN overrides the admin token file. Do not add a host, port or token key to this file.
To rotate a machine’s credential, revoke and re-enroll it, then replace the file on that machine:
brigadestack fleet nodes revoke <NODE_ID>
brigadestack fleet nodes add <NODE_ID> --label <HOSTNAME>
The hub refuses a revoked token with HTTP 401 at once. Events the machine produces in between stay in its local spool and go out on the next report or brigadestack fleet flush, once the new token is in place.
Windows client
Use the same [fleet] keys on Windows. The client expands ~ in node_token_file, so the same config works in a native Windows user profile:
New-Item -ItemType Directory -Force "$env:USERPROFILE\.brigade" | Out-Null
Copy-Item <NODE_TOKEN_FILE> "$env:USERPROFILE\.brigade\fleet-node.token"
@'
[fleet]
hub_url = "https://<HUB_TAILNET_FQDN>"
node_token_file = "~/.brigade/fleet-node.token"
'@ | Set-Content -Encoding utf8 "$env:USERPROFILE\.brigade\fleet.toml"
icacls "$env:USERPROFILE\.brigade\fleet-node.token" /inheritance:r /grant:r "$env:USERNAME:(R)" | Out-Null
brigadestack fleet status
Keep the token file readable only by the Windows user who runs Brigade. brigadestack fleet flush is the manual retry command on Windows too.
Check a client
Once a machine has a node identity and an active run, inspect the hub from it:
brigadestack fleet status
brigadestack fleet status --json
fleet status queries GET /status and shows node, repository, run id, seat and harness, state, and age. It hides finished runs unless you add --all. brigadestack fleet flush retries the local spool at once:
brigadestack fleet flush
Reporting is best effort. A failed POST is appended under the Brigade home directory at fleet-spool/<node_id>.jsonl. It never blocks the journal writer or fails the local run. A later report or brigadestack fleet flush sends the queue in order.
Dashboard from a laptop or phone
The hub serves a read-only web view of the same data at https://<HUB_TAILNET_FQDN>/ (machine cards) and https://<HUB_TAILNET_FQDN>/view/repos (repository board). It accepts the admin token, never a node token. From a machine that holds the token, a bearer header works as it does for /status:
curl --silent --show-error \
-H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
https://<HUB_TAILNET_FQDN>/ | head
Sort and filter with query parameters, for example /?attention=1 (only failed, awaiting-approval or stale runs), /view/repos?sort=repo, /?node=<prefix>, or /?all=1 (include finished runs). The page refreshes every 10 seconds and works with JavaScript off.
To open it on a phone, the browser must reach the hub over HTTPS. The unit above already does this: the hub is bound to numeric loopback with --trust-forwarded-proto, and Tailscale Serve preserves the browser-facing Host and sets X-Forwarded-Proto to a single https. The hub trusts that header only from an immediate loopback peer and marks the cookie Secure.
Do not point a client at the direct backend URL http://<TAILSCALE_IPV4>:3774. Clients refuse remote cleartext hub URLs before they attach a bearer credential. Set every client’s hub_url to https://<HUB_TAILNET_FQDN>.
On the admin machine, mint a private, read-only enrollment URL that lasts five minutes:
BRIGADE_FLEET_HUB_URL=https://<HUB_TAILNET_FQDN> brigadestack fleet enroll --label "phone" --base-url https://<HUB_TAILNET_FQDN>
Open the printed URL on the device and confirm the form. Opening the page does not use up the code. Submitting the form does, once, and redirects to /deck with a host-only cookie that lasts 30 days. That cookie reads HTML only. It cannot write the roster or policy, and it cannot read JSON. Each device enrolls on its own.
For a lost device, list the browser sessions, find its label, and revoke each session id you do not want:
BRIGADE_FLEET_HUB_URL=https://<HUB_TAILNET_FQDN> brigadestack fleet sessions --dashboard
BRIGADE_FLEET_HUB_URL=https://<HUB_TAILNET_FQDN> brigadestack fleet sessions --dashboard --revoke SESSION_ID
Follow the next-page cursor when a listing passes 100 sessions, and list again from page one during an incident to catch new sessions. --all also shows records kept after access expires or a session is revoked. Rotating the admin token does not revoke browser sessions, so revoke each one. These calls need the current admin token. brigadestack fleet sessions with no flag still reports interactive editor presence.
Tailscale Serve identity authentication (optional)
The hub can trust the Tailscale-User-Login header that Tailscale Serve adds, and let dashboard routes (/, /deck, /deck/repos and the older /view/* boards) through without the admin token or a cookie. Start it with --trust-tailscale-identity.
Turn this on only when all three are true:
- The hub is bound to loopback (
127.0.0.1) and nothing else. Never use all interfaces or a routable address with this flag. - Dashboard requests reach the hub only through a Tailscale Serve proxy on the same host, which terminates Tailscale identity and strips any spoofed
Tailscale-User-Loginheader. The immediate TCP peer is then loopback. - The backend is never exposed to the tailnet or any other network without the proxy in front.
Every process inside the hub host or container is trusted in this mode, because any local process can reach the loopback backend and set the header. Run the hub in a dedicated host or container, and skip the flag when untrusted workloads share its network namespace.
The hub accepts a dashboard request only when the immediate peer is loopback and the header is present, non-empty, at most 256 characters, and free of control characters. It never logs or renders the identity value. JSON and API routes (/status, /claims, /nodes, /cloud, /models and every POST) still need a token. The identity header grants none of them.
An example ExecStart that binds loopback and trusts the proxy header:
[Service]
ExecStart=/usr/local/bin/brigade fleet serve --host 127.0.0.1 --port 3774 --db /var/lib/brigade/fleet-hub.db --trust-tailscale-identity
Pair it with a Serve rule on the same host that forwards https://<HUB_TAILNET_FQDN> to http://127.0.0.1:3774. Never point the rule at a routable or remote backend address.
8. Back up the SQLite database
Use SQLite’s online backup. Do not copy only the main file while the service runs, because WAL data may not have reached it yet.
Create /usr/local/sbin/brigade-fleet-backup:
#!/bin/sh
set -eu
backup_dir=/var/backups/brigade
stamp=$(date -u +%Y%m%dT%H%M%SZ)
umask 077
install -d -o root -g root -m 0700 "$backup_dir"
sqlite3 /var/lib/brigade/fleet-hub.db ".backup '$backup_dir/fleet-hub-$stamp.db'"
chmod 0600 "$backup_dir/fleet-hub-$stamp.db"
find "$backup_dir" -type f -name 'fleet-hub-*.db' -mtime +14 -delete
Write the script to a temporary file, then install it:
install -o root -g root -m 0750 <BACKUP_SCRIPT_FILE> /usr/local/sbin/brigade-fleet-backup
Write /etc/systemd/system/brigade-fleet-backup.service:
[Unit]
Description=Back up the Brigade fleet hub SQLite database
After=brigade-fleet.service
[Service]
Type=oneshot
UMask=0077
ExecStart=/usr/local/sbin/brigade-fleet-backup
Write /etc/systemd/system/brigade-fleet-backup.timer:
[Unit]
Description=Daily Brigade fleet hub SQLite backup
[Timer]
OnCalendar=*-*-* 03:15:00
Persistent=true
Unit=brigade-fleet-backup.service
[Install]
WantedBy=timers.target
Enable the timer and run it once:
systemctl daemon-reload
systemctl enable --now brigade-fleet-backup.timer
systemctl start brigade-fleet-backup.service
systemctl list-timers brigade-fleet-backup.timer --no-pager
ls -lh /var/backups/brigade/
Copy backups off the CT with your existing backup system. The local timer is not a substitute for an off-host copy.
If the backup directory already exists from an earlier setup, stop the timer, confirm the path is the dedicated backup directory, then tighten it and the existing backups:
systemctl stop brigade-fleet-backup.timer
test -d /var/backups/brigade && test ! -L /var/backups/brigade
chown root:root /var/backups/brigade
chmod 0700 /var/backups/brigade
find /var/backups/brigade -type f -name 'fleet-hub-*.db' -exec chmod 0600 {} +
systemctl start brigade-fleet-backup.timer
Upgrade and rollback
Before you change the Brigade ref, take a backup and note the current ref:
systemctl start brigade-fleet-backup.service
pipx list
systemctl status --no-pager brigade-fleet.service
Upgrade to a reviewed ref:
systemctl stop brigade-fleet.service
PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin \
pipx install --force \
"git+https://github.com/escoffier-labs/brigade.git@<NEW_BRIGADE_GIT_REF>"
systemctl start brigade-fleet.service
systemctl status --no-pager brigade-fleet.service
curl --fail --silent --show-error https://<HUB_TAILNET_FQDN>/health
Upgrade the hub before the clients. To roll the application back to a known-good ref, run the same sequence with that ref:
systemctl stop brigade-fleet.service
PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin \
pipx install --force \
"git+https://github.com/escoffier-labs/brigade.git@<KNOWN_GOOD_BRIGADE_GIT_REF>"
systemctl start brigade-fleet.service
journalctl -u brigade-fleet.service -n 50 --no-pager
Keep the SQLite file during an application rollback. A hub refuses a database whose PRAGMA user_version is newer than it knows. If an upgrade migrated the database and the older ref refuses it, restore the pre-upgrade backup from step 8, which loses the events and claims received since, or stay on the newer ref.
Move off a direct Tailnet HTTP endpoint
If you built the hub from an earlier version of this runbook with a direct Tailnet HTTP endpoint, move it before you upgrade clients. Stop the hub, confirm /var/lib/brigade is the dedicated data directory and not a symlink, and tighten its ownership and mode:
systemctl stop brigade-fleet.service
test -d /var/lib/brigade && test ! -L /var/lib/brigade
chown root:root /var/lib/brigade
chmod 0700 /var/lib/brigade
stat -c '%U:%G %a %n' /var/lib/brigade
Replace the service unit with the loopback-only ExecStart from step 5, then restart the hub and enable Tailscale Serve:
systemctl daemon-reload
systemctl start brigade-fleet.service
tailscale serve --bg 3774
tailscale serve status
curl --fail --silent --show-error https://<HUB_TAILNET_FQDN>/health
Then set each client to https://<HUB_TAILNET_FQDN>. Offline events and messages wait in local spools until the new URL is set.
Troubleshooting
POST /events or POST /claims returns HTTP 403
The hub accepted the token but it may not act as the node_id in the body. Either the client sent the admin token (no node_token_file is set, and the client logged a shared-token WARNING) to a hub without --allow-admin-writes, or the node token on the machine was minted for a different node_id than brigadestack node --machine --json reports there. Compare that output with brigadestack fleet nodes list on the operator machine, then re-enroll (step 7). Events made in the meantime stay in the client spool and go out on the next brigadestack fleet flush.
GET /status or POST /events returns HTTP 401
The request reached the hub, but the bearer token matched neither the admin token nor an enrolled, unrevoked node token. A revoked node token answers unauthorized: node token revoked, so re-enroll it (step 7). Otherwise check that the client points at the intended token file, and that /etc/brigade/fleet-hub.env holds the token you meant. After you change that file, restart the service:
systemctl restart brigade-fleet.service
systemctl show brigade-fleet.service --property=EnvironmentFiles --no-pager
Do not print the token to logs or compare it in shell history. A queued event can wait in the client spool while you fix the token, then go out with:
brigadestack fleet flush
Tailscale is down or the hub name does not resolve
Check the daemon, address, route and MagicDNS path:
systemctl status --no-pager tailscaled.service
tailscale status
tailscale ip -4
getent hosts brigade-hub
tailscale ping brigade-hub
If the daemon is stopped, restart it and check the hub again:
systemctl restart tailscaled.service
systemctl restart brigade-fleet.service
If the CT has no /dev/net/tun, apply the host-side TUN setting from step 1. Do not enable nesting as a workaround.
SQLite reports database is locked
The hub uses WAL mode, a 5-second SQLite busy timeout and a 10-second connection timeout. Look for competing writers and read the service log:
journalctl -u brigade-fleet.service --since '-15 min' --no-pager
fuser -v /var/lib/brigade/fleet-hub.db
sqlite3 /var/lib/brigade/fleet-hub.db 'PRAGMA integrity_check;'
Only the hub should write the database. Stop the service before any offline repair or manual replacement, and never delete the -wal or -shm files while it runs:
systemctl stop brigade-fleet.service
sqlite3 /var/lib/brigade/fleet-hub.db 'PRAGMA integrity_check;'
systemctl start brigade-fleet.service
If lock errors continue, check that the backup script uses .backup and not a second long-running writer, then look at the CT’s disk and memory pressure.
More in the GitHub runbook
These topics are for operators running a larger fleet. They stay in the GitHub runbook and are not repeated here: