Search BrigadeStack

Search BrigadeStack pages and docs.

BrigadeStack
Menu
Docs menu

Docs / Self-hosting

Run the backend on Proxmox

Run the backend, which tracks runs, claims, messages and the dashboard across your machines, in an unprivileged Debian container on Proxmox VE behind Tailscale Serve HTTPS.

This guide builds the backend as an unprivileged Debian LXC on Proxmox VE 9.x. The backend tracks runs, claims and messages across your machines and serves the dashboard, and the commands call it Fleet Hub, so this page says “the hub” from here on. The hub is a Brigade service listening on loopback only, and Tailscale Serve is the only way in. It keeps its live view in one SQLite database. Local journals stay authoritative, so an outage delays the aggregate view and makes clients spool events until it returns. For what the hub is and when to use it, read Run the backend first. The ExecStart lines here run the default engine, which has no shared memory. Shared memory needs extra flags: see Turn on shared memory.

Set up with your agent

Run the backend on Proxmox

Give this prompt to your coding agent.

Help me run the BrigadeStack backend so my machines share claims, runs and memory.

Follow https://brigadestack.com/docs/guides/fleet-hub-proxmox one step at a time. The CT runs on my Proxmox host; ask me for the host and storage names instead of guessing them.

Tell me what each step changes and on which machine. Keep the backend on a private network such as Tailscale. Never publish it to the internet or behind a public tunnel.

Create tokens on the hub host and store them in files only the service user can read. Never ask me to paste a token into this chat, and never put one in a command line or source file.

When the backend is up, run the health and status checks from the guide, then enroll this machine as a node and confirm it shows up.

Run tailscale serve status to print the hub’s HTTPS tailnet name. This guide calls it <HUB_TAILNET_FQDN>. Replace every other angle-bracket value before you run a command.

Target layout

  • Proxmox VE 9.x host.
  • A Debian 12 or Debian 13 CT template.
  • 1 vCPU, 512 MB RAM, 4 GB root disk.
  • An unprivileged CT with nesting disabled.
  • Tailscale inside the CT, hostname brigade-hub.
  • The hub bound to 127.0.0.1:3774, behind Tailscale Serve HTTPS.
  • The SQLite database at /var/lib/brigade/fleet-hub.db.
  • A root-owned, mode 0600 systemd environment file at /etc/brigade/fleet-hub.env.

The hub exposes these endpoints:

EndpointAuthPurpose
GET /healthnoneLiveness check
POST /eventsnode tokenAppend an event. The admin token works only with --allow-admin-writes.
GET /statusadmin or node tokenLatest non-terminal state per node and run. Add ?all=1 to include finished runs.
POST /claims, GET /claimsnode token to post, admin or node token to readRepository claims
GET /nodes, POST /nodesadmin token onlyNode enrollment
GET /, GET /view/machines, GET /view/reposadmin token or dashboard cookieThe Fleet dashboard
GET /deck/roster, POST /deck/rosteradmin Authorization header to saveThe roster page

The hub drops a repeated event, matched on (node_id, run_id, sequence, digest), so reposting after a lost response is safe.

Two bearer credentials exist. The admin token in /etc/brigade/fleet-hub.env is the control plane: it enrolls nodes, reads status and claims, and mints dashboard enrollment codes. Each machine’s node token, created by brigadestack fleet nodes add, is that machine’s identity. The hub derives node_id from it and refuses an event or claim for any other node with HTTP 403. The hub stores only a SHA-256 of each node token.

1. Create the CT on the Proxmox host

Download a Debian 12 or 13 standard CT template into the template storage first, and confirm the exact filename and storage names.

Run as root on the Proxmox host:

pct create <CTID> <TEMPLATE_STORAGE>:vztmpl/<DEBIAN_TEMPLATE>.tar.zst \
  --hostname brigade-hub \
  --ostype debian \
  --cores 1 \
  --memory 512 \
  --swap 512 \
  --rootfs <ROOT_STORAGE>:4 \
  --unprivileged 1 \
  --features nesting=0 \
  --net0 name=eth0,bridge=<PROXMOX_BRIDGE>,ip=<CT_IPV4/CIDR>,gw=<GATEWAY_IPV4>,type=veth \
  --onboot 1 \
  --start 0 \
  --ssh-public-keys <PUBLIC_KEY_FILE>

pct start <CTID>
pct enter <CTID>

The CT needs outbound access to the Tailscale package repository and the tailnet. Do not enable nesting for this service.

Tailscale normally needs /dev/net/tun. Check from inside the CT:

test -c /dev/net/tun && echo "TUN device present" || echo "TUN device missing"

If the device is missing, leave nesting disabled and add only the TUN device permission on the Proxmox host:

printf '%s\n' \
  'lxc.cgroup2.devices.allow: c 10:200 rwm' \
  'lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file' \
  >> /etc/pve/lxc/<CTID>.conf
pct restart <CTID>
pct enter <CTID>

2. Install Tailscale in the CT

Run as root inside the CT. The repository path comes from Debian’s codename, so the same commands work on Debian 12 and 13.

apt-get update
apt-get install -y ca-certificates curl gnupg

. /etc/os-release
curl -fsSL "https://pkgs.tailscale.com/stable/debian/${VERSION_CODENAME}.noarmor.gpg" \
  -o /usr/share/keyrings/tailscale-archive-keyring.gpg
curl -fsSL "https://pkgs.tailscale.com/stable/debian/${VERSION_CODENAME}.tailscale-keyring.list" \
  -o /etc/apt/sources.list.d/tailscale.list

apt-get update
apt-get install -y tailscale
systemctl enable --now tailscaled
tailscale up --authkey=tskey-auth-<PREAUTH_KEY> --hostname=brigade-hub --accept-dns=true
tailscale status
tailscale ip -4

The pre-auth key must be reusable, or have enough uses for the rebuilds you expect. Type it in the CT’s own shell. Do not put it in a repository, a chat, or a systemd unit. If you would rather not use a key, leave out --authkey=...: tailscale up then prints a login URL to approve in a browser.

The IPv4 address only shows that Tailscale is connected. The hub itself binds to loopback.

Confirm the MagicDNS name resolves from the CT and from one client machine:

getent hosts brigade-hub
tailscale ping brigade-hub

3. Install Python and Brigade

Debian 12 ships Python 3.11 and Debian 13 ships a newer one. Confirm the interpreter before you install the CLI.

python3 --version
apt-get install -y python3 python3-venv pipx sqlite3 curl
install -d -o root -g root -m 0755 /opt/pipx /usr/local/bin
python3 -c 'import sys; print(sys.version); raise SystemExit("Python 3.11+ required") if sys.version_info < (3, 11) else None'

PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin \
  pipx install --force \
  "git+https://github.com/escoffier-labs/brigade.git@<BRIGADE_GIT_REF>"

/usr/local/bin/brigade --version

Set <BRIGADE_GIT_REF> to a reviewed tag or commit, and use the same ref on the hub and on every client while you roll out a new build. This install creates the brigade entry point. It is the same CLI as brigadestack, and the service below runs /usr/local/bin/brigade.

4. Create the admin token file

The service reads the admin token from BRIGADE_FLEET_TOKEN in /etc/brigade/fleet-hub.env. Run this in the CT as root. It generates the token on the hub host and writes it straight to the file without printing it:

install -d -o root -g root -m 0750 /etc/brigade
install -d -o root -g root -m 0700 /var/lib/brigade
umask 077
generated_token="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')" && \
  [ -n "$generated_token" ] && \
  printf 'BRIGADE_FLEET_TOKEN=%s\n' "$generated_token" > /etc/brigade/fleet-hub.env
unset generated_token
chown root:root /etc/brigade/fleet-hub.env
chmod 0600 /etc/brigade/fleet-hub.env

If you keep the token in 1Password, read it into the file instead. This expects op to be signed in for the vault, with the item field named password:

install -d -o root -g root -m 0750 /etc/brigade
install -d -o root -g root -m 0700 /var/lib/brigade
umask 077
fleet_token="$(op read 'op://<VAULT>/brigade-fleet-hub/password')" && \
  printf 'BRIGADE_FLEET_TOKEN=%s\n' "$fleet_token" > /etc/brigade/fleet-hub.env
unset fleet_token
chown root:root /etc/brigade/fleet-hub.env
chmod 0600 /etc/brigade/fleet-hub.env

Check the file metadata, not its contents:

stat -c '%U:%G %a %n' /etc/brigade/fleet-hub.env

The output should include root:root 600. The hub does not persist the admin token in SQLite. It keeps node tokens only as SHA-256 digests.

The operator machine that enrolls nodes needs the same admin token in ~/.brigade/fleet.token, mode 0600. Move it with your usual secret channel and never through chat. One option, run on the operator machine, streams it over SSH without printing it:

umask 077
mkdir -p ~/.brigade
ssh root@<HUB_SSH_HOST> "sed -n 's/^BRIGADE_FLEET_TOKEN=//p' /etc/brigade/fleet-hub.env" > ~/.brigade/fleet.token

5. Install and start the service

Write /etc/systemd/system/brigade-fleet.service:

[Unit]
Description=Brigade fleet hub
After=network-online.target tailscaled.service
Wants=network-online.target
Requires=tailscaled.service

[Service]
Type=simple
User=root
WorkingDirectory=/var/lib/brigade
EnvironmentFile=/etc/brigade/fleet-hub.env
ExecStart=/usr/local/bin/brigade fleet serve --host 127.0.0.1 --port 3774 --db /var/lib/brigade/fleet-hub.db --trust-forwarded-proto
Restart=on-failure
RestartSec=5
PrivateTmp=true
NoNewPrivileges=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/brigade

[Install]
WantedBy=multi-user.target

--host is required. Never bind to all interfaces or a routable address: the hub must listen on loopback so Tailscale Serve is its only network frontend. The explicit --db path keeps the database under /var/lib/brigade, away from any user’s home directory. That directory must be owned by root with mode 0700.

A new hub requires a node token for every write. Add --allow-admin-writes only to keep clients that still share one token working during a migration.

Start it and put Tailscale Serve in front:

systemctl daemon-reload
systemctl enable --now brigade-fleet.service
systemctl status --no-pager brigade-fleet.service
journalctl -u brigade-fleet.service -n 50 --no-pager
tailscale serve --bg 3774
tailscale serve status

Tailscale Serve provides the HTTPS MagicDNS origin https://<HUB_TAILNET_FQDN> and proxies it to the local service. The startup log should name 127.0.0.1:3774 and /var/lib/brigade/fleet-hub.db. Keep the Serve proxy on this CT. Do not point it at the Tailscale interface or another host.

6. Verify health and authenticated status

Health needs no token:

curl --fail --silent --show-error https://<HUB_TAILNET_FQDN>/health

Expected response:

{"ok": true, "service": "brigade-fleet-hub"}

From the operator machine, check the protected status endpoint with the admin token:

curl --fail --silent --show-error \
  -H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
  https://<HUB_TAILNET_FQDN>/status

An empty result is valid before any client reports. HTTP 401 means the request reached the hub but the token did not match.

For a direct endpoint test, enroll a throwaway node and post one synthetic event with its token. Use a node and run id nobody could confuse with a real run, and revoke the node afterward:

smoke_token="$(curl --fail --silent --show-error \
  -X POST \
  -H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
  -H 'Content-Type: application/json' \
  --data '{"action":"add","node_id":"smoke-<NODE_ID>","label":"smoke test"}' \
  https://<HUB_TAILNET_FQDN>/nodes | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')"

curl --fail --silent --show-error \
  -X POST \
  -H "Authorization: Bearer $smoke_token" \
  -H 'Content-Type: application/json' \
  --data '{"node_id":"smoke-<NODE_ID>","run_id":"smoke-<RUN_ID>","repo":"<REPO_NAME>","seat":"<SEAT>","harness":"<HARNESS>","state":"run.started","ts":"<ISO8601_UTC>","sequence":1,"digest":"<EVENT_DIGEST>"}' \
  https://<HUB_TAILNET_FQDN>/events

curl --fail --silent --show-error \
  -H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
  https://<HUB_TAILNET_FQDN>/status

curl --fail --silent --show-error \
  -X POST \
  -H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
  -H 'Content-Type: application/json' \
  --data '{"action":"revoke","node_id":"smoke-<NODE_ID>"}' \
  https://<HUB_TAILNET_FQDN>/nodes
unset smoke_token

The first post to /events reports accepted: 1. Posting the same body again reports a duplicate, which shows the idempotency key works. Posting the same event with the admin token instead answers HTTP 403 on a hub without --allow-admin-writes, which shows node tokens are required.

7. Enroll and configure each client

Every machine needs its own node token. On the machine, print its identity. brigadestack node --machine creates it once under BRIGADE_HOME (~/.brigade/node.toml by default). Every event and claim is stamped with this identity, never a workspace-local one.

brigadestack node --machine --json | python3 -c 'import json,sys; print(json.load(sys.stdin)["node_id"])'

From the operator machine that holds the admin token ([fleet] token_file in its own ~/.brigade/fleet.toml, or BRIGADE_FLEET_TOKEN), enroll that identity. The token prints once and never again, because the hub keeps only its hash:

brigadestack fleet nodes add <NODE_ID> --label <HOSTNAME>
brigadestack fleet nodes list

Deliver the printed token to the machine through your normal secret-distribution process and install it as a file only that user can read, mode 0600. The admin token stays on the operator machine. A node does not need it.

mkdir -p ~/.brigade
install -m 0600 <NODE_TOKEN_FILE> ~/.brigade/fleet-node.token
chmod 0700 ~/.brigade

cat > ~/.brigade/fleet.toml <<'EOF'
[fleet]
hub_url = "https://<HUB_TAILNET_FQDN>"
node_token_file = "~/.brigade/fleet-node.token"
EOF

chmod 0600 ~/.brigade/fleet.toml

The config keys are [fleet] hub_url, [fleet] node_token_file (this machine’s identity), and optionally [fleet] token_file (the admin token, only where brigadestack fleet nodes runs). BRIGADE_FLEET_HUB_URL overrides hub_url, BRIGADE_FLEET_NODE_TOKEN overrides the node token file, and BRIGADE_FLEET_TOKEN overrides the admin token file. Do not add a host, port or token key to this file.

To rotate a machine’s credential, revoke and re-enroll it, then replace the file on that machine:

brigadestack fleet nodes revoke <NODE_ID>
brigadestack fleet nodes add <NODE_ID> --label <HOSTNAME>

The hub refuses a revoked token with HTTP 401 at once. Events the machine produces in between stay in its local spool and go out on the next report or brigadestack fleet flush, once the new token is in place.

Windows client

Use the same [fleet] keys on Windows. The client expands ~ in node_token_file, so the same config works in a native Windows user profile:

New-Item -ItemType Directory -Force "$env:USERPROFILE\.brigade" | Out-Null
Copy-Item <NODE_TOKEN_FILE> "$env:USERPROFILE\.brigade\fleet-node.token"
@'
[fleet]
hub_url = "https://<HUB_TAILNET_FQDN>"
node_token_file = "~/.brigade/fleet-node.token"
'@ | Set-Content -Encoding utf8 "$env:USERPROFILE\.brigade\fleet.toml"
icacls "$env:USERPROFILE\.brigade\fleet-node.token" /inheritance:r /grant:r "$env:USERNAME:(R)" | Out-Null
brigadestack fleet status

Keep the token file readable only by the Windows user who runs Brigade. brigadestack fleet flush is the manual retry command on Windows too.

Check a client

Once a machine has a node identity and an active run, inspect the hub from it:

brigadestack fleet status
brigadestack fleet status --json

fleet status queries GET /status and shows node, repository, run id, seat and harness, state, and age. It hides finished runs unless you add --all. brigadestack fleet flush retries the local spool at once:

brigadestack fleet flush

Reporting is best effort. A failed POST is appended under the Brigade home directory at fleet-spool/<node_id>.jsonl. It never blocks the journal writer or fails the local run. A later report or brigadestack fleet flush sends the queue in order.

Dashboard from a laptop or phone

The hub serves a read-only web view of the same data at https://<HUB_TAILNET_FQDN>/ (machine cards) and https://<HUB_TAILNET_FQDN>/view/repos (repository board). It accepts the admin token, never a node token. From a machine that holds the token, a bearer header works as it does for /status:

curl --silent --show-error \
  -H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
  https://<HUB_TAILNET_FQDN>/ | head

Sort and filter with query parameters, for example /?attention=1 (only failed, awaiting-approval or stale runs), /view/repos?sort=repo, /?node=<prefix>, or /?all=1 (include finished runs). The page refreshes every 10 seconds and works with JavaScript off.

To open it on a phone, the browser must reach the hub over HTTPS. The unit above already does this: the hub is bound to numeric loopback with --trust-forwarded-proto, and Tailscale Serve preserves the browser-facing Host and sets X-Forwarded-Proto to a single https. The hub trusts that header only from an immediate loopback peer and marks the cookie Secure.

Do not point a client at the direct backend URL http://<TAILSCALE_IPV4>:3774. Clients refuse remote cleartext hub URLs before they attach a bearer credential. Set every client’s hub_url to https://<HUB_TAILNET_FQDN>.

On the admin machine, mint a private, read-only enrollment URL that lasts five minutes:

BRIGADE_FLEET_HUB_URL=https://<HUB_TAILNET_FQDN> brigadestack fleet enroll --label "phone" --base-url https://<HUB_TAILNET_FQDN>

Open the printed URL on the device and confirm the form. Opening the page does not use up the code. Submitting the form does, once, and redirects to /deck with a host-only cookie that lasts 30 days. That cookie reads HTML only. It cannot write the roster or policy, and it cannot read JSON. Each device enrolls on its own.

For a lost device, list the browser sessions, find its label, and revoke each session id you do not want:

BRIGADE_FLEET_HUB_URL=https://<HUB_TAILNET_FQDN> brigadestack fleet sessions --dashboard
BRIGADE_FLEET_HUB_URL=https://<HUB_TAILNET_FQDN> brigadestack fleet sessions --dashboard --revoke SESSION_ID

Follow the next-page cursor when a listing passes 100 sessions, and list again from page one during an incident to catch new sessions. --all also shows records kept after access expires or a session is revoked. Rotating the admin token does not revoke browser sessions, so revoke each one. These calls need the current admin token. brigadestack fleet sessions with no flag still reports interactive editor presence.

Tailscale Serve identity authentication (optional)

The hub can trust the Tailscale-User-Login header that Tailscale Serve adds, and let dashboard routes (/, /deck, /deck/repos and the older /view/* boards) through without the admin token or a cookie. Start it with --trust-tailscale-identity.

Turn this on only when all three are true:

  1. The hub is bound to loopback (127.0.0.1) and nothing else. Never use all interfaces or a routable address with this flag.
  2. Dashboard requests reach the hub only through a Tailscale Serve proxy on the same host, which terminates Tailscale identity and strips any spoofed Tailscale-User-Login header. The immediate TCP peer is then loopback.
  3. The backend is never exposed to the tailnet or any other network without the proxy in front.

Every process inside the hub host or container is trusted in this mode, because any local process can reach the loopback backend and set the header. Run the hub in a dedicated host or container, and skip the flag when untrusted workloads share its network namespace.

The hub accepts a dashboard request only when the immediate peer is loopback and the header is present, non-empty, at most 256 characters, and free of control characters. It never logs or renders the identity value. JSON and API routes (/status, /claims, /nodes, /cloud, /models and every POST) still need a token. The identity header grants none of them.

An example ExecStart that binds loopback and trusts the proxy header:

[Service]
ExecStart=/usr/local/bin/brigade fleet serve --host 127.0.0.1 --port 3774 --db /var/lib/brigade/fleet-hub.db --trust-tailscale-identity

Pair it with a Serve rule on the same host that forwards https://<HUB_TAILNET_FQDN> to http://127.0.0.1:3774. Never point the rule at a routable or remote backend address.

8. Back up the SQLite database

Use SQLite’s online backup. Do not copy only the main file while the service runs, because WAL data may not have reached it yet.

Create /usr/local/sbin/brigade-fleet-backup:

#!/bin/sh
set -eu

backup_dir=/var/backups/brigade
stamp=$(date -u +%Y%m%dT%H%M%SZ)
umask 077
install -d -o root -g root -m 0700 "$backup_dir"
sqlite3 /var/lib/brigade/fleet-hub.db ".backup '$backup_dir/fleet-hub-$stamp.db'"
chmod 0600 "$backup_dir/fleet-hub-$stamp.db"
find "$backup_dir" -type f -name 'fleet-hub-*.db' -mtime +14 -delete

Write the script to a temporary file, then install it:

install -o root -g root -m 0750 <BACKUP_SCRIPT_FILE> /usr/local/sbin/brigade-fleet-backup

Write /etc/systemd/system/brigade-fleet-backup.service:

[Unit]
Description=Back up the Brigade fleet hub SQLite database
After=brigade-fleet.service

[Service]
Type=oneshot
UMask=0077
ExecStart=/usr/local/sbin/brigade-fleet-backup

Write /etc/systemd/system/brigade-fleet-backup.timer:

[Unit]
Description=Daily Brigade fleet hub SQLite backup

[Timer]
OnCalendar=*-*-* 03:15:00
Persistent=true
Unit=brigade-fleet-backup.service

[Install]
WantedBy=timers.target

Enable the timer and run it once:

systemctl daemon-reload
systemctl enable --now brigade-fleet-backup.timer
systemctl start brigade-fleet-backup.service
systemctl list-timers brigade-fleet-backup.timer --no-pager
ls -lh /var/backups/brigade/

Copy backups off the CT with your existing backup system. The local timer is not a substitute for an off-host copy.

If the backup directory already exists from an earlier setup, stop the timer, confirm the path is the dedicated backup directory, then tighten it and the existing backups:

systemctl stop brigade-fleet-backup.timer
test -d /var/backups/brigade && test ! -L /var/backups/brigade
chown root:root /var/backups/brigade
chmod 0700 /var/backups/brigade
find /var/backups/brigade -type f -name 'fleet-hub-*.db' -exec chmod 0600 {} +
systemctl start brigade-fleet-backup.timer

Upgrade and rollback

Before you change the Brigade ref, take a backup and note the current ref:

systemctl start brigade-fleet-backup.service
pipx list
systemctl status --no-pager brigade-fleet.service

Upgrade to a reviewed ref:

systemctl stop brigade-fleet.service
PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin \
  pipx install --force \
  "git+https://github.com/escoffier-labs/brigade.git@<NEW_BRIGADE_GIT_REF>"
systemctl start brigade-fleet.service
systemctl status --no-pager brigade-fleet.service
curl --fail --silent --show-error https://<HUB_TAILNET_FQDN>/health

Upgrade the hub before the clients. To roll the application back to a known-good ref, run the same sequence with that ref:

systemctl stop brigade-fleet.service
PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin \
  pipx install --force \
  "git+https://github.com/escoffier-labs/brigade.git@<KNOWN_GOOD_BRIGADE_GIT_REF>"
systemctl start brigade-fleet.service
journalctl -u brigade-fleet.service -n 50 --no-pager

Keep the SQLite file during an application rollback. A hub refuses a database whose PRAGMA user_version is newer than it knows. If an upgrade migrated the database and the older ref refuses it, restore the pre-upgrade backup from step 8, which loses the events and claims received since, or stay on the newer ref.

Move off a direct Tailnet HTTP endpoint

If you built the hub from an earlier version of this runbook with a direct Tailnet HTTP endpoint, move it before you upgrade clients. Stop the hub, confirm /var/lib/brigade is the dedicated data directory and not a symlink, and tighten its ownership and mode:

systemctl stop brigade-fleet.service
test -d /var/lib/brigade && test ! -L /var/lib/brigade
chown root:root /var/lib/brigade
chmod 0700 /var/lib/brigade
stat -c '%U:%G %a %n' /var/lib/brigade

Replace the service unit with the loopback-only ExecStart from step 5, then restart the hub and enable Tailscale Serve:

systemctl daemon-reload
systemctl start brigade-fleet.service
tailscale serve --bg 3774
tailscale serve status
curl --fail --silent --show-error https://<HUB_TAILNET_FQDN>/health

Then set each client to https://<HUB_TAILNET_FQDN>. Offline events and messages wait in local spools until the new URL is set.

Troubleshooting

POST /events or POST /claims returns HTTP 403

The hub accepted the token but it may not act as the node_id in the body. Either the client sent the admin token (no node_token_file is set, and the client logged a shared-token WARNING) to a hub without --allow-admin-writes, or the node token on the machine was minted for a different node_id than brigadestack node --machine --json reports there. Compare that output with brigadestack fleet nodes list on the operator machine, then re-enroll (step 7). Events made in the meantime stay in the client spool and go out on the next brigadestack fleet flush.

GET /status or POST /events returns HTTP 401

The request reached the hub, but the bearer token matched neither the admin token nor an enrolled, unrevoked node token. A revoked node token answers unauthorized: node token revoked, so re-enroll it (step 7). Otherwise check that the client points at the intended token file, and that /etc/brigade/fleet-hub.env holds the token you meant. After you change that file, restart the service:

systemctl restart brigade-fleet.service
systemctl show brigade-fleet.service --property=EnvironmentFiles --no-pager

Do not print the token to logs or compare it in shell history. A queued event can wait in the client spool while you fix the token, then go out with:

brigadestack fleet flush

Tailscale is down or the hub name does not resolve

Check the daemon, address, route and MagicDNS path:

systemctl status --no-pager tailscaled.service
tailscale status
tailscale ip -4
getent hosts brigade-hub
tailscale ping brigade-hub

If the daemon is stopped, restart it and check the hub again:

systemctl restart tailscaled.service
systemctl restart brigade-fleet.service

If the CT has no /dev/net/tun, apply the host-side TUN setting from step 1. Do not enable nesting as a workaround.

SQLite reports database is locked

The hub uses WAL mode, a 5-second SQLite busy timeout and a 10-second connection timeout. Look for competing writers and read the service log:

journalctl -u brigade-fleet.service --since '-15 min' --no-pager
fuser -v /var/lib/brigade/fleet-hub.db
sqlite3 /var/lib/brigade/fleet-hub.db 'PRAGMA integrity_check;'

Only the hub should write the database. Stop the service before any offline repair or manual replacement, and never delete the -wal or -shm files while it runs:

systemctl stop brigade-fleet.service
sqlite3 /var/lib/brigade/fleet-hub.db 'PRAGMA integrity_check;'
systemctl start brigade-fleet.service

If lock errors continue, check that the backup script uses .backup and not a second long-running writer, then look at the CT’s disk and memory pressure.

More in the GitHub runbook

These topics are for operators running a larger fleet. They stay in the GitHub runbook and are not repeated here: