# Run the backend on Proxmox

This guide builds the backend as an unprivileged Debian LXC on Proxmox VE 9.x. The backend tracks runs, claims and messages across your machines and serves the dashboard, and the commands call it Fleet Hub, so this page says "the hub" from here on. The hub is a Brigade service listening on loopback only, and Tailscale Serve is the only way in. It keeps its live view in one SQLite database. Local journals stay authoritative, so an outage delays the aggregate view and makes clients spool events until it returns. For what the hub is and when to use it, read [Run the backend](/docs/guides/fleet-hub) first. The `ExecStart` lines here run the default engine, which has no shared memory. Shared memory needs extra flags: see [Turn on shared memory](/docs/guides/fleet-hub#turn-on-shared-memory).

> Prompt for your coding agent:
>
> Help me run the BrigadeStack backend so my machines share claims, runs and memory.
>
> Follow https://brigadestack.com/docs/guides/fleet-hub-proxmox one step at a time. The CT runs on my Proxmox host; ask me for the host and storage names instead of guessing them.
>
> Tell me what each step changes and on which machine. Keep the backend on a private network such as Tailscale. Never publish it to the internet or behind a public tunnel.
>
> Create tokens on the hub host and store them in files only the service user can read. Never ask me to paste a token into this chat, and never put one in a command line or source file.
>
> When the backend is up, run the health and status checks from the guide, then enroll this machine as a node and confirm it shows up.

Run `tailscale serve status` to print the hub's HTTPS tailnet name. This guide calls it `<HUB_TAILNET_FQDN>`. Replace every other angle-bracket value before you run a command.

## Target layout

- Proxmox VE 9.x host.
- A Debian 12 or Debian 13 CT template.
- 1 vCPU, 512 MB RAM, 4 GB root disk.
- An unprivileged CT with nesting disabled.
- Tailscale inside the CT, hostname `brigade-hub`.
- The hub bound to `127.0.0.1:3774`, behind Tailscale Serve HTTPS.
- The SQLite database at `/var/lib/brigade/fleet-hub.db`.
- A root-owned, mode `0600` systemd environment file at `/etc/brigade/fleet-hub.env`.

The hub exposes these endpoints:

| Endpoint | Auth | Purpose |
|---|---|---|
| `GET /health` | none | Liveness check |
| `POST /events` | node token | Append an event. The admin token works only with `--allow-admin-writes`. |
| `GET /status` | admin or node token | Latest non-terminal state per node and run. Add `?all=1` to include finished runs. |
| `POST /claims`, `GET /claims` | node token to post, admin or node token to read | Repository claims |
| `GET /nodes`, `POST /nodes` | admin token only | Node enrollment |
| `GET /`, `GET /view/machines`, `GET /view/repos` | admin token or dashboard cookie | The Fleet dashboard |
| `GET /deck/roster`, `POST /deck/roster` | admin Authorization header to save | The roster page |

The hub drops a repeated event, matched on `(node_id, run_id, sequence, digest)`, so reposting after a lost response is safe.

Two bearer credentials exist. The **admin token** in `/etc/brigade/fleet-hub.env` is the control plane: it enrolls nodes, reads status and claims, and mints dashboard enrollment codes. Each machine's **node token**, created by `brigadestack fleet nodes add`, is that machine's identity. The hub derives `node_id` from it and refuses an event or claim for any other node with HTTP 403. The hub stores only a SHA-256 of each node token.

## 1. Create the CT on the Proxmox host

Download a Debian 12 or 13 standard CT template into the template storage first, and confirm the exact filename and storage names.

Run as root on the Proxmox host:

```bash
pct create <CTID> <TEMPLATE_STORAGE>:vztmpl/<DEBIAN_TEMPLATE>.tar.zst \
  --hostname brigade-hub \
  --ostype debian \
  --cores 1 \
  --memory 512 \
  --swap 512 \
  --rootfs <ROOT_STORAGE>:4 \
  --unprivileged 1 \
  --features nesting=0 \
  --net0 name=eth0,bridge=<PROXMOX_BRIDGE>,ip=<CT_IPV4/CIDR>,gw=<GATEWAY_IPV4>,type=veth \
  --onboot 1 \
  --start 0 \
  --ssh-public-keys <PUBLIC_KEY_FILE>

pct start <CTID>
pct enter <CTID>
```

The CT needs outbound access to the Tailscale package repository and the tailnet. Do not enable nesting for this service.

Tailscale normally needs `/dev/net/tun`. Check from inside the CT:

```bash
test -c /dev/net/tun && echo "TUN device present" || echo "TUN device missing"
```

If the device is missing, leave nesting disabled and add only the TUN device permission on the Proxmox host:

```bash
printf '%s\n' \
  'lxc.cgroup2.devices.allow: c 10:200 rwm' \
  'lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file' \
  >> /etc/pve/lxc/<CTID>.conf
pct restart <CTID>
pct enter <CTID>
```

## 2. Install Tailscale in the CT

Run as root inside the CT. The repository path comes from Debian's codename, so the same commands work on Debian 12 and 13.

```bash
apt-get update
apt-get install -y ca-certificates curl gnupg

. /etc/os-release
curl -fsSL "https://pkgs.tailscale.com/stable/debian/${VERSION_CODENAME}.noarmor.gpg" \
  -o /usr/share/keyrings/tailscale-archive-keyring.gpg
curl -fsSL "https://pkgs.tailscale.com/stable/debian/${VERSION_CODENAME}.tailscale-keyring.list" \
  -o /etc/apt/sources.list.d/tailscale.list

apt-get update
apt-get install -y tailscale
systemctl enable --now tailscaled
tailscale up --authkey=tskey-auth-<PREAUTH_KEY> --hostname=brigade-hub --accept-dns=true
tailscale status
tailscale ip -4
```

The pre-auth key must be reusable, or have enough uses for the rebuilds you expect. Type it in the CT's own shell. Do not put it in a repository, a chat, or a systemd unit. If you would rather not use a key, leave out `--authkey=...`: `tailscale up` then prints a login URL to approve in a browser.

The IPv4 address only shows that Tailscale is connected. The hub itself binds to loopback.

Confirm the MagicDNS name resolves from the CT and from one client machine:

```bash
getent hosts brigade-hub
tailscale ping brigade-hub
```

## 3. Install Python and Brigade

Debian 12 ships Python 3.11 and Debian 13 ships a newer one. Confirm the interpreter before you install the CLI.

```bash
python3 --version
apt-get install -y python3 python3-venv pipx sqlite3 curl
install -d -o root -g root -m 0755 /opt/pipx /usr/local/bin
python3 -c 'import sys; print(sys.version); raise SystemExit("Python 3.11+ required") if sys.version_info < (3, 11) else None'

PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin \
  pipx install --force \
  "git+https://github.com/escoffier-labs/brigade.git@<BRIGADE_GIT_REF>"

/usr/local/bin/brigade --version
```

Set `<BRIGADE_GIT_REF>` to a reviewed tag or commit, and use the same ref on the hub and on every client while you roll out a new build. This install creates the `brigade` entry point. It is the same CLI as `brigadestack`, and the service below runs `/usr/local/bin/brigade`.

## 4. Create the admin token file

The service reads the admin token from `BRIGADE_FLEET_TOKEN` in `/etc/brigade/fleet-hub.env`. Run this in the CT as root. It generates the token on the hub host and writes it straight to the file without printing it:

```bash
install -d -o root -g root -m 0750 /etc/brigade
install -d -o root -g root -m 0700 /var/lib/brigade
umask 077
generated_token="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')" && \
  [ -n "$generated_token" ] && \
  printf 'BRIGADE_FLEET_TOKEN=%s\n' "$generated_token" > /etc/brigade/fleet-hub.env
unset generated_token
chown root:root /etc/brigade/fleet-hub.env
chmod 0600 /etc/brigade/fleet-hub.env
```

If you keep the token in 1Password, read it into the file instead. This expects `op` to be signed in for the vault, with the item field named `password`:

```bash
install -d -o root -g root -m 0750 /etc/brigade
install -d -o root -g root -m 0700 /var/lib/brigade
umask 077
fleet_token="$(op read 'op://<VAULT>/brigade-fleet-hub/password')" && \
  printf 'BRIGADE_FLEET_TOKEN=%s\n' "$fleet_token" > /etc/brigade/fleet-hub.env
unset fleet_token
chown root:root /etc/brigade/fleet-hub.env
chmod 0600 /etc/brigade/fleet-hub.env
```

Check the file metadata, not its contents:

```bash
stat -c '%U:%G %a %n' /etc/brigade/fleet-hub.env
```

The output should include `root:root 600`. The hub does not persist the admin token in SQLite. It keeps node tokens only as SHA-256 digests.

The operator machine that enrolls nodes needs the same admin token in `~/.brigade/fleet.token`, mode `0600`. Move it with your usual secret channel and never through chat. One option, run on the operator machine, streams it over SSH without printing it:

```bash
umask 077
mkdir -p ~/.brigade
ssh root@<HUB_SSH_HOST> "sed -n 's/^BRIGADE_FLEET_TOKEN=//p' /etc/brigade/fleet-hub.env" > ~/.brigade/fleet.token
```

## 5. Install and start the service

Write `/etc/systemd/system/brigade-fleet.service`:

```ini
[Unit]
Description=Brigade fleet hub
After=network-online.target tailscaled.service
Wants=network-online.target
Requires=tailscaled.service

[Service]
Type=simple
User=root
WorkingDirectory=/var/lib/brigade
EnvironmentFile=/etc/brigade/fleet-hub.env
ExecStart=/usr/local/bin/brigade fleet serve --host 127.0.0.1 --port 3774 --db /var/lib/brigade/fleet-hub.db --trust-forwarded-proto
Restart=on-failure
RestartSec=5
PrivateTmp=true
NoNewPrivileges=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/brigade

[Install]
WantedBy=multi-user.target
```

`--host` is required. Never bind to all interfaces or a routable address: the hub must listen on loopback so Tailscale Serve is its only network frontend. The explicit `--db` path keeps the database under `/var/lib/brigade`, away from any user's home directory. That directory must be owned by root with mode `0700`.

A new hub requires a node token for every write. Add `--allow-admin-writes` only to keep clients that still share one token working during a migration.

Start it and put Tailscale Serve in front:

```bash
systemctl daemon-reload
systemctl enable --now brigade-fleet.service
systemctl status --no-pager brigade-fleet.service
journalctl -u brigade-fleet.service -n 50 --no-pager
tailscale serve --bg 3774
tailscale serve status
```

Tailscale Serve provides the HTTPS MagicDNS origin `https://<HUB_TAILNET_FQDN>` and proxies it to the local service. The startup log should name `127.0.0.1:3774` and `/var/lib/brigade/fleet-hub.db`. Keep the Serve proxy on this CT. Do not point it at the Tailscale interface or another host.

## 6. Verify health and authenticated status

Health needs no token:

```bash
curl --fail --silent --show-error https://<HUB_TAILNET_FQDN>/health
```

Expected response:

```json
{"ok": true, "service": "brigade-fleet-hub"}
```

From the operator machine, check the protected status endpoint with the admin token:

```bash
curl --fail --silent --show-error \
  -H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
  https://<HUB_TAILNET_FQDN>/status
```

An empty result is valid before any client reports. HTTP 401 means the request reached the hub but the token did not match.

For a direct endpoint test, enroll a throwaway node and post one synthetic event with its token. Use a node and run id nobody could confuse with a real run, and revoke the node afterward:

```bash
smoke_token="$(curl --fail --silent --show-error \
  -X POST \
  -H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
  -H 'Content-Type: application/json' \
  --data '{"action":"add","node_id":"smoke-<NODE_ID>","label":"smoke test"}' \
  https://<HUB_TAILNET_FQDN>/nodes | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')"

curl --fail --silent --show-error \
  -X POST \
  -H "Authorization: Bearer $smoke_token" \
  -H 'Content-Type: application/json' \
  --data '{"node_id":"smoke-<NODE_ID>","run_id":"smoke-<RUN_ID>","repo":"<REPO_NAME>","seat":"<SEAT>","harness":"<HARNESS>","state":"run.started","ts":"<ISO8601_UTC>","sequence":1,"digest":"<EVENT_DIGEST>"}' \
  https://<HUB_TAILNET_FQDN>/events

curl --fail --silent --show-error \
  -H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
  https://<HUB_TAILNET_FQDN>/status

curl --fail --silent --show-error \
  -X POST \
  -H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
  -H 'Content-Type: application/json' \
  --data '{"action":"revoke","node_id":"smoke-<NODE_ID>"}' \
  https://<HUB_TAILNET_FQDN>/nodes
unset smoke_token
```

The first post to `/events` reports `accepted: 1`. Posting the same body again reports a duplicate, which shows the idempotency key works. Posting the same event with the admin token instead answers HTTP 403 on a hub without `--allow-admin-writes`, which shows node tokens are required.

## 7. Enroll and configure each client

Every machine needs its own node token. On the machine, print its identity. `brigadestack node --machine` creates it once under `BRIGADE_HOME` (`~/.brigade/node.toml` by default). Every event and claim is stamped with this identity, never a workspace-local one.

```bash
brigadestack node --machine --json | python3 -c 'import json,sys; print(json.load(sys.stdin)["node_id"])'
```

From the operator machine that holds the admin token (`[fleet] token_file` in its own `~/.brigade/fleet.toml`, or `BRIGADE_FLEET_TOKEN`), enroll that identity. The token prints once and never again, because the hub keeps only its hash:

```bash
brigadestack fleet nodes add <NODE_ID> --label <HOSTNAME>
brigadestack fleet nodes list
```

Deliver the printed token to the machine through your normal secret-distribution process and install it as a file only that user can read, mode `0600`. The admin token stays on the operator machine. A node does not need it.

```bash
mkdir -p ~/.brigade
install -m 0600 <NODE_TOKEN_FILE> ~/.brigade/fleet-node.token
chmod 0700 ~/.brigade

cat > ~/.brigade/fleet.toml <<'EOF'
[fleet]
hub_url = "https://<HUB_TAILNET_FQDN>"
node_token_file = "~/.brigade/fleet-node.token"
EOF

chmod 0600 ~/.brigade/fleet.toml
```

The config keys are `[fleet] hub_url`, `[fleet] node_token_file` (this machine's identity), and optionally `[fleet] token_file` (the admin token, only where `brigadestack fleet nodes` runs). `BRIGADE_FLEET_HUB_URL` overrides `hub_url`, `BRIGADE_FLEET_NODE_TOKEN` overrides the node token file, and `BRIGADE_FLEET_TOKEN` overrides the admin token file. Do not add a `host`, `port` or `token` key to this file.

To rotate a machine's credential, revoke and re-enroll it, then replace the file on that machine:

```bash
brigadestack fleet nodes revoke <NODE_ID>
brigadestack fleet nodes add <NODE_ID> --label <HOSTNAME>
```

The hub refuses a revoked token with HTTP 401 at once. Events the machine produces in between stay in its local spool and go out on the next report or `brigadestack fleet flush`, once the new token is in place.

### Windows client

Use the same `[fleet]` keys on Windows. The client expands `~` in `node_token_file`, so the same config works in a native Windows user profile:

```powershell
New-Item -ItemType Directory -Force "$env:USERPROFILE\.brigade" | Out-Null
Copy-Item <NODE_TOKEN_FILE> "$env:USERPROFILE\.brigade\fleet-node.token"
@'
[fleet]
hub_url = "https://<HUB_TAILNET_FQDN>"
node_token_file = "~/.brigade/fleet-node.token"
'@ | Set-Content -Encoding utf8 "$env:USERPROFILE\.brigade\fleet.toml"
icacls "$env:USERPROFILE\.brigade\fleet-node.token" /inheritance:r /grant:r "$env:USERNAME:(R)" | Out-Null
brigadestack fleet status
```

Keep the token file readable only by the Windows user who runs Brigade. `brigadestack fleet flush` is the manual retry command on Windows too.

### Check a client

Once a machine has a node identity and an active run, inspect the hub from it:

```bash
brigadestack fleet status
brigadestack fleet status --json
```

`fleet status` queries `GET /status` and shows node, repository, run id, seat and harness, state, and age. It hides finished runs unless you add `--all`. `brigadestack fleet flush` retries the local spool at once:

```bash
brigadestack fleet flush
```

Reporting is best effort. A failed POST is appended under the Brigade home directory at `fleet-spool/<node_id>.jsonl`. It never blocks the journal writer or fails the local run. A later report or `brigadestack fleet flush` sends the queue in order.

### Dashboard from a laptop or phone

The hub serves a read-only web view of the same data at `https://<HUB_TAILNET_FQDN>/` (machine cards) and `https://<HUB_TAILNET_FQDN>/view/repos` (repository board). It accepts the admin token, never a node token. From a machine that holds the token, a bearer header works as it does for `/status`:

```bash
curl --silent --show-error \
  -H "Authorization: Bearer $(<~/.brigade/fleet.token)" \
  https://<HUB_TAILNET_FQDN>/ | head
```

Sort and filter with query parameters, for example `/?attention=1` (only failed, awaiting-approval or stale runs), `/view/repos?sort=repo`, `/?node=<prefix>`, or `/?all=1` (include finished runs). The page refreshes every 10 seconds and works with JavaScript off.

To open it on a phone, the browser must reach the hub over HTTPS. The unit above already does this: the hub is bound to numeric loopback with `--trust-forwarded-proto`, and Tailscale Serve preserves the browser-facing Host and sets `X-Forwarded-Proto` to a single `https`. The hub trusts that header only from an immediate loopback peer and marks the cookie Secure.

Do not point a client at the direct backend URL `http://<TAILSCALE_IPV4>:3774`. Clients refuse remote cleartext hub URLs before they attach a bearer credential. Set every client's `hub_url` to `https://<HUB_TAILNET_FQDN>`.

On the admin machine, mint a private, read-only enrollment URL that lasts five minutes:

```bash
BRIGADE_FLEET_HUB_URL=https://<HUB_TAILNET_FQDN> brigadestack fleet enroll --label "phone" --base-url https://<HUB_TAILNET_FQDN>
```

Open the printed URL on the device and confirm the form. Opening the page does not use up the code. Submitting the form does, once, and redirects to `/deck` with a host-only cookie that lasts 30 days. That cookie reads HTML only. It cannot write the roster or policy, and it cannot read JSON. Each device enrolls on its own.

For a lost device, list the browser sessions, find its label, and revoke each session id you do not want:

```bash
BRIGADE_FLEET_HUB_URL=https://<HUB_TAILNET_FQDN> brigadestack fleet sessions --dashboard
BRIGADE_FLEET_HUB_URL=https://<HUB_TAILNET_FQDN> brigadestack fleet sessions --dashboard --revoke SESSION_ID
```

Follow the next-page cursor when a listing passes 100 sessions, and list again from page one during an incident to catch new sessions. `--all` also shows records kept after access expires or a session is revoked. Rotating the admin token does not revoke browser sessions, so revoke each one. These calls need the current admin token. `brigadestack fleet sessions` with no flag still reports interactive editor presence.

### Tailscale Serve identity authentication (optional)

The hub can trust the `Tailscale-User-Login` header that Tailscale Serve adds, and let dashboard routes (`/`, `/deck`, `/deck/repos` and the older `/view/*` boards) through without the admin token or a cookie. Start it with `--trust-tailscale-identity`.

Turn this on only when all three are true:

1. The hub is bound to loopback (`127.0.0.1`) and nothing else. Never use all interfaces or a routable address with this flag.
2. Dashboard requests reach the hub only through a Tailscale Serve proxy on the same host, which terminates Tailscale identity and strips any spoofed `Tailscale-User-Login` header. The immediate TCP peer is then loopback.
3. The backend is never exposed to the tailnet or any other network without the proxy in front.

Every process inside the hub host or container is trusted in this mode, because any local process can reach the loopback backend and set the header. Run the hub in a dedicated host or container, and skip the flag when untrusted workloads share its network namespace.

The hub accepts a dashboard request only when the immediate peer is loopback and the header is present, non-empty, at most 256 characters, and free of control characters. It never logs or renders the identity value. JSON and API routes (`/status`, `/claims`, `/nodes`, `/cloud`, `/models` and every `POST`) still need a token. The identity header grants none of them.

An example `ExecStart` that binds loopback and trusts the proxy header:

```ini
[Service]
ExecStart=/usr/local/bin/brigade fleet serve --host 127.0.0.1 --port 3774 --db /var/lib/brigade/fleet-hub.db --trust-tailscale-identity
```

Pair it with a Serve rule on the same host that forwards `https://<HUB_TAILNET_FQDN>` to `http://127.0.0.1:3774`. Never point the rule at a routable or remote backend address.

## 8. Back up the SQLite database

Use SQLite's online backup. Do not copy only the main file while the service runs, because WAL data may not have reached it yet.

Create `/usr/local/sbin/brigade-fleet-backup`:

```sh
#!/bin/sh
set -eu

backup_dir=/var/backups/brigade
stamp=$(date -u +%Y%m%dT%H%M%SZ)
umask 077
install -d -o root -g root -m 0700 "$backup_dir"
sqlite3 /var/lib/brigade/fleet-hub.db ".backup '$backup_dir/fleet-hub-$stamp.db'"
chmod 0600 "$backup_dir/fleet-hub-$stamp.db"
find "$backup_dir" -type f -name 'fleet-hub-*.db' -mtime +14 -delete
```

Write the script to a temporary file, then install it:

```bash
install -o root -g root -m 0750 <BACKUP_SCRIPT_FILE> /usr/local/sbin/brigade-fleet-backup
```

Write `/etc/systemd/system/brigade-fleet-backup.service`:

```ini
[Unit]
Description=Back up the Brigade fleet hub SQLite database
After=brigade-fleet.service

[Service]
Type=oneshot
UMask=0077
ExecStart=/usr/local/sbin/brigade-fleet-backup
```

Write `/etc/systemd/system/brigade-fleet-backup.timer`:

```ini
[Unit]
Description=Daily Brigade fleet hub SQLite backup

[Timer]
OnCalendar=*-*-* 03:15:00
Persistent=true
Unit=brigade-fleet-backup.service

[Install]
WantedBy=timers.target
```

Enable the timer and run it once:

```bash
systemctl daemon-reload
systemctl enable --now brigade-fleet-backup.timer
systemctl start brigade-fleet-backup.service
systemctl list-timers brigade-fleet-backup.timer --no-pager
ls -lh /var/backups/brigade/
```

Copy backups off the CT with your existing backup system. The local timer is not a substitute for an off-host copy.

If the backup directory already exists from an earlier setup, stop the timer, confirm the path is the dedicated backup directory, then tighten it and the existing backups:

```bash
systemctl stop brigade-fleet-backup.timer
test -d /var/backups/brigade && test ! -L /var/backups/brigade
chown root:root /var/backups/brigade
chmod 0700 /var/backups/brigade
find /var/backups/brigade -type f -name 'fleet-hub-*.db' -exec chmod 0600 {} +
systemctl start brigade-fleet-backup.timer
```

## Upgrade and rollback

Before you change the Brigade ref, take a backup and note the current ref:

```bash
systemctl start brigade-fleet-backup.service
pipx list
systemctl status --no-pager brigade-fleet.service
```

Upgrade to a reviewed ref:

```bash
systemctl stop brigade-fleet.service
PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin \
  pipx install --force \
  "git+https://github.com/escoffier-labs/brigade.git@<NEW_BRIGADE_GIT_REF>"
systemctl start brigade-fleet.service
systemctl status --no-pager brigade-fleet.service
curl --fail --silent --show-error https://<HUB_TAILNET_FQDN>/health
```

Upgrade the hub before the clients. To roll the application back to a known-good ref, run the same sequence with that ref:

```bash
systemctl stop brigade-fleet.service
PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin \
  pipx install --force \
  "git+https://github.com/escoffier-labs/brigade.git@<KNOWN_GOOD_BRIGADE_GIT_REF>"
systemctl start brigade-fleet.service
journalctl -u brigade-fleet.service -n 50 --no-pager
```

Keep the SQLite file during an application rollback. A hub refuses a database whose `PRAGMA user_version` is newer than it knows. If an upgrade migrated the database and the older ref refuses it, restore the pre-upgrade backup from step 8, which loses the events and claims received since, or stay on the newer ref.

### Move off a direct Tailnet HTTP endpoint

If you built the hub from an earlier version of this runbook with a direct Tailnet HTTP endpoint, move it before you upgrade clients. Stop the hub, confirm `/var/lib/brigade` is the dedicated data directory and not a symlink, and tighten its ownership and mode:

```bash
systemctl stop brigade-fleet.service
test -d /var/lib/brigade && test ! -L /var/lib/brigade
chown root:root /var/lib/brigade
chmod 0700 /var/lib/brigade
stat -c '%U:%G %a %n' /var/lib/brigade
```

Replace the service unit with the loopback-only `ExecStart` from step 5, then restart the hub and enable Tailscale Serve:

```bash
systemctl daemon-reload
systemctl start brigade-fleet.service
tailscale serve --bg 3774
tailscale serve status
curl --fail --silent --show-error https://<HUB_TAILNET_FQDN>/health
```

Then set each client to `https://<HUB_TAILNET_FQDN>`. Offline events and messages wait in local spools until the new URL is set.

## Troubleshooting

### `POST /events` or `POST /claims` returns HTTP 403

The hub accepted the token but it may not act as the `node_id` in the body. Either the client sent the admin token (no `node_token_file` is set, and the client logged a shared-token `WARNING`) to a hub without `--allow-admin-writes`, or the node token on the machine was minted for a different `node_id` than `brigadestack node --machine --json` reports there. Compare that output with `brigadestack fleet nodes list` on the operator machine, then re-enroll (step 7). Events made in the meantime stay in the client spool and go out on the next `brigadestack fleet flush`.

### `GET /status` or `POST /events` returns HTTP 401

The request reached the hub, but the bearer token matched neither the admin token nor an enrolled, unrevoked node token. A revoked node token answers `unauthorized: node token revoked`, so re-enroll it (step 7). Otherwise check that the client points at the intended token file, and that `/etc/brigade/fleet-hub.env` holds the token you meant. After you change that file, restart the service:

```bash
systemctl restart brigade-fleet.service
systemctl show brigade-fleet.service --property=EnvironmentFiles --no-pager
```

Do not print the token to logs or compare it in shell history. A queued event can wait in the client spool while you fix the token, then go out with:

```bash
brigadestack fleet flush
```

### Tailscale is down or the hub name does not resolve

Check the daemon, address, route and MagicDNS path:

```bash
systemctl status --no-pager tailscaled.service
tailscale status
tailscale ip -4
getent hosts brigade-hub
tailscale ping brigade-hub
```

If the daemon is stopped, restart it and check the hub again:

```bash
systemctl restart tailscaled.service
systemctl restart brigade-fleet.service
```

If the CT has no `/dev/net/tun`, apply the host-side TUN setting from step 1. Do not enable nesting as a workaround.

### SQLite reports `database is locked`

The hub uses WAL mode, a 5-second SQLite busy timeout and a 10-second connection timeout. Look for competing writers and read the service log:

```bash
journalctl -u brigade-fleet.service --since '-15 min' --no-pager
fuser -v /var/lib/brigade/fleet-hub.db
sqlite3 /var/lib/brigade/fleet-hub.db 'PRAGMA integrity_check;'
```

Only the hub should write the database. Stop the service before any offline repair or manual replacement, and never delete the `-wal` or `-shm` files while it runs:

```bash
systemctl stop brigade-fleet.service
sqlite3 /var/lib/brigade/fleet-hub.db 'PRAGMA integrity_check;'
systemctl start brigade-fleet.service
```

If lock errors continue, check that the backup script uses `.backup` and not a second long-running writer, then look at the CT's disk and memory pressure.

## More in the GitHub runbook

These topics are for operators running a larger fleet. They stay in the [GitHub runbook](https://github.com/escoffier-labs/brigade/blob/main/docs/runbooks/fleet-hub-proxmox.md) and are not repeated here:

- [External agent principals](https://github.com/escoffier-labs/brigade/blob/main/docs/runbooks/fleet-hub-proxmox.md#external-agent-principals)
- [Fleet MCP component](https://github.com/escoffier-labs/brigade/blob/main/docs/runbooks/fleet-hub-proxmox.md#fleet-mcp-station)
- [Operator questions](https://github.com/escoffier-labs/brigade/blob/main/docs/runbooks/fleet-hub-proxmox.md#operator-questions)
- [Migration from the shared token](https://github.com/escoffier-labs/brigade/blob/main/docs/runbooks/fleet-hub-proxmox.md#migration-from-the-shared-token)
- [Compact the event journal](https://github.com/escoffier-labs/brigade/blob/main/docs/runbooks/fleet-hub-proxmox.md#9-compact-the-event-journal-optional)