diff --git a/.claude/skills/xce-deploy-ha/SKILL.md b/.claude/skills/xce-deploy-ha/SKILL.md index 5b610212..65540cf2 100644 --- a/.claude/skills/xce-deploy-ha/SKILL.md +++ b/.claude/skills/xce-deploy-ha/SKILL.md @@ -43,6 +43,10 @@ disown`) before relying on subagent dispatch for unattended use. `deploy-ha.sh` never runs this itself; without it, Step 3 simply skips generating `scripts/start-apiserver.sh` (soft-skip, not an error), same as the existing worker/scheduler convention. +- Run `/xce-build worker` so both worker binaries exist in the same build + dir as `$BINARY`: `xcecd_virtual` (default) and `xcecd` (the real, + root-gated production daemon). Step 3 auto-picks which one + `scripts/start-worker.sh` targets — see Step 4b. No data directory needed: TLS cert/key and `cluster.bin` are both generated directly into the deploy dir (`$TOP`) by `deploy-ha.sh` itself, not read from @@ -101,6 +105,7 @@ to the intended interpreter). kill -9 $(pgrep xce_controller) 2>/dev/null || true kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true kill -9 $(pgrep xce_scheduler) 2>/dev/null || true +sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root — plain kill can't touch it ``` ## Step 3 — Deploy cluster @@ -143,10 +148,14 @@ This: (standbys receive config from primary via `OPCode_replConfigPush`) - Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`) and `cleanup.sh` - Also generates `$TOP/scripts/start-worker.sh` and `$TOP/scripts/start-scheduler.sh` - **if** `xcecd_virtual`/`xce_scheduler` exist alongside `$BINARY` (same - build dir) — soft-skipped with a warning, not a hard error, if either is - missing (so `/xce-test-ha`, which never builds worker/scheduler and calls - this same script, is unaffected). See Step 4b. + **if** the relevant binary exists alongside `$BINARY` (same build dir) — + soft-skipped with a warning, not a hard error, if missing (so + `/xce-test-ha`, which never builds worker/scheduler and calls this same + script, is unaffected). Which worker binary — `xcecd_virtual` or the real + `xcecd` — `start-worker.sh` targets is **auto-detected** per deploy + (`/run/systemd/system` present → real `xcecd`; absent, e.g. this sandbox → + `xcecd_virtual`), overridable with `XCE_REAL_WORKER=0|1` before Step 3. See + Step 4b. - Also extracts a standalone xceadmin into `$TOP/xceadmin-site` and generates `$TOP/scripts/xceadmin` **if** a wheel exists at `/xceadmin/dist/*.whl` — same soft-skip-with-warning @@ -189,19 +198,67 @@ so there is nothing to keep in sync by hand: "$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v & # scheduler is a cluster singleton, no name ``` -- **`start-worker.sh`** runs `xcecd_virtual`, not production `xcecd` — it - skips the root-only sqlite/db-recovery and domain-socket subsystems, so it - runs as an ordinary user with no `sudo` (this session/skill cannot `sudo` - anyway). Logs land at `$TOP/logs/xcecd-.log`. Start more than one by - passing a different name: `start-worker.sh worker-2`. To point at a real - `xcecd` instead (needs root — run it yourself outside this skill), set - `XCE_WORKER_BIN=/path/to/xcecd` before invoking the script. +- **`start-worker.sh`** runs whichever binary Step 3 auto-detected — + `xcecd_virtual` (ordinary user, no `sudo`) in this sandbox, or the real, + production `xcecd` (wrapped in `sudo` by the generated script itself) on a + host where `/run/systemd/system` exists. See "Real worker" below for what + changes when it's the real binary. - **`start-scheduler.sh`** runs `xce_scheduler`; its identity is fixed internally to `xce_scheduler` (no name argument). Logs to `$TOP/logs/scheduler.log`. -- Verify a worker joined: `grep "Successfully sent .* load indices" "$TOP/logs/xcecd-worker-1.log"`. +- Verify a worker joined: `grep "Successfully sent .* load indices" "$TOP/logs/xcecd-worker-1.log"` + (virtual — logs are `xcecd-.log`) or + `sudo grep "Successfully sent .* load indices" "$TOP/logs/xcecd.log"` (real + — always just `xcecd.log`, not per-name, **and root-owned `640`, not + world-readable** — a plain `grep` as yourself gets `Permission denied`; + see below). - Verify the scheduler joined: `grep "sched_init: joined cluster" "$TOP/logs/scheduler.log"`. +### Real worker (production `xcecd`) + +Only relevant when Step 3 picked the real binary (`/run/systemd/system` +present, or `XCE_REAL_WORKER=1` forced it). Two things are structurally +different from `xcecd_virtual`, both because of how the production binary +itself is built, not this skill's choice: + +- **Single instance only.** `xcecd`'s `db_path` (`/var/run/xcecd.db`), + `domain_sock_file` (`/var/run/.xcecd.sock`), and log file + (`$TOP/logs/xcecd.log`) are all hardcoded in `worker/src/main.c` — unlike + `xcecd_virtual`, `-n`/the name argument does **not** change any of them + (it only sets the HA cluster identity used to join `--peers`). Only run + one real worker per host at a time; you cannot do a real-worker + equivalent of `start-worker.sh worker-2`. +- **Job dispatch needs a real systemd+dbus host.** Actually running a job + goes through `sd_bus_open_system()` → + `org.freedesktop.systemd1.Manager.StartTransientUnit` + (`worker/src/xcecd_cgroups.c`) — a live call to the system dbus, answered + by a real systemd manager. That's exactly the `/run/systemd/system` signal + Step 3 auto-detects on, so on a host that picked the real binary this + should just work; if you forced `XCE_REAL_WORKER=1` somewhere that signal + is absent (e.g. this sandbox), the daemon still starts, joins the + cluster, and exercises its root-gated sqlite db-init/recovery and domain + socket — but job *submission* will fail at the dispatch step regardless of + `sudo`. + +The generated `start-worker.sh` itself handles the rest — `sudo mkdir -p +/var/run` (one-time preflight; `xcecd` exits immediately with "Failed to +initialize database" if that directory doesn't exist, since neither +hardcoded path is a flag) and wrapping the daemon exec in `sudo`. Nothing +extra to do at call time beyond what's shown above, **provided `sudo` is +passwordless** (confirmed `sudo -n true` in this sandbox) — on a host where +it isn't, run this step interactively, not via the unattended `Agent`-tool +dispatch described at the top of this file (a password prompt there would +just hang). + +Verify it's running as root: `ps -o pid,user,cmd -C xcecd`. + +**Reading its log needs `sudo` too** — confirmed live: `$TOP/logs/xcecd.log` +comes out owned `root:root`, mode `640` (unlike `xcecd_worker.log` and the +domain socket, both of which the daemon explicitly `chmod`s `0666`; the main +log file gets no such treatment). A plain `grep`/`tail` against it as +yourself fails with `Permission denied` — use `sudo grep ...`/`sudo tail +...` instead, same as the `sudo kill` already needed to stop the process. + ## Step 4c — Use xceadmin against this deployment (optional) `deploy-ha.sh` also installs a **standalone** `xceadmin` at `$TOP/xceadmin-site` @@ -333,6 +390,7 @@ Stop all instances, clean up state, and restart without redeploying: kill -9 $(pgrep xce_controller) 2>/dev/null || true kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true kill -9 $(pgrep xce_scheduler) 2>/dev/null || true +sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root # 2. Run the generated cleanup script # Keeps: controller/ctrl-1/conf/* (primary config + current pointer), $TOP/certs/ @@ -361,6 +419,14 @@ sleep 0.5 - All `controller/ctrl-{2..N}/conf/` — standbys rebuild config from primary's `OPCode_replConfigPush` - All `logs/*.log` — clean log slate +**Not touched by cleanup.sh, if you ran the real worker:** `/var/run/xcecd.db` +and `/var/run/.xcecd.sock` live outside `$TOP` entirely (hardcoded, not +parameterized by this deploy), so they persist across `cleanup.sh` runs — +that's what lets the real worker's db-recovery path actually be exercised on +restart. Remove them yourself (`sudo rm -f /var/run/xcecd.db +/var/run/.xcecd.sock`) if you specifically want a from-scratch real-worker +state. + If you need a full redeploy (different peers or new binary), run steps 3–5 instead. ## Troubleshooting @@ -373,8 +439,14 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins | `election_timeout` loops without primary | All nodes have `config_tag=0` (cold-start) or network issue | Check peers string matches actual listen ports | | `ha-term.bin` causes "already voted" on restart | Old state from previous run | Run clean restart above | | `SSL_do_handshake: Broken pipe` flooding from a worker/scheduler | A typo'd hostname in the `--peers` you passed (e.g. `local` instead of `localhost`) — the connect reaches SSL before failing, so it doesn't error out cleanly | Diff the failing `--peers` string against `$TOP/scripts/start.sh`'s (they must be byte-identical); or just use `start-worker.sh`/`start-scheduler.sh`, which always carry the exact string the controllers were deployed with | -| `start-worker.sh`/`start-scheduler.sh` not generated | `xcecd_virtual`/`xce_scheduler` don't exist in `$BINARY`'s build dir (deploy-ha.sh logs a `Warning:`, not an error) | `/xce-build worker` and/or `/xce-build scheduler`, then redeploy (Step 3) | -| `xcecd should be run as root` | You substituted real `xcecd` via `XCE_WORKER_BIN` | Either run it yourself with `sudo` outside this skill, or drop back to the default `xcecd_virtual` (no `sudo` needed, skips root-only subsystems) | +| `start-worker.sh`/`start-scheduler.sh` not generated | Whichever worker binary auto-detection (or `XCE_REAL_WORKER`) selected — `xcecd_virtual` or `xcecd` — doesn't exist in `$BINARY`'s build dir, or `xce_scheduler` doesn't (deploy-ha.sh logs a `Warning:`, not an error) | `/xce-build worker` and/or `/xce-build scheduler`, then redeploy (Step 3) — `/xce-build worker` produces both `xcecd_virtual` and `xcecd` together | +| `xcecd should be run as root` | `XCE_WORKER_BIN` was pointed at real `xcecd` directly (bypassing the generated `start-worker.sh`, which already wraps it in `sudo`) | Invoke `start-worker.sh` itself rather than the binary directly, or prefix your own invocation with `sudo` | +| Real worker exits immediately: `Failed to initialize database` / `Failed to open database` | `/var/run` doesn't exist on this host — `xcecd` hardcodes `db_path=/var/run/xcecd.db`, not a flag | `start-worker.sh`'s real-worker branch already runs `sudo mkdir -p /var/run`; if you invoked `xcecd` directly instead, run that yourself first | +| Job submitted to a real worker never starts / dispatch silently fails | Real `xcecd` dispatches via `sd_bus_open_system()` → `org.freedesktop.systemd1.Manager.StartTransientUnit` — no system dbus / systemd manager reachable (no `/run/dbus/system_bus_socket`, no `dbus-daemon`/`systemctl`) | Not fixable inside this skill on a host without a real systemd manager; the real worker still exercises root-gated startup (sqlite db init/recovery, domain socket, controller registration/reconnect) there, just not job execution — use `xcecd_virtual` (`XCE_REAL_WORKER=0`) for anything that needs jobs to actually run | +| Two real workers won't start / second clobbers the first | Real `xcecd`'s `db_path`/`domain_sock_file`/log file are hardcoded, not parameterized by `-n` like `xcecd_virtual` is | Only run one real-worker instance per host; use `xcecd_virtual` (`XCE_REAL_WORKER=0`) for multiple named workers | +| `kill -9 $(pgrep xcecd)` doesn't stop the real worker | It's running as root (via `sudo`); an unprivileged `kill` against another user's PID fails silently | Use `sudo kill -9 $(pgrep -x xcecd)` (already in Step 2 / Clean restart above) | +| `grep`/`tail` against `$TOP/logs/xcecd.log` says `Permission denied` | Confirmed live: the real worker's log file comes out `root:root`, mode `640` — not `chmod`'d permissive like `xcecd_worker.log`/the domain socket are | Prefix with `sudo` (`sudo grep ...`/`sudo tail ...`) | +| Deploy picked the wrong worker binary (real when you wanted virtual, or vice versa) | Auto-detection follows `/run/systemd/system`, which may not match what you actually want on this host | Set `XCE_REAL_WORKER=0` or `XCE_REAL_WORKER=1` before Step 3 to force it, then redeploy | | A generated script fails with `bad interpreter` or silently uses the wrong bash | Redeploying with a stale `$BASH_BIN` captured outside `nix develop`, or a shell that shadows `bash` | Re-resolve `BASH_BIN=$(nix develop ./src --command which bash)` right before Step 3 and re-deploy | | `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER=""` before the `exec` line | | `scripts/xceadmin` not generated | No wheel in `$BINARY`'s build dir (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build xceadmin` (add `--incremental` if you don't want to lose other already-built binaries in that dir — see `/xce-build`'s SKILL.md), then redeploy (Step 3) | diff --git a/src/tools/deploy-ha.sh b/src/tools/deploy-ha.sh index 004d82bb..7b62665d 100755 --- a/src/tools/deploy-ha.sh +++ b/src/tools/deploy-ha.sh @@ -85,9 +85,48 @@ fi # only: xce-test-ha's soak driver also calls this script and never builds # either, so a missing worker/scheduler binary must not fail the deploy -- # it just skips generating that one start script, with a warning. -WORKER_BIN="$BINARY_BUILD_DIR/worker/xcecd_virtual" +# +# Which of the two sibling worker binaries this deploy's start-worker.sh +# targets is auto-detected by default, overridable with XCE_REAL_WORKER=0/1: +# xcecd_virtual: skips the root-only sqlite db-recovery and domain-socket +# subsystems, runs as an ordinary user, no sudo needed. +# xcecd (the production daemon): worker/src/main.c's check_user() +# unconditionally requires geteuid()==0, so the generated +# start-worker.sh below wraps its exec in `sudo` and preflights +# `sudo mkdir -p /var/run` (xcecd hardcodes db_path=/var/run/xcecd.db +# and domain_sock_file=/var/run/.xcecd.sock -- neither is a flag). +# Also, actually dispatching a job goes through sd_bus_open_system() -> +# org.freedesktop.systemd1.Manager.StartTransientUnit +# (worker/src/xcecd_cgroups.c) -- it needs a real systemd manager +# answering on the system dbus, not just root. +# +# Auto-detection: /run/systemd/system existing is systemd's own sd_booted(3) +# signal for "a real systemd instance is running as this host's manager" -- +# true on an actual host/VM, false in sandboxes that boot under something +# else (verified 2026-08-19: this dev sandbox has no /run at all, PID 1 is +# s6-svscan, no dbus-daemon/systemctl even under `nix develop`). Real job +# dispatch needs that systemd manager reachable regardless of root, so +# picking xcecd_virtual when it's absent -- and xcecd when it's present -- +# means this one deploy-ha.sh works unmodified on both a throwaway sandbox +# and a real HA test host. XCE_REAL_WORKER=0/1 overrides the auto-detect in +# either direction (e.g. force xcecd_virtual on a real host, or force xcecd +# on a sandbox to exercise its root-gated startup path even though job +# dispatch there will still fail). +if [ -n "${XCE_REAL_WORKER:-}" ]; then + USE_REAL_WORKER="$XCE_REAL_WORKER" +elif [ -d /run/systemd/system ]; then + USE_REAL_WORKER=1 +else + USE_REAL_WORKER=0 +fi +if [ "$USE_REAL_WORKER" = "1" ]; then + WORKER_BIN="$BINARY_BUILD_DIR/worker/xcecd" +else + WORKER_BIN="$BINARY_BUILD_DIR/worker/xcecd_virtual" +fi +echo "Worker binary: $([ "$USE_REAL_WORKER" = "1" ] && echo "xcecd (real, root via sudo)" || echo "xcecd_virtual (no root)") -- override with XCE_REAL_WORKER=0/1" if [ ! -x "$WORKER_BIN" ]; then - echo "Warning: no xcecd_virtual found at $WORKER_BIN -- skipping scripts/start-worker.sh generation (run '/xce-build worker' first if you need it)" + echo "Warning: no $(basename "$WORKER_BIN") found at $WORKER_BIN -- skipping scripts/start-worker.sh generation (run '/xce-build worker' first if you need it)" WORKER_BIN="" fi SCHED_BIN="$BINARY_BUILD_DIR/scheduler/xce_scheduler" @@ -229,15 +268,29 @@ chmod +x "$TOP/scripts/cleanup.sh" # Generate worker start script (component-failover-adoption's --peers path): # same $PEERS_STR the controllers themselves were just given, so a worker # started via this script fails over across the identical candidate list. -# xcecd_virtual (not production xcecd) is the default -- it skips the -# root-only sqlite/db-recovery and domain-socket subsystems, so it runs as -# an ordinary user in this same dev/CI environment with no sudo, which is -# what this skill's "local testing" scope calls for. Override with -# XCE_WORKER_BIN=/path/to/xcecd for a real worker if you specifically need -# job execution / cgroups (requires root -- run it yourself, outside this -# script, since this skill/session cannot sudo). +# Which binary WORKER_BIN points at (xcecd_virtual or xcecd -- auto-detected +# or XCE_REAL_WORKER-overridden above) was decided once, at deploy time; +# XCE_WORKER_BIN can still override the path at run time (e.g. to point at a +# differently-built xcecd), but does NOT flip the sudo/preflight branch below +# -- that follows USE_REAL_WORKER as resolved for THIS deploy, since a +# run-time path swap doesn't change whether the binary needs root. if [ -n "$WORKER_BIN" ]; then - cat > "$TOP/scripts/start-worker.sh" << EOF + if [ "$USE_REAL_WORKER" = "1" ]; then + cat > "$TOP/scripts/start-worker.sh" << EOF +#!$GEN_BASH +set -e +NAME=\${1:-real-worker} +[ \$# -gt 0 ] && shift +WORKER_BIN="\${XCE_WORKER_BIN:-$WORKER_BIN}" +mkdir -p $TOP/logs +# xcecd (production) requires root (worker/src/main.c:check_user()) and +# hardcodes db_path=/var/run/xcecd.db + domain_sock_file=/var/run/.xcecd.sock +# -- neither is a flag, so /var/run must exist before it will start. +sudo mkdir -p /var/run +sudo "\$WORKER_BIN" --peers "$PEERS_STR" -n "\$NAME" -l $TOP/logs \$* +EOF + else + cat > "$TOP/scripts/start-worker.sh" << EOF #!$GEN_BASH set -e NAME=\${1:-worker-1} @@ -246,6 +299,7 @@ WORKER_BIN="\${XCE_WORKER_BIN:-$WORKER_BIN}" mkdir -p $TOP/logs "\$WORKER_BIN" --peers "$PEERS_STR" -n "\$NAME" -l $TOP/logs \$* EOF + fi chmod +x "$TOP/scripts/start-worker.sh" fi