feat(xce-deploy-ha): support real xcecd worker deployment alongside virtual
Adds auto-detection of real vs virtual worker based on systemd availability (/run/systemd/system), plus an XCE_REAL_WORKER override. Real xcecd requires root (systemd D-Bus transient units, hardcoded /var/run paths) and is used opportunistically now that passwordless sudo is available in the sandbox; defaults to xcecd_virtual when systemd/D-Bus isn't available.
This commit is contained in:
parent
c9acd502c0
commit
83e61eecc8
2 changed files with 150 additions and 24 deletions
|
|
@ -43,6 +43,10 @@ disown`) before relying on subagent dispatch for unattended use.
|
|||
`deploy-ha.sh` never runs this itself; without it, Step 3 simply skips
|
||||
generating `scripts/start-apiserver.sh` (soft-skip, not an error), same as
|
||||
the existing worker/scheduler convention.
|
||||
- Run `/xce-build worker` so both worker binaries exist in the same build
|
||||
dir as `$BINARY`: `xcecd_virtual` (default) and `xcecd` (the real,
|
||||
root-gated production daemon). Step 3 auto-picks which one
|
||||
`scripts/start-worker.sh` targets — see Step 4b.
|
||||
|
||||
No data directory needed: TLS cert/key and `cluster.bin` are both generated
|
||||
directly into the deploy dir (`$TOP`) by `deploy-ha.sh` itself, not read from
|
||||
|
|
@ -101,6 +105,7 @@ to the intended interpreter).
|
|||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
|
||||
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
|
||||
sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root — plain kill can't touch it
|
||||
```
|
||||
|
||||
## Step 3 — Deploy cluster
|
||||
|
|
@ -143,10 +148,14 @@ This:
|
|||
(standbys receive config from primary via `OPCode_replConfigPush`)
|
||||
- Generates `$TOP/scripts/start.sh` (with `--workdir $TOP/controller --certdir $TOP/certs`) and `cleanup.sh`
|
||||
- Also generates `$TOP/scripts/start-worker.sh` and `$TOP/scripts/start-scheduler.sh`
|
||||
**if** `xcecd_virtual`/`xce_scheduler` exist alongside `$BINARY` (same
|
||||
build dir) — soft-skipped with a warning, not a hard error, if either is
|
||||
missing (so `/xce-test-ha`, which never builds worker/scheduler and calls
|
||||
this same script, is unaffected). See Step 4b.
|
||||
**if** the relevant binary exists alongside `$BINARY` (same build dir) —
|
||||
soft-skipped with a warning, not a hard error, if missing (so
|
||||
`/xce-test-ha`, which never builds worker/scheduler and calls this same
|
||||
script, is unaffected). Which worker binary — `xcecd_virtual` or the real
|
||||
`xcecd` — `start-worker.sh` targets is **auto-detected** per deploy
|
||||
(`/run/systemd/system` present → real `xcecd`; absent, e.g. this sandbox →
|
||||
`xcecd_virtual`), overridable with `XCE_REAL_WORKER=0|1` before Step 3. See
|
||||
Step 4b.
|
||||
- Also extracts a standalone xceadmin into `$TOP/xceadmin-site` and generates
|
||||
`$TOP/scripts/xceadmin` **if** a wheel exists at
|
||||
`<same build dir>/xceadmin/dist/*.whl` — same soft-skip-with-warning
|
||||
|
|
@ -189,19 +198,67 @@ so there is nothing to keep in sync by hand:
|
|||
"$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v & # scheduler is a cluster singleton, no name
|
||||
```
|
||||
|
||||
- **`start-worker.sh`** runs `xcecd_virtual`, not production `xcecd` — it
|
||||
skips the root-only sqlite/db-recovery and domain-socket subsystems, so it
|
||||
runs as an ordinary user with no `sudo` (this session/skill cannot `sudo`
|
||||
anyway). Logs land at `$TOP/logs/xcecd-<name>.log`. Start more than one by
|
||||
passing a different name: `start-worker.sh worker-2`. To point at a real
|
||||
`xcecd` instead (needs root — run it yourself outside this skill), set
|
||||
`XCE_WORKER_BIN=/path/to/xcecd` before invoking the script.
|
||||
- **`start-worker.sh`** runs whichever binary Step 3 auto-detected —
|
||||
`xcecd_virtual` (ordinary user, no `sudo`) in this sandbox, or the real,
|
||||
production `xcecd` (wrapped in `sudo` by the generated script itself) on a
|
||||
host where `/run/systemd/system` exists. See "Real worker" below for what
|
||||
changes when it's the real binary.
|
||||
- **`start-scheduler.sh`** runs `xce_scheduler`; its identity is fixed
|
||||
internally to `xce_scheduler` (no name argument). Logs to
|
||||
`$TOP/logs/scheduler.log`.
|
||||
- Verify a worker joined: `grep "Successfully sent .* load indices" "$TOP/logs/xcecd-worker-1.log"`.
|
||||
- Verify a worker joined: `grep "Successfully sent .* load indices" "$TOP/logs/xcecd-worker-1.log"`
|
||||
(virtual — logs are `xcecd-<name>.log`) or
|
||||
`sudo grep "Successfully sent .* load indices" "$TOP/logs/xcecd.log"` (real
|
||||
— always just `xcecd.log`, not per-name, **and root-owned `640`, not
|
||||
world-readable** — a plain `grep` as yourself gets `Permission denied`;
|
||||
see below).
|
||||
- Verify the scheduler joined: `grep "sched_init: joined cluster" "$TOP/logs/scheduler.log"`.
|
||||
|
||||
### Real worker (production `xcecd`)
|
||||
|
||||
Only relevant when Step 3 picked the real binary (`/run/systemd/system`
|
||||
present, or `XCE_REAL_WORKER=1` forced it). Two things are structurally
|
||||
different from `xcecd_virtual`, both because of how the production binary
|
||||
itself is built, not this skill's choice:
|
||||
|
||||
- **Single instance only.** `xcecd`'s `db_path` (`/var/run/xcecd.db`),
|
||||
`domain_sock_file` (`/var/run/.xcecd.sock`), and log file
|
||||
(`$TOP/logs/xcecd.log`) are all hardcoded in `worker/src/main.c` — unlike
|
||||
`xcecd_virtual`, `-n`/the name argument does **not** change any of them
|
||||
(it only sets the HA cluster identity used to join `--peers`). Only run
|
||||
one real worker per host at a time; you cannot do a real-worker
|
||||
equivalent of `start-worker.sh worker-2`.
|
||||
- **Job dispatch needs a real systemd+dbus host.** Actually running a job
|
||||
goes through `sd_bus_open_system()` →
|
||||
`org.freedesktop.systemd1.Manager.StartTransientUnit`
|
||||
(`worker/src/xcecd_cgroups.c`) — a live call to the system dbus, answered
|
||||
by a real systemd manager. That's exactly the `/run/systemd/system` signal
|
||||
Step 3 auto-detects on, so on a host that picked the real binary this
|
||||
should just work; if you forced `XCE_REAL_WORKER=1` somewhere that signal
|
||||
is absent (e.g. this sandbox), the daemon still starts, joins the
|
||||
cluster, and exercises its root-gated sqlite db-init/recovery and domain
|
||||
socket — but job *submission* will fail at the dispatch step regardless of
|
||||
`sudo`.
|
||||
|
||||
The generated `start-worker.sh` itself handles the rest — `sudo mkdir -p
|
||||
/var/run` (one-time preflight; `xcecd` exits immediately with "Failed to
|
||||
initialize database" if that directory doesn't exist, since neither
|
||||
hardcoded path is a flag) and wrapping the daemon exec in `sudo`. Nothing
|
||||
extra to do at call time beyond what's shown above, **provided `sudo` is
|
||||
passwordless** (confirmed `sudo -n true` in this sandbox) — on a host where
|
||||
it isn't, run this step interactively, not via the unattended `Agent`-tool
|
||||
dispatch described at the top of this file (a password prompt there would
|
||||
just hang).
|
||||
|
||||
Verify it's running as root: `ps -o pid,user,cmd -C xcecd`.
|
||||
|
||||
**Reading its log needs `sudo` too** — confirmed live: `$TOP/logs/xcecd.log`
|
||||
comes out owned `root:root`, mode `640` (unlike `xcecd_worker.log` and the
|
||||
domain socket, both of which the daemon explicitly `chmod`s `0666`; the main
|
||||
log file gets no such treatment). A plain `grep`/`tail` against it as
|
||||
yourself fails with `Permission denied` — use `sudo grep ...`/`sudo tail
|
||||
...` instead, same as the `sudo kill` already needed to stop the process.
|
||||
|
||||
## Step 4c — Use xceadmin against this deployment (optional)
|
||||
|
||||
`deploy-ha.sh` also installs a **standalone** `xceadmin` at `$TOP/xceadmin-site`
|
||||
|
|
@ -333,6 +390,7 @@ Stop all instances, clean up state, and restart without redeploying:
|
|||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
|
||||
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
|
||||
sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true # real worker runs as root
|
||||
|
||||
# 2. Run the generated cleanup script
|
||||
# Keeps: controller/ctrl-1/conf/* (primary config + current pointer), $TOP/certs/
|
||||
|
|
@ -361,6 +419,14 @@ sleep 0.5
|
|||
- All `controller/ctrl-{2..N}/conf/` — standbys rebuild config from primary's `OPCode_replConfigPush`
|
||||
- All `logs/*.log` — clean log slate
|
||||
|
||||
**Not touched by cleanup.sh, if you ran the real worker:** `/var/run/xcecd.db`
|
||||
and `/var/run/.xcecd.sock` live outside `$TOP` entirely (hardcoded, not
|
||||
parameterized by this deploy), so they persist across `cleanup.sh` runs —
|
||||
that's what lets the real worker's db-recovery path actually be exercised on
|
||||
restart. Remove them yourself (`sudo rm -f /var/run/xcecd.db
|
||||
/var/run/.xcecd.sock`) if you specifically want a from-scratch real-worker
|
||||
state.
|
||||
|
||||
If you need a full redeploy (different peers or new binary), run steps 3–5 instead.
|
||||
|
||||
## Troubleshooting
|
||||
|
|
@ -373,8 +439,14 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins
|
|||
| `election_timeout` loops without primary | All nodes have `config_tag=0` (cold-start) or network issue | Check peers string matches actual listen ports |
|
||||
| `ha-term.bin` causes "already voted" on restart | Old state from previous run | Run clean restart above |
|
||||
| `SSL_do_handshake: Broken pipe` flooding from a worker/scheduler | A typo'd hostname in the `--peers` you passed (e.g. `local` instead of `localhost`) — the connect reaches SSL before failing, so it doesn't error out cleanly | Diff the failing `--peers` string against `$TOP/scripts/start.sh`'s (they must be byte-identical); or just use `start-worker.sh`/`start-scheduler.sh`, which always carry the exact string the controllers were deployed with |
|
||||
| `start-worker.sh`/`start-scheduler.sh` not generated | `xcecd_virtual`/`xce_scheduler` don't exist in `$BINARY`'s build dir (deploy-ha.sh logs a `Warning:`, not an error) | `/xce-build worker` and/or `/xce-build scheduler`, then redeploy (Step 3) |
|
||||
| `xcecd should be run as root` | You substituted real `xcecd` via `XCE_WORKER_BIN` | Either run it yourself with `sudo` outside this skill, or drop back to the default `xcecd_virtual` (no `sudo` needed, skips root-only subsystems) |
|
||||
| `start-worker.sh`/`start-scheduler.sh` not generated | Whichever worker binary auto-detection (or `XCE_REAL_WORKER`) selected — `xcecd_virtual` or `xcecd` — doesn't exist in `$BINARY`'s build dir, or `xce_scheduler` doesn't (deploy-ha.sh logs a `Warning:`, not an error) | `/xce-build worker` and/or `/xce-build scheduler`, then redeploy (Step 3) — `/xce-build worker` produces both `xcecd_virtual` and `xcecd` together |
|
||||
| `xcecd should be run as root` | `XCE_WORKER_BIN` was pointed at real `xcecd` directly (bypassing the generated `start-worker.sh`, which already wraps it in `sudo`) | Invoke `start-worker.sh` itself rather than the binary directly, or prefix your own invocation with `sudo` |
|
||||
| Real worker exits immediately: `Failed to initialize database` / `Failed to open database` | `/var/run` doesn't exist on this host — `xcecd` hardcodes `db_path=/var/run/xcecd.db`, not a flag | `start-worker.sh`'s real-worker branch already runs `sudo mkdir -p /var/run`; if you invoked `xcecd` directly instead, run that yourself first |
|
||||
| Job submitted to a real worker never starts / dispatch silently fails | Real `xcecd` dispatches via `sd_bus_open_system()` → `org.freedesktop.systemd1.Manager.StartTransientUnit` — no system dbus / systemd manager reachable (no `/run/dbus/system_bus_socket`, no `dbus-daemon`/`systemctl`) | Not fixable inside this skill on a host without a real systemd manager; the real worker still exercises root-gated startup (sqlite db init/recovery, domain socket, controller registration/reconnect) there, just not job execution — use `xcecd_virtual` (`XCE_REAL_WORKER=0`) for anything that needs jobs to actually run |
|
||||
| Two real workers won't start / second clobbers the first | Real `xcecd`'s `db_path`/`domain_sock_file`/log file are hardcoded, not parameterized by `-n` like `xcecd_virtual` is | Only run one real-worker instance per host; use `xcecd_virtual` (`XCE_REAL_WORKER=0`) for multiple named workers |
|
||||
| `kill -9 $(pgrep xcecd)` doesn't stop the real worker | It's running as root (via `sudo`); an unprivileged `kill` against another user's PID fails silently | Use `sudo kill -9 $(pgrep -x xcecd)` (already in Step 2 / Clean restart above) |
|
||||
| `grep`/`tail` against `$TOP/logs/xcecd.log` says `Permission denied` | Confirmed live: the real worker's log file comes out `root:root`, mode `640` — not `chmod`'d permissive like `xcecd_worker.log`/the domain socket are | Prefix with `sudo` (`sudo grep ...`/`sudo tail ...`) |
|
||||
| Deploy picked the wrong worker binary (real when you wanted virtual, or vice versa) | Auto-detection follows `/run/systemd/system`, which may not match what you actually want on this host | Set `XCE_REAL_WORKER=0` or `XCE_REAL_WORKER=1` before Step 3 to force it, then redeploy |
|
||||
| A generated script fails with `bad interpreter` or silently uses the wrong bash | Redeploying with a stale `$BASH_BIN` captured outside `nix develop`, or a shell that shadows `bash` | Re-resolve `BASH_BIN=$(nix develop ./src --command which bash)` right before Step 3 and re-deploy |
|
||||
| `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER="<first --peers candidate>"` before the `exec` line |
|
||||
| `scripts/xceadmin` not generated | No wheel in `$BINARY`'s build dir (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build xceadmin` (add `--incremental` if you don't want to lose other already-built binaries in that dir — see `/xce-build`'s SKILL.md), then redeploy (Step 3) |
|
||||
|
|
|
|||
|
|
@ -85,9 +85,48 @@ fi
|
|||
# only: xce-test-ha's soak driver also calls this script and never builds
|
||||
# either, so a missing worker/scheduler binary must not fail the deploy --
|
||||
# it just skips generating that one start script, with a warning.
|
||||
WORKER_BIN="$BINARY_BUILD_DIR/worker/xcecd_virtual"
|
||||
#
|
||||
# Which of the two sibling worker binaries this deploy's start-worker.sh
|
||||
# targets is auto-detected by default, overridable with XCE_REAL_WORKER=0/1:
|
||||
# xcecd_virtual: skips the root-only sqlite db-recovery and domain-socket
|
||||
# subsystems, runs as an ordinary user, no sudo needed.
|
||||
# xcecd (the production daemon): worker/src/main.c's check_user()
|
||||
# unconditionally requires geteuid()==0, so the generated
|
||||
# start-worker.sh below wraps its exec in `sudo` and preflights
|
||||
# `sudo mkdir -p /var/run` (xcecd hardcodes db_path=/var/run/xcecd.db
|
||||
# and domain_sock_file=/var/run/.xcecd.sock -- neither is a flag).
|
||||
# Also, actually dispatching a job goes through sd_bus_open_system() ->
|
||||
# org.freedesktop.systemd1.Manager.StartTransientUnit
|
||||
# (worker/src/xcecd_cgroups.c) -- it needs a real systemd manager
|
||||
# answering on the system dbus, not just root.
|
||||
#
|
||||
# Auto-detection: /run/systemd/system existing is systemd's own sd_booted(3)
|
||||
# signal for "a real systemd instance is running as this host's manager" --
|
||||
# true on an actual host/VM, false in sandboxes that boot under something
|
||||
# else (verified 2026-08-19: this dev sandbox has no /run at all, PID 1 is
|
||||
# s6-svscan, no dbus-daemon/systemctl even under `nix develop`). Real job
|
||||
# dispatch needs that systemd manager reachable regardless of root, so
|
||||
# picking xcecd_virtual when it's absent -- and xcecd when it's present --
|
||||
# means this one deploy-ha.sh works unmodified on both a throwaway sandbox
|
||||
# and a real HA test host. XCE_REAL_WORKER=0/1 overrides the auto-detect in
|
||||
# either direction (e.g. force xcecd_virtual on a real host, or force xcecd
|
||||
# on a sandbox to exercise its root-gated startup path even though job
|
||||
# dispatch there will still fail).
|
||||
if [ -n "${XCE_REAL_WORKER:-}" ]; then
|
||||
USE_REAL_WORKER="$XCE_REAL_WORKER"
|
||||
elif [ -d /run/systemd/system ]; then
|
||||
USE_REAL_WORKER=1
|
||||
else
|
||||
USE_REAL_WORKER=0
|
||||
fi
|
||||
if [ "$USE_REAL_WORKER" = "1" ]; then
|
||||
WORKER_BIN="$BINARY_BUILD_DIR/worker/xcecd"
|
||||
else
|
||||
WORKER_BIN="$BINARY_BUILD_DIR/worker/xcecd_virtual"
|
||||
fi
|
||||
echo "Worker binary: $([ "$USE_REAL_WORKER" = "1" ] && echo "xcecd (real, root via sudo)" || echo "xcecd_virtual (no root)") -- override with XCE_REAL_WORKER=0/1"
|
||||
if [ ! -x "$WORKER_BIN" ]; then
|
||||
echo "Warning: no xcecd_virtual found at $WORKER_BIN -- skipping scripts/start-worker.sh generation (run '/xce-build worker' first if you need it)"
|
||||
echo "Warning: no $(basename "$WORKER_BIN") found at $WORKER_BIN -- skipping scripts/start-worker.sh generation (run '/xce-build worker' first if you need it)"
|
||||
WORKER_BIN=""
|
||||
fi
|
||||
SCHED_BIN="$BINARY_BUILD_DIR/scheduler/xce_scheduler"
|
||||
|
|
@ -229,15 +268,29 @@ chmod +x "$TOP/scripts/cleanup.sh"
|
|||
# Generate worker start script (component-failover-adoption's --peers path):
|
||||
# same $PEERS_STR the controllers themselves were just given, so a worker
|
||||
# started via this script fails over across the identical candidate list.
|
||||
# xcecd_virtual (not production xcecd) is the default -- it skips the
|
||||
# root-only sqlite/db-recovery and domain-socket subsystems, so it runs as
|
||||
# an ordinary user in this same dev/CI environment with no sudo, which is
|
||||
# what this skill's "local testing" scope calls for. Override with
|
||||
# XCE_WORKER_BIN=/path/to/xcecd for a real worker if you specifically need
|
||||
# job execution / cgroups (requires root -- run it yourself, outside this
|
||||
# script, since this skill/session cannot sudo).
|
||||
# Which binary WORKER_BIN points at (xcecd_virtual or xcecd -- auto-detected
|
||||
# or XCE_REAL_WORKER-overridden above) was decided once, at deploy time;
|
||||
# XCE_WORKER_BIN can still override the path at run time (e.g. to point at a
|
||||
# differently-built xcecd), but does NOT flip the sudo/preflight branch below
|
||||
# -- that follows USE_REAL_WORKER as resolved for THIS deploy, since a
|
||||
# run-time path swap doesn't change whether the binary needs root.
|
||||
if [ -n "$WORKER_BIN" ]; then
|
||||
cat > "$TOP/scripts/start-worker.sh" << EOF
|
||||
if [ "$USE_REAL_WORKER" = "1" ]; then
|
||||
cat > "$TOP/scripts/start-worker.sh" << EOF
|
||||
#!$GEN_BASH
|
||||
set -e
|
||||
NAME=\${1:-real-worker}
|
||||
[ \$# -gt 0 ] && shift
|
||||
WORKER_BIN="\${XCE_WORKER_BIN:-$WORKER_BIN}"
|
||||
mkdir -p $TOP/logs
|
||||
# xcecd (production) requires root (worker/src/main.c:check_user()) and
|
||||
# hardcodes db_path=/var/run/xcecd.db + domain_sock_file=/var/run/.xcecd.sock
|
||||
# -- neither is a flag, so /var/run must exist before it will start.
|
||||
sudo mkdir -p /var/run
|
||||
sudo "\$WORKER_BIN" --peers "$PEERS_STR" -n "\$NAME" -l $TOP/logs \$*
|
||||
EOF
|
||||
else
|
||||
cat > "$TOP/scripts/start-worker.sh" << EOF
|
||||
#!$GEN_BASH
|
||||
set -e
|
||||
NAME=\${1:-worker-1}
|
||||
|
|
@ -246,6 +299,7 @@ WORKER_BIN="\${XCE_WORKER_BIN:-$WORKER_BIN}"
|
|||
mkdir -p $TOP/logs
|
||||
"\$WORKER_BIN" --peers "$PEERS_STR" -n "\$NAME" -l $TOP/logs \$*
|
||||
EOF
|
||||
fi
|
||||
chmod +x "$TOP/scripts/start-worker.sh"
|
||||
fi
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue