From 8579e3cefcc366671e710a685ce716f4a84a18a3 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 3 Oct 2026 08:25:13 +0000 Subject: [PATCH] docs(controller-peers-env-fallback): note XCE_CONTROLLER_PEERS in the HA-deploy and multi-host skill docs (task 6.2) Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/skills/xce-deploy-ha/SKILL.md | 18 ++++++++++++++---- .claude/skills/xce-test-multihost/SKILL.md | 14 ++++++++++++-- .../controller-peers-env-fallback/tasks.md | 2 +- 3 files changed, 27 insertions(+), 7 deletions(-) diff --git a/.claude/skills/xce-deploy-ha/SKILL.md b/.claude/skills/xce-deploy-ha/SKILL.md index 2cdd1be4..eb9e7caf 100644 --- a/.claude/skills/xce-deploy-ha/SKILL.md +++ b/.claude/skills/xce-deploy-ha/SKILL.md @@ -238,7 +238,12 @@ both fail over across the whole candidate list on their own), start a worker and/or the scheduler with the **identical `--peers` configuration** the controllers were deployed with — `deploy-ha.sh` bakes the exact same `ctrl-1=host:port,ctrl-2=host:port,...` string into both generated scripts, -so there is nothing to keep in sync by hand: +so there is nothing to keep in sync by hand. (The daemons can also take the +list from `XCE_CONTROLLER_PEERS` when started without `--peers`, but the +generated `start.sh`/`start-worker.sh`/`start-scheduler.sh` still pass +`--peers` explicitly, so the variable has no effect on them; see +`.kiro/steering/build-deployment.md`, "Controller Peer List from the +Environment".) ```bash "$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv & # name defaults to worker-1 @@ -327,10 +332,15 @@ a warning if no wheel exists in the same build dir as `$BINARY`, same as - **Defaults `--controller` to this deployment's own first `--peers` candidate**, not xceadmin's generic bare fallback (`localhost:9991`, unrelated to any real deployment's actual ports). The wrapper exports + `XCE_CONTROLLER_PEERS=""` and `XCEADMIN_CONTROLLER=""`, which every `--controller`-accepting xceadmin subcommand (`config push`, `scheduler - record start`/`stop`/`status`) reads via `envvar=` fallback — an explicit - `--controller host:port` on the command line still overrides it. Before + record start`/`stop`/`status`) reads via `envvar=` fallback, in that + order — an explicit `--controller host:port` on the command line still + overrides both. Pinning `XCE_CONTROLLER_PEERS` means a value left in your + shell for another cluster cannot redirect this wrapper (see + `.kiro/steering/build-deployment.md`, "Controller Peer List from the + Environment"). Before this (bug-1566), a bare `scripts/xceadmin scheduler record status` with no `--controller` silently tried port 9991 and failed with a connection- refused that had nothing to do with the actual deployment. @@ -515,7 +525,7 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins | `grep`/`tail` against `$TOP/logs/xcecd.log` says `Permission denied` | Confirmed live: the real worker's log file comes out `root:root`, mode `640` — not `chmod`'d permissive like `xcecd_worker.log`/the domain socket are | Prefix with `sudo` (`sudo grep ...`/`sudo tail ...`) | | Deploy picked the wrong worker binary (real when you wanted virtual, or vice versa) | Auto-detection follows `/run/systemd/system`, which may not match what you actually want on this host | Set `XCE_REAL_WORKER=0` or `XCE_REAL_WORKER=1` before Step 3 to force it, then redeploy | | A generated script fails with `bad interpreter` or silently uses the wrong bash | Redeploying with a stale `$BASH_BIN` captured outside `nix develop`, or a shell that shadows `bash` | Re-resolve `BASH_BIN=$(nix develop ./src --command which bash)` right before Step 3 and re-deploy | -| `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER=""` before the `exec` line | +| `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER=""` and `export XCE_CONTROLLER_PEERS=""` before the `exec` line (if only the first is there, an ambient `XCE_CONTROLLER_PEERS` in your shell takes precedence and points it elsewhere — redeploy) | | `scripts/xceadmin` not generated | No wheel in `$BINARY`'s build dir (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build xceadmin` (add `--incremental` if you don't want to lose other already-built binaries in that dir — see `/xce-build`'s SKILL.md), then redeploy (Step 3) | | `scripts/start-apiserver.sh` not generated | No binary at `/frontend/apiserver/xce_apiserver` or `/tmp/apiserver` (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build apiserver`, then redeploy (Step 3) — or place a binary at `/tmp/apiserver` (legacy path, not guaranteed to be the same build generation) | | Asked to use `xce_test` against this deployment but `/harness/xce_test` doesn't exist | Harness never built — `/xce-build controller` no longer produces `xce_test` as a side effect, and `deploy-ha.sh` doesn't build or reference it either (see Prerequisites) | `/xce-build harness`, then invoke `/harness/xce_test` directly against this deployment's `--peers` (no wrapper script is generated for it, unlike worker/scheduler/xceadmin) | diff --git a/.claude/skills/xce-test-multihost/SKILL.md b/.claude/skills/xce-test-multihost/SKILL.md index 25d1e365..4245ab43 100644 --- a/.claude/skills/xce-test-multihost/SKILL.md +++ b/.claude/skills/xce-test-multihost/SKILL.md @@ -66,7 +66,10 @@ Traces to `.kiro/specs/resreq-maxtasks-wiring/design.md`'s named workers individually via the scripts `deploy-ha.sh` generates (`start.sh ctrl-1`, `start-scheduler.sh`, `start-worker.sh ` twice with two different names) -- deliberately NOT `start-all.sh`, which only - ever starts a single worker hardcoded to the name `worker-1`. A + ever starts a single worker hardcoded to the name `worker-1`. (Those + generated scripts pass `--peers` explicitly, so `XCE_CONTROLLER_PEERS` + -- the daemons' environment fallback, see + `.kiro/steering/build-deployment.md` -- has no effect on this run.) A `trap`-based cleanup is registered before the first process starts, so it fires on every exit path (a completed run, a mid-deployment abort, or an external SIGINT/SIGTERM), matching `xce-test-resources`'s convention. @@ -244,6 +247,10 @@ processor/slot capacity: `worker/src/main.c`'s `getopt_long()` table is `xcecd_virtual` means it skips real job execution/cgroups, **not** that it fakes a different processor count -- each worker reports the real host's actual core count (confirmed live in this sandbox: `nproc` reports 22). +(`--peers` can also come from `XCE_CONTROLLER_PEERS` when the flag is +absent -- see `.kiro/steering/build-deployment.md` -- but this skill's +generated `start-worker.sh` always passes `--peers`, so that does not +apply here.) `compute_task_count()` in `run-multihost-verify.sh` therefore queries `nproc` live at test-run time (never hardcoded) and picks the test job's @@ -350,7 +357,10 @@ allocation decision, not a copy of the plain job's. - `xce-deploy-ha`'s `deploy-ha.sh` and its generated `start.sh` / `start-scheduler.sh` / `start-worker.sh` (single-controller-node path, - two named worker instances against the same `--peers` config). + two named worker instances against the same `--peers` config). These + scripts, and this script's own `xce_test` sessions (`set masters`), pass + the controller list explicitly, so `XCE_CONTROLLER_PEERS` (see + `.kiro/steering/build-deployment.md`) does not change this verification. - Controller, scheduler, worker, and harness (`xce_test`) binaries in the same build dir (see Prerequisites above). diff --git a/.kiro/specs/controller-peers-env-fallback/tasks.md b/.kiro/specs/controller-peers-env-fallback/tasks.md index d60b2fdd..3b82596c 100644 --- a/.kiro/specs/controller-peers-env-fallback/tasks.md +++ b/.kiro/specs/controller-peers-env-fallback/tasks.md @@ -120,7 +120,7 @@ - _Requirements: 3.1_ - _Boundary: steering docs_ -- [ ] 6.2 (P) Note the variable in the HA-deploy and multi-host skill docs +- [x] 6.2 (P) Note the variable in the HA-deploy and multi-host skill docs - HA-deploy skill: the variable's availability, that generated daemon scripts still pass `--peers`, and that the xceadmin wrapper now exports the variable (description and troubleshooting row) - Multi-host skill: the same availability note at its three `--peers` locations, without implying any change to its generated invocations - Done when both skill documents carry the notes at the listed locations