diff --git a/.claude/skills/xce-test-ha/SKILL.md b/.claude/skills/xce-test-ha/SKILL.md index 8c243a66..21c18334 100644 --- a/.claude/skills/xce-test-ha/SKILL.md +++ b/.claude/skills/xce-test-ha/SKILL.md @@ -65,7 +65,7 @@ dir, an integer positional is rounds. So the release-binary short run is: # 2-round chaotic soak on real-fs release binaries (no nix shell needed for the # release build), positional rounds + dir: XCE_HA_BIN=$PWD/src/build.release/controller/xce_controller \ -XCE_HA_XT=$PWD/src/build.release/controller/xce_test \ +XCE_HA_XT=$PWD/src/build.release/harness/xce_test \ bash .claude/skills/xce-test-ha/run-ha-soak.sh --chaotic 2 ~/tmp/ha-1p5 ``` @@ -465,21 +465,32 @@ bash .claude/skills/xce-test-ha/soak-verdict-selftest.sh ## Step 1 — Ensure binaries are built -The test needs `xce_controller` and `xce_test`: +The test needs `xce_controller` (controller component) and `xce_test` +(standalone `harness` component, decoupled from the controller build since +xce-harness-component task 3.1 — a controller-only build no longer produces +it): ```bash BIN=src/xce.build/controller/xce_controller -XT=src/xce.build/controller/xce_test +XT=src/xce.build/harness/xce_test ls -lh "$BIN" "$XT" ``` -If either is missing, build them first with the `/xce-build` skill: +If either is missing, build the corresponding component with the +`/xce-build` skill. Both land in the same build dir (`src/xce.build/` by +default), but `xce-build.sh` wipes that whole dir on every invocation unless +told otherwise (bug-1565) — building a second component after the first +needs `--incremental` or it silently deletes the one you just built: ``` /xce-build controller +/xce-build harness --incremental ``` -(Do NOT proceed to the soak until both binaries exist.) +(Do NOT proceed to the soak until both binaries exist.) `run-ha-soak.sh` +itself also checks for both at startup and fails fast with the exact +`/xce-build` command to run — see the Troubleshooting table below — but +confirming here first avoids a fresh build dir surprising you mid-deploy. ## Step 2 — Run the soak driver @@ -542,7 +553,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK | `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` | | `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, `array-chaos-retry`, `config-upgrade`, or `config-upgrade-wal-trim` (its `--flag` wins) | `sequential` | | `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `/src/xce.build/controller/xce_controller` | -| `XCE_HA_XT` | `xce_test` binary | `/src/xce.build/controller/xce_test` | +| `XCE_HA_XT` | `xce_test` binary | `/src/xce.build/harness/xce_test` | | `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` | | `XCE_HA_JOBS` | jobs per submit batch | `3` | | `XCE_HA_WAITMS` | `wait-primary` budget per probe | `30000` | @@ -647,7 +658,10 @@ each iteration: 2. if exit == 0 and log has "PASS:" → GOAL REACHED, stop 3. else: read the FAIL line + $XCE_HA_TOP/logs to find the root cause 4. fix the underlying bug in src/ (controller or libxce) - 5. /xce-build controller (rebuild the binaries the next run uses) + 5. /xce-build controller (rebuild the binaries the next run uses -- this + wipes the build dir, so also rerun `/xce-build harness --incremental` + if xce_test was already built there, or the next iteration will fail + the harness precondition check instead of the actual regression) 6. repeat ``` @@ -687,6 +701,7 @@ printf 'waldump %s/controller/ctrl-1/data\nquit\n' "$XCE_HA_TOP" \ | Symptom | Cause | Fix | |---|---|---| | `controller binary missing` | not built | `/xce-build controller` | +| `xce_test missing` | harness not built — decoupled from the controller build (xce-harness-component task 3.1), so `/xce-build controller` alone no longer produces it | `/xce-build harness --incremental` if a controller build already occupies the build dir (see Step 1), else plain `/xce-build harness` | | `no primary elected within ~90s via client probe` | cold-start election stalled / port clash (NOT a log-buffer artifact — detection is client-probe based) | check `logs/ctrl-*.log`; ensure ports 9991-9993 free | | `primary elected in logs but client could not reach it` | client discovery (bug-854 regression) | check `client.c` rotation fix is in the binary | | `a committed job was LOST after failover` | data-safety regression (bug-828 class) | inspect WAL via `waldump`; this is a real FAIL | diff --git a/.claude/skills/xce-test-ha/run-ha-soak.sh b/.claude/skills/xce-test-ha/run-ha-soak.sh index a8040b01..04bb9fe4 100755 --- a/.claude/skills/xce-test-ha/run-ha-soak.sh +++ b/.claude/skills/xce-test-ha/run-ha-soak.sh @@ -323,7 +323,12 @@ esac # non-empty dir that carries this marker (see "Step 2: deploy" below). MARKER_NAME=".xce-ha-deploy" BIN="${XCE_HA_BIN:-$REPO/src/xce.build/controller/xce_controller}" -XT="${XCE_HA_XT:-$REPO/src/xce.build/controller/xce_test}" +# xce-harness-component task 3.1: xce_test now builds from the standalone +# `src/harness/` component (its own `/xce-build harness` entry), not as a +# side effect of `/xce-build controller` -- its default path here follows +# that move. XCE_HA_XT still lets a caller point at any build dir/layout +# (e.g. a release build) explicitly. +XT="${XCE_HA_XT:-$REPO/src/xce.build/harness/xce_test}" DEPLOY="$REPO/src/tools/deploy-ha.sh" # Rounds precedence: bare-integer positional (ROUNDS_CLI) > $XCE_HA_ROUNDS > 6. @@ -658,7 +663,13 @@ verify_worker_resync_jobs_final() { # ── Preconditions ──────────────────────────────────────────────────────────── [ -x "$BIN" ] || { echo "[xce-test-ha] FAIL: controller binary missing: $BIN (run /xce-build controller)" >&2; exit 2; } -[ -x "$XT" ] || { echo "[xce-test-ha] FAIL: xce_test missing: $XT (run /xce-build controller)" >&2; exit 2; } +# xce-harness-component task 3.1 decoupled xce_test from the controller build -- +# it is the soak's own driver client (every kill/verify/probe step below runs +# through xtest()), so a missing binary here is fatal, not soft-skippable like +# the optional worker/scheduler coverage above. Fail with the harness-specific +# build command instead of a raw spawn/connect error the first time xtest() +# tries to exec a nonexistent path. +[ -x "$XT" ] || { echo "[xce-test-ha] FAIL: xce_test missing: $XT (run /xce-build harness)" >&2; exit 2; } # TLS cert/key and cluster.bin are no longer preconditions here at all -- # deploy-ha.sh (Step 2 below) generates both fresh directly into $TOP on # every run (openssl self-signed cert; cluster.bin from src/configs/*.yaml diff --git a/.kiro/specs/xce-harness-component/tasks.md b/.kiro/specs/xce-harness-component/tasks.md index d2c548a1..8e4237a0 100644 --- a/.kiro/specs/xce-harness-component/tasks.md +++ b/.kiro/specs/xce-harness-component/tasks.md @@ -37,7 +37,7 @@ - In `src/tools/xce-build.sh`, drop `controller/xce_test` from the `controller)` case's compile target list; add a new `harness)` case following the identical three-part `setup_build` / `meson compile` / conditional `meson test --suite harness` template already used by `scheduler)`/`worker)` - Observable: `/xce-build harness` on a fresh build dir (no prior controller build) produces the `xce_test` binary and runs the `harness` suite; `/xce-build controller` alone does not produce `xce_test` - _Requirements: 2.1, 2.2, 2.3, 2.4_ -- [ ] 3.2 (P) Update the HA soak workflow's harness prerequisite +- [x] 3.2 (P) Update the HA soak workflow's harness prerequisite - Update `.claude/skills/xce-test-ha/run-ha-soak.sh` and `SKILL.md` so the soak ensures the harness is built (or fails with a clear message) instead of assuming a controller build produced it - Observable: running the workflow from a build dir where only controller was built either builds the harness itself or reports a clear, actionable message instead of a raw spawn/connect failure - _Requirements: 5.1, 5.4_