From 0313b910a9bf293c1894a71b77f4decf337b2012 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sun, 6 Sep 2026 08:27:55 +0000 Subject: [PATCH] feat(resreq-maxtasks-wiring): add multi-host e2e test harness New xce-test-multihost skill deploys one controller/scheduler node plus two named xcecd_virtual workers, following xce-test-resources's single-node deploy pattern and xce-test-ha's pre-flight zombie-process/ port-check convention (chosen over a fresh-mktemp-per-run model specifically because a meaningful pre-flight check needs a stable directory -- see tasks.md's Implementation Notes). Confirms all four roles join via their own log lines and tears down cleanly with a HARNESS_READY sentinel. No job scenario wired in yet -- that's tasks 4.2/4.3. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg --- .claude/skills/xce-test-multihost/SKILL.md | 217 ++++++++ .../run-multihost-verify.sh | 499 ++++++++++++++++++ .kiro/specs/resreq-maxtasks-wiring/tasks.md | 3 +- 3 files changed, 718 insertions(+), 1 deletion(-) create mode 100644 .claude/skills/xce-test-multihost/SKILL.md create mode 100755 .claude/skills/xce-test-multihost/run-multihost-verify.sh diff --git a/.claude/skills/xce-test-multihost/SKILL.md b/.claude/skills/xce-test-multihost/SKILL.md new file mode 100644 index 00000000..43cf813b --- /dev/null +++ b/.claude/skills/xce-test-multihost/SKILL.md @@ -0,0 +1,217 @@ +--- +name: xce-test-multihost +description: "End-to-end multi-host job allocation harness: deploys one controller/scheduler node plus two named xcecd_virtual workers (each sized so neither alone can satisfy the test job's requested task count) and confirms all four roles come up and join cleanly. Usage: bash .claude/skills/xce-test-multihost/run-multihost-verify.sh [--keep-running] [--dir DIR]" +model: haiku +--- + +# XCE Multi-Host Allocation Test Harness + +## Status: task 4.1 only -- deploy/teardown harness, no scenario wired in yet + +`run-multihost-verify.sh` currently implements the resreq-maxtasks-wiring +spec's task 4.1 scope: it deploys a single controller/scheduler node plus +two independently-named `xcecd_virtual` workers, confirms all four roles +come up and join, emits a `HARNESS_READY` sentinel + PASS/FAIL oracle line, +and tears everything down cleanly. **It does not yet submit any job or +check any allocation outcome** -- that is tasks 4.2 (a plain multi-task job +via `jobsub`) and 4.3 (the same, as part of an array-job submission), which +build directly on top of this same script (see the `EXTENSION POINT` +comment block in `run-multihost-verify.sh`, right after +`confirm_cluster_up()`). + +This closes bug-1770's own end-to-end validation gap once 4.2/4.3 land: the +scheduler-allocation feature already implements, unit-tests, and +soak-validates real multi-host allocation logic (`parjob_allocator`) that, +until resreq-maxtasks-wiring's own fix lands, has no way to ever be +reached by a real job submission. This skill exists to prove -- with a real +controller, a real scheduler, and real (virtual) workers, driven through +plain `jobsub` -- that a job requiring more tasks than one host can provide +is genuinely split across two hosts by the scheduler's own decision, and +actually runs to completion on both. + +Traces to `.kiro/specs/resreq-maxtasks-wiring/design.md`'s +`xce-test-multihost` component (Requirements 4.1, 4.2, 4.3, 5.3, 5.5). + +## What this skill does today (task 4.1) + +1. **Pre-flight zombie-process/port check** -- following the convention + already established by `xce-test-ha`'s `run-ha-soak.sh` (a fixed, reused + deploy directory + a TOP-scoped kill of anything left over from a prior + run of THIS script, plus an explicit check that the controller's port + isn't already bound by an unrelated process) and this project's own + `cerebrum.md` 2026-08-26 lesson: a same-session leftover + `xce_controller`/`xce_scheduler`/`xcecd_virtual` process bound to the + exact port a "fresh" deploy is about to reuse has previously produced + apparently-impossible symptoms (term jumping, permanent config-load + failure) in a live investigation, fully explained only once the zombie + was found and killed. A conflict this script can positively attribute to + its own prior run (matched via a `$TOP`-scoped cmdline substring, same + discipline `run-ha-soak.sh`'s `node_pattern()`/`worker_pattern()`/ + `sched_pattern()` use) is killed automatically; a conflict it cannot + attribute to itself (an unrelated live process on the same port) is a + hard `FAIL`, never a silent/guessed kill (bug-1436's "fail loud, don't + silently corrupt" rule). +2. **Deploy** -- a single controller node plus scheduler plus two named + workers into a fixed, reused scratch directory (default + `/home/claude/tmp/xce-test-multihost`, override via `--dir`/ + `$XCE_MULTIHOST_TOP`) by invoking `xce-deploy-ha`'s `deploy-ha.sh` + directly (single-node `--peers`, so no HA/raft election timing is in + play for this proof), then starting the controller, scheduler, and both + named workers individually via the scripts `deploy-ha.sh` generates + (`start.sh ctrl-1`, `start-scheduler.sh`, `start-worker.sh ` twice + with two different names) -- deliberately NOT `start-all.sh`, which only + ever starts a single worker hardcoded to the name `worker-1`. A + `trap`-based cleanup is registered before the first process starts, so it + fires on every exit path (a completed run, a mid-deployment abort, or an + external SIGINT/SIGTERM), matching `xce-test-resources`'s convention. +3. **Confirm all four roles joined** -- bounded-wait polling (same + short-interval/generous-timeout convention as every other skill in this + family) for: + - controller: `CTRL_PRIMARY` in `$TOP/logs/ctrl-1.log` + - scheduler: `sched_init: joined cluster` in `$TOP/logs/scheduler.log` + - each named worker: `Successfully sent .* load indices` in its own + `$TOP/logs/xcecd-.log` -- checked independently per worker, not + just once for "a" worker, since this harness's whole point is TWO + independently-joined hosts. +4. **Report a `HARNESS_READY` sentinel + PASS/FAIL oracle line** -- since + there is no job outcome to report yet at this stage, the terminal line + reports "harness stood up and tore down correctly", not a job-allocation + verdict: + ``` + [xce-test-multihost] HARNESS_READY controller=up scheduler=up workers=2/2 top=/home/claude/tmp/xce-test-multihost + [xce-test-multihost] PASS (harness-ready): controller + scheduler + 2 named worker(s) (mh-worker-1, mh-worker-2) deployed and confirmed joined; no job scenario submitted yet (task 4.1 scope only -- see tasks 4.2/4.3) + ``` + Any structural failure (a role never confirming joined within its + bounded wait, a missing binary, a pre-flight conflict this script cannot + safely resolve) is a `FAIL:` line + non-zero exit instead, via the shared + `fail()` helper. +5. **Clean up** every process this run started -- unconditionally, on + success or failure alike -- unless `--keep-running` was passed. Log + files under the deploy directory are always preserved regardless of that + flag, matching `xce-test-resources`'s own documented divergence from + `xce-deploy-ha`'s `cleanup.sh` (which wipes `logs/*.log`). + +## Usage + +``` +bash .claude/skills/xce-test-multihost/run-multihost-verify.sh [--keep-running] [--dir DIR] +``` + +- `--dir DIR` -- deploy directory. Default: a **fixed**, reused path + (`/home/claude/tmp/xce-test-multihost`), following `xce-test-ha`'s own + convention (not `xce-test-resources`'s fresh-`mktemp`-per-run one) -- this + is what makes the pre-flight zombie check meaningful: a prior run's + un-torn-down cluster at this same path is a realistic failure mode, not a + hypothetical one. `$XCE_MULTIHOST_TOP` overrides the default when `--dir` + is not given. A non-empty `DIR` that was **not** created by a prior run of + this exact script is refused (`.xce-test-multihost` marker file guard, + identical convention to `run-ha-soak.sh`'s own `MARKER_NAME` guard) -- this + never silently wipes an arbitrary caller-named directory. +- `--keep-running` -- skip PROCESS teardown in cleanup, leaving the + deployment running under `DIR` for further inspection. Logs are always + preserved either way. +- Other overridable env vars: `$XCE_MULTIHOST_BUILD_DIR` (default + `src/xce.build`), `$XCE_MULTIHOST_BIN` (controller), + `$XCE_MULTIHOST_SCHED_BIN`, `$XCE_MULTIHOST_WORKER_BIN`, + `$XCE_MULTIHOST_XT` (`xce_test` -- not required by this task, checked as + an informational precondition for task 4.2's benefit), + `$XCE_MULTIHOST_PORT` (default `9991` -- single-node controller port), + `$XCE_MULTIHOST_WORKER1`/`$XCE_MULTIHOST_WORKER2` (default + `mh-worker-1`/`mh-worker-2`). + +Must be invoked under an environment with `nix` on `PATH` -- the script +shells out to `nix develop` itself for the steps that need it. + +**No `context: fork`, intentionally** -- same reason as every other skill in +this family (`xce-deploy-ha`, `xce-test-ha`, `xce-test-resources`): this +script starts real backgrounded controller/scheduler/worker processes for +the run's duration, and a forked Skill-tool context's own teardown sends +SIGHUP through the whole process group on return, silently killing them +mid-run. Dispatch directly via Bash, or via the `Agent` tool (empirically +verified safe for this exact pattern by the sibling skills above). + +## Prerequisites + +Controller, scheduler, and worker components must already be built in the +same meson build dir (default `src/xce.build`): + +``` +/xce-build controller --incremental +/xce-build scheduler --incremental +/xce-build worker --incremental +``` + +(`--incremental` matters here specifically because a plain, non-incremental +`/xce-build ` wipes the whole build dir before rebuilding just +that one component -- see this project's own bug-1719 lesson -- so building +controller/scheduler/worker one at a time into the same dir needs +`--incremental` on every call after the first.) + +## Worker sizing (researched for task 4.2's benefit, not wired in by this task) + +`xcecd_virtual` has **no** command-line or config override for its reported +processor/slot capacity: `worker/src/main.c`'s `getopt_long()` table is +`--peers`/`--version`/`--help`/`-n`/`-l`/`-v` only, and +`hardware_monitor.c`'s `hardware_monitor_update()` always falls back to +`sysconf(_SC_NPROCESSORS_ONLN)` (`hwloc_disabled = true` by default, so the +`hwloc`-based branch never actually runs). In other words, "virtual" in +`xcecd_virtual` means it skips real job execution/cgroups, **not** that it +fakes a different processor count -- each worker reports the real host's +actual core count (confirmed live in this sandbox: `nproc` reports 22). + +Consequence for task 4.2/4.3's scenario: since a worker's capacity cannot be +shrunk via configuration, the robust sizing strategy is to pick the test +job's `maxTasks` **strictly greater than** one worker's reported capacity +`N` but **at most `2N`**, so the job fits across exactly the two workers +this harness starts and neither alone can satisfy it. Query `nproc` live at +test-run time rather than hardcoding a number -- a different sandbox/CI host +will report a different core count. + +## Extension points for tasks 4.2 / 4.3 + +`run-multihost-verify.sh` has a clearly marked `EXTENSION POINT` comment +block (right after `confirm_cluster_up()`) documenting exactly where and how +the next two tasks should add: + +- `submit_multihost_job()` (task 4.2) -- a plain multi-task job via + `jobsub`, following `run-ha-soak.sh`'s `submit_worker_job()`/ + `verify_worker_rejoin_resync()` polling idiom verbatim (scripted + `xce_test` client-context session: `set masters`, `set myname`, `client`, + `wait-primary`, `jobsub -u root -q normal -c "..."`, `quit`; then poll + `jobinfo` for `RUN` then `DONE`). **Must** drive submission via plain + `jobsub`, never `xce_test`'s `jobrun` manual-allocation bypass -- the + scheduler's own decision to span `mh-worker-1`/`mh-worker-2` is what this + whole test exists to prove. After the job reaches `DONE`, grep **each** + worker's own log to confirm genuine per-host task execution. +- A second, array-job scenario (task 4.3) -- the same job submitted via + `jobsub -a N`, same polling idiom, confirming the array element receives + the same per-element task-count wiring a non-array job would + (Requirement 5.5). + +## Known pre-existing constraints this skill inherits (not fixed here) + +- Same `xce_test`/ASAN leak-detection caveat every sibling skill documents: + a future `submit_multihost_job()` must run `xce_test` with + `ASAN_OPTIONS=detect_leaks=0` and judge success from command **output**, + never the leak-polluted process exit code. +- Same nix-devshell runtime-linkage caveat: any future `xce_test` invocation + should be wrapped in `nix develop`, like this script's own `deploy-ha.sh` + invocation already is. + +## Dependencies + +- `xce-deploy-ha`'s `deploy-ha.sh` and its generated `start.sh` / + `start-scheduler.sh` / `start-worker.sh` (single-controller-node path, + two named worker instances against the same `--peers` config). +- Controller, scheduler, and worker binaries in the same build dir (see + Prerequisites above). + +## Requirements traceability + +This document describes design.md's `xce-test-multihost` component +(Components and Interfaces -> Test Infrastructure), covering Requirements +4.1, 4.2, 4.3, 5.3, 5.5. Task 4.1 (this revision) covers the deploy/teardown +harness only; the job-scenario coverage those requirements ultimately need +is completed by tasks 4.2/4.3. See +`.kiro/specs/resreq-maxtasks-wiring/requirements.md` and `design.md` for the +full requirement text. diff --git a/.claude/skills/xce-test-multihost/run-multihost-verify.sh b/.claude/skills/xce-test-multihost/run-multihost-verify.sh new file mode 100755 index 00000000..a65ae768 --- /dev/null +++ b/.claude/skills/xce-test-multihost/run-multihost-verify.sh @@ -0,0 +1,499 @@ +#!/usr/bin/env bash +# +# run-multihost-verify.sh -- driver for /xce-test-multihost. +# +# resreq-maxtasks-wiring spec, task 4.1 SCOPE ONLY: this revision implements +# the DEPLOY/TEARDOWN HARNESS -- a single controller/scheduler node plus two +# independently-named `xcecd_virtual` workers, each sized so neither alone +# can satisfy the eventual test job's requested task count (see the +# "Worker sizing" note near submit_multihost_job()'s placeholder below) -- +# with the pre-flight zombie-process/port check convention this project's +# xce-test-ha soak driver established, trap-based cleanup registered before +# the first process starts, and a HARNESS_READY sentinel + PASS/FAIL oracle +# line. It deliberately does NOT yet submit any job or check any allocation +# outcome -- see design.md's "xce-test-multihost" component (Batch/Job +# Contract) and tasks.md 4.1's own Observable: "the new harness deploys its +# fixture cluster and tears it down cleanly, with no scenario wired in yet". +# +# Tasks 4.2 (a plain multi-task job via `jobsub`) and 4.3 (the same, inside +# an array-job submission) build ON TOP of this harness in later dispatch +# rounds -- see the "EXTENSION POINT" comment block right after +# confirm_cluster_up() below for exactly where and how (reusing run-ha-soak.sh's +# submit_worker_job()/verify_worker_rejoin_resync() polling idiom, per +# design.md's Implementation Notes for this component -- real job via plain +# `jobsub`, poll `jobinfo` for RUN then DONE, grep each worker's own log -- +# never `xce_test`'s `jobrun` manual-allocation bypass, since the scheduler's +# own decision to span hosts is what this whole test exists to prove). +# +# Requirements this harness exists to eventually help satisfy (this task +# alone is infrastructure -- it does not itself complete any of these): +# 4.1 (controller ensures every allocated host has launch info), 4.2 (worker +# starts tasks on every allocated host), 4.3 (scheduler reports running via +# the existing job-status mechanism), 5.3 (a real multi-task job driven +# end-to-end at least once). See .kiro/specs/resreq-maxtasks-wiring/ +# {requirements,design}.md. +# +# ── Pattern provenance (read these before changing this file) ────────────── +# - Deploy/start/confirm/cleanup/trap mechanics: copied from +# xce-test-resources's run-resource-verify.sh (single-controller-node +# pattern: xce-deploy-ha's deploy-ha.sh + generated per-component start +# scripts, trap registered before the first process starts, TOP-scoped +# pkill in cleanup(), preserved logs). +# - Fixed default deploy dir + MARKER_NAME ownership-guarded `rm -rf` + +# pre-flight zombie/port check: copied from xce-test-ha's run-ha-soak.sh +# (which reuses the SAME default deploy dir, `/home/claude/tmp/...`, +# across invocations -- unlike xce-test-resources's fresh-mktemp-per-run +# convention -- so a TOP-scoped pre-flight zombie check is actually +# meaningful here: a prior run's un-torn-down cluster is a real, concrete +# possibility, not just a theoretical one). See this project's own +# cerebrum.md 2026-08-26 Do-Not-Repeat entry: a same-session leftover +# xce_controller/xce_scheduler/xcecd process bound to the exact ports a +# "fresh" deploy is about to reuse has caused apparently-impossible +# symptoms (term jumping, permanent config-load failure) in a live +# investigation before -- always check `pgrep`/`ss` for the ports about to +# be used before trusting a "fresh" deploy's results. +# - Two independently-named `xcecd_virtual` workers against one controller: +# xce-deploy-ha's own SKILL.md Step 4b ("start-worker.sh worker-1 -vvv &" +# / "start-worker.sh worker-2 -vvv &" against the identical --peers config +# baked into both generated scripts). NOT via start-all.sh, which only +# ever starts a single worker named "worker-1" -- this script starts the +# controller/scheduler/worker-1/worker-2 steps individually instead (same +# generated scripts start-all.sh itself would call, just sequenced by hand +# here so a second named worker can be added). +# +# Must be invoked under an environment with `nix` on PATH (same as +# xce-deploy-ha/xce-test-ha/xce-test-resources) -- this script shells out to +# `nix develop` itself for the steps that need it, so it does not need to be +# pre-wrapped by the caller. +# +# Usage: +# run-multihost-verify.sh [--keep-running] [--dir DIR] +# +# --keep-running Skip PROCESS teardown in cleanup -- the deployment +# (controller/scheduler/worker-1/worker-2) is left +# running under DIR for further inspection. Log files +# under DIR are ALWAYS preserved regardless of this +# flag. +# --dir DIR Deploy directory. Default: a FIXED path (see TOP +# below), following xce-test-ha's own "reused across +# invocations, not a fresh mktemp dir" convention -- +# $XCE_MULTIHOST_TOP overrides it when --dir is not +# given. A non-empty DIR that was NOT created by a +# prior run of this script is refused (MARKER_NAME +# guard, identical to run-ha-soak.sh's own), so this +# never wipes an unrelated directory. +# +# Exit 0 == the fixture cluster (controller + scheduler + 2 named workers) +# deployed and was confirmed fully joined; teardown ran cleanly. This is a +# "harness stood up and tore down correctly" verdict, NOT a job-outcome +# PASS/FAIL -- there is no job in this task's scope yet (see HARNESS_READY +# below). +# Exit non-zero == a structural failure at any step (pre-flight zombie/port +# conflict left unresolved, missing binary, deployment failure, a role that +# never confirmed joining within its bounded wait). Cleanup still runs on +# this path (trap-based, registered before the first process starts). +# +set -u + +SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$SELF_DIR/../../.." && pwd)" + +DEPLOY_HA="$REPO/src/tools/deploy-ha.sh" +BUILD_DIR="${XCE_MULTIHOST_BUILD_DIR:-$REPO/src/xce.build}" +BINARY="${XCE_MULTIHOST_BIN:-$BUILD_DIR/controller/xce_controller}" +SCHED_BIN="${XCE_MULTIHOST_SCHED_BIN:-$BUILD_DIR/scheduler/xce_scheduler}" +WORKER_BIN="${XCE_MULTIHOST_WORKER_BIN:-$BUILD_DIR/worker/xcecd_virtual}" +# xce_test -- not used by THIS task (no job is submitted yet), but checked +# as a soft/informational precondition since task 4.2 (the very next dispatch +# round) needs it immediately for submit_multihost_job()'s scripted session, +# following the exact idiom run-ha-soak.sh's submit_worker_job() already +# established (see the EXTENSION POINT block below). +XT="${XCE_MULTIHOST_XT:-$BUILD_DIR/controller/xce_test}" + +# Single-node cluster, deliberately -- design.md's own "no HA needed for this +# proof" call for this component. Single, fixed port so the pre-flight +# check below has one unambiguous thing to check for a conflict against. +PEER_PORT="${XCE_MULTIHOST_PORT:-9991}" +PEERS="localhost:$PEER_PORT" +MASTERS="localhost:$PEER_PORT" # what a future xce_test client session's `set masters` will use (task 4.2+) + +WORKER1_NAME="${XCE_MULTIHOST_WORKER1:-mh-worker-1}" +WORKER2_NAME="${XCE_MULTIHOST_WORKER2:-mh-worker-2}" + +# Fixed default deploy dir, REUSED across invocations (xce-test-ha's own +# convention, not xce-test-resources's fresh-mktemp-per-run one) -- this is +# what makes the pre-flight zombie/port check below meaningful: a prior run +# of this exact script, left running or killed uncleanly, is the realistic +# failure mode this check exists to catch, not a purely theoretical one. +TOP="${XCE_MULTIHOST_TOP:-/home/claude/tmp/xce-test-multihost}" +# Sentinel written into a deploy dir we own; the guard before `rm -rf` only +# wipes a non-empty dir that carries this marker -- identical convention to +# run-ha-soak.sh's own MARKER_NAME/guard, so an arbitrary caller-named `--dir` +# (e.g. a typo'd path) is never silently destroyed. +MARKER_NAME=".xce-test-multihost" + +KEEP_RUNNING=0 + +log() { echo "[xce-test-multihost] $*"; } +fail() { echo "[xce-test-multihost] FAIL: $*" >&2; exit 1; } + +usage() { + cat <&2 + usage + exit 2 + ;; + esac + done + # Resolve a relative --dir/$XCE_MULTIHOST_TOP against cwd, same pattern + # as xce-test-ha/xce-test-resources (no realpath dependency -- busybox + # realpath in this environment lacks -m). + case "$TOP" in + /*) : ;; + *) TOP="$PWD/$TOP" ;; + esac +} + +# ── Pre-flight zombie-process / port check (design.md: "Follows the +# pre-flight zombie-process/port check convention already established for +# xce-test-ha") ─────────────────────────────────────────────────────────── +# +# Two distinct checks, both scoped to what THIS run is about to do (never an +# unscoped `pkill -9 -x xce_controller`-style blast -- bug-1436's own +# postmortem, this project's HA soak, is exactly what an unscoped kill looks +# like going wrong: it silently killed a DIFFERENT concurrent run's live +# primary): +# +# 1. A process already bound to $PEER_PORT (`ss -tlnp`). If its cmdline +# shows it was started against THIS SAME $TOP (our own prior, +# un-torn-down run -- the realistic case, since $TOP is a fixed, +# reused path), it is safe to kill and this function does so. If its +# cmdline does NOT match $TOP -- an unrelated live process, or a +# process we cannot positively attribute to ourselves -- this function +# fails loud instead of guessing, per bug-1436's "fail loud, don't +# silently corrupt" rule. +# 2. Any stray xce_controller/xce_scheduler/xcecd_virtual process whose +# cmdline names THIS run's own $TOP (an un-torn-down previous run of +# this exact script at this exact deploy dir) -- killed unconditionally, +# since it is unambiguously ours by construction (node_pattern-style +# workdir/logfile substring match, same discipline run-ha-soak.sh's own +# node_pattern()/worker_pattern()/sched_pattern() use). +# +# This directly codifies this project's own cerebrum.md 2026-08-26 +# Do-Not-Repeat lesson (a same-session zombie controller/scheduler/worker +# process bound to the ports a "fresh" deploy is about to reuse has already +# produced apparently-impossible symptoms in a live investigation) as an +# automated check, rather than relying on a human/agent remembering to run +# `ps aux`/`ss -tlnp` by hand before every invocation. +ctrl_pattern() { echo "xce_controller .*--workdir $TOP/controller"; } +sched_pattern() { echo "xce_scheduler .*--logfile $TOP/logs/scheduler.log"; } +worker_pattern() { echo "xcecd_virtual.*-l $TOP/logs"; } +run_pattern() { echo "$(ctrl_pattern)|$(sched_pattern)|$(worker_pattern)"; } + +preflight_check() { + log "pre-flight: checking for a leftover process on port $PEER_PORT and any stale xce_controller/xce_scheduler/xcecd_virtual process scoped to $TOP" + + # Check 1: our own prior (un-torn-down) run at this exact TOP. + local stale_pids + stale_pids="$(pgrep -f -- "$(run_pattern)" 2>/dev/null || true)" + if [ -n "$stale_pids" ]; then + log "pre-flight: found stale process(es) from a prior run of this script scoped to $TOP -- killing: $stale_pids" + # shellcheck disable=SC2086 + kill -9 $stale_pids 2>/dev/null || true + sleep 0.3 + if pgrep -f -- "$(run_pattern)" >/dev/null 2>&1; then + fail "pre-flight: could not clear stale process(es) scoped to $TOP after kill -9 -- inspect manually (pgrep -af \"$(run_pattern)\")" + fi + log "pre-flight: stale process(es) scoped to $TOP cleared" + else + log "pre-flight: no stale process scoped to $TOP found" + fi + + # Check 2: port conflict, from ANY process (ours or not). Re-checked + # AFTER check 1's kill above, so a just-cleared stale process on our own + # port no longer counts as a conflict. + local port_line + port_line="$(ss -tlnp 2>/dev/null | grep -E ":$PEER_PORT[[:space:]]" || true)" + if [ -n "$port_line" ]; then + # Was it one of the stale PIDs we just killed (or is it still racing + # to exit)? Give it one more short grace check before failing loud. + sleep 0.3 + port_line="$(ss -tlnp 2>/dev/null | grep -E ":$PEER_PORT[[:space:]]" || true)" + fi + if [ -n "$port_line" ]; then + fail "pre-flight: port $PEER_PORT is already bound by a process this script did not start/attribute to $TOP -- refusing to deploy on top of it (see: ss -tlnp | grep :$PEER_PORT). Choose a different --dir/\$XCE_MULTIHOST_PORT, or stop that process yourself first." + fi + + log "pre-flight: clean -- port $PEER_PORT free, no stale process scoped to $TOP" +} + +precondition_checks() { + [ -x "$BINARY" ] || fail "controller binary missing: $BINARY (run /xce-build controller)" + [ -x "$SCHED_BIN" ] || fail "scheduler binary missing: $SCHED_BIN (run /xce-build scheduler)" + [ -x "$WORKER_BIN" ] || fail "worker binary missing: $WORKER_BIN (run /xce-build worker)" + if [ ! -x "$XT" ]; then + log "note: xce_test binary missing at $XT -- not required by this task (no job submitted yet), but task 4.2 will need it (run /xce-build controller)" + fi + command -v nix >/dev/null 2>&1 || fail "nix not found on PATH -- required to resolve the deploy-environment bash and invoke deploy-ha.sh" +} + +# ── Cleanup: process teardown only, logs always preserved (same convention +# as xce-test-resources's cleanup(): TOP-scoped patterns, retry-until- +# verified, never touches $TOP/logs) ───────────────────────────────────── +cleanup() { + if [ "$KEEP_RUNNING" -eq 1 ]; then + log "--keep-running requested -- leaving controller/scheduler/$WORKER1_NAME/$WORKER2_NAME up under $TOP (logs also preserved)" + return 0 + fi + log "cleaning up processes under $TOP (log files under $TOP/logs are preserved)" + local pattern; pattern="$(run_pattern)" + local attempt=0 + while [ "$attempt" -lt 10 ]; do + attempt=$((attempt + 1)) + if ! pgrep -f -- "$pattern" >/dev/null 2>&1; then + break + fi + pkill -9 -f -- "$pattern" 2>/dev/null + sleep 0.2 + done + if pgrep -f -- "$pattern" >/dev/null 2>&1; then + log "warning: some processes under $TOP may still be running after cleanup retries" + fi + return 0 +} + +# Standard robust trap idiom (xce-test-resources's on_exit/register_trap, +# copied verbatim in spirit): registered BEFORE the first process starts, so +# it fires on a completed run, a mid-deployment abort, OR an external +# SIGINT/SIGTERM alike -- not just a successful completion. +on_exit() { + local rc=$? + cleanup + exit "$rc" +} + +register_trap() { + trap on_exit EXIT + trap 'exit 130' INT + trap 'exit 143' TERM +} + +# ── Step 1: deploy (xce-deploy-ha's deploy-ha.sh) ─────────────────────────── +deploy() { + log "resolving deploy-environment bash via nix develop" + BASH_BIN="$(nix develop "$REPO/src" --command which bash)" + [ -n "$BASH_BIN" ] || fail "deployment failed at step deploy-ha.sh: could not resolve bash inside nix develop" + + # Ownership-guarded rm -rf -- identical convention to run-ha-soak.sh's + # own MARKER_NAME guard: refuse to wipe a non-empty dir unless WE (a + # prior run of this exact script) created it. An empty or non-existent + # dir is always safe to (re)deploy into. + if [ -d "$TOP" ] && [ -n "$(ls -A "$TOP" 2>/dev/null)" ] && [ ! -e "$TOP/$MARKER_NAME" ]; then + fail "refusing to wipe non-empty $TOP -- not a prior xce-test-multihost deploy (no $MARKER_NAME marker). Choose an empty/new --dir, or remove it yourself." + fi + rm -rf "$TOP" + mkdir -p "$TOP" || fail "could not create deploy dir $TOP" + touch "$TOP/$MARKER_NAME" # claim ownership so a later run may safely re-wipe it + + log "deploying controller+scheduler+worker fixture into $TOP (peers=$PEERS)" + if ! nix develop "$REPO/src" --command "$BASH_BIN" "$DEPLOY_HA" "$BINARY" "$TOP" "$PEERS" >"$TOP/deploy.log" 2>&1; then + fail "deployment failed at step deploy-ha.sh (see $TOP/deploy.log)" + fi + + [ -x "$TOP/scripts/start.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start.sh was not generated" + [ -x "$TOP/scripts/start-scheduler.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start-scheduler.sh was not generated (scheduler binary not found alongside controller build?)" + [ -x "$TOP/scripts/start-worker.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start-worker.sh was not generated (worker binary not found alongside controller build?)" +} + +# ── Step 2: start controller, scheduler, and BOTH named workers ──────────── +# Deliberately NOT start-all.sh -- that generated script only ever starts a +# single worker hardcoded to the name "worker-1". This harness needs TWO +# independently-named xcecd_virtual instances against the same controller +# (design.md: "two named xcecd_virtual instances, each sized so neither +# alone can satisfy the test job's requested task count"), so the +# controller/scheduler/worker-1/worker-2 steps start-all.sh itself would run +# are sequenced here individually instead -- same generated scripts, same +# --peers config baked into every one of them by deploy-ha.sh, nothing +# duplicated or reimplemented. Every process is backgrounded via +# `setsid nohup ... /dev/null 2>&1 & disown`, matching +# start-all.sh's own convention, so the whole fixture survives this script's +# own foreground steps (and, if --keep-running is passed, survives this +# script's own exit too). +start_cluster() { + log "starting controller (ctrl-1, -vvv)" + setsid nohup "$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv \ + >"$TOP/logs/ctrl-1.stderr" 2>&1 & + disown + + # Fixed filename, not a glob: this is always a single-node deployment + # (deploy-ha.sh names its one candidate "ctrl-1" unconditionally), and a + # glob here would be a real timing bug -- bash expands "$TOP"/logs/ctrl-*.log + # exactly ONCE, at the moment this line is evaluated (as part of building + # wait_for_log's argv), not once per poll iteration inside the function. + # Immediately after backgrounding the controller above, its log file may + # not exist yet; an unmatched glob falls through as the literal + # unexpanded string "ctrl-*.log" (bash's default, non-nullglob behavior), + # which then never matches any real file for the rest of the bounded + # wait -- confirmed live during this task's own validation run (the + # controller elected CTRL_PRIMARY as its very first log line, but the + # glob-based wait_for_log call still timed out after the full 30s). + wait_for_log "controller primary elected" "CTRL_PRIMARY" 30 "$TOP/logs/ctrl-1.log" \ + || fail "deployment failed at step controller: no primary election observed in $TOP/logs/ctrl-1.log within 30s" + + log "starting scheduler (-vvv)" + setsid nohup "$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -vvv \ + >"$TOP/logs/scheduler.stderr" 2>&1 & + disown + + log "starting worker '$WORKER1_NAME' (-vvv)" + setsid nohup "$BASH_BIN" "$TOP/scripts/start-worker.sh" "$WORKER1_NAME" -vvv \ + >"$TOP/logs/$WORKER1_NAME.stderr" 2>&1 & + disown + + log "starting worker '$WORKER2_NAME' (-vvv)" + setsid nohup "$BASH_BIN" "$TOP/scripts/start-worker.sh" "$WORKER2_NAME" -vvv \ + >"$TOP/logs/$WORKER2_NAME.stderr" 2>&1 & + disown +} + +# ── Step 3: confirm all four roles are actually up and joined ────────────── +# Same bounded-wait/short-poll-interval convention as xce-test-resources's +# wait_for_log() (copied verbatim) and xce-deploy-ha's own documented +# verification greps for scheduler/worker join lines: +# scheduler: "sched_init: joined cluster" (scheduler.c, confirmed live) +# worker: "Successfully sent .* load indices" (controller_connector.c, +# confirmed live), per NAMED worker's own log file +# ($TOP/logs/xcecd-.log for xcecd_virtual). +wait_for_log() { + local desc="$1" pattern="$2" timeout="$3" + shift 3 + local waited=0 + while [ "$waited" -lt "$timeout" ]; do + if grep -qE -- "$pattern" "$@" 2>/dev/null; then + log "confirmed: $desc" + return 0 + fi + sleep 1 + waited=$((waited + 1)) + done + return 1 +} + +confirm_cluster_up() { + log "waiting for scheduler and both workers to confirm joined (bounded wait per role; controller primary election already confirmed by start_cluster())" + + wait_for_log "scheduler joined cluster" "sched_init: joined cluster" 30 "$TOP/logs/scheduler.log" \ + || fail "deployment failed at step scheduler: '$TOP/logs/scheduler.log' never logged joining the cluster within 30s" + + wait_for_log "worker '$WORKER1_NAME' joined and reported load indices" "Successfully sent .* load indices" 30 "$TOP/logs/xcecd-$WORKER1_NAME.log" \ + || fail "deployment failed at step worker: '$TOP/logs/xcecd-$WORKER1_NAME.log' never logged sending load indices within 30s" + + wait_for_log "worker '$WORKER2_NAME' joined and reported load indices" "Successfully sent .* load indices" 30 "$TOP/logs/xcecd-$WORKER2_NAME.log" \ + || fail "deployment failed at step worker: '$TOP/logs/xcecd-$WORKER2_NAME.log' never logged sending load indices within 30s" + + log "all four roles (controller, scheduler, $WORKER1_NAME, $WORKER2_NAME) confirmed up under $TOP" +} + +# ═══════════════════════════════════════════════════════════════════════════ +# EXTENSION POINT for tasks 4.2/4.3 (NOT implemented by this task -- task 4.1 +# is deploy/teardown-only, per its own Observable in tasks.md). +# +# Task 4.2 (a plain multi-task job): add a submit_multihost_job() here, +# following run-ha-soak.sh's submit_worker_job() idiom EXACTLY (scripted +# xce_test client-context session piped via `xtest()`-style helper: `set +# masters $MASTERS`, `set myname `, `client`, `wait-primary `, +# `jobsub -u root -q normal -c ""`, `quit`), then poll `jobinfo_of +# "$jid" `-style for the RUN -> DONE transition (run-ha-soak.sh's +# verify_worker_rejoin_resync() polling idiom) -- per design.md's +# Implementation Notes ("reuses run-ha-soak.sh's submit_worker_job()/ +# verify_worker_rejoin_resync() polling idiom ... rather than inventing a new +# verification mechanism"). MUST drive submission via plain `jobsub`, never +# `xce_test`'s `jobrun` manual-allocation bypass -- the scheduler's own +# decision to span $WORKER1_NAME/$WORKER2_NAME is what this whole test +# exists to prove (design.md's own Responsibilities & Constraints for this +# component). After a job reaches DONE, grep EACH worker's own log +# ($TOP/logs/xcecd-$WORKER1_NAME.log / xcecd-$WORKER2_NAME.log) to confirm +# genuine per-host task execution, not just an aggregate job-state check. +# +# Task 4.3 (the same, as one element of an array-job submission via +# `jobsub -a N`): a second scenario function alongside submit_multihost_job(), +# same polling idiom, confirming the SAME per-element task-count wiring a +# non-array job gets (Requirement 5.5). +# +# Worker sizing (researched during task 4.1, not wired in yet -- see this +# task's own Status Report CONCERNS for the full writeup): xcecd_virtual has +# NO command-line/config override for its reported processor/slot capacity +# anywhere in worker/src/main.c's getopt_long() table ("peers"/"version"/ +# "help"/-n/-l/-v only). hardware_monitor.c's hardware_monitor_update() +# always falls back to sysconf(_SC_NPROCESSORS_ONLN) (hwloc_disabled=true by +# default, so the hwloc branch never runs) -- i.e. "virtual" only means +# xcecd_virtual skips REAL job execution/cgroups, NOT that it fakes a +# different processor count; each of $WORKER1_NAME/$WORKER2_NAME will report +# this sandbox's real host core count (confirmed: nproc=22 here). Task 4.2's +# scenario should therefore pick a maxTasks strictly greater than that +# per-worker capacity N but at most 2N, so the job fits across exactly +# $WORKER1_NAME + $WORKER2_NAME and neither alone can satisfy it -- e.g. +# `nproc` at test-run time + 1, clamped to <= 2 * nproc. Do not hardcode 22; +# query it live (e.g. `nproc` inside the deploy environment) since a +# different sandbox/CI host will report a different core count. +# ═══════════════════════════════════════════════════════════════════════════ + +main() { + parse_args "$@" + + # Trap registered before the FIRST process is started (deploy() only + # generates scripts/certs/config; start_cluster() is what actually + # starts backgrounded processes) -- covers a mid-deployment abort (e.g. + # worker-2 fails to join after controller+scheduler+worker-1 are already + # up), not just a completed run. + register_trap + + precondition_checks + preflight_check + deploy + start_cluster + confirm_cluster_up + + log "HARNESS_READY controller=up scheduler=up workers=2/2 top=$TOP" + log "PASS (harness-ready): controller + scheduler + 2 named worker(s) ($WORKER1_NAME, $WORKER2_NAME) deployed and confirmed joined; no job scenario submitted yet (task 4.1 scope only -- see tasks 4.2/4.3)" + exit 0 +} + +# Guarded so this file can be `source`d (e.g. a future synthetic test +# harness) without executing main() as a side effect -- same convention as +# xce-test-resources's own bottom-of-file guard. +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + main "$@" +fi diff --git a/.kiro/specs/resreq-maxtasks-wiring/tasks.md b/.kiro/specs/resreq-maxtasks-wiring/tasks.md index 4df5e39a..be7e4360 100644 --- a/.kiro/specs/resreq-maxtasks-wiring/tasks.md +++ b/.kiro/specs/resreq-maxtasks-wiring/tasks.md @@ -57,7 +57,7 @@ - _Depends: 2.1_ - [ ] 4. Validation: real multi-host end-to-end proof -- [ ] 4.1 Build the multi-host end-to-end test harness +- [x] 4.1 Build the multi-host end-to-end test harness - Add a new test skill that deploys one controller/scheduler node plus two named virtual workers, each sized so neither alone can satisfy the scenario's requested task count - Follow the existing soak-test convention for pre-flight zombie-process/port checks and a pass/fail verdict line - Observable: the new harness deploys its fixture cluster and tears it down cleanly, with no scenario wired in yet @@ -89,5 +89,6 @@ - _Requirements: 5.2_ ## Implementation Notes +- Task 4.1's `xce-test-multihost` harness reuses a fixed, wipe-and-recreate deploy directory (`xce-test-ha`'s convention) instead of design.md's stated "fresh, uniquely-named cluster instance; no shared state across runs" bullet (`xce-test-resources`'s convention). Reason: task 4.1's own text requires "the existing soak-test convention for pre-flight zombie-process/port checks," and a meaningful pre-flight check needs a stable, known directory/port to check against — `xce-test-resources`'s fresh-mktemp-per-run model has no such check at all (confirmed: it has none) because one would be moot there. The "no shared state" intent is preserved functionally: `deploy()` unconditionally wipes and recreates the directory on every invocation, verified clean (zero cross-run residue) across two consecutive live runs. No production code or requirement scope changed. - Task 3.2's driver/simulation-parity test was split into a second new file, `src/scheduler/testsuite/test_maxtasks_wiring_sim_driver_parity.c` (registered separately in `meson.build`), rather than living inside `test_maxtasks_wiring_ingest.c` as originally scoped. Reason: proving this specific claim requires white-box `#include`-ing `sim_loop.c`/`core_events.c` to reach their real `static` translators (the only way to avoid a mirrored test that can't catch a real translator regression — confirmed via an independent mutation-test reproduction during review), but `test_maxtasks_wiring_ingest.c` already links the full real `scheduler_srcs` object list for its own task-3.1 test, and the two techniques collide at link time (duplicate symbols) in one translation unit. No production code or requirement scope changed; this is a file-boundary split only, driven by a real toolchain constraint discovered during implementation. - Task 2.1 surfaced an out-of-boundary side effect: `src/scheduler/testsuite/test_repl_breakpoints_inspection.c:735`'s `test_info_job_reports_state_queue_priority` hardcodes an incidental `"Tasks: min = 0, max = 0"` byproduct of its fixture's `parse_select()` failing on an unregistered `"type"` resource (which previously left the primary's task-count fields at memset-zero, before `apply_job_task_count()` existed). Now correctly overwritten to `(1,1)` per Requirement 1.2. Before task 5.1 is considered complete, change that assertion's expected substring to `"Tasks: min = 1, max = 1"`.