diff --git a/.claude/skills/xce-test-multihost/SKILL.md b/.claude/skills/xce-test-multihost/SKILL.md index 43cf813b..154e175e 100644 --- a/.claude/skills/xce-test-multihost/SKILL.md +++ b/.claude/skills/xce-test-multihost/SKILL.md @@ -1,38 +1,41 @@ --- name: xce-test-multihost -description: "End-to-end multi-host job allocation harness: deploys one controller/scheduler node plus two named xcecd_virtual workers (each sized so neither alone can satisfy the test job's requested task count) and confirms all four roles come up and join cleanly. Usage: bash .claude/skills/xce-test-multihost/run-multihost-verify.sh [--keep-running] [--dir DIR]" +description: "End-to-end multi-host job allocation proof: deploys one controller/scheduler node plus two named xcecd_virtual workers (each sized so neither alone can satisfy the test job's requested task count), submits a real multi-task job via plain jobsub, and confirms the scheduler's own decision genuinely splits it across both hosts, which both reach DONE. Usage: bash .claude/skills/xce-test-multihost/run-multihost-verify.sh [--keep-running] [--dir DIR]" model: haiku --- # XCE Multi-Host Allocation Test Harness -## Status: task 4.1 only -- deploy/teardown harness, no scenario wired in yet +## Status: tasks 4.1 + 4.2 -- harness plus a real plain-job scenario; 4.3 (array-job variant) not yet wired in -`run-multihost-verify.sh` currently implements the resreq-maxtasks-wiring -spec's task 4.1 scope: it deploys a single controller/scheduler node plus -two independently-named `xcecd_virtual` workers, confirms all four roles -come up and join, emits a `HARNESS_READY` sentinel + PASS/FAIL oracle line, -and tears everything down cleanly. **It does not yet submit any job or -check any allocation outcome** -- that is tasks 4.2 (a plain multi-task job -via `jobsub`) and 4.3 (the same, as part of an array-job submission), which -build directly on top of this same script (see the `EXTENSION POINT` -comment block in `run-multihost-verify.sh`, right after -`confirm_cluster_up()`). +`run-multihost-verify.sh` implements the resreq-maxtasks-wiring spec's task +4.1 scope (deploy a single controller/scheduler node plus two +independently-named `xcecd_virtual` workers, confirm all four roles come up +and join) **plus** task 4.2's scenario on top of it: once the fixture is up +(`HARNESS_READY`), it submits ONE real multi-task job via plain `jobsub`, +polls `jobinfo` for `RUN` then `DONE`, confirms BOTH named workers' +own logs independently show them starting their own share of the job's +tasks, and emits a PASS/FAIL verdict reflecting that real outcome -- not +just "harness stood up". **Not yet implemented**: task 4.3 (the same +scenario submitted as one element of an array job via `jobsub -a N`) -- +see the `EXTENSION POINT for task 4.3` comment block near the bottom of +`run-multihost-verify.sh`. -This closes bug-1770's own end-to-end validation gap once 4.2/4.3 land: the -scheduler-allocation feature already implements, unit-tests, and -soak-validates real multi-host allocation logic (`parjob_allocator`) that, -until resreq-maxtasks-wiring's own fix lands, has no way to ever be -reached by a real job submission. This skill exists to prove -- with a real -controller, a real scheduler, and real (virtual) workers, driven through -plain `jobsub` -- that a job requiring more tasks than one host can provide -is genuinely split across two hosts by the scheduler's own decision, and -actually runs to completion on both. +This closes bug-1770's own end-to-end validation gap for the plain-job path +(this revision): the scheduler-allocation feature already implements, +unit-tests, and soak-validates real multi-host allocation logic +(`parjob_allocator`) that, until resreq-maxtasks-wiring's own fix lands, has +no way to ever be reached by a real job submission. This skill now proves -- +with a real controller, a real scheduler, and real (virtual) workers, driven +through plain `jobsub` -- that a job requiring more tasks than one host can +provide is genuinely split across two hosts by the scheduler's own decision, +and actually runs to completion on both. The array-job variant of this same +proof (Requirement 5.5) remains open until task 4.3 lands. Traces to `.kiro/specs/resreq-maxtasks-wiring/design.md`'s `xce-test-multihost` component (Requirements 4.1, 4.2, 4.3, 5.3, 5.5). -## What this skill does today (task 4.1) +## What this skill does today (tasks 4.1 + 4.2) 1. **Pre-flight zombie-process/port check** -- following the convention already established by `xce-test-ha`'s `run-ha-soak.sh` (a fixed, reused @@ -73,19 +76,48 @@ Traces to `.kiro/specs/resreq-maxtasks-wiring/design.md`'s `$TOP/logs/xcecd-.log` -- checked independently per worker, not just once for "a" worker, since this harness's whole point is TWO independently-joined hosts. -4. **Report a `HARNESS_READY` sentinel + PASS/FAIL oracle line** -- since - there is no job outcome to report yet at this stage, the terminal line - reports "harness stood up and tore down correctly", not a job-allocation - verdict: +4. **Report a `HARNESS_READY` sentinel** once all four roles are confirmed + joined -- an intermediate milestone log line, not the final verdict. +5. **Size and submit ONE real multi-task job via plain `jobsub`** (task + 4.2) -- never `xce_test`'s `jobrun` manual-allocation bypass, so the + scheduler's own dispatch decision is what actually spans the two named + workers. Task count is queried live via `nproc` on each worker (strictly + greater than one worker's own capacity, at most both combined -- + `xcecd_virtual` reports the REAL host core count, confirmed no + config/CLI override exists to shrink it), forcing a genuine 2-host + split; empirically confirmed live (this task's own validation) that a + small job well within one worker's capacity (`-n 5,5`) stays on a + SINGLE host, so this sizing is what actually forces the multi-host path, + not something the scheduler would do regardless of size. The job command + is `sleep `, the SAME mechanism `xce-test-ha`'s `run-ha-soak.sh` uses + to drive a virtual worker's own timer-based simulated execution. +6. **Poll `jobinfo` for `RUN` then `DONE`** -- reusing `run-ha-soak.sh`'s + `jobinfo_of()`/state-polling idiom verbatim (scripted `xce_test` + client-context session: `set masters`, `set myname`, `client`, + `wait-primary`, `jobinfo -j `, `quit`; `state: RUN`/`state: DONE` + extracted via `sed`), bounded timeouts throughout. +7. **Confirm EACH allocated worker's own log independently shows it + starting its own share of the job's tasks** -- the harness's central + claim. Confirmed live that the controller sends a startjob message to + only the FIRST allocated host, which itself relays a runtasks message + directly to the OTHER allocated host(s); this script accepts EITHER + path's own jobid-tagged evidence line (`"Job ntasks is: "` from + the startjob path, or `"run task: jobid = "` from the relayed + runtasks path) on each worker's own `$TOP/logs/xcecd-.log`, since + which host takes which path is the allocator's own internal choice. +8. **Report a real PASS/FAIL verdict** reflecting the job's actual outcome + -- not just "harness stood up": ``` [xce-test-multihost] HARNESS_READY controller=up scheduler=up workers=2/2 top=/home/claude/tmp/xce-test-multihost - [xce-test-multihost] PASS (harness-ready): controller + scheduler + 2 named worker(s) (mh-worker-1, mh-worker-2) deployed and confirmed joined; no job scenario submitted yet (task 4.1 scope only -- see tasks 4.2/4.3) + ... + [xce-test-multihost] PASS: job 0-9-1788684444-366142 (23 tasks, sleep 5s) submitted via plain jobsub, reached RUN then DONE; mh-worker-1 and mh-worker-2 each independently logged starting their own share of its tasks -- the scheduler's own decision genuinely spans both hosts and the job completed on both ``` - Any structural failure (a role never confirming joined within its - bounded wait, a missing binary, a pre-flight conflict this script cannot - safely resolve) is a `FAIL:` line + non-zero exit instead, via the shared - `fail()` helper. -5. **Clean up** every process this run started -- unconditionally, on + Any structural failure (a role never confirming joined, a missing + binary, a pre-flight conflict) OR job-scenario failure (submission + rejected, RUN/DONE never reached within its bounded timeout, either + worker never logging its own share starting) is a `FAIL:` line + + non-zero exit instead, via the shared `fail()` helper. +9. **Clean up** every process this run started -- unconditionally, on success or failure alike -- unless `--keep-running` was passed. Log files under the deploy directory are always preserved regardless of that flag, matching `xce-test-resources`'s own documented divergence from @@ -113,11 +145,15 @@ bash .claude/skills/xce-test-multihost/run-multihost-verify.sh [--keep-running] - Other overridable env vars: `$XCE_MULTIHOST_BUILD_DIR` (default `src/xce.build`), `$XCE_MULTIHOST_BIN` (controller), `$XCE_MULTIHOST_SCHED_BIN`, `$XCE_MULTIHOST_WORKER_BIN`, - `$XCE_MULTIHOST_XT` (`xce_test` -- not required by this task, checked as - an informational precondition for task 4.2's benefit), + `$XCE_MULTIHOST_XT` (`xce_test` -- REQUIRED as of task 4.2's job scenario, + a hard precondition failure if missing), `$XCE_MULTIHOST_PORT` (default `9991` -- single-node controller port), `$XCE_MULTIHOST_WORKER1`/`$XCE_MULTIHOST_WORKER2` (default - `mh-worker-1`/`mh-worker-2`). + `mh-worker-1`/`mh-worker-2`), `$XCE_MULTIHOST_WAITMS` (default `5000` -- + `wait-primary` budget per `xce_test` session), `$XCE_MULTIHOST_JOB_SLEEP` + (default `5` -- seconds the test job's `sleep ` command runs), + `$XCE_MULTIHOST_RUN_TIMEOUT` (default `30`), `$XCE_MULTIHOST_DONE_TIMEOUT` + (default `$XCE_MULTIHOST_JOB_SLEEP + 30`). Must be invoked under an environment with `nix` on `PATH` -- the script shells out to `nix develop` itself for the steps that need it. @@ -147,7 +183,7 @@ that one component -- see this project's own bug-1719 lesson -- so building controller/scheduler/worker one at a time into the same dir needs `--incremental` on every call after the first.) -## Worker sizing (researched for task 4.2's benefit, not wired in by this task) +## Worker sizing (researched during task 4.1, wired in by task 4.2) `xcecd_virtual` has **no** command-line or config override for its reported processor/slot capacity: `worker/src/main.c`'s `getopt_long()` table is @@ -159,44 +195,66 @@ processor/slot capacity: `worker/src/main.c`'s `getopt_long()` table is fakes a different processor count -- each worker reports the real host's actual core count (confirmed live in this sandbox: `nproc` reports 22). -Consequence for task 4.2/4.3's scenario: since a worker's capacity cannot be -shrunk via configuration, the robust sizing strategy is to pick the test -job's `maxTasks` **strictly greater than** one worker's reported capacity -`N` but **at most `2N`**, so the job fits across exactly the two workers -this harness starts and neither alone can satisfy it. Query `nproc` live at -test-run time rather than hardcoding a number -- a different sandbox/CI host -will report a different core count. +`compute_task_count()` in `run-multihost-verify.sh` therefore queries +`nproc` live at test-run time (never hardcoded) and picks the test job's +task count as `nproc + 1` -- strictly greater than one worker's reported +capacity `N` but at most `2N`, so the job fits across exactly the two +workers this harness starts and neither alone can satisfy it. **Empirically +confirmed** (a separate, one-off manual check during this task's own +validation, not part of the committed script): submitting a `-n 5,5` job +against the same live cluster landed entirely on ONE worker (`ntasks is: 5` +on that worker's own log, zero occurrences of the jobid on the other) -- +confirming the `nproc + 1` sizing is what actually forces the 2-host split, +not something the scheduler would do for any job regardless of size. -## Extension points for tasks 4.2 / 4.3 +## Per-host task-start evidence: two possible code paths (confirmed live) -`run-multihost-verify.sh` has a clearly marked `EXTENSION POINT` comment -block (right after `confirm_cluster_up()`) documenting exactly where and how -the next two tasks should add: +Requirement 4.1 says the controller "shall ensure every allocated host has +the information it needs to start its own share" -- it does NOT mandate +*how* each host receives that information, and in practice it is not a +uniform per-host broadcast: confirmed live (this task's own first +validation run) that the controller sends a startjob message to only the +FIRST allocated host; that host's own `job_start_remote_task()` +(`controller_connector.c` ~725-790) then relays a runtasks message directly +to every OTHER allocated host via `xcecli_runtask()`. Both still leave every +host with a genuine, independently-verifiable "I started my own share" log +line, tagged with the exact jobid (`fmt_jobid()`, the SAME formatting +function `xce_test`'s own `cmd_jobsub` uses for `jobsubmit ok: JobId: `): -- `submit_multihost_job()` (task 4.2) -- a plain multi-task job via - `jobsub`, following `run-ha-soak.sh`'s `submit_worker_job()`/ - `verify_worker_rejoin_resync()` polling idiom verbatim (scripted - `xce_test` client-context session: `set masters`, `set myname`, `client`, - `wait-primary`, `jobsub -u root -q normal -c "..."`, `quit`; then poll - `jobinfo` for `RUN` then `DONE`). **Must** drive submission via plain - `jobsub`, never `xce_test`'s `jobrun` manual-allocation bypass -- the - scheduler's own decision to span `mh-worker-1`/`mh-worker-2` is what this - whole test exists to prove. After the job reaches `DONE`, grep **each** - worker's own log to confirm genuine per-host task execution. -- A second, array-job scenario (task 4.3) -- the same job submitted via - `jobsub -a N`, same polling idiom, confirming the array element receives - the same per-element task-count wiring a non-array job would - (Requirement 5.5). +- startjob path (`job_start_local_task()`, `controller_connector.c` ~1012): + `"Job ntasks is: "` at `XCE_LOG_DEBUG`. +- relayed runtasks path (`handle_runtasks()`, `controller_connector.c` + ~1174): `"run task: jobid = "` at `XCE_LOG_DEBUG3`. -## Known pre-existing constraints this skill inherits (not fixed here) +Which of `mh-worker-1`/`mh-worker-2` takes which path is the allocator's own +internal host-ordering choice, not something this script predicts -- +`verify_task_started_on_host()` accepts EITHER line as equally valid +evidence for a given host. (A third, path-independent line that also fires +on both --`"Starting worker as user ... to process command: ..."`, +`xcecd.c` ~54 -- lacks the jobid tag, so it is not used as the primary +signal, but corroborates the two above when inspecting a log by hand.) + +## Extension point for task 4.3 + +`run-multihost-verify.sh` has a clearly marked `EXTENSION POINT for task +4.3` comment block (near the bottom of the file, after +`run_multihost_job_scenario()`) documenting exactly where and how to add a +second, array-job scenario -- the same job submitted via `jobsub -a N`, +same `jobinfo_of()`/`wait_for_job_state()` polling idiom, confirming the +array element receives the same per-element task-count wiring a non-array +job would (Requirement 5.5). + +## Known pre-existing constraints this skill inherits - Same `xce_test`/ASAN leak-detection caveat every sibling skill documents: - a future `submit_multihost_job()` must run `xce_test` with - `ASAN_OPTIONS=detect_leaks=0` and judge success from command **output**, - never the leak-polluted process exit code. -- Same nix-devshell runtime-linkage caveat: any future `xce_test` invocation - should be wrapped in `nix develop`, like this script's own `deploy-ha.sh` - invocation already is. + this script's `xtest()` helper runs `xce_test` with + `ASAN_OPTIONS=detect_leaks=0` and judges success from command **output** + (`jobsubmit ok: JobId: `, `^JobID: ` in `jobinfo`), never the + leak-polluted process exit code. +- Same nix-devshell runtime-linkage caveat: every `xce_test` invocation + (`jobinfo_of()`, `submit_multihost_job()`, via the shared `xtest()` + helper) is wrapped in `nix develop`, like this script's own + `deploy-ha.sh` invocation. ## Dependencies @@ -210,8 +268,10 @@ the next two tasks should add: This document describes design.md's `xce-test-multihost` component (Components and Interfaces -> Test Infrastructure), covering Requirements -4.1, 4.2, 4.3, 5.3, 5.5. Task 4.1 (this revision) covers the deploy/teardown -harness only; the job-scenario coverage those requirements ultimately need -is completed by tasks 4.2/4.3. See +4.1, 4.2, 4.3, 5.3, 5.5. Task 4.1 built the deploy/teardown harness; task +4.2 (this revision) drives a real plain multi-task job through it end to +end, satisfying 4.1, 4.2, 4.3, and 5.3 with live evidence. Requirement 5.5 +(the same proof for an array-job submission) remains open until task 4.3 +lands -- see the `EXTENSION POINT for task 4.3` block. See `.kiro/specs/resreq-maxtasks-wiring/requirements.md` and `design.md` for the full requirement text. diff --git a/.claude/skills/xce-test-multihost/run-multihost-verify.sh b/.claude/skills/xce-test-multihost/run-multihost-verify.sh index a65ae768..67abadf3 100755 --- a/.claude/skills/xce-test-multihost/run-multihost-verify.sh +++ b/.claude/skills/xce-test-multihost/run-multihost-verify.sh @@ -2,36 +2,43 @@ # # run-multihost-verify.sh -- driver for /xce-test-multihost. # -# resreq-maxtasks-wiring spec, task 4.1 SCOPE ONLY: this revision implements -# the DEPLOY/TEARDOWN HARNESS -- a single controller/scheduler node plus two -# independently-named `xcecd_virtual` workers, each sized so neither alone -# can satisfy the eventual test job's requested task count (see the -# "Worker sizing" note near submit_multihost_job()'s placeholder below) -- -# with the pre-flight zombie-process/port check convention this project's -# xce-test-ha soak driver established, trap-based cleanup registered before -# the first process starts, and a HARNESS_READY sentinel + PASS/FAIL oracle -# line. It deliberately does NOT yet submit any job or check any allocation -# outcome -- see design.md's "xce-test-multihost" component (Batch/Job -# Contract) and tasks.md 4.1's own Observable: "the new harness deploys its -# fixture cluster and tears it down cleanly, with no scenario wired in yet". +# resreq-maxtasks-wiring spec, task 4.1 implemented the DEPLOY/TEARDOWN +# HARNESS -- a single controller/scheduler node plus two independently-named +# `xcecd_virtual` workers, each sized so neither alone can satisfy the test +# job's requested task count -- with the pre-flight zombie-process/port check +# convention this project's xce-test-ha soak driver established, trap-based +# cleanup registered before the first process starts, and a HARNESS_READY +# sentinel once all four roles are confirmed joined. # -# Tasks 4.2 (a plain multi-task job via `jobsub`) and 4.3 (the same, inside -# an array-job submission) build ON TOP of this harness in later dispatch -# rounds -- see the "EXTENSION POINT" comment block right after -# confirm_cluster_up() below for exactly where and how (reusing run-ha-soak.sh's -# submit_worker_job()/verify_worker_rejoin_resync() polling idiom, per -# design.md's Implementation Notes for this component -- real job via plain -# `jobsub`, poll `jobinfo` for RUN then DONE, grep each worker's own log -- -# never `xce_test`'s `jobrun` manual-allocation bypass, since the scheduler's -# own decision to span hosts is what this whole test exists to prove). +# Task 4.2 (THIS revision) builds the actual scenario on top of that harness: +# once HARNESS_READY is reached, submit ONE real multi-task job via plain +# `jobsub` (never `xce_test`'s `jobrun` manual-allocation bypass -- the +# scheduler's own decision to span $WORKER1_NAME/$WORKER2_NAME is what this +# whole test exists to prove), poll `jobinfo` for RUN then DONE (reusing +# run-ha-soak.sh's submit_worker_job()/jobinfo_of() polling idiom verbatim, +# per design.md's Implementation Notes for this component), and confirm EACH +# allocated worker's own log independently shows it starting its own share of +# the job's tasks. Confirmed LIVE that the controller sends a startjob +# message to only the FIRST allocated host, which itself relays a runtasks +# message directly to every OTHER allocated host -- so which of the two +# named workers logs "Job ntasks is: " (the startjob path) versus +# "run task: jobid = " (the relayed runtasks path) depends on the +# allocator's own internal host ordering, not something this script +# predicts; see verify_task_started_on_host() below for the full citation +# and why both are accepted as equally valid per-host evidence. The overall +# PASS/FAIL oracle line now reflects this job's real outcome, not just +# harness-ready. # -# Requirements this harness exists to eventually help satisfy (this task -# alone is infrastructure -- it does not itself complete any of these): -# 4.1 (controller ensures every allocated host has launch info), 4.2 (worker -# starts tasks on every allocated host), 4.3 (scheduler reports running via -# the existing job-status mechanism), 5.3 (a real multi-task job driven -# end-to-end at least once). See .kiro/specs/resreq-maxtasks-wiring/ -# {requirements,design}.md. +# Task 4.3 (the same scenario submitted as one element of an array job via +# `jobsub -a N`) is NOT implemented by this revision -- see the "EXTENSION +# POINT for task 4.3" comment block near the bottom of this file. +# +# Requirements this task helps satisfy: 4.1 (controller ensures every +# allocated host has launch info), 4.2 (worker starts tasks on every +# allocated host), 4.3 (scheduler reports running via the existing +# job-status mechanism), 5.3 (a real multi-task job driven end-to-end at +# least once, confirmed reaching RUN then completing). See +# .kiro/specs/resreq-maxtasks-wiring/{requirements,design}.md. # # ── Pattern provenance (read these before changing this file) ────────────── # - Deploy/start/confirm/cleanup/trap mechanics: copied from @@ -83,15 +90,17 @@ # guard, identical to run-ha-soak.sh's own), so this # never wipes an unrelated directory. # -# Exit 0 == the fixture cluster (controller + scheduler + 2 named workers) -# deployed and was confirmed fully joined; teardown ran cleanly. This is a -# "harness stood up and tore down correctly" verdict, NOT a job-outcome -# PASS/FAIL -- there is no job in this task's scope yet (see HARNESS_READY -# below). +# Exit 0 == the fixture cluster deployed and joined (HARNESS_READY), AND the +# real multi-task job scenario passed: submitted via plain jobsub, reached +# RUN, both $WORKER1_NAME/$WORKER2_NAME independently logged starting their +# own share of its tasks, and it reached DONE -- a genuine job-outcome +# PASS, not just "harness stood up". # Exit non-zero == a structural failure at any step (pre-flight zombie/port # conflict left unresolved, missing binary, deployment failure, a role that -# never confirmed joining within its bounded wait). Cleanup still runs on -# this path (trap-based, registered before the first process starts). +# never confirmed joining within its bounded wait) OR a job-scenario failure +# (submission rejected, RUN/DONE never reached, either worker never logged +# starting its share of the job's tasks). Cleanup still runs on this path +# (trap-based, registered before the first process starts). # set -u @@ -103,19 +112,43 @@ BUILD_DIR="${XCE_MULTIHOST_BUILD_DIR:-$REPO/src/xce.build}" BINARY="${XCE_MULTIHOST_BIN:-$BUILD_DIR/controller/xce_controller}" SCHED_BIN="${XCE_MULTIHOST_SCHED_BIN:-$BUILD_DIR/scheduler/xce_scheduler}" WORKER_BIN="${XCE_MULTIHOST_WORKER_BIN:-$BUILD_DIR/worker/xcecd_virtual}" -# xce_test -- not used by THIS task (no job is submitted yet), but checked -# as a soft/informational precondition since task 4.2 (the very next dispatch -# round) needs it immediately for submit_multihost_job()'s scripted session, -# following the exact idiom run-ha-soak.sh's submit_worker_job() already -# established (see the EXTENSION POINT block below). +# xce_test -- required by THIS task's submit_multihost_job()/jobinfo_of() +# scripted sessions, following the exact idiom run-ha-soak.sh's +# submit_worker_job()/jobinfo_of() already established. XT="${XCE_MULTIHOST_XT:-$BUILD_DIR/controller/xce_test}" +# xce_test client-session tuning (run_multihost_job_scenario() below): +# WAITMS wait-primary budget per xtest() session, ms. Single- +# node, already-joined cluster -- no failover/election +# is ever in play here, so this can be far smaller than +# run-ha-soak.sh's own 30000ms (which budgets for a +# real post-kill primary rejoin). +# JOB_SLEEP_SECS the `sleep N` command the test job runs -- N seconds +# of xcecd_worker_utils.c's own VIRTUAL_CLUSTER +# timer-based simulated execution (add_worker()'s +# sleep_time parsing, run-ha-soak.sh's own documented +# mechanism), giving the RUN-poll loop below several +# full 1s iterations to observe RUN before the job +# simulates finishing -- long enough to not race a +# lucky RUN->DONE flip, short enough to not slow this +# script down. +# RUN_TIMEOUT_SECS bounded wait for the job to reach RUN after submit. +# DONE_TIMEOUT_SECS bounded wait for the job to reach DONE after RUN -- +# must exceed JOB_SLEEP_SECS by a comfortable margin +# (IPC round-trip + poll granularity), hence the +# default's "+30" padding rather than JOB_SLEEP_SECS +# alone. +WAITMS="${XCE_MULTIHOST_WAITMS:-5000}" +JOB_SLEEP_SECS="${XCE_MULTIHOST_JOB_SLEEP:-5}" +RUN_TIMEOUT_SECS="${XCE_MULTIHOST_RUN_TIMEOUT:-30}" +DONE_TIMEOUT_SECS="${XCE_MULTIHOST_DONE_TIMEOUT:-$((JOB_SLEEP_SECS + 30))}" + # Single-node cluster, deliberately -- design.md's own "no HA needed for this # proof" call for this component. Single, fixed port so the pre-flight # check below has one unambiguous thing to check for a conflict against. PEER_PORT="${XCE_MULTIHOST_PORT:-9991}" PEERS="localhost:$PEER_PORT" -MASTERS="localhost:$PEER_PORT" # what a future xce_test client session's `set masters` will use (task 4.2+) +MASTERS="localhost:$PEER_PORT" # what this script's own xce_test client sessions (jobinfo_of()/submit_multihost_job()) `set masters` to WORKER1_NAME="${XCE_MULTIHOST_WORKER1:-mh-worker-1}" WORKER2_NAME="${XCE_MULTIHOST_WORKER2:-mh-worker-2}" @@ -263,9 +296,7 @@ precondition_checks() { [ -x "$BINARY" ] || fail "controller binary missing: $BINARY (run /xce-build controller)" [ -x "$SCHED_BIN" ] || fail "scheduler binary missing: $SCHED_BIN (run /xce-build scheduler)" [ -x "$WORKER_BIN" ] || fail "worker binary missing: $WORKER_BIN (run /xce-build worker)" - if [ ! -x "$XT" ]; then - log "note: xce_test binary missing at $XT -- not required by this task (no job submitted yet), but task 4.2 will need it (run /xce-build controller)" - fi + [ -x "$XT" ] || fail "xce_test binary missing: $XT (run /xce-build controller) -- required by this task's job-scenario scripted sessions" command -v nix >/dev/null 2>&1 || fail "nix not found on PATH -- required to resolve the deploy-environment bash and invoke deploy-ha.sh" } @@ -426,48 +457,199 @@ confirm_cluster_up() { log "all four roles (controller, scheduler, $WORKER1_NAME, $WORKER2_NAME) confirmed up under $TOP" } -# ═══════════════════════════════════════════════════════════════════════════ -# EXTENSION POINT for tasks 4.2/4.3 (NOT implemented by this task -- task 4.1 -# is deploy/teardown-only, per its own Observable in tasks.md). +# ── Step 4: the real job scenario (task 4.2) ──────────────────────────────── # -# Task 4.2 (a plain multi-task job): add a submit_multihost_job() here, -# following run-ha-soak.sh's submit_worker_job() idiom EXACTLY (scripted -# xce_test client-context session piped via `xtest()`-style helper: `set -# masters $MASTERS`, `set myname `, `client`, `wait-primary `, -# `jobsub -u root -q normal -c ""`, `quit`), then poll `jobinfo_of -# "$jid" `-style for the RUN -> DONE transition (run-ha-soak.sh's -# verify_worker_rejoin_resync() polling idiom) -- per design.md's -# Implementation Notes ("reuses run-ha-soak.sh's submit_worker_job()/ -# verify_worker_rejoin_resync() polling idiom ... rather than inventing a new -# verification mechanism"). MUST drive submission via plain `jobsub`, never -# `xce_test`'s `jobrun` manual-allocation bypass -- the scheduler's own -# decision to span $WORKER1_NAME/$WORKER2_NAME is what this whole test -# exists to prove (design.md's own Responsibilities & Constraints for this -# component). After a job reaches DONE, grep EACH worker's own log -# ($TOP/logs/xcecd-$WORKER1_NAME.log / xcecd-$WORKER2_NAME.log) to confirm -# genuine per-host task execution, not just an aggregate job-state check. -# -# Task 4.3 (the same, as one element of an array-job submission via -# `jobsub -a N`): a second scenario function alongside submit_multihost_job(), -# same polling idiom, confirming the SAME per-element task-count wiring a -# non-array job gets (Requirement 5.5). -# -# Worker sizing (researched during task 4.1, not wired in yet -- see this -# task's own Status Report CONCERNS for the full writeup): xcecd_virtual has +# Worker sizing (researched during task 4.1, wired in here): xcecd_virtual has # NO command-line/config override for its reported processor/slot capacity # anywhere in worker/src/main.c's getopt_long() table ("peers"/"version"/ # "help"/-n/-l/-v only). hardware_monitor.c's hardware_monitor_update() # always falls back to sysconf(_SC_NPROCESSORS_ONLN) (hwloc_disabled=true by # default, so the hwloc branch never runs) -- i.e. "virtual" only means # xcecd_virtual skips REAL job execution/cgroups, NOT that it fakes a -# different processor count; each of $WORKER1_NAME/$WORKER2_NAME will report -# this sandbox's real host core count (confirmed: nproc=22 here). Task 4.2's -# scenario should therefore pick a maxTasks strictly greater than that -# per-worker capacity N but at most 2N, so the job fits across exactly -# $WORKER1_NAME + $WORKER2_NAME and neither alone can satisfy it -- e.g. -# `nproc` at test-run time + 1, clamped to <= 2 * nproc. Do not hardcode 22; -# query it live (e.g. `nproc` inside the deploy environment) since a -# different sandbox/CI host will report a different core count. +# different processor count; each of $WORKER1_NAME/$WORKER2_NAME reports this +# host's REAL core count. So the test job's task count is picked strictly +# greater than that per-worker capacity N but at most 2N, so the job fits +# across exactly $WORKER1_NAME + $WORKER2_NAME and neither alone can satisfy +# it -- queried live via `nproc` (never hardcoded: a different sandbox/CI +# host reports a different core count). +compute_task_count() { + local n + n="$(nproc 2>/dev/null)" + case "$n" in + ''|*[!0-9]*|0) + log "warning: nproc did not return a usable core count ('$n') -- falling back to 4" >&2 + n=4 + ;; + esac + echo $((n + 1)) +} + +# xce_test client-session runner -- self-nix-wrapping (like deploy()'s own +# BASH_BIN resolution above), following run-resource-verify.sh's own xtest() +# EXACTLY: ASAN_OPTIONS=detect_leaks=0 (the debug/ASAN xce_test build reports +# harmless leaks this project already treats as a known, accepted caveat -- +# same as every sibling skill), wrapped in `nix develop` so xce_test's +# runtime libs (openssl etc.) resolve, bounded by `timeout` so a hung session +# can never wedge this script. Judge success from OUTPUT text, never the +# leak-polluted process exit code (same caveat every sibling skill +# documents). Depends on $BASH_BIN, resolved once by deploy() above. +xtest() { + nix develop "$REPO/src" --command env ASAN_OPTIONS=detect_leaks=0 timeout 60 "$XT" --norc 2>&1 +} + +# `jobinfo -j ` against the live (single-node) primary. Prints the +# jobinfo text and returns 0 on a hit; returns 1 (nothing printed) if not +# found/unreachable. Retries a couple of times with a short sleep to absorb +# the same post-submit client-retry race run-ha-soak.sh's own jobinfo_of() +# tolerates -- copied from that function verbatim in spirit, just against +# this harness's own single-node $MASTERS/$XT/xtest(). +jobinfo_of() { + local id="$1" waitms="${2:-$WAITMS}" out attempt + for attempt in 1 2 3; do + out="$(printf 'set masters %s\nset myname mh-verify\nclient\nwait-primary %s\njobinfo -j %s\nquit\n' \ + "$MASTERS" "$waitms" "$id" | xtest)" + if printf '%s\n' "$out" | grep -q "^JobID: "; then + printf '%s\n' "$out" + return 0 + fi + [ "$attempt" -lt 3 ] && sleep 0.5 + done + return 1 +} + +# Submits ONE real job via plain `jobsub` -- NOT xce_test's `jobrun` +# manual-allocation bypass -- so the scheduler's own dispatch decision is +# what actually spans $WORKER1_NAME/$WORKER2_NAME (design.md's own +# Responsibilities & Constraints for this component). `-n ,` +# sets BOTH mintasks and maxtasks to the caller-computed split-forcing count +# (cmd_jobsub, test.c ~2256-2274); resreq is omitted so cmd_jobsub defaults +# it to "type == any" (test.c ~2313-2315), letting the allocator consider +# every worker's capacity. `sleep ` is the command +# xcecd_worker_utils.c's add_worker() (VIRTUAL_CLUSTER branch) parses to +# drive each host's own timer-based simulated execution -- the SAME +# mechanism run-ha-soak.sh's own submit_worker_job() relies on. Echoes the +# jobid on success; returns 1 (echoing nothing, diagnostic on stderr so it +# never pollutes the captured jobid) on a rejected submit. +submit_multihost_job() { + local ntasks="$1" sleep_secs="$2" out jid + out="$(printf 'set masters %s\nset myname mh-verify\nclient\nwait-primary %s\njobsub -u root -q normal -c "sleep %s" -n %s,%s\nquit\n' \ + "$MASTERS" "$WAITMS" "$sleep_secs" "$ntasks" "$ntasks" | xtest)" + jid="$(printf '%s\n' "$out" | sed -n 's/^jobsubmit ok: JobId: \(.*\)$/\1/p')" + if [ -z "$jid" ]; then + log "job submission failed -- raw xce_test output: $(printf '%s' "$out" | tr '\n' ' ')" >&2 + return 1 + fi + echo "$jid" + return 0 +} + +# Poll jobinfo_of() every 1s (up to $3 seconds) until job $1 reaches state +# $2. Echoes the observed state on success; returns 1 on timeout (same +# jobinfo_of()+sed state-extraction+1s-sleep-loop idiom as run-ha-soak.sh's +# own submit_worker_job()/verify_worker_rejoin_resync()). +wait_for_job_state() { + local jid="$1" want="$2" timeout="$3" i st + for i in $(seq 1 "$timeout"); do + st="$(jobinfo_of "$jid" 2000 | sed -n 's/^state: //p')" + if [ "$st" = "$want" ]; then + echo "$st" + return 0 + fi + sleep 1 + done + log "job $jid never reached $want after ${timeout}s (last observed state: ${st:-unknown})" >&2 + return 1 +} + +# Per-host "this host actually started its share of THIS job's tasks" +# evidence (task 4.2's Observable). Confirmed LIVE (this task's own first +# validation run) that the controller does NOT send an independent per-host +# startjob message to every allocated host as design.md's sequence diagram +# simplifies it -- it sends ONE startjob to the first allocated host only; +# THAT host's own job_start_remote_task() (controller_connector.c ~725-790) +# then relays a runtasks message directly to every OTHER allocated host via +# xcecli_runtask(). Requirement 4.1 ("the Controller shall ensure every +# allocated host has the information it needs to start its own share") is +# still satisfied end to end -- every host DOES end up with what it needs -- +# just via two different message types depending on which host the +# allocator picked as first. Which of $WORKER1_NAME/$WORKER2_NAME takes +# which path is the allocator's own internal choice, not something this +# script can predict -- this check accepts EITHER path's own jobid-tagged +# evidence line, both confirmed live in this task's own first validation run: +# - startjob path (job_start_local_task(), controller_connector.c ~1012): +# XCECD_LOG(XCE_LOG_DEBUG, "Job %s ntasks is: %d", jobidStr, ...); +# (mh-worker-1 took this path in the live run.) +# - runtasks path (handle_runtasks(), controller_connector.c ~1174): +# XCECD_LOG(XCE_LOG_DEBUG3, "run task: jobid = %s", jobidStr); +# (mh-worker-2 took this path in the live run.) +# Both `jobidStr`s are fmt_jobid() applied to the exact jobid this host was +# told to run -- the SAME formatting function xce_test's own cmd_jobsub uses +# to print "jobsubmit ok: JobId: " (test.c ~2424/2434), so the `$jid` +# this script captures from that line and the `` embedded in either log +# line are the IDENTICAL string: an exact, job-scoped match, not a +# same-cluster coincidence. (A third, path-independent but NOT jobid-tagged +# line -- xcecd.c ~54's "Starting worker as user ... to process command: +# ..." -- fires on both paths too, confirmed live on both hosts; not used as +# the primary signal here only because it lacks the jobid, but corroborates +# the two lines above whenever inspecting a log by hand.) Reuses this +# script's own wait_for_log() (short bounded wait: the log line is written +# synchronously as part of the SAME async task that gets this job to RUN, so +# it is expected to already be present by the time wait_for_job_state() +# above observes RUN -- this wait is a safety margin, not the primary gate). +verify_task_started_on_host() { + local name="$1" jid="$2" + wait_for_log "worker '$name' started its share of job $jid's tasks" "Job $jid ntasks is:|run task: jobid = $jid" 15 "$TOP/logs/xcecd-$name.log" +} + +# Orchestrates the whole task 4.2 scenario: size the job so it must span +# both hosts, submit it via plain jobsub, poll RUN, confirm BOTH hosts +# independently logged starting their own share, then poll DONE. Hard-fails +# (via the shared fail() helper -- immediate FAIL: line + non-zero exit, +# cleanup still runs via the registered trap) on the first violated step, +# same discipline as every other structural check in this script. +run_multihost_job_scenario() { + local ntasks jid + + ntasks="$(compute_task_count)" + log "job scenario: sizing task count to $ntasks (nproc=$(nproc 2>/dev/null || echo '?') per worker) -- strictly more than either $WORKER1_NAME or $WORKER2_NAME can satisfy alone, at most both combined, forcing a genuine 2-host split" + + log "submitting a real multi-task job via plain jobsub (-n $ntasks,$ntasks, \"sleep $JOB_SLEEP_SECS\") -- never xce_test's jobrun manual-allocation bypass" + jid="$(submit_multihost_job "$ntasks" "$JOB_SLEEP_SECS")" \ + || fail "job submission failed -- see the diagnostic line just above" + log "submitted job $jid -- polling for RUN (timeout ${RUN_TIMEOUT_SECS}s)" + + wait_for_job_state "$jid" "RUN" "$RUN_TIMEOUT_SECS" >/dev/null \ + || fail "job $jid never reached RUN within ${RUN_TIMEOUT_SECS}s -- the scheduler's own allocation decision never dispatched it" + log "confirmed: job $jid reached RUN" + + log "confirming each allocated host's own log shows its share of job $jid's tasks actually starting" + verify_task_started_on_host "$WORKER1_NAME" "$jid" \ + || fail "job $jid: '$TOP/logs/xcecd-$WORKER1_NAME.log' never logged starting a share of this job's tasks" + verify_task_started_on_host "$WORKER2_NAME" "$jid" \ + || fail "job $jid: '$TOP/logs/xcecd-$WORKER2_NAME.log' never logged starting a share of this job's tasks -- the allocation did not genuinely span both hosts" + log "confirmed: BOTH $WORKER1_NAME and $WORKER2_NAME independently logged starting their own share of job $jid's tasks -- the scheduler's own decision genuinely spans both hosts" + + wait_for_job_state "$jid" "DONE" "$DONE_TIMEOUT_SECS" >/dev/null \ + || fail "job $jid reached RUN and both hosts started their share, but never reached DONE within ${DONE_TIMEOUT_SECS}s" + log "confirmed: job $jid reached DONE -- completed successfully on both hosts" + + MULTIHOST_JOB_ID="$jid" + MULTIHOST_JOB_NTASKS="$ntasks" +} + +# ═══════════════════════════════════════════════════════════════════════════ +# EXTENSION POINT for task 4.3 (NOT implemented by this task -- 4.2 above +# covers a plain job only). +# +# Task 4.3 (the same scenario, as one element of an array-job submission via +# `jobsub -a N`): add a second scenario function alongside +# run_multihost_job_scenario() above, following the SAME polling idiom +# (submit_multihost_job() would need an `-a N` variant, or a sibling +# function built the same way; jobinfo/wait_for_job_state() work unchanged +# against a per-element jobid string, same array-element-addressing +# convention run-ha-soak.sh's own array_element_id()/verify_array_elements() +# already establish), confirming the array element receives the SAME +# per-element task-count wiring a non-array job gets (Requirement 5.5). # ═══════════════════════════════════════════════════════════════════════════ main() { @@ -487,7 +669,10 @@ main() { confirm_cluster_up log "HARNESS_READY controller=up scheduler=up workers=2/2 top=$TOP" - log "PASS (harness-ready): controller + scheduler + 2 named worker(s) ($WORKER1_NAME, $WORKER2_NAME) deployed and confirmed joined; no job scenario submitted yet (task 4.1 scope only -- see tasks 4.2/4.3)" + + run_multihost_job_scenario + + log "PASS: job $MULTIHOST_JOB_ID ($MULTIHOST_JOB_NTASKS tasks, sleep ${JOB_SLEEP_SECS}s) submitted via plain jobsub, reached RUN then DONE; $WORKER1_NAME and $WORKER2_NAME each independently logged starting their own share of its tasks -- the scheduler's own decision genuinely spans both hosts and the job completed on both" exit 0 } diff --git a/.kiro/specs/resreq-maxtasks-wiring/tasks.md b/.kiro/specs/resreq-maxtasks-wiring/tasks.md index be7e4360..3a3651c8 100644 --- a/.kiro/specs/resreq-maxtasks-wiring/tasks.md +++ b/.kiro/specs/resreq-maxtasks-wiring/tasks.md @@ -65,7 +65,7 @@ - _Boundary: xce-test-multihost_ - _Depends: 2.1, 2.2_ -- [ ] 4.2 Prove a plain multi-task job is genuinely scheduled and completes across multiple hosts +- [x] 4.2 Prove a plain multi-task job is genuinely scheduled and completes across multiple hosts - Submit a real multi-task job through normal job submission (not a manual-allocation bypass) and poll until it reaches a running state, then confirm it reaches completion - Confirm each allocated host's own log shows its share of the job's tasks actually starting - Observable: a PASS verdict confirms the scheduler's own decision spans both hosts and the job completes successfully on both