feat(resreq-maxtasks-wiring): submit real multi-task job across hosts

Extends xce-test-multihost's harness to submit one real job via plain
jobsub, sized (nproc+1 tasks, queried live) so it must span both
mh-worker-1 and mh-worker-2, poll jobinfo for RUN then DONE, and
confirm each allocated host's own log independently shows it starting
its share of the job's tasks.

The two allocated hosts take genuinely different code paths: the
controller sends startjob only to the head-node host
(dispatch_a_job_(), job.c); that host's job_start_remote_task()
relays a runtasks request back through the controller
(do_runtasks()/run_tasks_on_host(), task.c) to the other host, which
independently calls submit_fork_task(). The harness's per-host
verification accepts either line as legitimate evidence for that
host, since which host lands as head node is an allocator-internal
detail this test correctly doesn't hardcode.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg

.claude/skills/xce-test-multihost/SKILL.md

@@ -1,38 +1,41 @@
 ---
 name: xce-test-multihost
-description: "End-to-end multi-host job allocation harness: deploys one controller/scheduler node plus two named xcecd_virtual workers (each sized so neither alone can satisfy the test job's requested task count) and confirms all four roles come up and join cleanly. Usage: bash .claude/skills/xce-test-multihost/run-multihost-verify.sh [--keep-running] [--dir DIR]"
+description: "End-to-end multi-host job allocation proof: deploys one controller/scheduler node plus two named xcecd_virtual workers (each sized so neither alone can satisfy the test job's requested task count), submits a real multi-task job via plain jobsub, and confirms the scheduler's own decision genuinely splits it across both hosts, which both reach DONE. Usage: bash .claude/skills/xce-test-multihost/run-multihost-verify.sh [--keep-running] [--dir DIR]"
 model: haiku
 ---
 
 # XCE Multi-Host Allocation Test Harness
 
-## Status: task 4.1 only -- deploy/teardown harness, no scenario wired in yet
+## Status: tasks 4.1 + 4.2 -- harness plus a real plain-job scenario; 4.3 (array-job variant) not yet wired in
 
-`run-multihost-verify.sh` currently implements the resreq-maxtasks-wiring
-spec's task 4.1 scope: it deploys a single controller/scheduler node plus
-two independently-named `xcecd_virtual` workers, confirms all four roles
-come up and join, emits a `HARNESS_READY` sentinel + PASS/FAIL oracle line,
-and tears everything down cleanly. **It does not yet submit any job or
-check any allocation outcome** -- that is tasks 4.2 (a plain multi-task job
-via `jobsub`) and 4.3 (the same, as part of an array-job submission), which
-build directly on top of this same script (see the `EXTENSION POINT`
-comment block in `run-multihost-verify.sh`, right after
-`confirm_cluster_up()`).
+`run-multihost-verify.sh` implements the resreq-maxtasks-wiring spec's task
+4.1 scope (deploy a single controller/scheduler node plus two
+independently-named `xcecd_virtual` workers, confirm all four roles come up
+and join) **plus** task 4.2's scenario on top of it: once the fixture is up
+(`HARNESS_READY`), it submits ONE real multi-task job via plain `jobsub`,
+polls `jobinfo` for `RUN` then `DONE`, confirms BOTH named workers'
+own logs independently show them starting their own share of the job's
+tasks, and emits a PASS/FAIL verdict reflecting that real outcome -- not
+just "harness stood up". **Not yet implemented**: task 4.3 (the same
+scenario submitted as one element of an array job via `jobsub -a N`) --
+see the `EXTENSION POINT for task 4.3` comment block near the bottom of
+`run-multihost-verify.sh`.
 
-This closes bug-1770's own end-to-end validation gap once 4.2/4.3 land: the
-scheduler-allocation feature already implements, unit-tests, and
-soak-validates real multi-host allocation logic (`parjob_allocator`) that,
-until resreq-maxtasks-wiring's own fix lands, has no way to ever be
-reached by a real job submission. This skill exists to prove -- with a real
-controller, a real scheduler, and real (virtual) workers, driven through
-plain `jobsub` -- that a job requiring more tasks than one host can provide
-is genuinely split across two hosts by the scheduler's own decision, and
-actually runs to completion on both.
+This closes bug-1770's own end-to-end validation gap for the plain-job path
+(this revision): the scheduler-allocation feature already implements,
+unit-tests, and soak-validates real multi-host allocation logic
+(`parjob_allocator`) that, until resreq-maxtasks-wiring's own fix lands, has
+no way to ever be reached by a real job submission. This skill now proves --
+with a real controller, a real scheduler, and real (virtual) workers, driven
+through plain `jobsub` -- that a job requiring more tasks than one host can
+provide is genuinely split across two hosts by the scheduler's own decision,
+and actually runs to completion on both. The array-job variant of this same
+proof (Requirement 5.5) remains open until task 4.3 lands.
 
 Traces to `.kiro/specs/resreq-maxtasks-wiring/design.md`'s
 `xce-test-multihost` component (Requirements 4.1, 4.2, 4.3, 5.3, 5.5).
 
-## What this skill does today (task 4.1)
+## What this skill does today (tasks 4.1 + 4.2)
 
 1. **Pre-flight zombie-process/port check** -- following the convention
    already established by `xce-test-ha`'s `run-ha-soak.sh` (a fixed, reused
@@ -73,19 +76,48 @@ Traces to `.kiro/specs/resreq-maxtasks-wiring/design.md`'s
      `$TOP/logs/xcecd-<name>.log` -- checked independently per worker, not
      just once for "a" worker, since this harness's whole point is TWO
      independently-joined hosts.
-4. **Report a `HARNESS_READY` sentinel + PASS/FAIL oracle line** -- since
-   there is no job outcome to report yet at this stage, the terminal line
-   reports "harness stood up and tore down correctly", not a job-allocation
-   verdict:
+4. **Report a `HARNESS_READY` sentinel** once all four roles are confirmed
+   joined -- an intermediate milestone log line, not the final verdict.
+5. **Size and submit ONE real multi-task job via plain `jobsub`** (task
+   4.2) -- never `xce_test`'s `jobrun` manual-allocation bypass, so the
+   scheduler's own dispatch decision is what actually spans the two named
+   workers. Task count is queried live via `nproc` on each worker (strictly
+   greater than one worker's own capacity, at most both combined --
+   `xcecd_virtual` reports the REAL host core count, confirmed no
+   config/CLI override exists to shrink it), forcing a genuine 2-host
+   split; empirically confirmed live (this task's own validation) that a
+   small job well within one worker's capacity (`-n 5,5`) stays on a
+   SINGLE host, so this sizing is what actually forces the multi-host path,
+   not something the scheduler would do regardless of size. The job command
+   is `sleep <N>`, the SAME mechanism `xce-test-ha`'s `run-ha-soak.sh` uses
+   to drive a virtual worker's own timer-based simulated execution.
+6. **Poll `jobinfo` for `RUN` then `DONE`** -- reusing `run-ha-soak.sh`'s
+   `jobinfo_of()`/state-polling idiom verbatim (scripted `xce_test`
+   client-context session: `set masters`, `set myname`, `client`,
+   `wait-primary`, `jobinfo -j <id>`, `quit`; `state: RUN`/`state: DONE`
+   extracted via `sed`), bounded timeouts throughout.
+7. **Confirm EACH allocated worker's own log independently shows it
+   starting its own share of the job's tasks** -- the harness's central
+   claim. Confirmed live that the controller sends a startjob message to
+   only the FIRST allocated host, which itself relays a runtasks message
+   directly to the OTHER allocated host(s); this script accepts EITHER
+   path's own jobid-tagged evidence line (`"Job <id> ntasks is: <n>"` from
+   the startjob path, or `"run task: jobid = <id>"` from the relayed
+   runtasks path) on each worker's own `$TOP/logs/xcecd-<name>.log`, since
+   which host takes which path is the allocator's own internal choice.
+8. **Report a real PASS/FAIL verdict** reflecting the job's actual outcome
+   -- not just "harness stood up":
    ```
    [xce-test-multihost] HARNESS_READY controller=up scheduler=up workers=2/2 top=/home/claude/tmp/xce-test-multihost
-   [xce-test-multihost] PASS (harness-ready): controller + scheduler + 2 named worker(s) (mh-worker-1, mh-worker-2) deployed and confirmed joined; no job scenario submitted yet (task 4.1 scope only -- see tasks 4.2/4.3)
+   ...
+   [xce-test-multihost] PASS: job 0-9-1788684444-366142 (23 tasks, sleep 5s) submitted via plain jobsub, reached RUN then DONE; mh-worker-1 and mh-worker-2 each independently logged starting their own share of its tasks -- the scheduler's own decision genuinely spans both hosts and the job completed on both
    ```
-   Any structural failure (a role never confirming joined within its
-   bounded wait, a missing binary, a pre-flight conflict this script cannot
-   safely resolve) is a `FAIL:` line + non-zero exit instead, via the shared
-   `fail()` helper.
-5. **Clean up** every process this run started -- unconditionally, on
+   Any structural failure (a role never confirming joined, a missing
+   binary, a pre-flight conflict) OR job-scenario failure (submission
+   rejected, RUN/DONE never reached within its bounded timeout, either
+   worker never logging its own share starting) is a `FAIL:` line +
+   non-zero exit instead, via the shared `fail()` helper.
+9. **Clean up** every process this run started -- unconditionally, on
    success or failure alike -- unless `--keep-running` was passed. Log
    files under the deploy directory are always preserved regardless of that
    flag, matching `xce-test-resources`'s own documented divergence from
@@ -113,11 +145,15 @@ bash .claude/skills/xce-test-multihost/run-multihost-verify.sh [--keep-running]
 - Other overridable env vars: `$XCE_MULTIHOST_BUILD_DIR` (default
   `src/xce.build`), `$XCE_MULTIHOST_BIN` (controller),
   `$XCE_MULTIHOST_SCHED_BIN`, `$XCE_MULTIHOST_WORKER_BIN`,
-  `$XCE_MULTIHOST_XT` (`xce_test` -- not required by this task, checked as
-  an informational precondition for task 4.2's benefit),
+  `$XCE_MULTIHOST_XT` (`xce_test` -- REQUIRED as of task 4.2's job scenario,
+  a hard precondition failure if missing),
   `$XCE_MULTIHOST_PORT` (default `9991` -- single-node controller port),
   `$XCE_MULTIHOST_WORKER1`/`$XCE_MULTIHOST_WORKER2` (default
-  `mh-worker-1`/`mh-worker-2`).
+  `mh-worker-1`/`mh-worker-2`), `$XCE_MULTIHOST_WAITMS` (default `5000` --
+  `wait-primary` budget per `xce_test` session), `$XCE_MULTIHOST_JOB_SLEEP`
+  (default `5` -- seconds the test job's `sleep <N>` command runs),
+  `$XCE_MULTIHOST_RUN_TIMEOUT` (default `30`), `$XCE_MULTIHOST_DONE_TIMEOUT`
+  (default `$XCE_MULTIHOST_JOB_SLEEP + 30`).
 
 Must be invoked under an environment with `nix` on `PATH` -- the script
 shells out to `nix develop` itself for the steps that need it.
@@ -147,7 +183,7 @@ that one component -- see this project's own bug-1719 lesson -- so building
 controller/scheduler/worker one at a time into the same dir needs
 `--incremental` on every call after the first.)
 
-## Worker sizing (researched for task 4.2's benefit, not wired in by this task)
+## Worker sizing (researched during task 4.1, wired in by task 4.2)
 
 `xcecd_virtual` has **no** command-line or config override for its reported
 processor/slot capacity: `worker/src/main.c`'s `getopt_long()` table is
@@ -159,44 +195,66 @@ processor/slot capacity: `worker/src/main.c`'s `getopt_long()` table is
 fakes a different processor count -- each worker reports the real host's
 actual core count (confirmed live in this sandbox: `nproc` reports 22).
 
-Consequence for task 4.2/4.3's scenario: since a worker's capacity cannot be
-shrunk via configuration, the robust sizing strategy is to pick the test
-job's `maxTasks` **strictly greater than** one worker's reported capacity
-`N` but **at most `2N`**, so the job fits across exactly the two workers
-this harness starts and neither alone can satisfy it. Query `nproc` live at
-test-run time rather than hardcoding a number -- a different sandbox/CI host
-will report a different core count.
+`compute_task_count()` in `run-multihost-verify.sh` therefore queries
+`nproc` live at test-run time (never hardcoded) and picks the test job's
+task count as `nproc + 1` -- strictly greater than one worker's reported
+capacity `N` but at most `2N`, so the job fits across exactly the two
+workers this harness starts and neither alone can satisfy it. **Empirically
+confirmed** (a separate, one-off manual check during this task's own
+validation, not part of the committed script): submitting a `-n 5,5` job
+against the same live cluster landed entirely on ONE worker (`ntasks is: 5`
+on that worker's own log, zero occurrences of the jobid on the other) --
+confirming the `nproc + 1` sizing is what actually forces the 2-host split,
+not something the scheduler would do for any job regardless of size.
 
-## Extension points for tasks 4.2 / 4.3
+## Per-host task-start evidence: two possible code paths (confirmed live)
 
-`run-multihost-verify.sh` has a clearly marked `EXTENSION POINT` comment
-block (right after `confirm_cluster_up()`) documenting exactly where and how
-the next two tasks should add:
+Requirement 4.1 says the controller "shall ensure every allocated host has
+the information it needs to start its own share" -- it does NOT mandate
+*how* each host receives that information, and in practice it is not a
+uniform per-host broadcast: confirmed live (this task's own first
+validation run) that the controller sends a startjob message to only the
+FIRST allocated host; that host's own `job_start_remote_task()`
+(`controller_connector.c` ~725-790) then relays a runtasks message directly
+to every OTHER allocated host via `xcecli_runtask()`. Both still leave every
+host with a genuine, independently-verifiable "I started my own share" log
+line, tagged with the exact jobid (`fmt_jobid()`, the SAME formatting
+function `xce_test`'s own `cmd_jobsub` uses for `jobsubmit ok: JobId: <id>`):
 
-- `submit_multihost_job()` (task 4.2) -- a plain multi-task job via
-  `jobsub`, following `run-ha-soak.sh`'s `submit_worker_job()`/
-  `verify_worker_rejoin_resync()` polling idiom verbatim (scripted
-  `xce_test` client-context session: `set masters`, `set myname`, `client`,
-  `wait-primary`, `jobsub -u root -q normal -c "..."`, `quit`; then poll
-  `jobinfo` for `RUN` then `DONE`). **Must** drive submission via plain
-  `jobsub`, never `xce_test`'s `jobrun` manual-allocation bypass -- the
-  scheduler's own decision to span `mh-worker-1`/`mh-worker-2` is what this
-  whole test exists to prove. After the job reaches `DONE`, grep **each**
-  worker's own log to confirm genuine per-host task execution.
-- A second, array-job scenario (task 4.3) -- the same job submitted via
-  `jobsub -a N`, same polling idiom, confirming the array element receives
-  the same per-element task-count wiring a non-array job would
-  (Requirement 5.5).
+- startjob path (`job_start_local_task()`, `controller_connector.c` ~1012):
+  `"Job <id> ntasks is: <n>"` at `XCE_LOG_DEBUG`.
+- relayed runtasks path (`handle_runtasks()`, `controller_connector.c`
+  ~1174): `"run task: jobid = <id>"` at `XCE_LOG_DEBUG3`.
 
-## Known pre-existing constraints this skill inherits (not fixed here)
+Which of `mh-worker-1`/`mh-worker-2` takes which path is the allocator's own
+internal host-ordering choice, not something this script predicts --
+`verify_task_started_on_host()` accepts EITHER line as equally valid
+evidence for a given host. (A third, path-independent line that also fires
+on both --`"Starting worker as user ... to process command: ..."`,
+`xcecd.c` ~54 -- lacks the jobid tag, so it is not used as the primary
+signal, but corroborates the two above when inspecting a log by hand.)
+
+## Extension point for task 4.3
+
+`run-multihost-verify.sh` has a clearly marked `EXTENSION POINT for task
+4.3` comment block (near the bottom of the file, after
+`run_multihost_job_scenario()`) documenting exactly where and how to add a
+second, array-job scenario -- the same job submitted via `jobsub -a N`,
+same `jobinfo_of()`/`wait_for_job_state()` polling idiom, confirming the
+array element receives the same per-element task-count wiring a non-array
+job would (Requirement 5.5).
+
+## Known pre-existing constraints this skill inherits
 
 - Same `xce_test`/ASAN leak-detection caveat every sibling skill documents:
-  a future `submit_multihost_job()` must run `xce_test` with
-  `ASAN_OPTIONS=detect_leaks=0` and judge success from command **output**,
-  never the leak-polluted process exit code.
-- Same nix-devshell runtime-linkage caveat: any future `xce_test` invocation
-  should be wrapped in `nix develop`, like this script's own `deploy-ha.sh`
-  invocation already is.
+  this script's `xtest()` helper runs `xce_test` with
+  `ASAN_OPTIONS=detect_leaks=0` and judges success from command **output**
+  (`jobsubmit ok: JobId: <id>`, `^JobID: ` in `jobinfo`), never the
+  leak-polluted process exit code.
+- Same nix-devshell runtime-linkage caveat: every `xce_test` invocation
+  (`jobinfo_of()`, `submit_multihost_job()`, via the shared `xtest()`
+  helper) is wrapped in `nix develop`, like this script's own
+  `deploy-ha.sh` invocation.
 
 ## Dependencies
 
@@ -210,8 +268,10 @@ the next two tasks should add:
 
 This document describes design.md's `xce-test-multihost` component
 (Components and Interfaces -> Test Infrastructure), covering Requirements
-4.1, 4.2, 4.3, 5.3, 5.5. Task 4.1 (this revision) covers the deploy/teardown
-harness only; the job-scenario coverage those requirements ultimately need
-is completed by tasks 4.2/4.3. See
+4.1, 4.2, 4.3, 5.3, 5.5. Task 4.1 built the deploy/teardown harness; task
+4.2 (this revision) drives a real plain multi-task job through it end to
+end, satisfying 4.1, 4.2, 4.3, and 5.3 with live evidence. Requirement 5.5
+(the same proof for an array-job submission) remains open until task 4.3
+lands -- see the `EXTENSION POINT for task 4.3` block. See
 `.kiro/specs/resreq-maxtasks-wiring/requirements.md` and `design.md` for the
 full requirement text.

.claude/skills/xce-test-multihost/run-multihost-verify.sh

@@ -2,36 +2,43 @@
 #
 # run-multihost-verify.sh -- driver for /xce-test-multihost.
 #
-# resreq-maxtasks-wiring spec, task 4.1 SCOPE ONLY: this revision implements
-# the DEPLOY/TEARDOWN HARNESS -- a single controller/scheduler node plus two
-# independently-named `xcecd_virtual` workers, each sized so neither alone
-# can satisfy the eventual test job's requested task count (see the
-# "Worker sizing" note near submit_multihost_job()'s placeholder below) --
-# with the pre-flight zombie-process/port check convention this project's
-# xce-test-ha soak driver established, trap-based cleanup registered before
-# the first process starts, and a HARNESS_READY sentinel + PASS/FAIL oracle
-# line. It deliberately does NOT yet submit any job or check any allocation
-# outcome -- see design.md's "xce-test-multihost" component (Batch/Job
-# Contract) and tasks.md 4.1's own Observable: "the new harness deploys its
-# fixture cluster and tears it down cleanly, with no scenario wired in yet".
+# resreq-maxtasks-wiring spec, task 4.1 implemented the DEPLOY/TEARDOWN
+# HARNESS -- a single controller/scheduler node plus two independently-named
+# `xcecd_virtual` workers, each sized so neither alone can satisfy the test
+# job's requested task count -- with the pre-flight zombie-process/port check
+# convention this project's xce-test-ha soak driver established, trap-based
+# cleanup registered before the first process starts, and a HARNESS_READY
+# sentinel once all four roles are confirmed joined.
 #
-# Tasks 4.2 (a plain multi-task job via `jobsub`) and 4.3 (the same, inside
-# an array-job submission) build ON TOP of this harness in later dispatch
-# rounds -- see the "EXTENSION POINT" comment block right after
-# confirm_cluster_up() below for exactly where and how (reusing run-ha-soak.sh's
-# submit_worker_job()/verify_worker_rejoin_resync() polling idiom, per
-# design.md's Implementation Notes for this component -- real job via plain
-# `jobsub`, poll `jobinfo` for RUN then DONE, grep each worker's own log --
-# never `xce_test`'s `jobrun` manual-allocation bypass, since the scheduler's
-# own decision to span hosts is what this whole test exists to prove).
+# Task 4.2 (THIS revision) builds the actual scenario on top of that harness:
+# once HARNESS_READY is reached, submit ONE real multi-task job via plain
+# `jobsub` (never `xce_test`'s `jobrun` manual-allocation bypass -- the
+# scheduler's own decision to span $WORKER1_NAME/$WORKER2_NAME is what this
+# whole test exists to prove), poll `jobinfo` for RUN then DONE (reusing
+# run-ha-soak.sh's submit_worker_job()/jobinfo_of() polling idiom verbatim,
+# per design.md's Implementation Notes for this component), and confirm EACH
+# allocated worker's own log independently shows it starting its own share of
+# the job's tasks. Confirmed LIVE that the controller sends a startjob
+# message to only the FIRST allocated host, which itself relays a runtasks
+# message directly to every OTHER allocated host -- so which of the two
+# named workers logs "Job <id> ntasks is: <n>" (the startjob path) versus
+# "run task: jobid = <id>" (the relayed runtasks path) depends on the
+# allocator's own internal host ordering, not something this script
+# predicts; see verify_task_started_on_host() below for the full citation
+# and why both are accepted as equally valid per-host evidence. The overall
+# PASS/FAIL oracle line now reflects this job's real outcome, not just
+# harness-ready.
 #
-# Requirements this harness exists to eventually help satisfy (this task
-# alone is infrastructure -- it does not itself complete any of these):
-# 4.1 (controller ensures every allocated host has launch info), 4.2 (worker
-# starts tasks on every allocated host), 4.3 (scheduler reports running via
-# the existing job-status mechanism), 5.3 (a real multi-task job driven
-# end-to-end at least once). See .kiro/specs/resreq-maxtasks-wiring/
-# {requirements,design}.md.
+# Task 4.3 (the same scenario submitted as one element of an array job via
+# `jobsub -a N`) is NOT implemented by this revision -- see the "EXTENSION
+# POINT for task 4.3" comment block near the bottom of this file.
+#
+# Requirements this task helps satisfy: 4.1 (controller ensures every
+# allocated host has launch info), 4.2 (worker starts tasks on every
+# allocated host), 4.3 (scheduler reports running via the existing
+# job-status mechanism), 5.3 (a real multi-task job driven end-to-end at
+# least once, confirmed reaching RUN then completing). See
+# .kiro/specs/resreq-maxtasks-wiring/{requirements,design}.md.
 #
 # ── Pattern provenance (read these before changing this file) ──────────────
 # - Deploy/start/confirm/cleanup/trap mechanics: copied from
@@ -83,15 +90,17 @@
 #                       guard, identical to run-ha-soak.sh's own), so this
 #                       never wipes an unrelated directory.
 #
-# Exit 0 == the fixture cluster (controller + scheduler + 2 named workers)
-# deployed and was confirmed fully joined; teardown ran cleanly. This is a
-# "harness stood up and tore down correctly" verdict, NOT a job-outcome
-# PASS/FAIL -- there is no job in this task's scope yet (see HARNESS_READY
-# below).
+# Exit 0 == the fixture cluster deployed and joined (HARNESS_READY), AND the
+# real multi-task job scenario passed: submitted via plain jobsub, reached
+# RUN, both $WORKER1_NAME/$WORKER2_NAME independently logged starting their
+# own share of its tasks, and it reached DONE -- a genuine job-outcome
+# PASS, not just "harness stood up".
 # Exit non-zero == a structural failure at any step (pre-flight zombie/port
 # conflict left unresolved, missing binary, deployment failure, a role that
-# never confirmed joining within its bounded wait). Cleanup still runs on
-# this path (trap-based, registered before the first process starts).
+# never confirmed joining within its bounded wait) OR a job-scenario failure
+# (submission rejected, RUN/DONE never reached, either worker never logged
+# starting its share of the job's tasks). Cleanup still runs on this path
+# (trap-based, registered before the first process starts).
 #
 set -u
 
@@ -103,19 +112,43 @@ BUILD_DIR="${XCE_MULTIHOST_BUILD_DIR:-$REPO/src/xce.build}"
 BINARY="${XCE_MULTIHOST_BIN:-$BUILD_DIR/controller/xce_controller}"
 SCHED_BIN="${XCE_MULTIHOST_SCHED_BIN:-$BUILD_DIR/scheduler/xce_scheduler}"
 WORKER_BIN="${XCE_MULTIHOST_WORKER_BIN:-$BUILD_DIR/worker/xcecd_virtual}"
-# xce_test -- not used by THIS task (no job is submitted yet), but checked
-# as a soft/informational precondition since task 4.2 (the very next dispatch
-# round) needs it immediately for submit_multihost_job()'s scripted session,
-# following the exact idiom run-ha-soak.sh's submit_worker_job() already
-# established (see the EXTENSION POINT block below).
+# xce_test -- required by THIS task's submit_multihost_job()/jobinfo_of()
+# scripted sessions, following the exact idiom run-ha-soak.sh's
+# submit_worker_job()/jobinfo_of() already established.
 XT="${XCE_MULTIHOST_XT:-$BUILD_DIR/controller/xce_test}"
 
+# xce_test client-session tuning (run_multihost_job_scenario() below):
+#   WAITMS             wait-primary budget per xtest() session, ms. Single-
+#                       node, already-joined cluster -- no failover/election
+#                       is ever in play here, so this can be far smaller than
+#                       run-ha-soak.sh's own 30000ms (which budgets for a
+#                       real post-kill primary rejoin).
+#   JOB_SLEEP_SECS      the `sleep N` command the test job runs -- N seconds
+#                       of xcecd_worker_utils.c's own VIRTUAL_CLUSTER
+#                       timer-based simulated execution (add_worker()'s
+#                       sleep_time parsing, run-ha-soak.sh's own documented
+#                       mechanism), giving the RUN-poll loop below several
+#                       full 1s iterations to observe RUN before the job
+#                       simulates finishing -- long enough to not race a
+#                       lucky RUN->DONE flip, short enough to not slow this
+#                       script down.
+#   RUN_TIMEOUT_SECS    bounded wait for the job to reach RUN after submit.
+#   DONE_TIMEOUT_SECS   bounded wait for the job to reach DONE after RUN --
+#                       must exceed JOB_SLEEP_SECS by a comfortable margin
+#                       (IPC round-trip + poll granularity), hence the
+#                       default's "+30" padding rather than JOB_SLEEP_SECS
+#                       alone.
+WAITMS="${XCE_MULTIHOST_WAITMS:-5000}"
+JOB_SLEEP_SECS="${XCE_MULTIHOST_JOB_SLEEP:-5}"
+RUN_TIMEOUT_SECS="${XCE_MULTIHOST_RUN_TIMEOUT:-30}"
+DONE_TIMEOUT_SECS="${XCE_MULTIHOST_DONE_TIMEOUT:-$((JOB_SLEEP_SECS + 30))}"
+
 # Single-node cluster, deliberately -- design.md's own "no HA needed for this
 # proof" call for this component. Single, fixed port so the pre-flight
 # check below has one unambiguous thing to check for a conflict against.
 PEER_PORT="${XCE_MULTIHOST_PORT:-9991}"
 PEERS="localhost:$PEER_PORT"
-MASTERS="localhost:$PEER_PORT"   # what a future xce_test client session's `set masters` will use (task 4.2+)
+MASTERS="localhost:$PEER_PORT"   # what this script's own xce_test client sessions (jobinfo_of()/submit_multihost_job()) `set masters` to
 
 WORKER1_NAME="${XCE_MULTIHOST_WORKER1:-mh-worker-1}"
 WORKER2_NAME="${XCE_MULTIHOST_WORKER2:-mh-worker-2}"
@@ -263,9 +296,7 @@ precondition_checks() {
     [ -x "$BINARY" ]    || fail "controller binary missing: $BINARY (run /xce-build controller)"
     [ -x "$SCHED_BIN" ] || fail "scheduler binary missing: $SCHED_BIN (run /xce-build scheduler)"
     [ -x "$WORKER_BIN" ] || fail "worker binary missing: $WORKER_BIN (run /xce-build worker)"
-    if [ ! -x "$XT" ]; then
-        log "note: xce_test binary missing at $XT -- not required by this task (no job submitted yet), but task 4.2 will need it (run /xce-build controller)"
-    fi
+    [ -x "$XT" ]         || fail "xce_test binary missing: $XT (run /xce-build controller) -- required by this task's job-scenario scripted sessions"
     command -v nix >/dev/null 2>&1 || fail "nix not found on PATH -- required to resolve the deploy-environment bash and invoke deploy-ha.sh"
 }
 
@@ -426,48 +457,199 @@ confirm_cluster_up() {
     log "all four roles (controller, scheduler, $WORKER1_NAME, $WORKER2_NAME) confirmed up under $TOP"
 }
 
-# ═══════════════════════════════════════════════════════════════════════════
-# EXTENSION POINT for tasks 4.2/4.3 (NOT implemented by this task -- task 4.1
-# is deploy/teardown-only, per its own Observable in tasks.md).
+# ── Step 4: the real job scenario (task 4.2) ────────────────────────────────
 #
-# Task 4.2 (a plain multi-task job): add a submit_multihost_job() here,
-# following run-ha-soak.sh's submit_worker_job() idiom EXACTLY (scripted
-# xce_test client-context session piped via `xtest()`-style helper: `set
-# masters $MASTERS`, `set myname <name>`, `client`, `wait-primary <ms>`,
-# `jobsub -u root -q normal -c "<cmd>"`, `quit`), then poll `jobinfo_of
-# "$jid" <timeout_ms>`-style for the RUN -> DONE transition (run-ha-soak.sh's
-# verify_worker_rejoin_resync() polling idiom) -- per design.md's
-# Implementation Notes ("reuses run-ha-soak.sh's submit_worker_job()/
-# verify_worker_rejoin_resync() polling idiom ... rather than inventing a new
-# verification mechanism"). MUST drive submission via plain `jobsub`, never
-# `xce_test`'s `jobrun` manual-allocation bypass -- the scheduler's own
-# decision to span $WORKER1_NAME/$WORKER2_NAME is what this whole test
-# exists to prove (design.md's own Responsibilities & Constraints for this
-# component). After a job reaches DONE, grep EACH worker's own log
-# ($TOP/logs/xcecd-$WORKER1_NAME.log / xcecd-$WORKER2_NAME.log) to confirm
-# genuine per-host task execution, not just an aggregate job-state check.
-#
-# Task 4.3 (the same, as one element of an array-job submission via
-# `jobsub -a N`): a second scenario function alongside submit_multihost_job(),
-# same polling idiom, confirming the SAME per-element task-count wiring a
-# non-array job gets (Requirement 5.5).
-#
-# Worker sizing (researched during task 4.1, not wired in yet -- see this
-# task's own Status Report CONCERNS for the full writeup): xcecd_virtual has
+# Worker sizing (researched during task 4.1, wired in here): xcecd_virtual has
 # NO command-line/config override for its reported processor/slot capacity
 # anywhere in worker/src/main.c's getopt_long() table ("peers"/"version"/
 # "help"/-n/-l/-v only). hardware_monitor.c's hardware_monitor_update()
 # always falls back to sysconf(_SC_NPROCESSORS_ONLN) (hwloc_disabled=true by
 # default, so the hwloc branch never runs) -- i.e. "virtual" only means
 # xcecd_virtual skips REAL job execution/cgroups, NOT that it fakes a
-# different processor count; each of $WORKER1_NAME/$WORKER2_NAME will report
-# this sandbox's real host core count (confirmed: nproc=22 here). Task 4.2's
-# scenario should therefore pick a maxTasks strictly greater than that
-# per-worker capacity N but at most 2N, so the job fits across exactly
-# $WORKER1_NAME + $WORKER2_NAME and neither alone can satisfy it -- e.g.
-# `nproc` at test-run time + 1, clamped to <= 2 * nproc. Do not hardcode 22;
-# query it live (e.g. `nproc` inside the deploy environment) since a
-# different sandbox/CI host will report a different core count.
+# different processor count; each of $WORKER1_NAME/$WORKER2_NAME reports this
+# host's REAL core count. So the test job's task count is picked strictly
+# greater than that per-worker capacity N but at most 2N, so the job fits
+# across exactly $WORKER1_NAME + $WORKER2_NAME and neither alone can satisfy
+# it -- queried live via `nproc` (never hardcoded: a different sandbox/CI
+# host reports a different core count).
+compute_task_count() {
+    local n
+    n="$(nproc 2>/dev/null)"
+    case "$n" in
+        ''|*[!0-9]*|0)
+            log "warning: nproc did not return a usable core count ('$n') -- falling back to 4" >&2
+            n=4
+            ;;
+    esac
+    echo $((n + 1))
+}
+
+# xce_test client-session runner -- self-nix-wrapping (like deploy()'s own
+# BASH_BIN resolution above), following run-resource-verify.sh's own xtest()
+# EXACTLY: ASAN_OPTIONS=detect_leaks=0 (the debug/ASAN xce_test build reports
+# harmless leaks this project already treats as a known, accepted caveat --
+# same as every sibling skill), wrapped in `nix develop` so xce_test's
+# runtime libs (openssl etc.) resolve, bounded by `timeout` so a hung session
+# can never wedge this script. Judge success from OUTPUT text, never the
+# leak-polluted process exit code (same caveat every sibling skill
+# documents). Depends on $BASH_BIN, resolved once by deploy() above.
+xtest() {
+    nix develop "$REPO/src" --command env ASAN_OPTIONS=detect_leaks=0 timeout 60 "$XT" --norc 2>&1
+}
+
+# `jobinfo -j <id>` against the live (single-node) primary. Prints the
+# jobinfo text and returns 0 on a hit; returns 1 (nothing printed) if not
+# found/unreachable. Retries a couple of times with a short sleep to absorb
+# the same post-submit client-retry race run-ha-soak.sh's own jobinfo_of()
+# tolerates -- copied from that function verbatim in spirit, just against
+# this harness's own single-node $MASTERS/$XT/xtest().
+jobinfo_of() {
+    local id="$1" waitms="${2:-$WAITMS}" out attempt
+    for attempt in 1 2 3; do
+        out="$(printf 'set masters %s\nset myname mh-verify\nclient\nwait-primary %s\njobinfo -j %s\nquit\n' \
+                "$MASTERS" "$waitms" "$id" | xtest)"
+        if printf '%s\n' "$out" | grep -q "^JobID: "; then
+            printf '%s\n' "$out"
+            return 0
+        fi
+        [ "$attempt" -lt 3 ] && sleep 0.5
+    done
+    return 1
+}
+
+# Submits ONE real job via plain `jobsub` -- NOT xce_test's `jobrun`
+# manual-allocation bypass -- so the scheduler's own dispatch decision is
+# what actually spans $WORKER1_NAME/$WORKER2_NAME (design.md's own
+# Responsibilities & Constraints for this component). `-n <ntasks>,<ntasks>`
+# sets BOTH mintasks and maxtasks to the caller-computed split-forcing count
+# (cmd_jobsub, test.c ~2256-2274); resreq is omitted so cmd_jobsub defaults
+# it to "type == any" (test.c ~2313-2315), letting the allocator consider
+# every worker's capacity. `sleep <secs>` is the command
+# xcecd_worker_utils.c's add_worker() (VIRTUAL_CLUSTER branch) parses to
+# drive each host's own timer-based simulated execution -- the SAME
+# mechanism run-ha-soak.sh's own submit_worker_job() relies on. Echoes the
+# jobid on success; returns 1 (echoing nothing, diagnostic on stderr so it
+# never pollutes the captured jobid) on a rejected submit.
+submit_multihost_job() {
+    local ntasks="$1" sleep_secs="$2" out jid
+    out="$(printf 'set masters %s\nset myname mh-verify\nclient\nwait-primary %s\njobsub -u root -q normal -c "sleep %s" -n %s,%s\nquit\n' \
+            "$MASTERS" "$WAITMS" "$sleep_secs" "$ntasks" "$ntasks" | xtest)"
+    jid="$(printf '%s\n' "$out" | sed -n 's/^jobsubmit ok: JobId: \(.*\)$/\1/p')"
+    if [ -z "$jid" ]; then
+        log "job submission failed -- raw xce_test output: $(printf '%s' "$out" | tr '\n' ' ')" >&2
+        return 1
+    fi
+    echo "$jid"
+    return 0
+}
+
+# Poll jobinfo_of() every 1s (up to $3 seconds) until job $1 reaches state
+# $2. Echoes the observed state on success; returns 1 on timeout (same
+# jobinfo_of()+sed state-extraction+1s-sleep-loop idiom as run-ha-soak.sh's
+# own submit_worker_job()/verify_worker_rejoin_resync()).
+wait_for_job_state() {
+    local jid="$1" want="$2" timeout="$3" i st
+    for i in $(seq 1 "$timeout"); do
+        st="$(jobinfo_of "$jid" 2000 | sed -n 's/^state: //p')"
+        if [ "$st" = "$want" ]; then
+            echo "$st"
+            return 0
+        fi
+        sleep 1
+    done
+    log "job $jid never reached $want after ${timeout}s (last observed state: ${st:-unknown})" >&2
+    return 1
+}
+
+# Per-host "this host actually started its share of THIS job's tasks"
+# evidence (task 4.2's Observable). Confirmed LIVE (this task's own first
+# validation run) that the controller does NOT send an independent per-host
+# startjob message to every allocated host as design.md's sequence diagram
+# simplifies it -- it sends ONE startjob to the first allocated host only;
+# THAT host's own job_start_remote_task() (controller_connector.c ~725-790)
+# then relays a runtasks message directly to every OTHER allocated host via
+# xcecli_runtask(). Requirement 4.1 ("the Controller shall ensure every
+# allocated host has the information it needs to start its own share") is
+# still satisfied end to end -- every host DOES end up with what it needs --
+# just via two different message types depending on which host the
+# allocator picked as first. Which of $WORKER1_NAME/$WORKER2_NAME takes
+# which path is the allocator's own internal choice, not something this
+# script can predict -- this check accepts EITHER path's own jobid-tagged
+# evidence line, both confirmed live in this task's own first validation run:
+#   - startjob path (job_start_local_task(), controller_connector.c ~1012):
+#       XCECD_LOG(XCE_LOG_DEBUG, "Job %s ntasks is: %d", jobidStr, ...);
+#     (mh-worker-1 took this path in the live run.)
+#   - runtasks path (handle_runtasks(), controller_connector.c ~1174):
+#       XCECD_LOG(XCE_LOG_DEBUG3, "run task: jobid = %s", jobidStr);
+#     (mh-worker-2 took this path in the live run.)
+# Both `jobidStr`s are fmt_jobid() applied to the exact jobid this host was
+# told to run -- the SAME formatting function xce_test's own cmd_jobsub uses
+# to print "jobsubmit ok: JobId: <id>" (test.c ~2424/2434), so the `$jid`
+# this script captures from that line and the `<id>` embedded in either log
+# line are the IDENTICAL string: an exact, job-scoped match, not a
+# same-cluster coincidence. (A third, path-independent but NOT jobid-tagged
+# line -- xcecd.c ~54's "Starting worker as user ... to process command:
+# ..." -- fires on both paths too, confirmed live on both hosts; not used as
+# the primary signal here only because it lacks the jobid, but corroborates
+# the two lines above whenever inspecting a log by hand.) Reuses this
+# script's own wait_for_log() (short bounded wait: the log line is written
+# synchronously as part of the SAME async task that gets this job to RUN, so
+# it is expected to already be present by the time wait_for_job_state()
+# above observes RUN -- this wait is a safety margin, not the primary gate).
+verify_task_started_on_host() {
+    local name="$1" jid="$2"
+    wait_for_log "worker '$name' started its share of job $jid's tasks" "Job $jid ntasks is:|run task: jobid = $jid" 15 "$TOP/logs/xcecd-$name.log"
+}
+
+# Orchestrates the whole task 4.2 scenario: size the job so it must span
+# both hosts, submit it via plain jobsub, poll RUN, confirm BOTH hosts
+# independently logged starting their own share, then poll DONE. Hard-fails
+# (via the shared fail() helper -- immediate FAIL: line + non-zero exit,
+# cleanup still runs via the registered trap) on the first violated step,
+# same discipline as every other structural check in this script.
+run_multihost_job_scenario() {
+    local ntasks jid
+
+    ntasks="$(compute_task_count)"
+    log "job scenario: sizing task count to $ntasks (nproc=$(nproc 2>/dev/null || echo '?') per worker) -- strictly more than either $WORKER1_NAME or $WORKER2_NAME can satisfy alone, at most both combined, forcing a genuine 2-host split"
+
+    log "submitting a real multi-task job via plain jobsub (-n $ntasks,$ntasks, \"sleep $JOB_SLEEP_SECS\") -- never xce_test's jobrun manual-allocation bypass"
+    jid="$(submit_multihost_job "$ntasks" "$JOB_SLEEP_SECS")" \
+        || fail "job submission failed -- see the diagnostic line just above"
+    log "submitted job $jid -- polling for RUN (timeout ${RUN_TIMEOUT_SECS}s)"
+
+    wait_for_job_state "$jid" "RUN" "$RUN_TIMEOUT_SECS" >/dev/null \
+        || fail "job $jid never reached RUN within ${RUN_TIMEOUT_SECS}s -- the scheduler's own allocation decision never dispatched it"
+    log "confirmed: job $jid reached RUN"
+
+    log "confirming each allocated host's own log shows its share of job $jid's tasks actually starting"
+    verify_task_started_on_host "$WORKER1_NAME" "$jid" \
+        || fail "job $jid: '$TOP/logs/xcecd-$WORKER1_NAME.log' never logged starting a share of this job's tasks"
+    verify_task_started_on_host "$WORKER2_NAME" "$jid" \
+        || fail "job $jid: '$TOP/logs/xcecd-$WORKER2_NAME.log' never logged starting a share of this job's tasks -- the allocation did not genuinely span both hosts"
+    log "confirmed: BOTH $WORKER1_NAME and $WORKER2_NAME independently logged starting their own share of job $jid's tasks -- the scheduler's own decision genuinely spans both hosts"
+
+    wait_for_job_state "$jid" "DONE" "$DONE_TIMEOUT_SECS" >/dev/null \
+        || fail "job $jid reached RUN and both hosts started their share, but never reached DONE within ${DONE_TIMEOUT_SECS}s"
+    log "confirmed: job $jid reached DONE -- completed successfully on both hosts"
+
+    MULTIHOST_JOB_ID="$jid"
+    MULTIHOST_JOB_NTASKS="$ntasks"
+}
+
+# ═══════════════════════════════════════════════════════════════════════════
+# EXTENSION POINT for task 4.3 (NOT implemented by this task -- 4.2 above
+# covers a plain job only).
+#
+# Task 4.3 (the same scenario, as one element of an array-job submission via
+# `jobsub -a N`): add a second scenario function alongside
+# run_multihost_job_scenario() above, following the SAME polling idiom
+# (submit_multihost_job() would need an `-a N` variant, or a sibling
+# function built the same way; jobinfo/wait_for_job_state() work unchanged
+# against a per-element jobid string, same array-element-addressing
+# convention run-ha-soak.sh's own array_element_id()/verify_array_elements()
+# already establish), confirming the array element receives the SAME
+# per-element task-count wiring a non-array job gets (Requirement 5.5).
 # ═══════════════════════════════════════════════════════════════════════════
 
 main() {
@@ -487,7 +669,10 @@ main() {
     confirm_cluster_up
 
     log "HARNESS_READY controller=up scheduler=up workers=2/2 top=$TOP"
-    log "PASS (harness-ready): controller + scheduler + 2 named worker(s) ($WORKER1_NAME, $WORKER2_NAME) deployed and confirmed joined; no job scenario submitted yet (task 4.1 scope only -- see tasks 4.2/4.3)"
+
+    run_multihost_job_scenario
+
+    log "PASS: job $MULTIHOST_JOB_ID ($MULTIHOST_JOB_NTASKS tasks, sleep ${JOB_SLEEP_SECS}s) submitted via plain jobsub, reached RUN then DONE; $WORKER1_NAME and $WORKER2_NAME each independently logged starting their own share of its tasks -- the scheduler's own decision genuinely spans both hosts and the job completed on both"
     exit 0
 }
 

.kiro/specs/resreq-maxtasks-wiring/tasks.md

@@ -65,7 +65,7 @@
   - _Boundary: xce-test-multihost_
   - _Depends: 2.1, 2.2_
 
-- [ ] 4.2 Prove a plain multi-task job is genuinely scheduled and completes across multiple hosts
+- [x] 4.2 Prove a plain multi-task job is genuinely scheduled and completes across multiple hosts
   - Submit a real multi-task job through normal job submission (not a manual-allocation bypass) and poll until it reaches a running state, then confirm it reaches completion
   - Confirm each allocated host's own log shows its share of the job's tasks actually starting
   - Observable: a PASS verdict confirms the scheduler's own decision spans both hosts and the job completes successfully on both