feat(resreq-maxtasks-wiring): add multi-host e2e test harness
New xce-test-multihost skill deploys one controller/scheduler node plus two named xcecd_virtual workers, following xce-test-resources's single-node deploy pattern and xce-test-ha's pre-flight zombie-process/ port-check convention (chosen over a fresh-mktemp-per-run model specifically because a meaningful pre-flight check needs a stable directory -- see tasks.md's Implementation Notes). Confirms all four roles join via their own log lines and tears down cleanly with a HARNESS_READY sentinel. No job scenario wired in yet -- that's tasks 4.2/4.3. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
This commit is contained in:
parent
d1849c94d8
commit
0313b910a9
3 changed files with 718 additions and 1 deletions
217
.claude/skills/xce-test-multihost/SKILL.md
Normal file
217
.claude/skills/xce-test-multihost/SKILL.md
Normal file
|
|
@ -0,0 +1,217 @@
|
|||
---
|
||||
name: xce-test-multihost
|
||||
description: "End-to-end multi-host job allocation harness: deploys one controller/scheduler node plus two named xcecd_virtual workers (each sized so neither alone can satisfy the test job's requested task count) and confirms all four roles come up and join cleanly. Usage: bash .claude/skills/xce-test-multihost/run-multihost-verify.sh [--keep-running] [--dir DIR]"
|
||||
model: haiku
|
||||
---
|
||||
|
||||
# XCE Multi-Host Allocation Test Harness
|
||||
|
||||
## Status: task 4.1 only -- deploy/teardown harness, no scenario wired in yet
|
||||
|
||||
`run-multihost-verify.sh` currently implements the resreq-maxtasks-wiring
|
||||
spec's task 4.1 scope: it deploys a single controller/scheduler node plus
|
||||
two independently-named `xcecd_virtual` workers, confirms all four roles
|
||||
come up and join, emits a `HARNESS_READY` sentinel + PASS/FAIL oracle line,
|
||||
and tears everything down cleanly. **It does not yet submit any job or
|
||||
check any allocation outcome** -- that is tasks 4.2 (a plain multi-task job
|
||||
via `jobsub`) and 4.3 (the same, as part of an array-job submission), which
|
||||
build directly on top of this same script (see the `EXTENSION POINT`
|
||||
comment block in `run-multihost-verify.sh`, right after
|
||||
`confirm_cluster_up()`).
|
||||
|
||||
This closes bug-1770's own end-to-end validation gap once 4.2/4.3 land: the
|
||||
scheduler-allocation feature already implements, unit-tests, and
|
||||
soak-validates real multi-host allocation logic (`parjob_allocator`) that,
|
||||
until resreq-maxtasks-wiring's own fix lands, has no way to ever be
|
||||
reached by a real job submission. This skill exists to prove -- with a real
|
||||
controller, a real scheduler, and real (virtual) workers, driven through
|
||||
plain `jobsub` -- that a job requiring more tasks than one host can provide
|
||||
is genuinely split across two hosts by the scheduler's own decision, and
|
||||
actually runs to completion on both.
|
||||
|
||||
Traces to `.kiro/specs/resreq-maxtasks-wiring/design.md`'s
|
||||
`xce-test-multihost` component (Requirements 4.1, 4.2, 4.3, 5.3, 5.5).
|
||||
|
||||
## What this skill does today (task 4.1)
|
||||
|
||||
1. **Pre-flight zombie-process/port check** -- following the convention
|
||||
already established by `xce-test-ha`'s `run-ha-soak.sh` (a fixed, reused
|
||||
deploy directory + a TOP-scoped kill of anything left over from a prior
|
||||
run of THIS script, plus an explicit check that the controller's port
|
||||
isn't already bound by an unrelated process) and this project's own
|
||||
`cerebrum.md` 2026-08-26 lesson: a same-session leftover
|
||||
`xce_controller`/`xce_scheduler`/`xcecd_virtual` process bound to the
|
||||
exact port a "fresh" deploy is about to reuse has previously produced
|
||||
apparently-impossible symptoms (term jumping, permanent config-load
|
||||
failure) in a live investigation, fully explained only once the zombie
|
||||
was found and killed. A conflict this script can positively attribute to
|
||||
its own prior run (matched via a `$TOP`-scoped cmdline substring, same
|
||||
discipline `run-ha-soak.sh`'s `node_pattern()`/`worker_pattern()`/
|
||||
`sched_pattern()` use) is killed automatically; a conflict it cannot
|
||||
attribute to itself (an unrelated live process on the same port) is a
|
||||
hard `FAIL`, never a silent/guessed kill (bug-1436's "fail loud, don't
|
||||
silently corrupt" rule).
|
||||
2. **Deploy** -- a single controller node plus scheduler plus two named
|
||||
workers into a fixed, reused scratch directory (default
|
||||
`/home/claude/tmp/xce-test-multihost`, override via `--dir`/
|
||||
`$XCE_MULTIHOST_TOP`) by invoking `xce-deploy-ha`'s `deploy-ha.sh`
|
||||
directly (single-node `--peers`, so no HA/raft election timing is in
|
||||
play for this proof), then starting the controller, scheduler, and both
|
||||
named workers individually via the scripts `deploy-ha.sh` generates
|
||||
(`start.sh ctrl-1`, `start-scheduler.sh`, `start-worker.sh <name>` twice
|
||||
with two different names) -- deliberately NOT `start-all.sh`, which only
|
||||
ever starts a single worker hardcoded to the name `worker-1`. A
|
||||
`trap`-based cleanup is registered before the first process starts, so it
|
||||
fires on every exit path (a completed run, a mid-deployment abort, or an
|
||||
external SIGINT/SIGTERM), matching `xce-test-resources`'s convention.
|
||||
3. **Confirm all four roles joined** -- bounded-wait polling (same
|
||||
short-interval/generous-timeout convention as every other skill in this
|
||||
family) for:
|
||||
- controller: `CTRL_PRIMARY` in `$TOP/logs/ctrl-1.log`
|
||||
- scheduler: `sched_init: joined cluster` in `$TOP/logs/scheduler.log`
|
||||
- each named worker: `Successfully sent .* load indices` in its own
|
||||
`$TOP/logs/xcecd-<name>.log` -- checked independently per worker, not
|
||||
just once for "a" worker, since this harness's whole point is TWO
|
||||
independently-joined hosts.
|
||||
4. **Report a `HARNESS_READY` sentinel + PASS/FAIL oracle line** -- since
|
||||
there is no job outcome to report yet at this stage, the terminal line
|
||||
reports "harness stood up and tore down correctly", not a job-allocation
|
||||
verdict:
|
||||
```
|
||||
[xce-test-multihost] HARNESS_READY controller=up scheduler=up workers=2/2 top=/home/claude/tmp/xce-test-multihost
|
||||
[xce-test-multihost] PASS (harness-ready): controller + scheduler + 2 named worker(s) (mh-worker-1, mh-worker-2) deployed and confirmed joined; no job scenario submitted yet (task 4.1 scope only -- see tasks 4.2/4.3)
|
||||
```
|
||||
Any structural failure (a role never confirming joined within its
|
||||
bounded wait, a missing binary, a pre-flight conflict this script cannot
|
||||
safely resolve) is a `FAIL:` line + non-zero exit instead, via the shared
|
||||
`fail()` helper.
|
||||
5. **Clean up** every process this run started -- unconditionally, on
|
||||
success or failure alike -- unless `--keep-running` was passed. Log
|
||||
files under the deploy directory are always preserved regardless of that
|
||||
flag, matching `xce-test-resources`'s own documented divergence from
|
||||
`xce-deploy-ha`'s `cleanup.sh` (which wipes `logs/*.log`).
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
bash .claude/skills/xce-test-multihost/run-multihost-verify.sh [--keep-running] [--dir DIR]
|
||||
```
|
||||
|
||||
- `--dir DIR` -- deploy directory. Default: a **fixed**, reused path
|
||||
(`/home/claude/tmp/xce-test-multihost`), following `xce-test-ha`'s own
|
||||
convention (not `xce-test-resources`'s fresh-`mktemp`-per-run one) -- this
|
||||
is what makes the pre-flight zombie check meaningful: a prior run's
|
||||
un-torn-down cluster at this same path is a realistic failure mode, not a
|
||||
hypothetical one. `$XCE_MULTIHOST_TOP` overrides the default when `--dir`
|
||||
is not given. A non-empty `DIR` that was **not** created by a prior run of
|
||||
this exact script is refused (`.xce-test-multihost` marker file guard,
|
||||
identical convention to `run-ha-soak.sh`'s own `MARKER_NAME` guard) -- this
|
||||
never silently wipes an arbitrary caller-named directory.
|
||||
- `--keep-running` -- skip PROCESS teardown in cleanup, leaving the
|
||||
deployment running under `DIR` for further inspection. Logs are always
|
||||
preserved either way.
|
||||
- Other overridable env vars: `$XCE_MULTIHOST_BUILD_DIR` (default
|
||||
`src/xce.build`), `$XCE_MULTIHOST_BIN` (controller),
|
||||
`$XCE_MULTIHOST_SCHED_BIN`, `$XCE_MULTIHOST_WORKER_BIN`,
|
||||
`$XCE_MULTIHOST_XT` (`xce_test` -- not required by this task, checked as
|
||||
an informational precondition for task 4.2's benefit),
|
||||
`$XCE_MULTIHOST_PORT` (default `9991` -- single-node controller port),
|
||||
`$XCE_MULTIHOST_WORKER1`/`$XCE_MULTIHOST_WORKER2` (default
|
||||
`mh-worker-1`/`mh-worker-2`).
|
||||
|
||||
Must be invoked under an environment with `nix` on `PATH` -- the script
|
||||
shells out to `nix develop` itself for the steps that need it.
|
||||
|
||||
**No `context: fork`, intentionally** -- same reason as every other skill in
|
||||
this family (`xce-deploy-ha`, `xce-test-ha`, `xce-test-resources`): this
|
||||
script starts real backgrounded controller/scheduler/worker processes for
|
||||
the run's duration, and a forked Skill-tool context's own teardown sends
|
||||
SIGHUP through the whole process group on return, silently killing them
|
||||
mid-run. Dispatch directly via Bash, or via the `Agent` tool (empirically
|
||||
verified safe for this exact pattern by the sibling skills above).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Controller, scheduler, and worker components must already be built in the
|
||||
same meson build dir (default `src/xce.build`):
|
||||
|
||||
```
|
||||
/xce-build controller --incremental
|
||||
/xce-build scheduler --incremental
|
||||
/xce-build worker --incremental
|
||||
```
|
||||
|
||||
(`--incremental` matters here specifically because a plain, non-incremental
|
||||
`/xce-build <component>` wipes the whole build dir before rebuilding just
|
||||
that one component -- see this project's own bug-1719 lesson -- so building
|
||||
controller/scheduler/worker one at a time into the same dir needs
|
||||
`--incremental` on every call after the first.)
|
||||
|
||||
## Worker sizing (researched for task 4.2's benefit, not wired in by this task)
|
||||
|
||||
`xcecd_virtual` has **no** command-line or config override for its reported
|
||||
processor/slot capacity: `worker/src/main.c`'s `getopt_long()` table is
|
||||
`--peers`/`--version`/`--help`/`-n`/`-l`/`-v` only, and
|
||||
`hardware_monitor.c`'s `hardware_monitor_update()` always falls back to
|
||||
`sysconf(_SC_NPROCESSORS_ONLN)` (`hwloc_disabled = true` by default, so the
|
||||
`hwloc`-based branch never actually runs). In other words, "virtual" in
|
||||
`xcecd_virtual` means it skips real job execution/cgroups, **not** that it
|
||||
fakes a different processor count -- each worker reports the real host's
|
||||
actual core count (confirmed live in this sandbox: `nproc` reports 22).
|
||||
|
||||
Consequence for task 4.2/4.3's scenario: since a worker's capacity cannot be
|
||||
shrunk via configuration, the robust sizing strategy is to pick the test
|
||||
job's `maxTasks` **strictly greater than** one worker's reported capacity
|
||||
`N` but **at most `2N`**, so the job fits across exactly the two workers
|
||||
this harness starts and neither alone can satisfy it. Query `nproc` live at
|
||||
test-run time rather than hardcoding a number -- a different sandbox/CI host
|
||||
will report a different core count.
|
||||
|
||||
## Extension points for tasks 4.2 / 4.3
|
||||
|
||||
`run-multihost-verify.sh` has a clearly marked `EXTENSION POINT` comment
|
||||
block (right after `confirm_cluster_up()`) documenting exactly where and how
|
||||
the next two tasks should add:
|
||||
|
||||
- `submit_multihost_job()` (task 4.2) -- a plain multi-task job via
|
||||
`jobsub`, following `run-ha-soak.sh`'s `submit_worker_job()`/
|
||||
`verify_worker_rejoin_resync()` polling idiom verbatim (scripted
|
||||
`xce_test` client-context session: `set masters`, `set myname`, `client`,
|
||||
`wait-primary`, `jobsub -u root -q normal -c "..."`, `quit`; then poll
|
||||
`jobinfo` for `RUN` then `DONE`). **Must** drive submission via plain
|
||||
`jobsub`, never `xce_test`'s `jobrun` manual-allocation bypass -- the
|
||||
scheduler's own decision to span `mh-worker-1`/`mh-worker-2` is what this
|
||||
whole test exists to prove. After the job reaches `DONE`, grep **each**
|
||||
worker's own log to confirm genuine per-host task execution.
|
||||
- A second, array-job scenario (task 4.3) -- the same job submitted via
|
||||
`jobsub -a N`, same polling idiom, confirming the array element receives
|
||||
the same per-element task-count wiring a non-array job would
|
||||
(Requirement 5.5).
|
||||
|
||||
## Known pre-existing constraints this skill inherits (not fixed here)
|
||||
|
||||
- Same `xce_test`/ASAN leak-detection caveat every sibling skill documents:
|
||||
a future `submit_multihost_job()` must run `xce_test` with
|
||||
`ASAN_OPTIONS=detect_leaks=0` and judge success from command **output**,
|
||||
never the leak-polluted process exit code.
|
||||
- Same nix-devshell runtime-linkage caveat: any future `xce_test` invocation
|
||||
should be wrapped in `nix develop`, like this script's own `deploy-ha.sh`
|
||||
invocation already is.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `xce-deploy-ha`'s `deploy-ha.sh` and its generated `start.sh` /
|
||||
`start-scheduler.sh` / `start-worker.sh` (single-controller-node path,
|
||||
two named worker instances against the same `--peers` config).
|
||||
- Controller, scheduler, and worker binaries in the same build dir (see
|
||||
Prerequisites above).
|
||||
|
||||
## Requirements traceability
|
||||
|
||||
This document describes design.md's `xce-test-multihost` component
|
||||
(Components and Interfaces -> Test Infrastructure), covering Requirements
|
||||
4.1, 4.2, 4.3, 5.3, 5.5. Task 4.1 (this revision) covers the deploy/teardown
|
||||
harness only; the job-scenario coverage those requirements ultimately need
|
||||
is completed by tasks 4.2/4.3. See
|
||||
`.kiro/specs/resreq-maxtasks-wiring/requirements.md` and `design.md` for the
|
||||
full requirement text.
|
||||
499
.claude/skills/xce-test-multihost/run-multihost-verify.sh
Executable file
499
.claude/skills/xce-test-multihost/run-multihost-verify.sh
Executable file
|
|
@ -0,0 +1,499 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# run-multihost-verify.sh -- driver for /xce-test-multihost.
|
||||
#
|
||||
# resreq-maxtasks-wiring spec, task 4.1 SCOPE ONLY: this revision implements
|
||||
# the DEPLOY/TEARDOWN HARNESS -- a single controller/scheduler node plus two
|
||||
# independently-named `xcecd_virtual` workers, each sized so neither alone
|
||||
# can satisfy the eventual test job's requested task count (see the
|
||||
# "Worker sizing" note near submit_multihost_job()'s placeholder below) --
|
||||
# with the pre-flight zombie-process/port check convention this project's
|
||||
# xce-test-ha soak driver established, trap-based cleanup registered before
|
||||
# the first process starts, and a HARNESS_READY sentinel + PASS/FAIL oracle
|
||||
# line. It deliberately does NOT yet submit any job or check any allocation
|
||||
# outcome -- see design.md's "xce-test-multihost" component (Batch/Job
|
||||
# Contract) and tasks.md 4.1's own Observable: "the new harness deploys its
|
||||
# fixture cluster and tears it down cleanly, with no scenario wired in yet".
|
||||
#
|
||||
# Tasks 4.2 (a plain multi-task job via `jobsub`) and 4.3 (the same, inside
|
||||
# an array-job submission) build ON TOP of this harness in later dispatch
|
||||
# rounds -- see the "EXTENSION POINT" comment block right after
|
||||
# confirm_cluster_up() below for exactly where and how (reusing run-ha-soak.sh's
|
||||
# submit_worker_job()/verify_worker_rejoin_resync() polling idiom, per
|
||||
# design.md's Implementation Notes for this component -- real job via plain
|
||||
# `jobsub`, poll `jobinfo` for RUN then DONE, grep each worker's own log --
|
||||
# never `xce_test`'s `jobrun` manual-allocation bypass, since the scheduler's
|
||||
# own decision to span hosts is what this whole test exists to prove).
|
||||
#
|
||||
# Requirements this harness exists to eventually help satisfy (this task
|
||||
# alone is infrastructure -- it does not itself complete any of these):
|
||||
# 4.1 (controller ensures every allocated host has launch info), 4.2 (worker
|
||||
# starts tasks on every allocated host), 4.3 (scheduler reports running via
|
||||
# the existing job-status mechanism), 5.3 (a real multi-task job driven
|
||||
# end-to-end at least once). See .kiro/specs/resreq-maxtasks-wiring/
|
||||
# {requirements,design}.md.
|
||||
#
|
||||
# ── Pattern provenance (read these before changing this file) ──────────────
|
||||
# - Deploy/start/confirm/cleanup/trap mechanics: copied from
|
||||
# xce-test-resources's run-resource-verify.sh (single-controller-node
|
||||
# pattern: xce-deploy-ha's deploy-ha.sh + generated per-component start
|
||||
# scripts, trap registered before the first process starts, TOP-scoped
|
||||
# pkill in cleanup(), preserved logs).
|
||||
# - Fixed default deploy dir + MARKER_NAME ownership-guarded `rm -rf` +
|
||||
# pre-flight zombie/port check: copied from xce-test-ha's run-ha-soak.sh
|
||||
# (which reuses the SAME default deploy dir, `/home/claude/tmp/...`,
|
||||
# across invocations -- unlike xce-test-resources's fresh-mktemp-per-run
|
||||
# convention -- so a TOP-scoped pre-flight zombie check is actually
|
||||
# meaningful here: a prior run's un-torn-down cluster is a real, concrete
|
||||
# possibility, not just a theoretical one). See this project's own
|
||||
# cerebrum.md 2026-08-26 Do-Not-Repeat entry: a same-session leftover
|
||||
# xce_controller/xce_scheduler/xcecd process bound to the exact ports a
|
||||
# "fresh" deploy is about to reuse has caused apparently-impossible
|
||||
# symptoms (term jumping, permanent config-load failure) in a live
|
||||
# investigation before -- always check `pgrep`/`ss` for the ports about to
|
||||
# be used before trusting a "fresh" deploy's results.
|
||||
# - Two independently-named `xcecd_virtual` workers against one controller:
|
||||
# xce-deploy-ha's own SKILL.md Step 4b ("start-worker.sh worker-1 -vvv &"
|
||||
# / "start-worker.sh worker-2 -vvv &" against the identical --peers config
|
||||
# baked into both generated scripts). NOT via start-all.sh, which only
|
||||
# ever starts a single worker named "worker-1" -- this script starts the
|
||||
# controller/scheduler/worker-1/worker-2 steps individually instead (same
|
||||
# generated scripts start-all.sh itself would call, just sequenced by hand
|
||||
# here so a second named worker can be added).
|
||||
#
|
||||
# Must be invoked under an environment with `nix` on PATH (same as
|
||||
# xce-deploy-ha/xce-test-ha/xce-test-resources) -- this script shells out to
|
||||
# `nix develop` itself for the steps that need it, so it does not need to be
|
||||
# pre-wrapped by the caller.
|
||||
#
|
||||
# Usage:
|
||||
# run-multihost-verify.sh [--keep-running] [--dir DIR]
|
||||
#
|
||||
# --keep-running Skip PROCESS teardown in cleanup -- the deployment
|
||||
# (controller/scheduler/worker-1/worker-2) is left
|
||||
# running under DIR for further inspection. Log files
|
||||
# under DIR are ALWAYS preserved regardless of this
|
||||
# flag.
|
||||
# --dir DIR Deploy directory. Default: a FIXED path (see TOP
|
||||
# below), following xce-test-ha's own "reused across
|
||||
# invocations, not a fresh mktemp dir" convention --
|
||||
# $XCE_MULTIHOST_TOP overrides it when --dir is not
|
||||
# given. A non-empty DIR that was NOT created by a
|
||||
# prior run of this script is refused (MARKER_NAME
|
||||
# guard, identical to run-ha-soak.sh's own), so this
|
||||
# never wipes an unrelated directory.
|
||||
#
|
||||
# Exit 0 == the fixture cluster (controller + scheduler + 2 named workers)
|
||||
# deployed and was confirmed fully joined; teardown ran cleanly. This is a
|
||||
# "harness stood up and tore down correctly" verdict, NOT a job-outcome
|
||||
# PASS/FAIL -- there is no job in this task's scope yet (see HARNESS_READY
|
||||
# below).
|
||||
# Exit non-zero == a structural failure at any step (pre-flight zombie/port
|
||||
# conflict left unresolved, missing binary, deployment failure, a role that
|
||||
# never confirmed joining within its bounded wait). Cleanup still runs on
|
||||
# this path (trap-based, registered before the first process starts).
|
||||
#
|
||||
set -u
|
||||
|
||||
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO="$(cd "$SELF_DIR/../../.." && pwd)"
|
||||
|
||||
DEPLOY_HA="$REPO/src/tools/deploy-ha.sh"
|
||||
BUILD_DIR="${XCE_MULTIHOST_BUILD_DIR:-$REPO/src/xce.build}"
|
||||
BINARY="${XCE_MULTIHOST_BIN:-$BUILD_DIR/controller/xce_controller}"
|
||||
SCHED_BIN="${XCE_MULTIHOST_SCHED_BIN:-$BUILD_DIR/scheduler/xce_scheduler}"
|
||||
WORKER_BIN="${XCE_MULTIHOST_WORKER_BIN:-$BUILD_DIR/worker/xcecd_virtual}"
|
||||
# xce_test -- not used by THIS task (no job is submitted yet), but checked
|
||||
# as a soft/informational precondition since task 4.2 (the very next dispatch
|
||||
# round) needs it immediately for submit_multihost_job()'s scripted session,
|
||||
# following the exact idiom run-ha-soak.sh's submit_worker_job() already
|
||||
# established (see the EXTENSION POINT block below).
|
||||
XT="${XCE_MULTIHOST_XT:-$BUILD_DIR/controller/xce_test}"
|
||||
|
||||
# Single-node cluster, deliberately -- design.md's own "no HA needed for this
|
||||
# proof" call for this component. Single, fixed port so the pre-flight
|
||||
# check below has one unambiguous thing to check for a conflict against.
|
||||
PEER_PORT="${XCE_MULTIHOST_PORT:-9991}"
|
||||
PEERS="localhost:$PEER_PORT"
|
||||
MASTERS="localhost:$PEER_PORT" # what a future xce_test client session's `set masters` will use (task 4.2+)
|
||||
|
||||
WORKER1_NAME="${XCE_MULTIHOST_WORKER1:-mh-worker-1}"
|
||||
WORKER2_NAME="${XCE_MULTIHOST_WORKER2:-mh-worker-2}"
|
||||
|
||||
# Fixed default deploy dir, REUSED across invocations (xce-test-ha's own
|
||||
# convention, not xce-test-resources's fresh-mktemp-per-run one) -- this is
|
||||
# what makes the pre-flight zombie/port check below meaningful: a prior run
|
||||
# of this exact script, left running or killed uncleanly, is the realistic
|
||||
# failure mode this check exists to catch, not a purely theoretical one.
|
||||
TOP="${XCE_MULTIHOST_TOP:-/home/claude/tmp/xce-test-multihost}"
|
||||
# Sentinel written into a deploy dir we own; the guard before `rm -rf` only
|
||||
# wipes a non-empty dir that carries this marker -- identical convention to
|
||||
# run-ha-soak.sh's own MARKER_NAME/guard, so an arbitrary caller-named `--dir`
|
||||
# (e.g. a typo'd path) is never silently destroyed.
|
||||
MARKER_NAME=".xce-test-multihost"
|
||||
|
||||
KEEP_RUNNING=0
|
||||
|
||||
log() { echo "[xce-test-multihost] $*"; }
|
||||
fail() { echo "[xce-test-multihost] FAIL: $*" >&2; exit 1; }
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $0 [--keep-running] [--dir DIR]
|
||||
|
||||
--keep-running leave controller/scheduler/worker-1/worker-2 processes
|
||||
running after this run instead of tearing them down. Logs
|
||||
are always preserved either way.
|
||||
--dir DIR deploy directory (default: $TOP, or \$XCE_MULTIHOST_TOP).
|
||||
Reused across invocations -- refuses to wipe a non-empty
|
||||
dir that wasn't created by a prior run of this script.
|
||||
EOF
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--keep-running)
|
||||
KEEP_RUNNING=1
|
||||
shift
|
||||
;;
|
||||
--dir)
|
||||
[ $# -ge 2 ] || fail "--dir requires an argument"
|
||||
TOP="$2"
|
||||
shift 2
|
||||
;;
|
||||
--dir=*)
|
||||
TOP="${1#--dir=}"
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "[xce-test-multihost] Unknown argument: $1" >&2
|
||||
usage
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
# Resolve a relative --dir/$XCE_MULTIHOST_TOP against cwd, same pattern
|
||||
# as xce-test-ha/xce-test-resources (no realpath dependency -- busybox
|
||||
# realpath in this environment lacks -m).
|
||||
case "$TOP" in
|
||||
/*) : ;;
|
||||
*) TOP="$PWD/$TOP" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# ── Pre-flight zombie-process / port check (design.md: "Follows the
|
||||
# pre-flight zombie-process/port check convention already established for
|
||||
# xce-test-ha") ───────────────────────────────────────────────────────────
|
||||
#
|
||||
# Two distinct checks, both scoped to what THIS run is about to do (never an
|
||||
# unscoped `pkill -9 -x xce_controller`-style blast -- bug-1436's own
|
||||
# postmortem, this project's HA soak, is exactly what an unscoped kill looks
|
||||
# like going wrong: it silently killed a DIFFERENT concurrent run's live
|
||||
# primary):
|
||||
#
|
||||
# 1. A process already bound to $PEER_PORT (`ss -tlnp`). If its cmdline
|
||||
# shows it was started against THIS SAME $TOP (our own prior,
|
||||
# un-torn-down run -- the realistic case, since $TOP is a fixed,
|
||||
# reused path), it is safe to kill and this function does so. If its
|
||||
# cmdline does NOT match $TOP -- an unrelated live process, or a
|
||||
# process we cannot positively attribute to ourselves -- this function
|
||||
# fails loud instead of guessing, per bug-1436's "fail loud, don't
|
||||
# silently corrupt" rule.
|
||||
# 2. Any stray xce_controller/xce_scheduler/xcecd_virtual process whose
|
||||
# cmdline names THIS run's own $TOP (an un-torn-down previous run of
|
||||
# this exact script at this exact deploy dir) -- killed unconditionally,
|
||||
# since it is unambiguously ours by construction (node_pattern-style
|
||||
# workdir/logfile substring match, same discipline run-ha-soak.sh's own
|
||||
# node_pattern()/worker_pattern()/sched_pattern() use).
|
||||
#
|
||||
# This directly codifies this project's own cerebrum.md 2026-08-26
|
||||
# Do-Not-Repeat lesson (a same-session zombie controller/scheduler/worker
|
||||
# process bound to the ports a "fresh" deploy is about to reuse has already
|
||||
# produced apparently-impossible symptoms in a live investigation) as an
|
||||
# automated check, rather than relying on a human/agent remembering to run
|
||||
# `ps aux`/`ss -tlnp` by hand before every invocation.
|
||||
ctrl_pattern() { echo "xce_controller .*--workdir $TOP/controller"; }
|
||||
sched_pattern() { echo "xce_scheduler .*--logfile $TOP/logs/scheduler.log"; }
|
||||
worker_pattern() { echo "xcecd_virtual.*-l $TOP/logs"; }
|
||||
run_pattern() { echo "$(ctrl_pattern)|$(sched_pattern)|$(worker_pattern)"; }
|
||||
|
||||
preflight_check() {
|
||||
log "pre-flight: checking for a leftover process on port $PEER_PORT and any stale xce_controller/xce_scheduler/xcecd_virtual process scoped to $TOP"
|
||||
|
||||
# Check 1: our own prior (un-torn-down) run at this exact TOP.
|
||||
local stale_pids
|
||||
stale_pids="$(pgrep -f -- "$(run_pattern)" 2>/dev/null || true)"
|
||||
if [ -n "$stale_pids" ]; then
|
||||
log "pre-flight: found stale process(es) from a prior run of this script scoped to $TOP -- killing: $stale_pids"
|
||||
# shellcheck disable=SC2086
|
||||
kill -9 $stale_pids 2>/dev/null || true
|
||||
sleep 0.3
|
||||
if pgrep -f -- "$(run_pattern)" >/dev/null 2>&1; then
|
||||
fail "pre-flight: could not clear stale process(es) scoped to $TOP after kill -9 -- inspect manually (pgrep -af \"$(run_pattern)\")"
|
||||
fi
|
||||
log "pre-flight: stale process(es) scoped to $TOP cleared"
|
||||
else
|
||||
log "pre-flight: no stale process scoped to $TOP found"
|
||||
fi
|
||||
|
||||
# Check 2: port conflict, from ANY process (ours or not). Re-checked
|
||||
# AFTER check 1's kill above, so a just-cleared stale process on our own
|
||||
# port no longer counts as a conflict.
|
||||
local port_line
|
||||
port_line="$(ss -tlnp 2>/dev/null | grep -E ":$PEER_PORT[[:space:]]" || true)"
|
||||
if [ -n "$port_line" ]; then
|
||||
# Was it one of the stale PIDs we just killed (or is it still racing
|
||||
# to exit)? Give it one more short grace check before failing loud.
|
||||
sleep 0.3
|
||||
port_line="$(ss -tlnp 2>/dev/null | grep -E ":$PEER_PORT[[:space:]]" || true)"
|
||||
fi
|
||||
if [ -n "$port_line" ]; then
|
||||
fail "pre-flight: port $PEER_PORT is already bound by a process this script did not start/attribute to $TOP -- refusing to deploy on top of it (see: ss -tlnp | grep :$PEER_PORT). Choose a different --dir/\$XCE_MULTIHOST_PORT, or stop that process yourself first."
|
||||
fi
|
||||
|
||||
log "pre-flight: clean -- port $PEER_PORT free, no stale process scoped to $TOP"
|
||||
}
|
||||
|
||||
precondition_checks() {
|
||||
[ -x "$BINARY" ] || fail "controller binary missing: $BINARY (run /xce-build controller)"
|
||||
[ -x "$SCHED_BIN" ] || fail "scheduler binary missing: $SCHED_BIN (run /xce-build scheduler)"
|
||||
[ -x "$WORKER_BIN" ] || fail "worker binary missing: $WORKER_BIN (run /xce-build worker)"
|
||||
if [ ! -x "$XT" ]; then
|
||||
log "note: xce_test binary missing at $XT -- not required by this task (no job submitted yet), but task 4.2 will need it (run /xce-build controller)"
|
||||
fi
|
||||
command -v nix >/dev/null 2>&1 || fail "nix not found on PATH -- required to resolve the deploy-environment bash and invoke deploy-ha.sh"
|
||||
}
|
||||
|
||||
# ── Cleanup: process teardown only, logs always preserved (same convention
|
||||
# as xce-test-resources's cleanup(): TOP-scoped patterns, retry-until-
|
||||
# verified, never touches $TOP/logs) ─────────────────────────────────────
|
||||
cleanup() {
|
||||
if [ "$KEEP_RUNNING" -eq 1 ]; then
|
||||
log "--keep-running requested -- leaving controller/scheduler/$WORKER1_NAME/$WORKER2_NAME up under $TOP (logs also preserved)"
|
||||
return 0
|
||||
fi
|
||||
log "cleaning up processes under $TOP (log files under $TOP/logs are preserved)"
|
||||
local pattern; pattern="$(run_pattern)"
|
||||
local attempt=0
|
||||
while [ "$attempt" -lt 10 ]; do
|
||||
attempt=$((attempt + 1))
|
||||
if ! pgrep -f -- "$pattern" >/dev/null 2>&1; then
|
||||
break
|
||||
fi
|
||||
pkill -9 -f -- "$pattern" 2>/dev/null
|
||||
sleep 0.2
|
||||
done
|
||||
if pgrep -f -- "$pattern" >/dev/null 2>&1; then
|
||||
log "warning: some processes under $TOP may still be running after cleanup retries"
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# Standard robust trap idiom (xce-test-resources's on_exit/register_trap,
|
||||
# copied verbatim in spirit): registered BEFORE the first process starts, so
|
||||
# it fires on a completed run, a mid-deployment abort, OR an external
|
||||
# SIGINT/SIGTERM alike -- not just a successful completion.
|
||||
on_exit() {
|
||||
local rc=$?
|
||||
cleanup
|
||||
exit "$rc"
|
||||
}
|
||||
|
||||
register_trap() {
|
||||
trap on_exit EXIT
|
||||
trap 'exit 130' INT
|
||||
trap 'exit 143' TERM
|
||||
}
|
||||
|
||||
# ── Step 1: deploy (xce-deploy-ha's deploy-ha.sh) ───────────────────────────
|
||||
deploy() {
|
||||
log "resolving deploy-environment bash via nix develop"
|
||||
BASH_BIN="$(nix develop "$REPO/src" --command which bash)"
|
||||
[ -n "$BASH_BIN" ] || fail "deployment failed at step deploy-ha.sh: could not resolve bash inside nix develop"
|
||||
|
||||
# Ownership-guarded rm -rf -- identical convention to run-ha-soak.sh's
|
||||
# own MARKER_NAME guard: refuse to wipe a non-empty dir unless WE (a
|
||||
# prior run of this exact script) created it. An empty or non-existent
|
||||
# dir is always safe to (re)deploy into.
|
||||
if [ -d "$TOP" ] && [ -n "$(ls -A "$TOP" 2>/dev/null)" ] && [ ! -e "$TOP/$MARKER_NAME" ]; then
|
||||
fail "refusing to wipe non-empty $TOP -- not a prior xce-test-multihost deploy (no $MARKER_NAME marker). Choose an empty/new --dir, or remove it yourself."
|
||||
fi
|
||||
rm -rf "$TOP"
|
||||
mkdir -p "$TOP" || fail "could not create deploy dir $TOP"
|
||||
touch "$TOP/$MARKER_NAME" # claim ownership so a later run may safely re-wipe it
|
||||
|
||||
log "deploying controller+scheduler+worker fixture into $TOP (peers=$PEERS)"
|
||||
if ! nix develop "$REPO/src" --command "$BASH_BIN" "$DEPLOY_HA" "$BINARY" "$TOP" "$PEERS" >"$TOP/deploy.log" 2>&1; then
|
||||
fail "deployment failed at step deploy-ha.sh (see $TOP/deploy.log)"
|
||||
fi
|
||||
|
||||
[ -x "$TOP/scripts/start.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start.sh was not generated"
|
||||
[ -x "$TOP/scripts/start-scheduler.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start-scheduler.sh was not generated (scheduler binary not found alongside controller build?)"
|
||||
[ -x "$TOP/scripts/start-worker.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start-worker.sh was not generated (worker binary not found alongside controller build?)"
|
||||
}
|
||||
|
||||
# ── Step 2: start controller, scheduler, and BOTH named workers ────────────
|
||||
# Deliberately NOT start-all.sh -- that generated script only ever starts a
|
||||
# single worker hardcoded to the name "worker-1". This harness needs TWO
|
||||
# independently-named xcecd_virtual instances against the same controller
|
||||
# (design.md: "two named xcecd_virtual instances, each sized so neither
|
||||
# alone can satisfy the test job's requested task count"), so the
|
||||
# controller/scheduler/worker-1/worker-2 steps start-all.sh itself would run
|
||||
# are sequenced here individually instead -- same generated scripts, same
|
||||
# --peers config baked into every one of them by deploy-ha.sh, nothing
|
||||
# duplicated or reimplemented. Every process is backgrounded via
|
||||
# `setsid nohup ... </dev/null >/dev/null 2>&1 & disown`, matching
|
||||
# start-all.sh's own convention, so the whole fixture survives this script's
|
||||
# own foreground steps (and, if --keep-running is passed, survives this
|
||||
# script's own exit too).
|
||||
start_cluster() {
|
||||
log "starting controller (ctrl-1, -vvv)"
|
||||
setsid nohup "$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv \
|
||||
</dev/null >>"$TOP/logs/ctrl-1.stderr" 2>&1 &
|
||||
disown
|
||||
|
||||
# Fixed filename, not a glob: this is always a single-node deployment
|
||||
# (deploy-ha.sh names its one candidate "ctrl-1" unconditionally), and a
|
||||
# glob here would be a real timing bug -- bash expands "$TOP"/logs/ctrl-*.log
|
||||
# exactly ONCE, at the moment this line is evaluated (as part of building
|
||||
# wait_for_log's argv), not once per poll iteration inside the function.
|
||||
# Immediately after backgrounding the controller above, its log file may
|
||||
# not exist yet; an unmatched glob falls through as the literal
|
||||
# unexpanded string "ctrl-*.log" (bash's default, non-nullglob behavior),
|
||||
# which then never matches any real file for the rest of the bounded
|
||||
# wait -- confirmed live during this task's own validation run (the
|
||||
# controller elected CTRL_PRIMARY as its very first log line, but the
|
||||
# glob-based wait_for_log call still timed out after the full 30s).
|
||||
wait_for_log "controller primary elected" "CTRL_PRIMARY" 30 "$TOP/logs/ctrl-1.log" \
|
||||
|| fail "deployment failed at step controller: no primary election observed in $TOP/logs/ctrl-1.log within 30s"
|
||||
|
||||
log "starting scheduler (-vvv)"
|
||||
setsid nohup "$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -vvv \
|
||||
</dev/null >>"$TOP/logs/scheduler.stderr" 2>&1 &
|
||||
disown
|
||||
|
||||
log "starting worker '$WORKER1_NAME' (-vvv)"
|
||||
setsid nohup "$BASH_BIN" "$TOP/scripts/start-worker.sh" "$WORKER1_NAME" -vvv \
|
||||
</dev/null >>"$TOP/logs/$WORKER1_NAME.stderr" 2>&1 &
|
||||
disown
|
||||
|
||||
log "starting worker '$WORKER2_NAME' (-vvv)"
|
||||
setsid nohup "$BASH_BIN" "$TOP/scripts/start-worker.sh" "$WORKER2_NAME" -vvv \
|
||||
</dev/null >>"$TOP/logs/$WORKER2_NAME.stderr" 2>&1 &
|
||||
disown
|
||||
}
|
||||
|
||||
# ── Step 3: confirm all four roles are actually up and joined ──────────────
|
||||
# Same bounded-wait/short-poll-interval convention as xce-test-resources's
|
||||
# wait_for_log() (copied verbatim) and xce-deploy-ha's own documented
|
||||
# verification greps for scheduler/worker join lines:
|
||||
# scheduler: "sched_init: joined cluster" (scheduler.c, confirmed live)
|
||||
# worker: "Successfully sent .* load indices" (controller_connector.c,
|
||||
# confirmed live), per NAMED worker's own log file
|
||||
# ($TOP/logs/xcecd-<name>.log for xcecd_virtual).
|
||||
wait_for_log() {
|
||||
local desc="$1" pattern="$2" timeout="$3"
|
||||
shift 3
|
||||
local waited=0
|
||||
while [ "$waited" -lt "$timeout" ]; do
|
||||
if grep -qE -- "$pattern" "$@" 2>/dev/null; then
|
||||
log "confirmed: $desc"
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
waited=$((waited + 1))
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
confirm_cluster_up() {
|
||||
log "waiting for scheduler and both workers to confirm joined (bounded wait per role; controller primary election already confirmed by start_cluster())"
|
||||
|
||||
wait_for_log "scheduler joined cluster" "sched_init: joined cluster" 30 "$TOP/logs/scheduler.log" \
|
||||
|| fail "deployment failed at step scheduler: '$TOP/logs/scheduler.log' never logged joining the cluster within 30s"
|
||||
|
||||
wait_for_log "worker '$WORKER1_NAME' joined and reported load indices" "Successfully sent .* load indices" 30 "$TOP/logs/xcecd-$WORKER1_NAME.log" \
|
||||
|| fail "deployment failed at step worker: '$TOP/logs/xcecd-$WORKER1_NAME.log' never logged sending load indices within 30s"
|
||||
|
||||
wait_for_log "worker '$WORKER2_NAME' joined and reported load indices" "Successfully sent .* load indices" 30 "$TOP/logs/xcecd-$WORKER2_NAME.log" \
|
||||
|| fail "deployment failed at step worker: '$TOP/logs/xcecd-$WORKER2_NAME.log' never logged sending load indices within 30s"
|
||||
|
||||
log "all four roles (controller, scheduler, $WORKER1_NAME, $WORKER2_NAME) confirmed up under $TOP"
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# EXTENSION POINT for tasks 4.2/4.3 (NOT implemented by this task -- task 4.1
|
||||
# is deploy/teardown-only, per its own Observable in tasks.md).
|
||||
#
|
||||
# Task 4.2 (a plain multi-task job): add a submit_multihost_job() here,
|
||||
# following run-ha-soak.sh's submit_worker_job() idiom EXACTLY (scripted
|
||||
# xce_test client-context session piped via `xtest()`-style helper: `set
|
||||
# masters $MASTERS`, `set myname <name>`, `client`, `wait-primary <ms>`,
|
||||
# `jobsub -u root -q normal -c "<cmd>"`, `quit`), then poll `jobinfo_of
|
||||
# "$jid" <timeout_ms>`-style for the RUN -> DONE transition (run-ha-soak.sh's
|
||||
# verify_worker_rejoin_resync() polling idiom) -- per design.md's
|
||||
# Implementation Notes ("reuses run-ha-soak.sh's submit_worker_job()/
|
||||
# verify_worker_rejoin_resync() polling idiom ... rather than inventing a new
|
||||
# verification mechanism"). MUST drive submission via plain `jobsub`, never
|
||||
# `xce_test`'s `jobrun` manual-allocation bypass -- the scheduler's own
|
||||
# decision to span $WORKER1_NAME/$WORKER2_NAME is what this whole test
|
||||
# exists to prove (design.md's own Responsibilities & Constraints for this
|
||||
# component). After a job reaches DONE, grep EACH worker's own log
|
||||
# ($TOP/logs/xcecd-$WORKER1_NAME.log / xcecd-$WORKER2_NAME.log) to confirm
|
||||
# genuine per-host task execution, not just an aggregate job-state check.
|
||||
#
|
||||
# Task 4.3 (the same, as one element of an array-job submission via
|
||||
# `jobsub -a N`): a second scenario function alongside submit_multihost_job(),
|
||||
# same polling idiom, confirming the SAME per-element task-count wiring a
|
||||
# non-array job gets (Requirement 5.5).
|
||||
#
|
||||
# Worker sizing (researched during task 4.1, not wired in yet -- see this
|
||||
# task's own Status Report CONCERNS for the full writeup): xcecd_virtual has
|
||||
# NO command-line/config override for its reported processor/slot capacity
|
||||
# anywhere in worker/src/main.c's getopt_long() table ("peers"/"version"/
|
||||
# "help"/-n/-l/-v only). hardware_monitor.c's hardware_monitor_update()
|
||||
# always falls back to sysconf(_SC_NPROCESSORS_ONLN) (hwloc_disabled=true by
|
||||
# default, so the hwloc branch never runs) -- i.e. "virtual" only means
|
||||
# xcecd_virtual skips REAL job execution/cgroups, NOT that it fakes a
|
||||
# different processor count; each of $WORKER1_NAME/$WORKER2_NAME will report
|
||||
# this sandbox's real host core count (confirmed: nproc=22 here). Task 4.2's
|
||||
# scenario should therefore pick a maxTasks strictly greater than that
|
||||
# per-worker capacity N but at most 2N, so the job fits across exactly
|
||||
# $WORKER1_NAME + $WORKER2_NAME and neither alone can satisfy it -- e.g.
|
||||
# `nproc` at test-run time + 1, clamped to <= 2 * nproc. Do not hardcode 22;
|
||||
# query it live (e.g. `nproc` inside the deploy environment) since a
|
||||
# different sandbox/CI host will report a different core count.
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
|
||||
# Trap registered before the FIRST process is started (deploy() only
|
||||
# generates scripts/certs/config; start_cluster() is what actually
|
||||
# starts backgrounded processes) -- covers a mid-deployment abort (e.g.
|
||||
# worker-2 fails to join after controller+scheduler+worker-1 are already
|
||||
# up), not just a completed run.
|
||||
register_trap
|
||||
|
||||
precondition_checks
|
||||
preflight_check
|
||||
deploy
|
||||
start_cluster
|
||||
confirm_cluster_up
|
||||
|
||||
log "HARNESS_READY controller=up scheduler=up workers=2/2 top=$TOP"
|
||||
log "PASS (harness-ready): controller + scheduler + 2 named worker(s) ($WORKER1_NAME, $WORKER2_NAME) deployed and confirmed joined; no job scenario submitted yet (task 4.1 scope only -- see tasks 4.2/4.3)"
|
||||
exit 0
|
||||
}
|
||||
|
||||
# Guarded so this file can be `source`d (e.g. a future synthetic test
|
||||
# harness) without executing main() as a side effect -- same convention as
|
||||
# xce-test-resources's own bottom-of-file guard.
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
|
|
@ -57,7 +57,7 @@
|
|||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 4. Validation: real multi-host end-to-end proof
|
||||
- [ ] 4.1 Build the multi-host end-to-end test harness
|
||||
- [x] 4.1 Build the multi-host end-to-end test harness
|
||||
- Add a new test skill that deploys one controller/scheduler node plus two named virtual workers, each sized so neither alone can satisfy the scenario's requested task count
|
||||
- Follow the existing soak-test convention for pre-flight zombie-process/port checks and a pass/fail verdict line
|
||||
- Observable: the new harness deploys its fixture cluster and tears it down cleanly, with no scenario wired in yet
|
||||
|
|
@ -89,5 +89,6 @@
|
|||
- _Requirements: 5.2_
|
||||
|
||||
## Implementation Notes
|
||||
- Task 4.1's `xce-test-multihost` harness reuses a fixed, wipe-and-recreate deploy directory (`xce-test-ha`'s convention) instead of design.md's stated "fresh, uniquely-named cluster instance; no shared state across runs" bullet (`xce-test-resources`'s convention). Reason: task 4.1's own text requires "the existing soak-test convention for pre-flight zombie-process/port checks," and a meaningful pre-flight check needs a stable, known directory/port to check against — `xce-test-resources`'s fresh-mktemp-per-run model has no such check at all (confirmed: it has none) because one would be moot there. The "no shared state" intent is preserved functionally: `deploy()` unconditionally wipes and recreates the directory on every invocation, verified clean (zero cross-run residue) across two consecutive live runs. No production code or requirement scope changed.
|
||||
- Task 3.2's driver/simulation-parity test was split into a second new file, `src/scheduler/testsuite/test_maxtasks_wiring_sim_driver_parity.c` (registered separately in `meson.build`), rather than living inside `test_maxtasks_wiring_ingest.c` as originally scoped. Reason: proving this specific claim requires white-box `#include`-ing `sim_loop.c`/`core_events.c` to reach their real `static` translators (the only way to avoid a mirrored test that can't catch a real translator regression — confirmed via an independent mutation-test reproduction during review), but `test_maxtasks_wiring_ingest.c` already links the full real `scheduler_srcs` object list for its own task-3.1 test, and the two techniques collide at link time (duplicate symbols) in one translation unit. No production code or requirement scope changed; this is a file-boundary split only, driven by a real toolchain constraint discovered during implementation.
|
||||
- Task 2.1 surfaced an out-of-boundary side effect: `src/scheduler/testsuite/test_repl_breakpoints_inspection.c:735`'s `test_info_job_reports_state_queue_priority` hardcodes an incidental `"Tasks: min = 0, max = 0"` byproduct of its fixture's `parse_select()` failing on an unregistered `"type"` resource (which previously left the primary's task-count fields at memset-zero, before `apply_job_task_count()` existed). Now correctly overwritten to `(1,1)` per Requirement 1.2. Before task 5.1 is considered complete, change that assertion's expected substring to `"Tasks: min = 1, max = 1"`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue