New xce-test-multihost skill deploys one controller/scheduler node
plus two named xcecd_virtual workers, following xce-test-resources's
single-node deploy pattern and xce-test-ha's pre-flight zombie-process/
port-check convention (chosen over a fresh-mktemp-per-run model
specifically because a meaningful pre-flight check needs a stable
directory -- see tasks.md's Implementation Notes). Confirms all four
roles join via their own log lines and tears down cleanly with a
HARNESS_READY sentinel. No job scenario wired in yet -- that's tasks
4.2/4.3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
test_parjob_e2e_multislot.c's standing "bug-1770" assertion now
expects the correct, real primary->minTasks/maxTasks (1/JOB_MAXTASKS)
instead of the buggy always-1 value it used to document. The
PATCH_MAXTASKS_FOR_BUG_1770 block that manually overwrote the
already-ingested resreq's task-count fields is removed entirely --
the full multi-host allocation scenario (2-host commit, per-host
slots) now runs end to end through unmodified real ingest. Header
comment updated to record that bug-1770 is fixed by this feature
rather than describing a live gap.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Extends test_maxtasks_wiring_ingest.c with two tests: (1) a resource-
request update never changes a job's already-dispatched task count,
because core_events.c's live JOB_UPDATE handler never rebuilds
job->schedobj (Requirement 1.4); (2) two otherwise-identical jobs
differing only in requested task count don't collide in the
unsatisfiable-resource-request cache on the fresh-ingest path
(Requirement 5.4).
Adds a new file, test_maxtasks_wiring_sim_driver_parity.c, proving the
simulation driver's real static translator chain
(sim_process_replay_event_message()/replay_event_to_core_event())
applies this feature's wiring identically to the real driver -- a
genuine white-box test of the real sim_loop.c code (following
test_sim_event_loop.c's established #include technique), not a mirror
of its logic. This was split into its own file because
test_maxtasks_wiring_ingest.c already links the full scheduler_srcs
object list for its own task-3.1 test, which collides at link time
with white-box #include-ing core_events.c/sim_loop.c in one
translation unit; see tasks.md's Implementation Notes.
No production code changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds test_maxtasks_wiring_ingest.c proving: (1) fresh ingest's
dispatched primary resreq reflects a real multi-task job's requested
count via the real core_build_job()/JOBS_PUSH path; (2) fresh ingest
and snapshot-restore ingest derive the identical task count for an
equivalent job, asserted both against expected constants and via
direct cross-comparison (Requirement 5.6); (3) a single-task job
reaches identical real host-selection/allocation behavior through the
live allocator dispatch chain, proving the feature is a true no-op
for maxTasks==1 (Requirements 3.1/3.2). No production code changed --
tasks 2.1/2.2 already wired the fix; this adds cross-call-site and
behavioral proof neither of those isolated tests attempted.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds white-box tests for job_start_remote_task()'s launch-type gate
(MPI types defer to their own remote-launch mechanism; default type
fans out via xcecli_runtask() across every allocated host, verified
via a #define-before-#include mock capturing taskrun/taskhost
contents) and create_mpi_cmdline()'s host-list/command construction
for openmpi and mpich. Zero production code changes -- this closes a
previously zero-coverage gap in existing, unmodified worker code
(Requirement 4.4), independent of this feature's resreq wiring work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
snapshot_load.c's job-restore path now calls apply_job_task_count()
with the job's already-restored mintasks/maxtasks, right after its
resreq parse succeeds and before snapshot_build_schedobj() reads the
resreq's task count for its fingerprint. This is the simulation/replay
driver's only live call site (xce_sched_core_load_snapshot(), reached
via --sim), independent of and parallel to task 2.1's fresh-ingest
wiring in core_events.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
core_build_job() now calls apply_job_task_count() right after its
resreq parse succeeds and before core_build_schedobj() reads the
resreq's task count for resreq_fp — so a freshly-submitted multi-task
job's dispatched primary resreq reflects its real requested min/max
task count instead of the previous always-one-task hardcode.
Tracked follow-up (not blocking, see tasks.md Implementation Notes):
test_repl_breakpoints_inspection.c:735 hardcodes an incidental
"Tasks: min = 0, max = 0" byproduct of a fixture's parse_select()
failure; now correctly reports (1,1) per Requirement 1.2. Fix folded
into task 5.1's full-suite regression pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds apply_job_task_count() to overwrite a standalone primary resreq's
minTasks/maxTasks with a job's real requested task count, no-op for
COMPOUND/ALTER nested primaries, with a defensive min>max clamp.
Closes the gap that lets scheduler-allocation's multi-host allocator
ever be reached (bug-1770). Wiring at live ingest call sites follows
in subsequent tasks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Validation-only: both sequential and chaotic HA soaks pass cleanly on
the final commit (zero job loss/divergence/stall), independently
re-run and confirmed by the reviewer. Worker-rejoin-resync coverage in
the soak is live evidence task 5.2's cache_clear_cb fix holds under
real primary failover. Standing caveat: bug-1770 means this soak (like
the rest of the feature) can only exercise seqjob's rewrap and the
cache dispatch/clear wiring -- parjob_allocator has no live production
path until that separate gap is fixed (Req 9.4).
This completes all 17 sub-tasks of the scheduler-allocation feature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Audit-only: confirmed all 4 pre-existing scheduler test files touched by
this feature (meson.build, test_core_events.c, test_job_update.c,
test_resource_accounting.c) contain only pure additions or link-scaffold
replacement, zero altered assertions; confirmed every existing
COMPOUND/ALTER-dispatch test is byte-identical to its pre-feature state
despite task 5.1's shared sched_make_primary_alloc_() change. Full suite:
74/75 OK, sole failure the pre-existing, unrelated bug-1758 (Req 9.2, 9.3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Drive a real multi-slot parallel job through the actual production
dispatch path -- real sched_init(), real queue/worker/job ingest via
xce_core_push_events(), a real scheduling session (exercising task 5.2's
cache-clear wiring), the real sched_allocator() thread, cache lookup,
parjob_allocator's real my_alloc(), host_iter, and commit -- and confirm
the resulting job allocation genuinely spans 2 hosts with correct
per-host slot counts (Req 4.1, 4.2, 4.3).
Discovered and logged bug-1770: parse_resreq_primary() unconditionally
hardcodes primary->minTasks/maxTasks to 1 regardless of what a job
actually requests, and nothing in the real ingest pipeline ever copies
the job's own submitted task count (correctly threaded from the
apiserver's num_processors/maxnum_processors all the way to
xce_sched_job_t.mintasks/maxtasks) onto the nested xce_primary_resreq_t
fields the allocator dispatch actually reads. This predates the feature
by ~7 months and is out of this spec's scope to fix, but it means
parjob_allocator has zero live production path until a dedicated
follow-up lands. The test compensates with one clearly-labeled patch of
the real, already-ingested resreq's task-count fields immediately after
real parsing; everything else in the pipeline is unmodified production
code.
Also logged bug-1771 (xce_sched_fcfs's fcfs_del_job() dereferences
job->queue->name with no NULL guard) discovered while building this
task's fixture; worked around by registering the job's queue before
push, mirroring the apiserver's own pre-submission queue validation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Clear the cache inside core_begin_session_internal() so a job cached as
unsatisfiable in one session is re-evaluated fresh in the next (Req 7.2).
An initial weak-symbol cast-based approach was rejected: it corrupted
xce_sched_sim_driver_t.alt_events_file, since core_begin_session_internal()
is shared between driver.c (real xce_scheduler_t) and sim_loop.c (a
structurally different, deliberately isolated sim driver struct with no
cache field at the same byte offset) -- the strong cache_clear() symbol
won unconditionally for both, invisible to ASAN and untested by any
test_sim_*.c file.
Fixed via a driver-registered callback: xce_sched_core_t gains a
cache_clear_cb hook (NULL by default, per xce_sched_core_init()'s
full-struct memset), set only in main.c's driver-mode branch --
mirroring the existing on_decision/xce_core_set_decision_cb() pattern.
--sim mode returns before that registration is ever reached, so its hook
stays NULL and core_begin_session_internal()'s guarded call is a no-op.
The real xce_scheduler_t-casting adapter lives in alloc_cache.c rather
than driver.c, since ~11 tests white-box-include driver.c without
linking alloc_cache.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Wrap sched_make_primary_alloc_() -- the single choke point all three
primary-resreq shapes (standalone, compound-nested, alter-nested via
compound delegation) already funnel through -- with a cache lookup
before allocation and a cache insert on exhaustion. On a hit, pend the
job immediately via the same make_pendreason()/add_job_preason() path
every exhaustion case already uses, without invoking either allocator
(Req 7.3, 7.4, 7.5).
A nested primary has no real per-primary schedobj, so
sched_primary_cache_obj_() synthesizes one on the stack per dispatch for
cache-key purposes; verified memory-safe since alloc_cache.c's
build_key_() copies the fingerprint/identity by value synchronously
before the stack frame unwinds.
Log bug-1768 (compound/alter-nested primaries never get real slot/rusage
availability checks -- seqjob/parjob's prepare_allocation hooks no-op
whenever ctx->schedobj->tag != XCE_RESREQ_PRIMARY) and bug-1769
(sched_make_alter_allocation() type-confuses xce_compound_schedobj_t*
as xce_sched_obj_t*, a ~112-byte heap over-read, currently untested).
Both are pre-existing and out of this feature's scope; discovered and
confirmed while building this task's dispatch tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Task 4.3's multi-host merge itself was already implemented and verified
by task 4.2. The one unproven clause -- job-level shared-resource
allocation computed once regardless of host count -- is now covered by
test_shared_resource_debited_once_regardless_of_host_count, using a
PER_JOB (not PER_TASK) global resource so a per-bucket-repeated
computation can't coincidentally sum to the correct total. Mutation
testing (a deliberately injected duplicate per-host shared debit)
confirms the test fails exactly as expected; production code required
no change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Replace parjob's return-0 stub with a real multi-host allocator: for each
candidate plan, walk buckets in the given (descending) order, claiming a
distinct host per bucket via a fresh host_iter excluding hosts already
claimed elsewhere in the same plan attempt. On a bucket failure, reverse
every earlier successful claim's provisional debit via
sched_ctx_uncommit_host() before advancing to the next plan (Req 4.2, 4.4,
4.5, 4.6, 4.7).
Fix parjob_prepare_allocation() to compute each bucket's want from that
bucket's own slot count (ctx->nslots) instead of a fixed value, and reset
ctx->alloc_req.host at the top of each bucket's scan so host_iter's
pre-filter can't inherit a stale want from a differently-sized prior
bucket (safe: sched_try_alloc_on_host_() always repopulates it from the
real per-bucket want before its own authoritative check runs).
Add a maxTasks<=1 guard matching Req 4 AC1's precondition, needed because
a snapshot-restored job with unpopulated maxTasks==0 (pre-existing,
tracked separately as bug-1767) would otherwise reach parjob's now-real
allocator instead of being silently ignored as before.
On full success, merges every claimed bucket into one xce_job_alloc_t via
make_job_alloc()'s existing runtime-sized host array, satisfying task
4.3's multi-host commit scope as well.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Add the real xce_alloc_plan_iter_t for parjob: for host-count k=1..maxTasks
(ascending, most-preferred-first), generate one balanced partition of
maxTasks into k buckets (floor/ceil split, remainder to the first rem
buckets), which is descending by construction. Pure function of maxTasks
alone -- never dereferences sched -- with a real pthread_testcancel()
cancellation point and stable post-exhaustion NULL (Req 2.1-2.6, 4.1).
my_alloc()/parjob_prepare_allocation() remain untouched stubs; wiring the
iterator into real allocation is task 4.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Add a dedicated equivalence suite (test_seqjob_rewrap_equivalence.c)
covering multi-host fixtures the trivial single-host test doesn't reach:
select-filter-only rejection, resource-only rejection, a combined
select-then-resource-then-success ordering, and per-host debit
correctness. The iteration-order test is grounded in the deleted
pre-rewrap sched_collect_workers() code path (verified via git show
against the parent commit) and the unmodified xlist.c primitive, not
merely in host_iter.c's own traversal, so it proves equivalence to the
old behavior rather than internal self-consistency.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Restructure seqjob's my_alloc() as a trivial single-bucket case over the
shared xce_alloc_plan_iter_t/xce_host_iter_t contracts, replacing its own
inline worker-collection loop. Guard seqjob_prepare_allocation() to
maxTasks == 1, closing the fan-out gap for the parallel allocator's later
fix (Req 9.1, 4.6).
Export sched_try_alloc_on_host_() from resource.c (drop static/va_list,
plain (worker, sched, ctx) signature) so seqjob calls the real per-host
try-alloc leaf instead of a private duplicate; delete the now-dead
sched_try_alloc_on_hosts() variadic wrapper. Also wires host_iter.c into
the production scheduler_srcs list (task 2.1 only added it to testsuite).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds alloc_cache.c: xce_cache_key_t, cache_key_ops (the corrected
three-way, 0-means-equal .cmp -- an inverted-boolean version was a
real bug found during design review), and cache_lookup/insert/clear
against a real kvtab_t (constructed matching xce_select.c's
sel_node_table_init() precedent). Cache values are a boxed reason-
category int, not a bare hit/miss sentinel, satisfying Req 7 AC5's
requirement that a hit reproduce the original pend-reason category.
unsatisfiable_cache lives on xce_scheduler_t per Req 7 AC1.
Task 2.2 (scheduler-allocation spec). Requirements 7.1, 7.3, 7.4, 7.5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds host_iter.c: the one shared xce_host_iter_t implementation,
lazily filtering sched->core.workers by select expression
(sched_meet_host_selection, unchanged) and resource availability
(sched_check_host_satisfy, unchanged), with a real cancellation
point per candidate visited. xce_alloc_plan_t gains a ctx field
(xce_sched_ctx_t*) so the iterator can reach the scheduler/select
expression/staged resource-want -- additive to task 1.1's approved
interface, NULL-guarded at init(). Test proves the core design
property this spec exists to establish: a host too small for one
search remains a valid candidate for a later, smaller search via a
fresh iterator instance over the same worker set.
Task 2.1 (scheduler-allocation spec). Requirements 3.1-3.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds sched_ctx_uncommit_host() (resource.c), the structural inverse
of sched_try_alloc_on_host_()'s existing XVEC_ADD_FLOAT commit call,
scoped to ctx->uncomitted only per requirements.md's Rollback scope
discussion. Test exercises the real sched_ctx_get_uncomit_host()
(schedobj.c) via white-box #include rather than a private
reimplementation, matching test_picker_config_fields.c's established
precedent.
Task 1.4 (scheduler-allocation spec). Requirement 4.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds xce_alloc_mode_t (XCE_ALLOC_NORMAL=0, XCE_ALLOC_PREEMPT, with
commented-out RESERVE/BACKFILL placeholders) and alloc_mode on
xce_sched_ctx_t, zero-defaulted by sched_ctx_init()'s existing
memset. Purely additive: nothing branches on the field yet -- this
spec only declares the extension point scheduler-preemption needs.
Task 1.3 (scheduler-allocation spec). Requirement 9.5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds rusage_hash to xce_primary_resreq_t (computed once by
parse_resreq_primary() via a new hash_rusages() helper, hashing
rusage content rather than raw struct bytes since resname is a
non-interned strdup'd pointer), the xce_resreq_fingerprint_t value
type, and resreq_fp on xce_primary_schedobj_t, populated once at real
job-ingest time in core_build_primary_schedobj().
Task 1.2 (scheduler-allocation spec). Requirement 7.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Defines xce_alloc_plan_t (partition of a job's task count into
per-host slot buckets), xce_alloc_plan_iter_t, and xce_host_iter_t
(init/next/fini vtables, stack-allocate-then-init-in-place convention
matching sched_ctx_init) as reusable abstractions any allocator can
adopt. Proven usable by a trivial no-op reference implementation and
a fresh-per-bucket host_iter composition test, no allocator wired in
yet.
Task 1.1 (scheduler-allocation spec). Requirements 2.1, 3.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Found during task 1.1 implementation: the Data Structure Changes
Summary declared next() returning xce_worker_t* (the wire/config
type), but the Composition Algorithm pseudocode in the same document
already used xce_sched_worker_t* (the scheduler-internal type with
resvals/resInUse/host_eval_ctx) for the identical call -- these are
genuinely different types in this codebase, confirmed via
make_sched_worker()'s xce_worker_t* -> xce_sched_worker_t* conversion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Extensive requirements review found and fixed a real correctness bug
(host_iter shared across plan buckets instead of re-scanned fresh per
bucket, which would silently drop solvable multi-host allocations),
a self-contradiction around parjob_prepare_allocation's scope, a
plan/host-iterator init calling-convention mismatch, and an inverted
kvtab .cmp polarity that would have made the unsatisfiable-resreq
cache permanently non-functional. brief.md brought back into sync
with requirements.md's corrected architecture.
design.md and tasks.md generated and validated (kiro-validate-design
GO after two fixes; independent task-graph sanity review PASS).
Two pre-existing, out-of-boundary gaps found along the way logged as
bug-1766/bug-1767.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Checkbox update for commit 6fabdba9 (deleted the file, but its
tasks.md diff wasn't included in that commit due to a git-add
pathspec issue on the already-git-rm'd path). Marks 4/4.1/4.2 done
and records the two out-of-boundary doc mentions found during
task 4.2's post-deletion review. All 12 tasks / 4 major groups are
now [x] -- the eliminate-frontend-makefile feature is complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Pure verification, no code changes. Confirmed all four checks against
one fresh build: /xce-build apiserver (migrated Go-test invocation,
zero make calls), meson test --suite apiserver (new backend CGO test
genuinely registered and passing), task 3.5's regression parity
(cited, same-session), and a fresh live-reference sweep (190 hits,
zero unaccounted -- corrects task 3.6's commit message, which said
188; verified byte-identical at both commits, a message-only miscount
not drift). Also independently reconfirmed the two failures blocking
a literal /xce-build all are exactly bug-1723/bug-1758 and nothing
else. Reviewer independently reproduced every check. Clear to proceed
to Makefile deletion (task 4.2).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Pure audit, no code changes. Full repo-wide grep (188 hits) fully
classified: every hit is either already fixed by tasks 3.1-3.4 or
falls into one of design.md's named exclusion classes (.kiro/specs/**
historical docs, vendor/**, dated historical records, roadmap.md's
own backlog entry, OpenWolf's logs, run.sh:12's retained placeholder).
Zero unaccounted hits. Independently re-run and confirmed exact match
by the reviewer. Also flips the "1." and "3." container-header
checkboxes now that all their children are done.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Pure verification, no code changes. Deployed a live single-node
cluster and ran tests/regression/run.sh against a Meson-built xcecli
binary, then again via the still-present Makefile path -- both
produced identical 28 passed/1 failed/1 skipped results. Reviewer
independently reproduced this three times and traced the one
pre-existing failure (jobs submit's job-ID format regex) through
source and git history to a 2026-05-07 commit, four months before
this spec existed -- logged as bug-1765 (not fixed, out of scope).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Comment-only edits in 4 files: test_ha_bootstrap.c/test_ha_ctrlhello.c
rename their "make test-integration" pointers to reference the
regression suite (tests/regression/run.sh) directly; apiserver/xcecli
cmd/main.go drop the Makefile clause from their ldflags-injection
comments, keeping only src/tools/go_build. Zero production/test
logic changed -- verified via full controller unit-test suite rerun
(1261 tests, 0 failures) and xcecli build.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Removes every live current-fact reference to src/frontend/Makefile
targets or Makefile-specific fixed paths across 11 docs (steering,
frontend/CLAUDE.md, root CLAUDE.md, deployment docs, and two skill
docs), replacing with the Meson-based equivalents established by
tasks 1-3.2: /xce-build apiserver, meson test --suite apiserver,
direct tests/regression/run.sh invocation.
Round-1 review rejected the first pass: xce-deploy-ha/SKILL.md had
over-corrected by deleting all mention of deploy-ha.sh's still-real
/tmp/apiserver legacy fallback (that script's discovery logic is
unchanged, unlike run-resource-verify.sh's, which genuinely lost its
fallback in task 3.1 -- xce-test-resources/SKILL.md's identical-
looking edit was correct for that reason). Fixed by restoring an
accurate, Makefile-free description of the real fallback at all four
touched sites. Re-reviewed and approved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Rewords deploy-ha.sh's block comment and two fallback echo lines, and
run.sh's XCE-not-found error, to stop naming the now-deleted
Makefile's build-server/build-cli targets, pointing at /xce-build
apiserver instead. Discovery/control-flow logic in both files is
unchanged (verified line-by-line: no if/elif/else lines touched);
run.sh's XCE default placeholder is untouched by design.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
APISERVER_BIN's default changes from the fixed /tmp/apiserver path
to $BUILD_DIR/frontend/apiserver/xce_apiserver, matching the sibling
BINARY/SCHED_BIN/XT convention exactly (no dual-path fallback, per
research.md's explicit Design Decision). Failure message now names
/xce-build apiserver instead of the Makefile target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Replaces both `cd frontend && make test` call sites (apiserver and
all cases) with direct `go test` invocations covering the identical
package set/flags, mirroring the xcecli case's already-Makefile-free
style. Verified: apiserver case end-to-end, matching the captured
pre-change baseline exactly; all case verified by proxy (byte-
identical replacement line, apiserver instance fully verified) since
a full /xce-build all run is currently blocked earlier in the
pipeline by two pre-existing, unrelated C-test bugs (bug-1723,
bug-1758) -- noted in Implementation Notes for task 4's awareness.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
One-time verification exercise, no permanent code change. Broke
apiserver_cgo_deps (c_capnproto_dep, not librtti_dep as design.md
suggests -- see Implementation Notes) and confirmed the new backend
test() genuinely fails with real capn_* undefined-reference link
errors, then confirmed it passes again fully restored. Independently
reproduced by the task reviewer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds a new `test('apiserver backend', ...)` in apiserver/meson.build
reusing the existing apiserver_cgo_deps-derived CGO flags (resolving
their @CURRENT_SOURCE_DIR@/@BUILD_ROOT@ placeholder tokens via
Meson's own .replace(), since this target invokes go_compiler
directly rather than through tools/go_build). Adds timeout: 300
(matching the existing controller test's convention) since a
cold-cache CGO+ASAN build exceeds Meson's 30s default.
Verified cold and warm cache; confirmed CGO_ENABLED=1 selects the
real backend/ CGO path, not the //go:build !cgo stub.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Requirements: 6 requirement areas covering single build path, unit/
backend/integration test migration, binary discovery, and doc
consistency. Design: 6 rounds of independent adversarial review to
convergence (2 real technical bugs caught -- unresolved Meson
placeholder tokens in CGO flags, invalid unquoted dict-literal keys --
plus a long tail of stale Makefile references across docs/scripts/
source comments, closed by a scope-rule restructure rather than
continued line-by-line patching). Tasks: 4 major / 12 sub-tasks,
task-graph sanity review passed after one repair round.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Expose the AppImage as a flake output via the nix-appimage input's
mkAppImage (passthru on the xce package) instead of requiring an
external 'nix bundle --bundler github:ralismark/nix-appimage' call.
Update README and build-deployment steering to match.
All three were 100% done but read as "in progress" from a naive
checkbox count -- same header-never-flipped drift as
5c82ad0b/b14e53f5/1abab949/8557b869:
- scheduler-sim-repl-verbosity: all 9 headers' children done
(last: task 10.2 final regression, commit e87c0b3f, then a
doc-gap follow-up a51790dc, 2026-09-02T21:55:50Z)
- scheduler-sim-reconcile-terminal-guard: both headers' children
done (last: task 2.2 REPL-level verification, commit 9dcffce8,
2026-09-03T08:31:37Z)
- scheduler-sim-step-verbosity: both headers' children done
(last: task 2.2 full regression, commit 56bd3103,
2026-09-03T05:15:45Z)
Checked all remaining headers, bumped all three to
implementation-complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
All 10 tasks were done (last: task 3.3 automation-regression check,
commit 8fc86ef4, 2026-08-26) but the 3 section-header checkboxes were
never flipped and spec.json.phase stayed at tasks-generated. Same
drift class as 5c82ad0b/b14e53f5/1abab949. Checked the 3 headers,
bumped phase to implementation-complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
- scheduler-sim-eager-reconciliation: all 9 tasks were done (last:
task 3.3 full regression, commit 446437bf, 2026-08-27) but the 3
section-header checkboxes were never flipped and phase stayed at
tasks-generated. Same drift class as 5c82ad0b/b14e53f5. Checked the
3 headers, bumped phase to implementation-complete.
- scheduler-allocation, scheduler-preemption: blocked_on named
scheduler-core and scheduler-dispatch-commit, both of which reached
implementation-complete weeks ago (2026-08-19, 2026-08-24). Annotated
as resolved rather than removed, so the historical block reason
stays visible. Requirements approval is still outstanding on both --
not touched, since that gate needs human review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
All 28/28 tasks were completed (last: task 4.4, commit b6a2f5ec) but
spec.json.phase was never advanced past tasks-generated. Flipped to
implementation-complete, matching the same drift class fixed for
5 other specs in 5c82ad0b.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Successor to scheduler-primary-lifecycle: a new per-node xce-node-manager
component (one per host, role-assigned: manager/worker/apiserver/
reverse-proxy) owns starting/stopping XCE components for its node's
role -- including starting/stopping the scheduler on a manager node in
reaction to that node's controller reporting itself primary vs.
standby over a local heartbeat (Unix domain socket, client logic
shared via libxce) -- rather than the controller directly
spawning/killing its own scheduler process. Carries forward
scheduler-primary-lifecycle's underlying requirements (bug-1760's
reactive-rejoin deadlock, wedge detection, graceful shutdown,
controller-resilience invariants).
Design review (kiro-validate-design) fixed 3 issues before GO: a
uniform-single-OS-user assumption that was factually wrong given
xce-deploy-ha's real worker daemon runs as root (resolved: root only
on worker-role nodes); Requirement 4.3's wedge-detection mechanism
verified directly against the actual scheduler code
(sched_job_picker's cbuf_get() does an unbounded wait when idle,
confirming the naive design would false-positive -- fixed by swapping
to the existing cbuf_timedget() bounded-deadline variant); and
PortCheckMonitor's TCP connect made explicitly non-blocking to match
HeartbeatListener's own discipline. Tasks not yet generated.
Checkpointing uncommitted spec work found in this shared worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Open-ended simulation-mode REPL diagnostics spec: first capability is
limit inspection (info limit <name|all>) showing configured setting
(scope, value/percent) and live accounted usage per instance. A second
capability (decision-reasoning trace for pend/dispatch outcomes) is
noted as a future seed, not yet in requirements.md.
Requirements generated but not yet approved; design/tasks not started.
Checkpointing uncommitted spec work found in this shared worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Requirements and design approved (design fixed the fork()-vs-posix_spawn
mutex-inheritance-deadlock risk product.md's own "no fork() for
snapshots" decision already rejected); tasks generated but never
approved. Superseded by xce-node-manager after the user decided the
controller should not directly own spawning/killing its own scheduler
process -- a new per-node xce-node-manager component takes over that
responsibility instead. Kept in place as a reference (notably the
fork()-vs-posix_spawn analysis); this spec's tasks.md must not be
implemented (see spec.json's superseded_note).
Checkpointing uncommitted spec work found in this shared worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch,
mirroring bug-1764's actual live-observed shape: a worker with one job
driven to a terminal state and two jobs genuinely still RUN, all real
allocations to the SAME worker. After reconciling a stray DISPATCH
entry for the terminal job's id, info worker excludes it while still
listing the two running jobs, and info summary's running count agrees.
Round 1 review rejected an initial version for using a mismatched host
string on the stray entry ("workerarr-stray" instead of "workerarr"),
which made the info worker/info summary assertions pass vacuously
regardless of whether the fix existed, since repl_info_worker_one()
filters assigned jobs by an exact host-string match. Round 2 fixed the
host string and confirmed via an independently-reproduced mutation
test (fix reverted, the test's own accounting-level pre-check
bypassed) that the REPL-level assertions now genuinely, independently
detect the bug.
design.md's Testing Strategy is corrected: it originally said no new
test was needed, reasoning the existing display-path suites already
covered this; tasks.md's own task 2.2 correctly required a
purpose-built test for this exact scenario instead.
Full scheduler test suite: 63/64 OK, sole failure the pre-existing,
unrelated bug-1758. This is the final task of the
scheduler-sim-reconcile-terminal-guard spec; all implementation tasks
(1.1-2.2) are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Verified (full regression build/test pass, an explicit by-name check
of all 18 tests in test_sim_reconcile_unaccounted.c and the JOBS_PUSH
resync-merge tests in test_core_events.c, and a diff-scope check
against the pre-feature baseline) that: test_core_events.c is
byte-identical to baseline, test_sim_reconcile_unaccounted.c's diff is
a pure addition with no existing assertion touched, and the fix's
total footprint is exactly the expected 8 files. The sole test
failure in the full suite (test_job_rusage_reconciles_mem_reservation_against_usage)
is confirmed pre-existing and unrelated (bug-1758). Read-only
validation task; no production or test code changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
sim_reconcile_unaccounted_jobs()'s per-job trigger check (sim_loop.c)
now also excludes JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state),
alongside the existing job->accounts != NULL skip. A terminal job is
never searched for a matching recorded DISPATCH entry, so a leftover
or duplicate DISPATCH decision from a later replay round can no
longer resurrect allocs/accounts on a job that has already completed
- the exact mechanism behind info worker's assigned-jobs listing
showing a finished job as still consuming a worker's resources.
The exclusion mirrors job_state_regresses_terminal()'s existing
"terminal is permanent" invariant (core_events.c) rather than
inventing a new rule. The function's doc comment gains a paragraph
clarifying the terminal exclusion is complementary to, not a reversal
of, the existing PEND/RUN state-independence rationale that motivated
this trigger's original widening.
Both RED-phase tests (task 1.1) pass; the pre-existing PEND
eager-capture regression test and core_events.c's JOBS_PUSH
resync-merge terminal-state tests are unaffected. The analogous
resync-merge branch was investigated and confirmed already correct.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
