Validation-only: both sequential and chaotic HA soaks pass cleanly on
the final commit (zero job loss/divergence/stall), independently
re-run and confirmed by the reviewer. Worker-rejoin-resync coverage in
the soak is live evidence task 5.2's cache_clear_cb fix holds under
real primary failover. Standing caveat: bug-1770 means this soak (like
the rest of the feature) can only exercise seqjob's rewrap and the
cache dispatch/clear wiring -- parjob_allocator has no live production
path until that separate gap is fixed (Req 9.4).
This completes all 17 sub-tasks of the scheduler-allocation feature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Audit-only: confirmed all 4 pre-existing scheduler test files touched by
this feature (meson.build, test_core_events.c, test_job_update.c,
test_resource_accounting.c) contain only pure additions or link-scaffold
replacement, zero altered assertions; confirmed every existing
COMPOUND/ALTER-dispatch test is byte-identical to its pre-feature state
despite task 5.1's shared sched_make_primary_alloc_() change. Full suite:
74/75 OK, sole failure the pre-existing, unrelated bug-1758 (Req 9.2, 9.3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Drive a real multi-slot parallel job through the actual production
dispatch path -- real sched_init(), real queue/worker/job ingest via
xce_core_push_events(), a real scheduling session (exercising task 5.2's
cache-clear wiring), the real sched_allocator() thread, cache lookup,
parjob_allocator's real my_alloc(), host_iter, and commit -- and confirm
the resulting job allocation genuinely spans 2 hosts with correct
per-host slot counts (Req 4.1, 4.2, 4.3).
Discovered and logged bug-1770: parse_resreq_primary() unconditionally
hardcodes primary->minTasks/maxTasks to 1 regardless of what a job
actually requests, and nothing in the real ingest pipeline ever copies
the job's own submitted task count (correctly threaded from the
apiserver's num_processors/maxnum_processors all the way to
xce_sched_job_t.mintasks/maxtasks) onto the nested xce_primary_resreq_t
fields the allocator dispatch actually reads. This predates the feature
by ~7 months and is out of this spec's scope to fix, but it means
parjob_allocator has zero live production path until a dedicated
follow-up lands. The test compensates with one clearly-labeled patch of
the real, already-ingested resreq's task-count fields immediately after
real parsing; everything else in the pipeline is unmodified production
code.
Also logged bug-1771 (xce_sched_fcfs's fcfs_del_job() dereferences
job->queue->name with no NULL guard) discovered while building this
task's fixture; worked around by registering the job's queue before
push, mirroring the apiserver's own pre-submission queue validation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Clear the cache inside core_begin_session_internal() so a job cached as
unsatisfiable in one session is re-evaluated fresh in the next (Req 7.2).
An initial weak-symbol cast-based approach was rejected: it corrupted
xce_sched_sim_driver_t.alt_events_file, since core_begin_session_internal()
is shared between driver.c (real xce_scheduler_t) and sim_loop.c (a
structurally different, deliberately isolated sim driver struct with no
cache field at the same byte offset) -- the strong cache_clear() symbol
won unconditionally for both, invisible to ASAN and untested by any
test_sim_*.c file.
Fixed via a driver-registered callback: xce_sched_core_t gains a
cache_clear_cb hook (NULL by default, per xce_sched_core_init()'s
full-struct memset), set only in main.c's driver-mode branch --
mirroring the existing on_decision/xce_core_set_decision_cb() pattern.
--sim mode returns before that registration is ever reached, so its hook
stays NULL and core_begin_session_internal()'s guarded call is a no-op.
The real xce_scheduler_t-casting adapter lives in alloc_cache.c rather
than driver.c, since ~11 tests white-box-include driver.c without
linking alloc_cache.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Wrap sched_make_primary_alloc_() -- the single choke point all three
primary-resreq shapes (standalone, compound-nested, alter-nested via
compound delegation) already funnel through -- with a cache lookup
before allocation and a cache insert on exhaustion. On a hit, pend the
job immediately via the same make_pendreason()/add_job_preason() path
every exhaustion case already uses, without invoking either allocator
(Req 7.3, 7.4, 7.5).
A nested primary has no real per-primary schedobj, so
sched_primary_cache_obj_() synthesizes one on the stack per dispatch for
cache-key purposes; verified memory-safe since alloc_cache.c's
build_key_() copies the fingerprint/identity by value synchronously
before the stack frame unwinds.
Log bug-1768 (compound/alter-nested primaries never get real slot/rusage
availability checks -- seqjob/parjob's prepare_allocation hooks no-op
whenever ctx->schedobj->tag != XCE_RESREQ_PRIMARY) and bug-1769
(sched_make_alter_allocation() type-confuses xce_compound_schedobj_t*
as xce_sched_obj_t*, a ~112-byte heap over-read, currently untested).
Both are pre-existing and out of this feature's scope; discovered and
confirmed while building this task's dispatch tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Task 4.3's multi-host merge itself was already implemented and verified
by task 4.2. The one unproven clause -- job-level shared-resource
allocation computed once regardless of host count -- is now covered by
test_shared_resource_debited_once_regardless_of_host_count, using a
PER_JOB (not PER_TASK) global resource so a per-bucket-repeated
computation can't coincidentally sum to the correct total. Mutation
testing (a deliberately injected duplicate per-host shared debit)
confirms the test fails exactly as expected; production code required
no change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Replace parjob's return-0 stub with a real multi-host allocator: for each
candidate plan, walk buckets in the given (descending) order, claiming a
distinct host per bucket via a fresh host_iter excluding hosts already
claimed elsewhere in the same plan attempt. On a bucket failure, reverse
every earlier successful claim's provisional debit via
sched_ctx_uncommit_host() before advancing to the next plan (Req 4.2, 4.4,
4.5, 4.6, 4.7).
Fix parjob_prepare_allocation() to compute each bucket's want from that
bucket's own slot count (ctx->nslots) instead of a fixed value, and reset
ctx->alloc_req.host at the top of each bucket's scan so host_iter's
pre-filter can't inherit a stale want from a differently-sized prior
bucket (safe: sched_try_alloc_on_host_() always repopulates it from the
real per-bucket want before its own authoritative check runs).
Add a maxTasks<=1 guard matching Req 4 AC1's precondition, needed because
a snapshot-restored job with unpopulated maxTasks==0 (pre-existing,
tracked separately as bug-1767) would otherwise reach parjob's now-real
allocator instead of being silently ignored as before.
On full success, merges every claimed bucket into one xce_job_alloc_t via
make_job_alloc()'s existing runtime-sized host array, satisfying task
4.3's multi-host commit scope as well.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Add the real xce_alloc_plan_iter_t for parjob: for host-count k=1..maxTasks
(ascending, most-preferred-first), generate one balanced partition of
maxTasks into k buckets (floor/ceil split, remainder to the first rem
buckets), which is descending by construction. Pure function of maxTasks
alone -- never dereferences sched -- with a real pthread_testcancel()
cancellation point and stable post-exhaustion NULL (Req 2.1-2.6, 4.1).
my_alloc()/parjob_prepare_allocation() remain untouched stubs; wiring the
iterator into real allocation is task 4.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Add a dedicated equivalence suite (test_seqjob_rewrap_equivalence.c)
covering multi-host fixtures the trivial single-host test doesn't reach:
select-filter-only rejection, resource-only rejection, a combined
select-then-resource-then-success ordering, and per-host debit
correctness. The iteration-order test is grounded in the deleted
pre-rewrap sched_collect_workers() code path (verified via git show
against the parent commit) and the unmodified xlist.c primitive, not
merely in host_iter.c's own traversal, so it proves equivalence to the
old behavior rather than internal self-consistency.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Restructure seqjob's my_alloc() as a trivial single-bucket case over the
shared xce_alloc_plan_iter_t/xce_host_iter_t contracts, replacing its own
inline worker-collection loop. Guard seqjob_prepare_allocation() to
maxTasks == 1, closing the fan-out gap for the parallel allocator's later
fix (Req 9.1, 4.6).
Export sched_try_alloc_on_host_() from resource.c (drop static/va_list,
plain (worker, sched, ctx) signature) so seqjob calls the real per-host
try-alloc leaf instead of a private duplicate; delete the now-dead
sched_try_alloc_on_hosts() variadic wrapper. Also wires host_iter.c into
the production scheduler_srcs list (task 2.1 only added it to testsuite).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds alloc_cache.c: xce_cache_key_t, cache_key_ops (the corrected
three-way, 0-means-equal .cmp -- an inverted-boolean version was a
real bug found during design review), and cache_lookup/insert/clear
against a real kvtab_t (constructed matching xce_select.c's
sel_node_table_init() precedent). Cache values are a boxed reason-
category int, not a bare hit/miss sentinel, satisfying Req 7 AC5's
requirement that a hit reproduce the original pend-reason category.
unsatisfiable_cache lives on xce_scheduler_t per Req 7 AC1.
Task 2.2 (scheduler-allocation spec). Requirements 7.1, 7.3, 7.4, 7.5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds host_iter.c: the one shared xce_host_iter_t implementation,
lazily filtering sched->core.workers by select expression
(sched_meet_host_selection, unchanged) and resource availability
(sched_check_host_satisfy, unchanged), with a real cancellation
point per candidate visited. xce_alloc_plan_t gains a ctx field
(xce_sched_ctx_t*) so the iterator can reach the scheduler/select
expression/staged resource-want -- additive to task 1.1's approved
interface, NULL-guarded at init(). Test proves the core design
property this spec exists to establish: a host too small for one
search remains a valid candidate for a later, smaller search via a
fresh iterator instance over the same worker set.
Task 2.1 (scheduler-allocation spec). Requirements 3.1-3.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds sched_ctx_uncommit_host() (resource.c), the structural inverse
of sched_try_alloc_on_host_()'s existing XVEC_ADD_FLOAT commit call,
scoped to ctx->uncomitted only per requirements.md's Rollback scope
discussion. Test exercises the real sched_ctx_get_uncomit_host()
(schedobj.c) via white-box #include rather than a private
reimplementation, matching test_picker_config_fields.c's established
precedent.
Task 1.4 (scheduler-allocation spec). Requirement 4.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds xce_alloc_mode_t (XCE_ALLOC_NORMAL=0, XCE_ALLOC_PREEMPT, with
commented-out RESERVE/BACKFILL placeholders) and alloc_mode on
xce_sched_ctx_t, zero-defaulted by sched_ctx_init()'s existing
memset. Purely additive: nothing branches on the field yet -- this
spec only declares the extension point scheduler-preemption needs.
Task 1.3 (scheduler-allocation spec). Requirement 9.5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds rusage_hash to xce_primary_resreq_t (computed once by
parse_resreq_primary() via a new hash_rusages() helper, hashing
rusage content rather than raw struct bytes since resname is a
non-interned strdup'd pointer), the xce_resreq_fingerprint_t value
type, and resreq_fp on xce_primary_schedobj_t, populated once at real
job-ingest time in core_build_primary_schedobj().
Task 1.2 (scheduler-allocation spec). Requirement 7.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Defines xce_alloc_plan_t (partition of a job's task count into
per-host slot buckets), xce_alloc_plan_iter_t, and xce_host_iter_t
(init/next/fini vtables, stack-allocate-then-init-in-place convention
matching sched_ctx_init) as reusable abstractions any allocator can
adopt. Proven usable by a trivial no-op reference implementation and
a fresh-per-bucket host_iter composition test, no allocator wired in
yet.
Task 1.1 (scheduler-allocation spec). Requirements 2.1, 3.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Found during task 1.1 implementation: the Data Structure Changes
Summary declared next() returning xce_worker_t* (the wire/config
type), but the Composition Algorithm pseudocode in the same document
already used xce_sched_worker_t* (the scheduler-internal type with
resvals/resInUse/host_eval_ctx) for the identical call -- these are
genuinely different types in this codebase, confirmed via
make_sched_worker()'s xce_worker_t* -> xce_sched_worker_t* conversion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Extensive requirements review found and fixed a real correctness bug
(host_iter shared across plan buckets instead of re-scanned fresh per
bucket, which would silently drop solvable multi-host allocations),
a self-contradiction around parjob_prepare_allocation's scope, a
plan/host-iterator init calling-convention mismatch, and an inverted
kvtab .cmp polarity that would have made the unsatisfiable-resreq
cache permanently non-functional. brief.md brought back into sync
with requirements.md's corrected architecture.
design.md and tasks.md generated and validated (kiro-validate-design
GO after two fixes; independent task-graph sanity review PASS).
Two pre-existing, out-of-boundary gaps found along the way logged as
bug-1766/bug-1767.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Checkbox update for commit 6fabdba9 (deleted the file, but its
tasks.md diff wasn't included in that commit due to a git-add
pathspec issue on the already-git-rm'd path). Marks 4/4.1/4.2 done
and records the two out-of-boundary doc mentions found during
task 4.2's post-deletion review. All 12 tasks / 4 major groups are
now [x] -- the eliminate-frontend-makefile feature is complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Pure verification, no code changes. Confirmed all four checks against
one fresh build: /xce-build apiserver (migrated Go-test invocation,
zero make calls), meson test --suite apiserver (new backend CGO test
genuinely registered and passing), task 3.5's regression parity
(cited, same-session), and a fresh live-reference sweep (190 hits,
zero unaccounted -- corrects task 3.6's commit message, which said
188; verified byte-identical at both commits, a message-only miscount
not drift). Also independently reconfirmed the two failures blocking
a literal /xce-build all are exactly bug-1723/bug-1758 and nothing
else. Reviewer independently reproduced every check. Clear to proceed
to Makefile deletion (task 4.2).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Pure audit, no code changes. Full repo-wide grep (188 hits) fully
classified: every hit is either already fixed by tasks 3.1-3.4 or
falls into one of design.md's named exclusion classes (.kiro/specs/**
historical docs, vendor/**, dated historical records, roadmap.md's
own backlog entry, OpenWolf's logs, run.sh:12's retained placeholder).
Zero unaccounted hits. Independently re-run and confirmed exact match
by the reviewer. Also flips the "1." and "3." container-header
checkboxes now that all their children are done.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Pure verification, no code changes. Deployed a live single-node
cluster and ran tests/regression/run.sh against a Meson-built xcecli
binary, then again via the still-present Makefile path -- both
produced identical 28 passed/1 failed/1 skipped results. Reviewer
independently reproduced this three times and traced the one
pre-existing failure (jobs submit's job-ID format regex) through
source and git history to a 2026-05-07 commit, four months before
this spec existed -- logged as bug-1765 (not fixed, out of scope).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Comment-only edits in 4 files: test_ha_bootstrap.c/test_ha_ctrlhello.c
rename their "make test-integration" pointers to reference the
regression suite (tests/regression/run.sh) directly; apiserver/xcecli
cmd/main.go drop the Makefile clause from their ldflags-injection
comments, keeping only src/tools/go_build. Zero production/test
logic changed -- verified via full controller unit-test suite rerun
(1261 tests, 0 failures) and xcecli build.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Removes every live current-fact reference to src/frontend/Makefile
targets or Makefile-specific fixed paths across 11 docs (steering,
frontend/CLAUDE.md, root CLAUDE.md, deployment docs, and two skill
docs), replacing with the Meson-based equivalents established by
tasks 1-3.2: /xce-build apiserver, meson test --suite apiserver,
direct tests/regression/run.sh invocation.
Round-1 review rejected the first pass: xce-deploy-ha/SKILL.md had
over-corrected by deleting all mention of deploy-ha.sh's still-real
/tmp/apiserver legacy fallback (that script's discovery logic is
unchanged, unlike run-resource-verify.sh's, which genuinely lost its
fallback in task 3.1 -- xce-test-resources/SKILL.md's identical-
looking edit was correct for that reason). Fixed by restoring an
accurate, Makefile-free description of the real fallback at all four
touched sites. Re-reviewed and approved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Rewords deploy-ha.sh's block comment and two fallback echo lines, and
run.sh's XCE-not-found error, to stop naming the now-deleted
Makefile's build-server/build-cli targets, pointing at /xce-build
apiserver instead. Discovery/control-flow logic in both files is
unchanged (verified line-by-line: no if/elif/else lines touched);
run.sh's XCE default placeholder is untouched by design.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
APISERVER_BIN's default changes from the fixed /tmp/apiserver path
to $BUILD_DIR/frontend/apiserver/xce_apiserver, matching the sibling
BINARY/SCHED_BIN/XT convention exactly (no dual-path fallback, per
research.md's explicit Design Decision). Failure message now names
/xce-build apiserver instead of the Makefile target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Replaces both `cd frontend && make test` call sites (apiserver and
all cases) with direct `go test` invocations covering the identical
package set/flags, mirroring the xcecli case's already-Makefile-free
style. Verified: apiserver case end-to-end, matching the captured
pre-change baseline exactly; all case verified by proxy (byte-
identical replacement line, apiserver instance fully verified) since
a full /xce-build all run is currently blocked earlier in the
pipeline by two pre-existing, unrelated C-test bugs (bug-1723,
bug-1758) -- noted in Implementation Notes for task 4's awareness.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
One-time verification exercise, no permanent code change. Broke
apiserver_cgo_deps (c_capnproto_dep, not librtti_dep as design.md
suggests -- see Implementation Notes) and confirmed the new backend
test() genuinely fails with real capn_* undefined-reference link
errors, then confirmed it passes again fully restored. Independently
reproduced by the task reviewer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds a new `test('apiserver backend', ...)` in apiserver/meson.build
reusing the existing apiserver_cgo_deps-derived CGO flags (resolving
their @CURRENT_SOURCE_DIR@/@BUILD_ROOT@ placeholder tokens via
Meson's own .replace(), since this target invokes go_compiler
directly rather than through tools/go_build). Adds timeout: 300
(matching the existing controller test's convention) since a
cold-cache CGO+ASAN build exceeds Meson's 30s default.
Verified cold and warm cache; confirmed CGO_ENABLED=1 selects the
real backend/ CGO path, not the //go:build !cgo stub.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Requirements: 6 requirement areas covering single build path, unit/
backend/integration test migration, binary discovery, and doc
consistency. Design: 6 rounds of independent adversarial review to
convergence (2 real technical bugs caught -- unresolved Meson
placeholder tokens in CGO flags, invalid unquoted dict-literal keys --
plus a long tail of stale Makefile references across docs/scripts/
source comments, closed by a scope-rule restructure rather than
continued line-by-line patching). Tasks: 4 major / 12 sub-tasks,
task-graph sanity review passed after one repair round.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Expose the AppImage as a flake output via the nix-appimage input's
mkAppImage (passthru on the xce package) instead of requiring an
external 'nix bundle --bundler github:ralismark/nix-appimage' call.
Update README and build-deployment steering to match.
All three were 100% done but read as "in progress" from a naive
checkbox count -- same header-never-flipped drift as
5c82ad0b/b14e53f5/1abab949/8557b869:
- scheduler-sim-repl-verbosity: all 9 headers' children done
(last: task 10.2 final regression, commit e87c0b3f, then a
doc-gap follow-up a51790dc, 2026-09-02T21:55:50Z)
- scheduler-sim-reconcile-terminal-guard: both headers' children
done (last: task 2.2 REPL-level verification, commit 9dcffce8,
2026-09-03T08:31:37Z)
- scheduler-sim-step-verbosity: both headers' children done
(last: task 2.2 full regression, commit 56bd3103,
2026-09-03T05:15:45Z)
Checked all remaining headers, bumped all three to
implementation-complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
All 10 tasks were done (last: task 3.3 automation-regression check,
commit 8fc86ef4, 2026-08-26) but the 3 section-header checkboxes were
never flipped and spec.json.phase stayed at tasks-generated. Same
drift class as 5c82ad0b/b14e53f5/1abab949. Checked the 3 headers,
bumped phase to implementation-complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
- scheduler-sim-eager-reconciliation: all 9 tasks were done (last:
task 3.3 full regression, commit 446437bf, 2026-08-27) but the 3
section-header checkboxes were never flipped and phase stayed at
tasks-generated. Same drift class as 5c82ad0b/b14e53f5. Checked the
3 headers, bumped phase to implementation-complete.
- scheduler-allocation, scheduler-preemption: blocked_on named
scheduler-core and scheduler-dispatch-commit, both of which reached
implementation-complete weeks ago (2026-08-19, 2026-08-24). Annotated
as resolved rather than removed, so the historical block reason
stays visible. Requirements approval is still outstanding on both --
not touched, since that gate needs human review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
All 28/28 tasks were completed (last: task 4.4, commit b6a2f5ec) but
spec.json.phase was never advanced past tasks-generated. Flipped to
implementation-complete, matching the same drift class fixed for
5 other specs in 5c82ad0b.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Successor to scheduler-primary-lifecycle: a new per-node xce-node-manager
component (one per host, role-assigned: manager/worker/apiserver/
reverse-proxy) owns starting/stopping XCE components for its node's
role -- including starting/stopping the scheduler on a manager node in
reaction to that node's controller reporting itself primary vs.
standby over a local heartbeat (Unix domain socket, client logic
shared via libxce) -- rather than the controller directly
spawning/killing its own scheduler process. Carries forward
scheduler-primary-lifecycle's underlying requirements (bug-1760's
reactive-rejoin deadlock, wedge detection, graceful shutdown,
controller-resilience invariants).
Design review (kiro-validate-design) fixed 3 issues before GO: a
uniform-single-OS-user assumption that was factually wrong given
xce-deploy-ha's real worker daemon runs as root (resolved: root only
on worker-role nodes); Requirement 4.3's wedge-detection mechanism
verified directly against the actual scheduler code
(sched_job_picker's cbuf_get() does an unbounded wait when idle,
confirming the naive design would false-positive -- fixed by swapping
to the existing cbuf_timedget() bounded-deadline variant); and
PortCheckMonitor's TCP connect made explicitly non-blocking to match
HeartbeatListener's own discipline. Tasks not yet generated.
Checkpointing uncommitted spec work found in this shared worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Open-ended simulation-mode REPL diagnostics spec: first capability is
limit inspection (info limit <name|all>) showing configured setting
(scope, value/percent) and live accounted usage per instance. A second
capability (decision-reasoning trace for pend/dispatch outcomes) is
noted as a future seed, not yet in requirements.md.
Requirements generated but not yet approved; design/tasks not started.
Checkpointing uncommitted spec work found in this shared worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Requirements and design approved (design fixed the fork()-vs-posix_spawn
mutex-inheritance-deadlock risk product.md's own "no fork() for
snapshots" decision already rejected); tasks generated but never
approved. Superseded by xce-node-manager after the user decided the
controller should not directly own spawning/killing its own scheduler
process -- a new per-node xce-node-manager component takes over that
responsibility instead. Kept in place as a reference (notably the
fork()-vs-posix_spawn analysis); this spec's tasks.md must not be
implemented (see spec.json's superseded_note).
Checkpointing uncommitted spec work found in this shared worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch,
mirroring bug-1764's actual live-observed shape: a worker with one job
driven to a terminal state and two jobs genuinely still RUN, all real
allocations to the SAME worker. After reconciling a stray DISPATCH
entry for the terminal job's id, info worker excludes it while still
listing the two running jobs, and info summary's running count agrees.
Round 1 review rejected an initial version for using a mismatched host
string on the stray entry ("workerarr-stray" instead of "workerarr"),
which made the info worker/info summary assertions pass vacuously
regardless of whether the fix existed, since repl_info_worker_one()
filters assigned jobs by an exact host-string match. Round 2 fixed the
host string and confirmed via an independently-reproduced mutation
test (fix reverted, the test's own accounting-level pre-check
bypassed) that the REPL-level assertions now genuinely, independently
detect the bug.
design.md's Testing Strategy is corrected: it originally said no new
test was needed, reasoning the existing display-path suites already
covered this; tasks.md's own task 2.2 correctly required a
purpose-built test for this exact scenario instead.
Full scheduler test suite: 63/64 OK, sole failure the pre-existing,
unrelated bug-1758. This is the final task of the
scheduler-sim-reconcile-terminal-guard spec; all implementation tasks
(1.1-2.2) are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Verified (full regression build/test pass, an explicit by-name check
of all 18 tests in test_sim_reconcile_unaccounted.c and the JOBS_PUSH
resync-merge tests in test_core_events.c, and a diff-scope check
against the pre-feature baseline) that: test_core_events.c is
byte-identical to baseline, test_sim_reconcile_unaccounted.c's diff is
a pure addition with no existing assertion touched, and the fix's
total footprint is exactly the expected 8 files. The sole test
failure in the full suite (test_job_rusage_reconciles_mem_reservation_against_usage)
is confirmed pre-existing and unrelated (bug-1758). Read-only
validation task; no production or test code changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
sim_reconcile_unaccounted_jobs()'s per-job trigger check (sim_loop.c)
now also excludes JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state),
alongside the existing job->accounts != NULL skip. A terminal job is
never searched for a matching recorded DISPATCH entry, so a leftover
or duplicate DISPATCH decision from a later replay round can no
longer resurrect allocs/accounts on a job that has already completed
- the exact mechanism behind info worker's assigned-jobs listing
showing a finished job as still consuming a worker's resources.
The exclusion mirrors job_state_regresses_terminal()'s existing
"terminal is permanent" invariant (core_events.c) rather than
inventing a new rule. The function's doc comment gains a paragraph
clarifying the terminal exclusion is complementary to, not a reversal
of, the existing PEND/RUN state-independence rationale that motivated
this trigger's original widening.
Both RED-phase tests (task 1.1) pass; the pre-existing PEND
eager-capture regression test and core_events.c's JOBS_PUSH
resync-merge terminal-state tests are unaffected. The analogous
resync-merge branch was investigated and confirmed already correct.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds two failing tests formalizing bug-1764's reproduction: a job
driven to JOB_STAT_DONE (and, separately, JOB_STAT_EXIT) via a real
xce_core_push_events() JOB_STATE_CHANGE call correctly has its
allocs/accounts nulled by the existing end-of-life cleanup, but
sim_reconcile_unaccounted_jobs() then wrongly resurrects both fields
when presented with a stray DISPATCH entry for that same jobid, since
its trigger checks only job->accounts == NULL with no state check.
The pre-existing eager-capture regression test,
test_reconcile_captures_non_run_state_job_when_dispatch_recorded,
already covers Requirement 1.2's "PEND job's early DISPATCH still
captured" case explicitly and needed no duplicate. No production code
touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Formalizes the fix for bug-1764 through this project's TDD process:
sim_reconcile_unaccounted_jobs() (sim_loop.c:664) triggers on
job->accounts == NULL alone, with no check on job->state -- a
deliberate widening from scheduler-sim-eager-reconciliation to catch
a PEND job's early-recorded DISPATCH before its RUN confirmation
replays, but one that never excluded jobs already at a terminal state
(DONE/EXIT). Since xce_core_push_events() correctly frees allocs/
accounts on a terminal transition (leaving accounts == NULL, the same
trigger signal), a later round's leftover/duplicate DISPATCH entry for
that job id gets misread as something to reconcile, resurrecting the
allocation on an already-dead job -- directly observable via info
worker's assigned-jobs listing showing a completed job as still
consuming a worker's resources.
Requirement 1 adds a terminal-state exclusion to the trigger check,
mirroring job_state_regresses_terminal()'s existing "terminal is
permanent" invariant (core_events.c). Requirement 2 captures the
resulting REPL-visible outcome without requiring any REPL code change.
The JOBS_PUSH resync-merge branch (core_events.c, bug-1734/bug-1738)
was investigated during design and confirmed to already guard against
this exact problem -- explicitly out of this spec's scope.
Generated via kiro-spec-quick with an inline requirements review gate,
an inline design review gate, and an independent task-graph sanity
review (one NEEDS_FIXES round, repaired: task 2.2's REPL-level
verification mechanism was underspecified, now pins down the exact
capture_dispatch()-driven harness and stray-DISPATCH-injection
technique to use).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Confirms the full scheduler test suite passes with no regressions
(63/64; sole failure is the pre-existing, unrelated bug-1758) and that
the feature works correctly at realistic scale. Built a synthetic
multi-round replay mirroring the original motivating scenario (a
JOB_PUSH immediately followed by several JOB_UPDATE/JOB_STATE_CHANGE
events for the same job, all sharing one rendered second/session) and
ran it through the real interactive binary in both step and continue
mode: a 6-event same-context run and a 4-event structural run each
collapsed to exactly one preamble line, an elapsed-time context change
and a session-only context change (same rendered second, different
session) each correctly forced a fresh preamble - confirming the
dedup key is genuinely the (timestamp, session) pair, not either
field alone. Continue mode's pre-existing, unrelated Event Echo Gate
correctly suppressed routine events throughout. A non-interactive
batch run of the same fixture produced the identical single decision
step/continue mode already showed, confirming no scheduling-decision
divergence.
This is the final task of the scheduler-sim-step-verbosity spec; all
implementation tasks (1.1-2.2) are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Verified (full regression build/test pass, a per-file diff of every
pre-existing REPL test file against the pre-feature baseline,
repl.c's command dispatch confirmed byte-identical, and a two-worktree
step-mode transcript diff against a synthetic fixture with both a
same-context and a differing-context event pair) that: no pre-existing
test assertion was touched, no top-level command changed, and the
transcript diff shows exactly the two same-context preamble lines
removed with every other line - including both events' own content -
byte-identical to the pre-feature baseline. The new cache fields
(last_echo_ts_human/last_echo_session) are read/written exclusively
inside repl_print_event_summary(), confirming the feature is
display-only and cannot affect scheduling decisions or simulation
state (Requirement 2.4). The sole test failure in the full suite
(test_job_rusage_reconciles_mem_reservation_against_usage) is
confirmed pre-existing and unrelated (bug-1758). Read-only validation
task; no production or test code changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
repl_print_event_summary() (repl_break.c) now compares the candidate
rendered timestamp and displayed session number against a new cache
on xce_sched_repl_state_s (repl_internal.h: last_echo_ts_human[32],
last_echo_session) before printing the "[sim] t=... session=..."
preamble. It prints and updates the cache only when either value has
changed, or no preamble has been printed yet this session (the cache's
zero-initialized default, which no real echoed event can produce);
otherwise the preamble is skipped and the cache is left untouched. The
event's own content line prints unconditionally either way
(Requirements 1.1-1.4, 2.2).
All 6 RED-phase tests (task 1.1) now pass, including the critical
raw-vs-rendered nuance test (Requirement 1.3). No regressions in the
Event Echo Gate suite or the sibling spec's breakpoints-inspection
suite (test_step_echoes_events_before_session_decisions specifically
confirmed still passing under the new dedup behavior).
repl_break.c grew from 949 to 980 lines (+31, 3.3% over the residual
already accepted in scheduler-sim-repl-verbosity task 9.2). Review
judged this small growth as accepted without a fresh empirical split
test: task 9.2's own split experiment (a larger +44-line growth)
already found extracting this file's lookup-table domain reaches only
838 lines (still over cap) while requiring 22 white-box test files'
#include lines to change - disproportionate churn this smaller
increment would only re-confirm.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 6 failing tests for the Event Echo Preamble Deduplicator
(Requirements 1.1-1.4, 2.2): a second echoed event sharing its
predecessor's rendered timestamp+session omits its preamble; two
events across a session boundary both print their own preamble; the
critical raw-vs-rendered nuance (Requirement 1.3) - two events whose
raw nanosecond timestamps genuinely differ but whose rendered,
second-granularity timestamp and session are identical still dedupe,
with a fixture-premise self-check proving the two raw values really
do render identically; the first echoed event in a fresh session
always prints its preamble; a structural event under continue mode
after suppressed routine events follows the same rule; event content
and ordering are unaffected regardless of the preamble decision.
Lands in a new test_repl_step_verbosity.c (1010 lines, 6 tests) rather
than extending test_repl_event_echo.c (705 lines) - proactive split
per this file family's established practice, since appending would
have pushed that file well past the project's ~1000-line trigger.
Registered in testsuite/meson.build. No production code touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Follow-on to scheduler-sim-repl-verbosity: collapses the REPL's
per-event "[sim] t=... session=..." preamble to print once per
distinct rendered (timestamp, session) run of consecutive echoed
events, instead of once per event, in both step mode and continue
mode's structural-event echo. Requirement 1.3 deliberately keys the
dedup on the rendered, second-granularity timestamp rather than the
raw nanosecond clock value, which ticks on every event and would
otherwise never repeat -- confirmed against a real user-supplied
transcript before writing requirements (see research.md). Requirement
2 preserves event selection, ordering, content, every other REPL
command, and all scheduling decisions/state exactly as
scheduler-sim-repl-verbosity's Requirement 5 already established.
Generated via kiro-spec-quick (single-pass, no approval gates) with
an inline requirements review gate, an inline design review gate, and
an independent task-graph sanity review (one NEEDS_FIXES round,
repaired: added file-size watch bullets to tasks 1.1/1.2 per
design.md's own housekeeping directives).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Found by /kiro-validate-impl's cross-task requirements-coverage and
design-alignment checks, both documentation-only (no code changes):
1. AC 6.5 (whole-number/no-trailing-.00 formatting) is implemented as
scoped to non-consumable (Load Metrics) values only - a consumable
resource's value=/used= pair deliberately keeps its unconditional
%.2f formatting to preserve bug-1601/1613/1727's already-pinned
display shape. This carve-out existed only as an inline code
comment (repl_info.c) and a test-file header comment, never
sanctioned in requirements.md/design.md itself. Amended AC 6.5's
text to state the scope explicitly and added a design.md Correction
note cross-referencing it.
2. design.md's File Structure Plan never documented task 6.1's proactive
split of the prompt-content tests into a new test_repl_prompt_context.c
(every other analogous split in this spec got a Correction note;
this one didn't), and task 9.1's later Correction note repeated the
stale claim that test_repl_execution_control.c "already carries" the
prompt-content tests - it doesn't, they moved to the new file at task
6.1. Added the missing task 6.1 Correction note and fixed both stale
references.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
