The oldest unresolved bug in this project's tracker (first sighted
2026-06-20): controller_unit_test intermittently hung for the full
300s meson timeout with no crash/ASAN report, never previously caught
live under gdb.
Live-reproduced this session (first attempt): thread 1 stuck forever
in commit_engine_fini() -> pthread_cond_destroy() (test_commit_barrier_
rollout.c's jap_teardown), while two fsync_worker threads sat parked
in pthread_cond_timedwait on the same persist_t. Root cause: several
test teardown functions called commit_engine_fini() BEFORE
persist_stop() -- the exact opposite of commit_engine_fini()'s own
documented contract. This let the live background fsync_worker thread
call back into commit_on_durable_advance() while commit_engine_fini()
was mid-destroy of the engine's lock/condvar -- a genuine TOCTOU race
with no synchronization between the two threads.
This is the same race already root-caused once for a SEGV symptom in
bug-1452 (2026-07-19, test_persist_truncate.c) and partially re-fixed
in bug-1687 (2026-08-25, cbrp_teardown) -- but never swept across all
affected files. Bug-1452's own remediation note explicitly flagged
test_demotion_convergence.c as a known, unfixed instance; that flag
went unactioned until now.
Fix: insert persist_stop() immediately before commit_engine_fini() at
21 confirmed-vulnerable sites across 12 files (test_apply_pump.c,
test_apply_pump_integration.c, test_commit_barrier_rollout.c,
test_commit_durability.c, test_commit_engine.c,
test_convergence_verification.c, test_demotion_convergence.c,
test_egress_filter.c, test_ha_role_lifecycle.c, test_job_array_fence.c,
test_job_array_replay_fence.c, test_p2_integration.c) -- the same
pattern bug-1452/1687 already established. persist_stop() is
idempotent (guarded by an atomic "already running" check), so each
fixture's own pre-existing persist_stop() call becomes a safe no-op.
No production code changed.
Known remaining exposure, documented but not fixed this session:
test_config_repl.c (~23 sites) and test_term_safety_adapter.c
(~5 sites) have the same ordering bug in their own teardown helpers,
flagged as follow-up.
Verified: full controller suite run 4 times total (once as part of
the fix, three further independent sequential runs) -- Ok:6 Fail:0
every time, 38-42s wall clock each run, no hang, nowhere near the
300s timeout. 1220/1220 individual controller_unit_test cases pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
XCE_CORE_EVENT_JOBS_PUSH's duplicate-jobid handling (sched_add_job()
EEXIST) assumed a redelivery is always a harmless no-op -- true for
bug-1714's array-push TOCTOU race (both copies start identically
PEND), but not for a scheduler reconnect's full resync
(on_scheduler_join() -> xcesched_push_jobs(), sched.c), which
re-sends every currently-committed job through this same path
regardless of how far its state has moved on since this scheduler
last saw it.
Live-reproduced on a real 3-node HA cluster + worker: the scheduler's
outbound xcecli connection repeatedly failed over under a 50-element
array-job load; each reconnect's resync redelivered jobs that had
already finished (PEND/RUN -> DONE) while disconnected -- confirmed
via ctrl-1.log's existing bug-1731 "c->sched.ep is NULL, notify
SKIPPED" diagnostic firing for the same jobids during the gap, making
the resync that job's only remaining chance to reach the scheduler.
Discarding it unconditionally permanently froze the tracked job at
its stale state: controller shows DONE, scheduler stuck at RUN/PEND
forever.
Fix: on EEXIST, look up the existing tracked job and merge in the
redelivery's state if different (same terminal-transition
accounts/allocs cleanup XCE_CORE_EVENT_JOB_STATE_CHANGE/JOB_UPDATE
already do), instead of silently dropping it. Only state is merged;
broader field reconciliation is out of scope, same as the existing
allocs/accounts scope note this code path already carries. Bundled a
requested logging fix at the same site (raw jobid -> fmt_jobid()).
Verified live end to end: rebuilt scheduler, reran the same load
against the same cluster (reconnect storm still occurred, proving the
fix doesn't depend on suppressing it), both test arrays now show
consistent DONE state matching the controller exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
The worker's rusage collector already measures and sends
"memory_swap"/"tasks_current" per job, but xce_host_runusage_t had no
field to receive them at all -- a complete dead end, not a naming
mismatch like bug-1735's cpu_time/mem_usage. Added mem_swap/
tasks_current fields (types.h), matching capnp fields on HostRunusage
(controller.capnp, new ordinals @3/@4) for WAL field-completeness, and
wired update_host_runusage() (misc.c) to parse them. Also fixed
merge_job_runusage()'s (job.c) existing-host merge branch, which only
copied cpu_time/mem_usage per host -- the two new fields would
otherwise have been silently dropped on WAL replay/catchup for a host
already seen.
Also computed peak_cpu_time as the running max of aggregate_job_
runusage()'s per-call cpu_time sum across every report a job has ever
received, guarding against a later report reducing it.
Deliberately NOT computed: peak_cpu_duration and the three efficiency
fields (mem_efficiency/avg_cpu_efficiency/peak_cpu_efficiency) -- these
need a "used vs. requested" ratio against a job's resource request (a
resreq expression tree, not a plain core/slot count) and/or an
unspecified duration semantic. A documented scope decision, not an
oversight -- see aggregate_job_runusage()'s doc comment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
test_recordstart_symlink_escape_replies_error expected "invalid record
file name" (the -EINVAL string), but sched_validate_record_filename()'s
canonical-path containment check has always, correctly, returned
-EACCES ("permission denied accessing record file") for a symlink
escaping record_dir -- confirmed against test_recording_path_validation.c's
own direct unit test for the identical scenario, which already asserts
-EACCES and has always passed.
Not environment-dependent as originally suspected (this project's
buglog carried it as a uid-1000-vs-root permission-check-ordering
theory across 7+ sightings since Aug 15) -- it fails identically
regardless of uid, since escape-containment and filesystem-permission
errors were never actually conflated in production code, only in this
test's assertion. No production code changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
All 11 leaf tasks (1.1-4.5) are checked done and the feature was
closed out by its own commits (5948e079, b64682dc); spec.json still
read "tasks-generated" -- same phase/reality lag class already fixed
for other specs this week.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
70+ testsuite/ files across controller/scheduler/lib call mkdtemp()
directly with no shared fixture helper, and none clean up after
themselves. Left unchecked this fills the sandbox disk (recurred three
times now: 2026-06-22, 2026-08-26, 2026-08-30, each larger than the
last -- 696GB at its worst), which then surfaces as unrelated
"Expected Non-NULL"-style cascade failures wherever the next test
happens to call mkdtemp()/persist_init(), easily mistaken for a real
regression.
Retrofitting cleanup into 70+ individual test files would be invasive
and risks breaking tests that inspect a fixture's contents across
multiple TEST_ functions in the same file. Every test run in this
project goes through xce-build.sh, so pruning once there -- age-gated
at 3 hours so a concurrent session's in-progress fixtures are never
touched -- is the lowest-risk single choke point.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Two independent gaps in misc.c/job.c, both silently zeroing every real
job's resource-usage data:
- update_host_runusage() matched worker rusage report resnames
"cputime"/"mem", but the real worker's collector
(collect_job_rusages(), xcecd_cgroups.c) sends "cpu_time_sec"/
"memory_rss" -- the two never matched, so cpu_time/mem_usage stayed
at their calloc()'d zero for every non-simulated job. Aligned both
sides, including xce_test's own simulated-worker fixture.
- update_job_rusages() had a standing FIXME: xce_job_runusage_t's
job-level max_mem/avg_mem/used_mem/cpu_time were never derived from
the per-host host_runusage[] entries. Added aggregate_job_runusage()
(sum/max/avg across hosts) and wired it in right after the per-host
update.
Added test_host_runusage.c (standalone unit test, mirrors
resval_typeresolve_unit_test's isolation pattern) plus the
controller_unit_test stub aggregate_job_runusage() needs since that
target excludes the real misc.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Generates/updates a rolling per-component change log from git history,
mirroring /weekly_report's date-window and continuity-check patterns but
organized by component (controller/worker/scheduler/etc.) instead of a
narrative weekly write-up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The scheduler-framework merge (2026-08-25) silently regressed the PEND
reason P1 work (4e2ac86/ee79179): the old fcfs.c became dead-on-default
(builtin_fcfs replaced it), resource.c/limit.c conflict resolution took
Rongsong's side, and the merge-fix only re-added the dispatchwindow FIELD
without the logic. After the merge, sched_queue_is_ready/sched_job_is_ready
were `return 1` stubs and only MATCH_FAIL/NO_RESOURCE were ever emitted.
Restore P1 on the active (framework) path, keeping every reason attach in
the allocator thread (bug-1521: the picker thread must not write
job->schedobj->reason — it races the allocator's MOVE_PTR):
- core.c: sched_job_is_ready (PEND & not suspended) and sched_queue_is_ready
(QUEUE_STAT_CLOSED/INACT_ADMIN/INACT_WINDOW gate) real implementations.
- scheduler.c: allocator loop injects sched_queue_status_reason() — queue
status first, then the dispatch window evaluated against the core's
virtual clock (xce_core_now/1e9) so live and sim modes agree; attaches
QUEUE_CLOSED/DRAINING/TIME_WINDOW and pends the job. Adds <time.h>.
- resource.c: skip UNAVAIL/CLOSED/CLOSED_BY_ADMIN workers + HOST_UNAVAIL
(restored from 4e2ac86).
- limit.c: attach LIMIT_GLOBAL/LIMIT_HOST + carry reason in
limit_job_election (restored from 4e2ac86).
Two further merge regressions found during cluster verification (the
framework core path never carried the queue dispatch window):
- core_events.c core_build_queue: duplicate q->dispatchwindow_config onto
xce_sched_queue_t.dispatchwindow (make_sched_queue did this; the core path
dropped it, so TIME_WINDOW could never fire on the default path).
- core_events.c core_upsert_queue update branch: refresh dispatchwindow on
QUEUE_UPDATE.
Regression test: test_core_ready_gates.c (white-box #include of core.c,
7 cases covering both gates), registered in scheduler testsuite meson.build.
Cluster-verified on t1/t2/t3 (nix xcecore closure + AppImage standard
deployment): night queue (window closed) -> main_reason=8 time_window;
all workers down -> main_reason=9 host_unavail with per-host detail;
closed queue -> job stays PEND; normal queue baseline -> DONE. Scheduler
suite 52/53 (1 pre-existing admin-request symlink-escape failure, unrelated).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Expose two backend capabilities MCP previously omitted:
- scheduler_checkpoint: control action "checkpoint" (CRIU)
- scheduler_restart: POST /xce/v1/jobs/:id/restart (re-queues as a new
job, resuming from the last checkpoint if one exists; returns new jobid)
schedulerclient gains CheckpointJob/RestartJob (RestartJobResult,
JobRestartRequest); Client interface updated. Both tools marked
destructive (confirm required). tools/list now reports 17 tools.
Unit tests added/updated in both modules — all pass. Backend endpoints
are existing apiserver routes; no backend changes in this commit.
Also syncs left_scheduler_work.md §八.4 (checkpoint/restart done; the
only remaining MCP tool gap is scheduler_modify, blocked on a controller
job-field-update opcode that does not exist yet).
Co-Authored-By: Claude Code <noreply@anthropic.com>
Two root causes, confirmed by direct code reading (previously misattributed
to an overlayfs sandbox fdatasync artifact):
conf_trim_old_versions() sorted config-version files by st_mtime (second
resolution). The test writes 3 versions 15ms apart -- well within one
second, so ties are the expected outcome, not a rare edge case. qsort()
isn't stable, so a tied ordering could push the protected current-pointer
file (skipped on deletion, never compensated) past keep_n, leaving one
extra file on disk ("Expected 2 Was 3"). Fixed by sorting on st_mtim
(nanosecond-resolution struct timespec) instead of st_mtime.
cr_setup() keyed its tmpdir on getpid()+time(NULL) (second resolution),
risking collision with a stale directory left by an earlier interrupted
run (ASAN abort, timeout, SIGKILL -- none reach cr_teardown()'s rm -rf)
reusing the same PID within the same second -- test_conf_get_saved_tag_
no_config then observed that directory as "not actually empty" and read
a leftover tag instead of the expected 0. Fixed with a monotonic counter
guaranteeing a unique directory per cr_setup() call.
TDD: RED reproduced on the first attempt (both tests failed with the
exact historical signature). GREEN confirmed across 16+ repeat runs
after the fix, zero other regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Found while chasing bug-1714's crash (root cause still open -- see that
entry): two related, independently-confirmed defects, not proven to be
bug-1714's own cause but real bugs on their own merits.
xce_core_push_events()'s JOBS_PUSH case never checked sched_add_job()'s
return value -- if key_add() failed (duplicate jobid), the freshly-built
xce_sched_job_t was handed to picker_iface->on_job_push() anyway, even
though it was never registered in core->jobs. An orphaned job the
picker knows about but sched_get_job() can never find again. Now checks
the return, frees the orphan (at core_build_job()'s own established
cleanup granularity -- xce_sched_job_t has no complete destructor
anywhere in this codebase, kill_sched_job() only frees ->rusages, so
this intentionally doesn't invent a more thorough one), and skips the
picker call.
This makes it safe to fix (not the other way around): sched_on_job_array()'s
per-element dedup check (handler.c) has a TOCTOU race -- a job is only
added to core->jobs later, by the driver thread, so a redelivered array
push arriving on the request thread before the driver catches up isn't
caught. Documented as a best-effort optimization, not a correctness
boundary -- the actual authoritative uniqueness check is sched_add_job()'s
key_add(), which the JOBS_PUSH fix above now handles correctly.
TDD: new test reproduces the duplicate-jobid path (RED: on_job_push
fired twice, Expected 1 Was 2), GREEN after the fix. Full scheduler
regression: 54/55, only the known bug-1561-adjacent flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Follow-up to ebe691c3 (HOOK_DETAIL_JOB_START/JOB_EXIT), same gap for job
failures: apply_committed_entry()'s WAL_JOB_FAIL case only ever fired
the generic HOOK_DETAIL_JOB_UPDATE, leaving sched_job_fail_fun (sched.c)
and apiserver_job_fail_fun (apiserver.c) dead -- the apiserver's
OnJobFinished was starved of data for job failures specifically (the
prior fix only closed this for clean finishes).
Fires HOOK_DETAIL_JOB_FAIL only for the WAL_JOB_FAIL_STAGE_FLAG branch,
which already constructs a real job_failinfo_t -- the legacy _pad==0
absolute-overwrite branch has no fail-specific data and must not
fabricate one, mirroring how the WAL_JOB_START fix only fires when
pidinfo was genuinely decoded. Unlike WAL_JOB_START, no gate-widening
was needed: apply_job_fail_transition() always returns
JOB_TRANSITION_APPLIED for a genuine new fail transition (confirmed by
reading it), so the existing result-gated early return already covers
this correctly.
TDD: RED confirmed (fix reverted, test failed "Expected 2 Was 1"),
GREEN after restoring it. Full regression: 1220 tests, only the known
bug-1561 flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
test_live_config_push_against_standby_returns_standby_rejection_not_redirect
locked in the OLD controller behavior (a standby flatly rejects
OP_upgradeConfig with no redirect fields). That gap is now closed:
myreq_handler's gate-rejection path calls fill_redirect_reply() on
every -XCE_ERR_STANDBY rejection (bug-1700), and handle_ctrlhello() now
actually populates primary_host/primary_port from the static --peers
list (bug-1701) -- a standby pushing config now redirects to the
primary and the push succeeds. Renamed to
test_live_config_push_against_standby_redirects_and_succeeds and
rewritten to assert the new behavior.
_assert_primary_committed() changed from a single-shot log read to a
poll loop (up to 5s) for the "config activated from committed
marker..." line -- accounts for the documented async gap between the
majority-ack-gated client reply and the primary's own local WAL-apply-
pump activation, a real race the old single-shot read could hit.
meson-build.sh: wire src/tools and each build profile's controller
subdir into the isolated repo-mirror the E2E tests run against, and
export XCEADMIN_BIN explicitly, so these live tests actually find real
binaries instead of always skipping.
Verified: xceadmin suite 288/288 (incl. both tests in this file);
controller suite 1220/1222 (only the pre-existing bug-1561 flake).
Updated admin-config-tool design.md and roadmap.md to mark the
dispatch-gate and redirect-wiring gaps RESOLVED, matching this test's
new assertions and the live evidence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
xce_sched_core_load_snapshot() left every restored worker's resInUse
(and core->sharedInUse) at zero -- zarray_init() in the workers-
restoration block clears them, and nothing re-summed against the
restored job allocations. A caller inspecting worker/resource state
between a snapshot load and the first scheduling session (e.g. the
REPL's `info worker` before the first `step`) saw a misleadingly empty
"used" total for jobs already known to be RUNNING. Added one
sched_account_jobs(core) call after workers/jobs are restored -- a
correct idempotent-from-zero re-derivation using the same
sched_charge_account() chain the next real session would use anyway.
Must not be called again without an intervening reset (accumulates,
does not assign) -- verified no such double-call exists in this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Requirements, design, research, and tasks all generated and approved
(ready_for_implementation: true). Implementation not yet started (0/8
tasks) -- saving the completed spec authoring; the 8-task
implementation is separate future work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
A JOB_EXITED status could mean either a deliberate, expected termination
(worker_signal_callback() only ever sends SIGTERM/SIGKILL to cancel/
force-kill a job) or a genuine unexpected crash signal. Both logged at
ERROR, drowning real crashes in expected-cancellation noise. Added
job_exit_log_level(): exit_code == 128+SIGTERM/SIGKILL logs at INFO
(matching process_exit_callback()'s existing plain-INFO treatment for
the same case), anything else stays ERROR. Verified: worker test suite
5/5 including the new test_uv_ipc_job_exit_severity coverage.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
The `worker)` case's meson compile target list omitted xcecd_virtual --
the no-root worker binary deploy-ha.sh/the xce-deploy-ha skill needs
whenever no real systemd manager is present (the common sandbox case).
/xce-build worker alone left it unbuilt, so deploy-ha.sh silently
soft-skipped generating start-worker.sh. Verified: xcecd_virtual now
builds with a fresh mtime, full worker suite still 5/5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
apply_committed_entry() (persist.c), the single seam that applies every
committed WAL entry, only ever fired the generic HOOK_DETAIL_JOB_UPDATE
for job-state changes -- HOOK_DETAIL_JOB_START/JOB_EXIT were
permanently dead code, since job.c only ever fires them at
HOOK_BEFORE_EVENT (for WAL-append purposes). This left sched_job_start_
fun/sched_job_exit_fun (sched.c) and apiserver_job_start_fun/
apiserver_job_exit_fun (apiserver.c) unreachable, and the Go apiserver's
already-wired OnJobFinished handler permanently starved of data -- a
finished job's state could stay stuck (e.g. showing RUN) in the
scheduler's own view and the apiserver never learned a job finished.
The wire format was already enriched by bug-1697/bug-1689 with the
pidinfo/exit_info data these hooks need, so this closes the gap rather
than requiring new wire work. WAL_JOB_FINISH now fires
HOOK_DETAIL_JOB_EXIT; WAL_JOB_START now fires HOOK_DETAIL_JOB_START,
gated on a call-scoped decode flag rather than the transition result
(WAL_JOB_START applies as a no-op duplicate for every real job in
production per bug-1697, so gating on the transition result would make
it dead code again). WAL_JOB_FAIL -> HOOK_DETAIL_JOB_FAIL has the same
gap and is left as a follow-up.
Also includes this investigation's supporting changes: a DEBUG3 log-
retention bump (main.c, both controller and scheduler) so race-condition
evidence survives a live burst without rotating out, and DEBUG3-gated
diagnostic trace logging at the two points a job-update notification
can fire or silently drop (job.c, sched.c).
Live-verified end to end on a real deployed cluster: scheduler-side
sched_job_state_change now fires for RUN and EXIT states, and the
apiserver's OnJobFinished -- previously never invoked in production --
now correctly updates job status on completion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
vtask_enq() published a task onto the shared dispatch queue via
dataq_enq() before finishing its own initialization (task->owner) -- a
concurrent vcpu could dequeue, run, and free the task before the
producer's own write executed, an ASAN-confirmed UAF that crashed 2 of
3 controllers under a heavy burst-array-job HA workload. Moved
task->owner = vtaskmgr before the publish, mirroring the endpoint_ref-
before-publish discipline already established two lines above it
(bug-1298). Adds a stress-test regression guard (test_vtask_enq_race.c).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
XCE_CORE_EVENT_JOB_UPDATE (core_events.c) is the only currently-live path
a controller job-finish notification reaches, but unlike its sibling
XCE_CORE_EVENT_JOB_STATE_CHANGE case, it never released job->accounts/
job->allocs on a terminal transition, leaking them for the scheduler
process's lifetime. Mirrors the existing JOB_STATE_CHANGE cleanup.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds the previously-uncommitted spec.json/requirements.md/design.md/research.md
for scheduler-snapshot-job-allocation (feature now fully implemented across
tasks 1.1-4.5) and the in-progress xce-test-cli-ux spec's requirements.md/
spec.json. Updates brief.md, a security.md note on an unrelated deferred RBAC
persistence gap found during this feature's cross-component survey, and syncs
OpenWolf's anatomy/memory/token-ledger/daemon-log bookkeeping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.5: scheduler dimension-4 field-completeness audit run against the
finished implementation -- clean, no undocumented gap. Corrects the one
remaining stale premise in snapshot.c's "internal_state contents" comment
(claimed job->allocs/accounts were still unrestored; they are now, per
task 2.2) while preserving its still-valid sharedInUse/resInUse rationale.
All tasks (1.1-4.5) for scheduler-snapshot-job-allocation are complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.4: the "Scope decisions" header comment previously stated a
job's allocs/accounts are "not captured" -- update it to describe the
now-implemented dup_job_alloc()/make_job_account() restoration path,
and add a note documenting job->rusages as a known, still-uncaptured
gap requiring a future capnp schema change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.3: independent full scheduler+lib suite rerun confirms zero new
failures beyond pre-existing unrelated bug-1560 and bug-1723; scope
confined to allocation/schedobj paths with zero diff to snapshot.c or
the reconciliation test files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.2: extend test_e2e_original_motivation_long_replay with PREEXIST_ID
(job 70013, dedicated worker worker-e2e-pre) -- a job submitted,
live-dispatched, and confirmed RUN entirely before sched_record_start(),
so its dispatch decision can never appear in the replay stream. Proves
its "used=9.00" resource contribution is correct starting from the very
first replayed round, recoverable only via snapshot-load allocation
restoration (tasks 2.1/2.2), not replay-based reconciliation -- the
complementary case to the file's existing ANCHOR_ID scenario.
Completes and fixes a pre-existing uncommitted WIP addendum to this file:
removed a redundant enqueue_queue_update() call that was masking a
genuine, unrelated latent bug in core_upsert_queue() (rtti makeString()
pointer freed via plain free(), same class as bug-1503/bug-1600) --
noted in tasks.md's Implementation Notes as a follow-up, not fixed here
(test-only task boundary).
Verified via fault injection: gating snapshot_load.c's allocation
restore reproduces the exact pre-fix failure at this test's new
assertion, then cleanly reverts.
Requirements: 1.1, 1.2, 1.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BSiC2cPVH2VSBdtFq4P4ju
Task 4.1: new test proving a snapshot-restored PENDING job (valid
resreq) reaches the real, live sched_allocator() thread's
sched_make_allocation() entry point without crashing -- the exact
bug-1569-class failure task 1.2 closed for the snapshot-load path.
New file (not an extension of test_snapshot_save_load.c) since driving
the real allocator needs scheduler.c's full dependency closure, too
heavy for that file's deliberately lean core-only TU. Follows the
already-established pattern from test_scheduler_connect.c /
test_scheduler_backpressure.c / test_scheduler_dispatch_commit.c: link
the real scheduler.c, wrap xcecli_start/xcecli_join, start a real
sched_init(), feed jobs directly into core.jbuf to drive the live
allocator thread. Wired into meson.build as a new executable/test.
Verified with two fault-injection experiments (reverted after):
disabling task 1.2's schedobj construction call makes this test fail
cleanly via its own assertion; going further and removing that
assertion reproduces a genuine ASAN-caught SEGV in the real allocator
path -- concrete proof this test would have caught bug-1569's failure
shape. Both polling loops (jbuf drain, decision wait) are bounded
(2s timeout, 5ms poll) -- no hang risk; confirmed non-flaky across 10
consecutive runs.
Requirements: 2.2, 2.4
Task 3.2: adds test_snapshot_load_leaves_running_job_without_allocation_at_zero_resinuse,
covering the specific Req 3.5 scenario the existing PENDING-job test didn't
-- a RUNNING job whose allocation was never captured (simulating a
pre-feature-saved snapshot). With a real worker/resource present (making
the assertion non-vacuous), confirms: load succeeds with no error,
job->schedobj is still valid (task 1.2, independent of allocation data),
job->allocs/accounts stay NULL, and resInUse correctly reads 0.0 via the
real accounting chain -- a clean degrade, no special-casing, matching
today's pre-feature behavior exactly.
Test-only task; no production code changed. Per this spec's own
"pre-release product" scope note, no archived legacy snapshot file is
needed as a fixture -- a freshly-built wire job with allocs unset
exercises the identical code path.
Requirements: 3.5
Task 3.1: comprehensive integration test proving tasks 1.1/1.2/2.1/2.2
work together end to end -- a RUNNING job (multi-host, multi-resource,
plus a genuine shared-resource-pool component) and a PENDING job saved
together and loaded into a second, independent core. Asserts:
- Both jobs get a valid, non-NULL schedobj in the restored core.
- The running job's allocation/accounting content-match the source,
including the shared-pool component.
- Worker resInUse and core sharedInUse, computed via the real
sched_charge_account() chain (white-box included from resource.c,
following the established test_resource_accounting.c/test.c
link-only-stub convention), match exactly between source and
restored cores -- the actual proof of Req 1.2/1.3, not just that
job->accounts holds the right values.
- Destroying the source core's job list afterward does not corrupt or
crash the restored core (Req 1.4's independence guarantee exercised
via real teardown, not just pointer-distinctness checks).
Test-only task; no production code changed. Independently verified via
a live RED/GREEN toggle against task 2.2's load-side restoration block.
Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.3, 2.4
Task 2.2: xce_sched_core_load_snapshot() now restores job->allocs via
dup_job_alloc() (never aliasing the wire job's own copy, since it
belongs to a const, potentially-shared snapshot struct) and rebuilds
job->accounts via the existing make_job_account(), positioned right
after task 1.2's schedobj construction and before sched_add_job().
Independent of resreq parse success -- not nested inside that guard.
Absent allocation data is the normal case: no WARN, no error.
New tests prove a full cross-core round trip (save core_a, load into
independent core_b): restored allocs/accounts content-match the
source, with three-way pointer independence (restored vs source-core
vs saved-wire) proving the deep-copy guarantee end to end. A companion
test confirms a job with no allocation loads cleanly.
Note: this task's implementation was completed directly after two
implementer subagent dispatches hit transient API interruptions; one
auto-resumed and reported a false READY_FOR_REVIEW (claiming 9/9
passing tests while the actual production code was never written,
only the test file). Caught by fresh-evidence verification before
trusting the report -- confirmed via an independent build+test run
showing a genuine failure, then fixed and re-verified by an
independent reviewer who reproduced RED-to-GREEN itself.
Requirements: 1.1, 1.2, 1.3, 1.4, 3.5
Task 2.1: build_wire_job() now deep-copies job->allocs into the wire
job's already-existing allocs field via dup_job_alloc() (task 1.1) when
the live job has an allocation, leaving it unset otherwise -- matching
how every other optional field in this function is handled. Never
aliases job->allocs.
New tests prove: a running job with a multi-host allocation saves with
matching content (host names, resource values) and a pointer distinct
from the live job's own allocation; a job with no allocation saves with
the wire field left NULL, no error.
Requirements: 1.1
Task 1.2: snapshot_load.c gains a local, static field-for-field port of
core_events.c's core_build_schedobj()/core_build_primary_schedobj()/
core_build_compound_schedobj()/core_build_alter_schedobj() (the bug-1569
fix for the JOBS_PUSH path). Every job restored from a snapshot whose
resreq parsed successfully now gets a valid, non-NULL job->schedobj,
closing the same NULL-deref crash class for the snapshot-load path.
Guarded on job->resreq != NULL so the pre-existing, unrelated
resreq-parse-failure path is untouched (still leaves schedobj NULL,
same as today -- not fixed here, per design.md's documented boundary).
New tests prove non-NULL schedobj with correct tag/union member for
primary, compound, and alter resource requests, plus a regression test
confirming the parse-failure path's behavior is unchanged.
Requirements: 2.1, 2.2, 2.3, 2.4
The scheduler-framework merge (4e894d21) reverted the bug-1507 fix from
c091a8da: job_start_local_task fell back to nobody only when execuser was
EMPTY, not when it does not exist on the worker host — a submit-user missing
from /etc/passwd makes systemd fail the worker unit with EXIT_USER (217) and
every job dies. Restored the getpwnam() existence check (+ #include <pwd.h>).
All 21 worker tests pass; fallback log message verified in the built xcecd.
Also records the merge-overwrite lesson in cerebrum Do-Not-Repeat: after any
branch-sync merge, re-verify each previously-fixed bug's marker symbol
survived (git show <merge>:<file> | grep -c <symbol> vs <merge>^).
Task 1.1: new deep-copy for xce_job_alloc_t composed from existing
make_alloc/make_host_alloc/make_job_alloc, explicitly preserving each
allocation's original from/duration rather than accepting make_alloc's
own time(NULL) stamp. TDD: 6 new unit tests (single-host, multi-host,
multi-resource-per-host, shared-resource-pool, independence-after-free,
from/duration fidelity), all passing under ASAN.
Also logs bug-1723 (pre-existing, unrelated leak in two bug-1722 test
fixtures, confirmed via git-stash A/B rebuild to be byte-for-byte
identical before and after this change) -- documentation only, not fixed
here.
Requirements: 1.1, 1.4
Extends the save/restore-specific gate added moments ago with a
broader, self-maintaining mechanism: rather than enumerating each
xce-audit-invariants dimension individually in the design-review
files (which would drift out of sync as new dimensions get added --
already up to 9 for controller, 4 for scheduler), both gates now
instruct walking the relevant component's whole dimension list against
the draft design. Catches design-time-checkable invariants (async I/O
thread-role placement, fail-loud/never-self-exit process lifecycle,
Raft/TLA+ safety checklist, commit-before-materialize ordering,
redirect-completeness for new opcodes, driver/sim state-equivalence)
before implementation rather than only via a later audit or live soak.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Prompted by rediscovering the same bug class in two places this
session: the controller's WAL already has this documented for one
mechanism (persistence.md's APPLY_CLEAN/bug-1678 section, xce-audit-
invariants controller dimension 7) but the scheduler's separate
snapshot save/load mechanism had the identical gap (job->allocs, this
session's scheduler-snapshot-job-allocation brief) with no equivalent
tracking. Closes that gap and makes the invariant self-reinforcing
going forward rather than something that has to be independently
rediscovered per persistence mechanism:
- xce-audit-invariants: new scheduler dimension 4 ("Field-complete
snapshot save/load parity"), sibling of controller dimension 7,
scoped to snapshot.c/snapshot_wire.c/snapshot_load.c.
- persistence.md: new "Snapshot Save/Load Field-Completeness
(Scheduler)" section cross-referencing the WAL-side bug-1678 writeup
and the new scheduler dimension.
- CLAUDE.md: new top-level Key Rules bullet stating the invariant
plainly, plus adding persistence.md to the steering-file pointer
table (it was missing).
- kiro-spec-design's design-review-gate.md (the mandatory pre-write
gate) and kiro-validate-design's design-review.md (the optional deep
review): new conditional review sections -- applies only when a
design touches a save/restore mechanism, requires enumerating every
persisted field's capture status rather than trusting an aggregate
"state looks right" check. Catches this class of gap at design time,
before implementation, rather than only via a later audit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
make_job_account() allocates account->accounts via calloc(1,
sizeof(xce_host_account_t*)) -- room for exactly 1 host pointer -- but
its own loop writes account->accounts[i] for i up to jalloc->nhosts-1,
overflowing the heap allocation for any job spanning more than one
host. Already reachable from live production dispatch (core_events.c,
schedobj.c, scheduler.c) for any multi-host job -- found while
verifying viability of a proposed scheduler snapshot-allocation-capture
fix, not from a user report.
Sibling function make_host_allocs() has the identical bug shape
(calloc(1, ...) vs a loop writing up to nhosts-1); every current call
site happens to pass nhosts=1 so it has never actually overflowed in
production, but it's the same live footgun in a public exported
helper. Fixed alongside since it's the identical bug class in the same
file.
New test_xcealloc.c drives a 2-host allocation through both functions,
reproducing the exact ASAN heap-buffer-overflow at xcealloc.c:188
pre-fix (RED), passing post-fix (GREEN). Both calloc(1, ...) sites
changed to size the allocation by the actual host count.
Full regression: libxce 303/303 (was 301, +2 new), scheduler 53/54 (1
pre-existing unrelated bug-1560 flake), controller 1211/1213 (2
pre-existing unrelated bug-1561 overlayfs flakes) -- zero new
failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
/kiro-discovery investigation of removing src/frontend/Makefile in favor
of Meson as the sole build system. Records the quick investigation's
findings (5 boundary seams, current-state evidence that build-cli/
build-server are already dead from /xce-build's perspective and
build-server is already broken today) and the user's confirmed decision
to fully remove the Makefile rather than keep a compat wrapper.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
CLAUDE.md: replace the stale @.wolf/OPENWOLF.md import with a pointer to
the openwolf skill / rules file, matching how the project's OpenWolf
setup actually resolves the protocol now.
api-standards.md: add checklist item 6 for new primary-only opcodes --
confirm ha_dispatch_gate's -XCE_ERR_STANDBY rejection actually populates
a redirect via fill_redirect_reply(), not just that the client has a
generic redirect-follower. Directly informed by bug-1700, where the
identical gap was independently rediscovered for OPCode_schedAdmin after
already being documented once for OPCode_upgradeConfig.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
