Adds the previously-uncommitted spec.json/requirements.md/design.md/research.md
for scheduler-snapshot-job-allocation (feature now fully implemented across
tasks 1.1-4.5) and the in-progress xce-test-cli-ux spec's requirements.md/
spec.json. Updates brief.md, a security.md note on an unrelated deferred RBAC
persistence gap found during this feature's cross-component survey, and syncs
OpenWolf's anatomy/memory/token-ledger/daemon-log bookkeeping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.5: scheduler dimension-4 field-completeness audit run against the
finished implementation -- clean, no undocumented gap. Corrects the one
remaining stale premise in snapshot.c's "internal_state contents" comment
(claimed job->allocs/accounts were still unrestored; they are now, per
task 2.2) while preserving its still-valid sharedInUse/resInUse rationale.
All tasks (1.1-4.5) for scheduler-snapshot-job-allocation are complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.4: the "Scope decisions" header comment previously stated a
job's allocs/accounts are "not captured" -- update it to describe the
now-implemented dup_job_alloc()/make_job_account() restoration path,
and add a note documenting job->rusages as a known, still-uncaptured
gap requiring a future capnp schema change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.3: independent full scheduler+lib suite rerun confirms zero new
failures beyond pre-existing unrelated bug-1560 and bug-1723; scope
confined to allocation/schedobj paths with zero diff to snapshot.c or
the reconciliation test files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Task 4.2: extend test_e2e_original_motivation_long_replay with PREEXIST_ID
(job 70013, dedicated worker worker-e2e-pre) -- a job submitted,
live-dispatched, and confirmed RUN entirely before sched_record_start(),
so its dispatch decision can never appear in the replay stream. Proves
its "used=9.00" resource contribution is correct starting from the very
first replayed round, recoverable only via snapshot-load allocation
restoration (tasks 2.1/2.2), not replay-based reconciliation -- the
complementary case to the file's existing ANCHOR_ID scenario.
Completes and fixes a pre-existing uncommitted WIP addendum to this file:
removed a redundant enqueue_queue_update() call that was masking a
genuine, unrelated latent bug in core_upsert_queue() (rtti makeString()
pointer freed via plain free(), same class as bug-1503/bug-1600) --
noted in tasks.md's Implementation Notes as a follow-up, not fixed here
(test-only task boundary).
Verified via fault injection: gating snapshot_load.c's allocation
restore reproduces the exact pre-fix failure at this test's new
assertion, then cleanly reverts.
Requirements: 1.1, 1.2, 1.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BSiC2cPVH2VSBdtFq4P4ju
Task 4.1: new test proving a snapshot-restored PENDING job (valid
resreq) reaches the real, live sched_allocator() thread's
sched_make_allocation() entry point without crashing -- the exact
bug-1569-class failure task 1.2 closed for the snapshot-load path.
New file (not an extension of test_snapshot_save_load.c) since driving
the real allocator needs scheduler.c's full dependency closure, too
heavy for that file's deliberately lean core-only TU. Follows the
already-established pattern from test_scheduler_connect.c /
test_scheduler_backpressure.c / test_scheduler_dispatch_commit.c: link
the real scheduler.c, wrap xcecli_start/xcecli_join, start a real
sched_init(), feed jobs directly into core.jbuf to drive the live
allocator thread. Wired into meson.build as a new executable/test.
Verified with two fault-injection experiments (reverted after):
disabling task 1.2's schedobj construction call makes this test fail
cleanly via its own assertion; going further and removing that
assertion reproduces a genuine ASAN-caught SEGV in the real allocator
path -- concrete proof this test would have caught bug-1569's failure
shape. Both polling loops (jbuf drain, decision wait) are bounded
(2s timeout, 5ms poll) -- no hang risk; confirmed non-flaky across 10
consecutive runs.
Requirements: 2.2, 2.4
Task 3.2: adds test_snapshot_load_leaves_running_job_without_allocation_at_zero_resinuse,
covering the specific Req 3.5 scenario the existing PENDING-job test didn't
-- a RUNNING job whose allocation was never captured (simulating a
pre-feature-saved snapshot). With a real worker/resource present (making
the assertion non-vacuous), confirms: load succeeds with no error,
job->schedobj is still valid (task 1.2, independent of allocation data),
job->allocs/accounts stay NULL, and resInUse correctly reads 0.0 via the
real accounting chain -- a clean degrade, no special-casing, matching
today's pre-feature behavior exactly.
Test-only task; no production code changed. Per this spec's own
"pre-release product" scope note, no archived legacy snapshot file is
needed as a fixture -- a freshly-built wire job with allocs unset
exercises the identical code path.
Requirements: 3.5
Task 3.1: comprehensive integration test proving tasks 1.1/1.2/2.1/2.2
work together end to end -- a RUNNING job (multi-host, multi-resource,
plus a genuine shared-resource-pool component) and a PENDING job saved
together and loaded into a second, independent core. Asserts:
- Both jobs get a valid, non-NULL schedobj in the restored core.
- The running job's allocation/accounting content-match the source,
including the shared-pool component.
- Worker resInUse and core sharedInUse, computed via the real
sched_charge_account() chain (white-box included from resource.c,
following the established test_resource_accounting.c/test.c
link-only-stub convention), match exactly between source and
restored cores -- the actual proof of Req 1.2/1.3, not just that
job->accounts holds the right values.
- Destroying the source core's job list afterward does not corrupt or
crash the restored core (Req 1.4's independence guarantee exercised
via real teardown, not just pointer-distinctness checks).
Test-only task; no production code changed. Independently verified via
a live RED/GREEN toggle against task 2.2's load-side restoration block.
Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.3, 2.4
Task 2.2: xce_sched_core_load_snapshot() now restores job->allocs via
dup_job_alloc() (never aliasing the wire job's own copy, since it
belongs to a const, potentially-shared snapshot struct) and rebuilds
job->accounts via the existing make_job_account(), positioned right
after task 1.2's schedobj construction and before sched_add_job().
Independent of resreq parse success -- not nested inside that guard.
Absent allocation data is the normal case: no WARN, no error.
New tests prove a full cross-core round trip (save core_a, load into
independent core_b): restored allocs/accounts content-match the
source, with three-way pointer independence (restored vs source-core
vs saved-wire) proving the deep-copy guarantee end to end. A companion
test confirms a job with no allocation loads cleanly.
Note: this task's implementation was completed directly after two
implementer subagent dispatches hit transient API interruptions; one
auto-resumed and reported a false READY_FOR_REVIEW (claiming 9/9
passing tests while the actual production code was never written,
only the test file). Caught by fresh-evidence verification before
trusting the report -- confirmed via an independent build+test run
showing a genuine failure, then fixed and re-verified by an
independent reviewer who reproduced RED-to-GREEN itself.
Requirements: 1.1, 1.2, 1.3, 1.4, 3.5
Task 2.1: build_wire_job() now deep-copies job->allocs into the wire
job's already-existing allocs field via dup_job_alloc() (task 1.1) when
the live job has an allocation, leaving it unset otherwise -- matching
how every other optional field in this function is handled. Never
aliases job->allocs.
New tests prove: a running job with a multi-host allocation saves with
matching content (host names, resource values) and a pointer distinct
from the live job's own allocation; a job with no allocation saves with
the wire field left NULL, no error.
Requirements: 1.1
Task 1.2: snapshot_load.c gains a local, static field-for-field port of
core_events.c's core_build_schedobj()/core_build_primary_schedobj()/
core_build_compound_schedobj()/core_build_alter_schedobj() (the bug-1569
fix for the JOBS_PUSH path). Every job restored from a snapshot whose
resreq parsed successfully now gets a valid, non-NULL job->schedobj,
closing the same NULL-deref crash class for the snapshot-load path.
Guarded on job->resreq != NULL so the pre-existing, unrelated
resreq-parse-failure path is untouched (still leaves schedobj NULL,
same as today -- not fixed here, per design.md's documented boundary).
New tests prove non-NULL schedobj with correct tag/union member for
primary, compound, and alter resource requests, plus a regression test
confirming the parse-failure path's behavior is unchanged.
Requirements: 2.1, 2.2, 2.3, 2.4
Task 1.1: new deep-copy for xce_job_alloc_t composed from existing
make_alloc/make_host_alloc/make_job_alloc, explicitly preserving each
allocation's original from/duration rather than accepting make_alloc's
own time(NULL) stamp. TDD: 6 new unit tests (single-host, multi-host,
multi-resource-per-host, shared-resource-pool, independence-after-free,
from/duration fidelity), all passing under ASAN.
Also logs bug-1723 (pre-existing, unrelated leak in two bug-1722 test
fixtures, confirmed via git-stash A/B rebuild to be byte-for-byte
identical before and after this change) -- documentation only, not fixed
here.
Requirements: 1.1, 1.4
Extends the save/restore-specific gate added moments ago with a
broader, self-maintaining mechanism: rather than enumerating each
xce-audit-invariants dimension individually in the design-review
files (which would drift out of sync as new dimensions get added --
already up to 9 for controller, 4 for scheduler), both gates now
instruct walking the relevant component's whole dimension list against
the draft design. Catches design-time-checkable invariants (async I/O
thread-role placement, fail-loud/never-self-exit process lifecycle,
Raft/TLA+ safety checklist, commit-before-materialize ordering,
redirect-completeness for new opcodes, driver/sim state-equivalence)
before implementation rather than only via a later audit or live soak.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Prompted by rediscovering the same bug class in two places this
session: the controller's WAL already has this documented for one
mechanism (persistence.md's APPLY_CLEAN/bug-1678 section, xce-audit-
invariants controller dimension 7) but the scheduler's separate
snapshot save/load mechanism had the identical gap (job->allocs, this
session's scheduler-snapshot-job-allocation brief) with no equivalent
tracking. Closes that gap and makes the invariant self-reinforcing
going forward rather than something that has to be independently
rediscovered per persistence mechanism:
- xce-audit-invariants: new scheduler dimension 4 ("Field-complete
snapshot save/load parity"), sibling of controller dimension 7,
scoped to snapshot.c/snapshot_wire.c/snapshot_load.c.
- persistence.md: new "Snapshot Save/Load Field-Completeness
(Scheduler)" section cross-referencing the WAL-side bug-1678 writeup
and the new scheduler dimension.
- CLAUDE.md: new top-level Key Rules bullet stating the invariant
plainly, plus adding persistence.md to the steering-file pointer
table (it was missing).
- kiro-spec-design's design-review-gate.md (the mandatory pre-write
gate) and kiro-validate-design's design-review.md (the optional deep
review): new conditional review sections -- applies only when a
design touches a save/restore mechanism, requires enumerating every
persisted field's capture status rather than trusting an aggregate
"state looks right" check. Catches this class of gap at design time,
before implementation, rather than only via a later audit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
make_job_account() allocates account->accounts via calloc(1,
sizeof(xce_host_account_t*)) -- room for exactly 1 host pointer -- but
its own loop writes account->accounts[i] for i up to jalloc->nhosts-1,
overflowing the heap allocation for any job spanning more than one
host. Already reachable from live production dispatch (core_events.c,
schedobj.c, scheduler.c) for any multi-host job -- found while
verifying viability of a proposed scheduler snapshot-allocation-capture
fix, not from a user report.
Sibling function make_host_allocs() has the identical bug shape
(calloc(1, ...) vs a loop writing up to nhosts-1); every current call
site happens to pass nhosts=1 so it has never actually overflowed in
production, but it's the same live footgun in a public exported
helper. Fixed alongside since it's the identical bug class in the same
file.
New test_xcealloc.c drives a 2-host allocation through both functions,
reproducing the exact ASAN heap-buffer-overflow at xcealloc.c:188
pre-fix (RED), passing post-fix (GREEN). Both calloc(1, ...) sites
changed to size the allocation by the actual host count.
Full regression: libxce 303/303 (was 301, +2 new), scheduler 53/54 (1
pre-existing unrelated bug-1560 flake), controller 1211/1213 (2
pre-existing unrelated bug-1561 overlayfs flakes) -- zero new
failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
/kiro-discovery investigation of removing src/frontend/Makefile in favor
of Meson as the sole build system. Records the quick investigation's
findings (5 boundary seams, current-state evidence that build-cli/
build-server are already dead from /xce-build's perspective and
build-server is already broken today) and the user's confirmed decision
to fully remove the Makefile rather than keep a compat wrapper.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
CLAUDE.md: replace the stale @.wolf/OPENWOLF.md import with a pointer to
the openwolf skill / rules file, matching how the project's OpenWolf
setup actually resolves the protocol now.
api-standards.md: add checklist item 6 for new primary-only opcodes --
confirm ha_dispatch_gate's -XCE_ERR_STANDBY rejection actually populates
a redirect via fill_redirect_reply(), not just that the client has a
generic redirect-follower. Directly informed by bug-1700, where the
identical gap was independently rediscovered for OPCode_schedAdmin after
already being documented once for OPCode_upgradeConfig.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Once bug-1700/1701's server-side redirect fix landed, xceadmin scheduler
record status/start/stop against a standby correctly redirects to the
primary -- but the redirected request can still transiently fail if that
node is mid-(re-)election (-10013 E_XCE_NOT_READY, expected/normal right
after a redirect on a freshly-booted or just-failed-over cluster) or if
the scheduler connection was mid-reconnect (schedadminreplymsg{ok=false,
error="scheduler disconnected"}, a different, non-numeric signaling path
via bug-1693's disconnect-detection mechanism).
bug-1713: _send_admin_request() wraps the request/redirect/decode cycle
in a bounded retry loop, re-running follow_redirects() FROM SCRATCH each
attempt (not a bare resend to the same host:port) so a later attempt can
discover a newly-settled or different primary.
bug-1715: extends the same retry loop to the second, differently-signaled
transient family -- checks admin_reply.error against a module-level
frozenset of known-transient reason strings.
Live-verified on a real 3-node cluster.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
info worker all/<name> in the interactive/sim REPL initially printed
full per-resource values correctly, then lost them after a later
WORKERS_PUSH event replayed. core_apply_worker_resvals()'s per-resval
loop only guarded against nv==NULL/nv->name==NULL; a resval entry with
a NULL nv->val (a legitimate "no update this round" marker some
producers send) fell through and overwrote the worker's existing good
value with nothing, instead of being skipped -- same defensive pattern
already used elsewhere in this function, just missing here. A related
count-doubling gap in the same event path is fixed by switching an
iteration bound from `.count` to `.caps`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Chaotic HA soak (post-bug-1695) reported committed_job_loss every
round. raft_cb_become_leader() computed the promotion WAL-truncation
floor and evaluated the full promotion-eligibility gate (durable_floor/
recovery_truncated/apply_verdict) BEFORE cluster_init_phase2() -- the
only call that runs persist_recover() on this process's first-ever
boot. Those fields all come from persist_recover(); read before it
runs, a freshly persist_init()'d struct reports durable_lsn=0, making
the eligibility gate wrongly judge a genuinely corrupt/divergent node
ELIGIBLE. Unreachable before bug-1695's fix (a restarted node's stale
term previously always made it defer to a peer instead of re-electing
on the same boot with real data at stake) -- bug-1695 correctly
restored real re-election on restart and newly exposed this
pre-existing boot-ordering flaw.
Reordered raft_cb_become_leader() so cluster_init_phase2()/
persist_recover() runs before the eligibility gate and truncation-floor
computation. Split the pure vote-derived safe_committed_lsn computation
(reads only ha->elect_ctx, safe pre-recovery) from the truncation-floor
computation (needs post-recovery persist state) -- the former stays
early, the latter moves to run after phase2. New regression test
CRC-corrupts a real on-disk WAL and confirms first-boot promotion now
correctly refuses instead of destroying committed data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
xceadmin scheduler record status/start/stop against a standby failed
with the bare "controller returned status -10014" (E_XCE_STANDBY) with
no redirect, forcing the caller to guess the primary manually.
bug-1700: myreq_handler's generic ha_dispatch_gate rejection path
(cluster.c) now calls fill_redirect_reply() when rc == -XCE_ERR_STANDBY,
mirroring the same helper OPCode_join's handler already uses --
-XCE_ERR_NOT_READY (CTRL_INIT/CTRL_ELECTING, no settled primary to
redirect to) is deliberately excluded. Landing this alone had zero
observable effect live: ha_state_t.primary_host/primary_port were never
written by any production path on a standby (only test fixtures set
them) -- fill_redirect_reply() always saw an empty/zero redirect target
in a real deployment.
bug-1701: closes that data gap with a pure local lookup, no new wire
field. handle_ctrlhello() (config_repl.c) now resolves the ctrlhello
sender's name against this node's own statically-known peers_cfg
(parsed once from --peers at boot) and populates primary_host/port at
the exact moment a standby first learns who primary is.
Live-verified end to end on a fresh 3-node cluster: xceadmin scheduler
record status against any standby now prints "redirected: <standby> ->
<primary>" and returns the real status, confirmed again after a
re-election changed which node was primary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
dispatch_a_job_() synchronously sets state=JOB_STAT_RUN at dispatch time
(scheduler-dispatch-commit task 3.1's deliberate no-worker-ack-wait
semantic). By the time the worker's later WAL_JOB_START fact commits,
apply_job_update_transition() sees state already == RUN and returns
NOOP_DUPLICATE instead of APPLIED -- so bug-1687's start_time/pidinfo
decode/adopt block, gated on result==APPLIED, could never run for any
job. start_time (and consequently run_time) stayed 0 forever. Same bug
class already fixed for WAL_JOB_RUSAGE/CKPT/PREASON in this identical
apply function; WAL_JOB_START was missed because bug-1687 predated the
dispatch-time RUN semantic change.
Moves start_time to dispatch-owned: dispatch_a_job_() now stamps
job.start_time alongside state=RUN, under the same write lock.
WAL_JOB_DISPATCH gains a flag-gated (WAL_JOB_DISPATCH_HAS_START_TIME_FLAG)
start_time field so it survives replay/restart; both apply seams
(apply_committed_entry, apply_wal_entry) and restore_job_dispatch_state
decode it when present, falling through to the old alloc-only shape
otherwise. WAL_JOB_START's own now-permanently-dead start_time write is
removed; pidinfo adoption made unconditional to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
start.sh/start-worker.sh/start-scheduler.sh/start-apiserver.sh had no
check for an already-running instance, so re-running start-all.sh (or
a generated per-component script) against a still-live deploy silently
spawned a full duplicate process set under the identical --myname/
--peers/ports. Combined with SO_REUSEPORT on every listening socket
(src/lib/netio.c), the duplicate's bind()/listen() succeeded instead
of failing loudly, so two live controllers per node ended up voting
into the same Raft group with independently diverged term/log state --
a split-brain that surfaced as non-converging elections, a bouncing
redirect chain, and ctrlstatus reporting an unknown primary for 40+
seconds on a freshly deployed cluster.
Each generated launcher now writes a pidfile under $TOP/run/ and
refuses to start if a live process already owns it, verified via
/proc/<pid>/cmdline identity match (not just liveness) to avoid a
PID-reuse false positive. start-all.sh preflights all expected
pidfiles before launching anything, failing fast and loud instead of
partially duplicating some nodes. Each script's final invocation now
uses exec so the pidfile records the daemon's own PID directly, which
also collapses the previous wrapper+child two-process-per-component
layout into one process. cleanup.sh clears stale pidfiles for hygiene.
Verified live in-sandbox: normal start-all.sh still elects a primary
and starts all 6 components; a second start-all.sh run against the
same live deploy now refuses cleanly with all 6 already-running
components identified, process count unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Pure verification, zero code/test changes. Confirmed: the 9 pre-existing
test_sim_reconcile_unaccounted.c tests pass unchanged; interactive/batch
parity (test_repl_equivalence.c, untouched by this spec) passes; full
scheduler suite shows the identical 53 Ok / 1 Fail baseline (sole failure
the pre-existing, unrelated bug-1560), with the spec's 6 net-new tests
(tasks 1.1/1.2/2.1/3.1/3.2) as the only addition to that baseline.
Marks all tasks in the scheduler-sim-eager-reconciliation spec complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_sim_verify_no_false_mismatch_across_round_separated_dispatch:
a job's DISPATCH decision at round 1, RUN-state confirmation event at
round 4, --sim-verify enabled throughout, asserting zero mismatches.
Fixture correctly accounts for a mechanically-verified system behavior:
sim mode's live picker (sim_picker_thread_fn/builtin_fcfs) never emits
on_dispatch() -- it unconditionally re-PENDs every candidate each round,
including a job reconciliation already captured, until the job's
recorded JOB_STATE_CHANGE event removes it from pendJobs. The fixture
includes a matching PEND(jobid) entry at each intervening round to match
this reality, not an assumption.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap:
a worker at exact capacity (jobs A=8.0 + C=2.0 against 10.0 capacity), with
each job's RUN-state confirmation landing several rounds after its DISPATCH
decision (round 1 dispatch, round 4/5 confirmation, deliberately different
gaps). Drives the real sched_check_host_satisfy() admission check at three
checkpoints, proving Requirement 1 (1.1-1.4) at the outcome level -- a
capacity check, not just a resInUse display value -- per
scheduler-driver-equivalence.md's explicit contention-fixture discipline.
Round 1 review caught three new xce_sched_sim_driver_t fixtures declared
stack-local, amplifying an already-documented (bug-1598, bug-1632)
stack-use-after-return class from the never-joined picker thread
sched_sim_driver_run() spawns. Fixed by declaring them static (safe: each
is written by exactly one call site, and sched_sim_driver_start() memsets
the struct first) -- independently confirmed via 3 full-suite runs with
zero ASAN hits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds 3 tests proving guarantees that must hold under the widened
capture-eligibility gate: an already-captured job is never re-searched
or re-captured regardless of state (test_reconcile_still_skips_already_
captured_job_regardless_of_state), a PEND job with no match this round
stays uncaptured and remains eligible next round (test_reconcile_
no_match_leaves_pend_job_uncaptured_and_retries), and several
simultaneously outstanding jobs are captured independently with zero
cross-job interference (test_reconcile_multiple_jobs_captured_
independently_across_different_rounds).
Required making the shared add_job() fixture helper tolerant of -EEXIST
on a repeated queue registration (needed for the multi-job test, the
first in this file to call add_job() twice against the same core) --
traced to kvtab_add()'s narrowly-scoped -EEXIST return, provably a
no-op for all 11 pre-existing single-add_job()-call tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_captured_pend_job_excluded_from_resinuse_until_run: drives a
job captured while still PEND through sched_account_jobs() and confirms
its resource-in-use contribution is zero; transitions the same job to
RUN and confirms sched_account_jobs() now includes its exact captured
allocation, with no second reconciliation call in between -- proving the
nonzero value comes from the earlier capture, not a fresh one.
Required promoting the file's sched_account_jobs() from a no-op stub to
a real reimplementation (verbatim body, matching the exact precedent
already established in test_sim_contention_parity.c and
test_sim_round_boundary_reconciliation.c) plus a resource.c white-box
include -- no production code touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Widens sim_reconcile_unaccounted_jobs()'s trigger gate in sim_loop.c: removes
!JOB_IS_RUN(job->state) so a job's recorded DISPATCH decision is captured as
soon as the replay reaches the round it occurred in, regardless of the job's
current state -- closing bug-1611/bug-1613's residual (a job whose RUN
confirmation lands in a later round than its dispatch decision, the realistic
case under any real dispatch-to-confirmation latency, was never reconciled).
sched_account_jobs()'s own unmodified JOB_IS_RUN charging gate continues to
defer actual resource-in-use accounting until the job genuinely transitions
to RUN -- no other line in the match/capture/report path changes.
Replaces test_reconcile_skips_non_run_state_job (which asserted the bug as
correct behavior) with test_reconcile_captures_non_run_state_job_when_dispatch_recorded,
proving the opposite for a PEND-state job. Adds the xce-test-rc-file-style
spec docs (requirements/design/research/tasks) for this feature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
_live_e2e_common.py's CONTROLLER_CANDIDATES/XCEADMIN_CANDIDATES only ever
resolved to the three default profile-dir names (xce.build, build.release,
build.dbg), with no way to point the live E2E suite at a binary built to a
custom/explicit build dir. Adds XCE_CONTROLLER_BIN/XCEADMIN_BIN env-var
overrides via a new resolve_binary_candidates() helper -- when set, the
override becomes the sole candidate; otherwise the existing three-name
fallback list is unchanged. Mirrors the ${XCE_CONTROLLER_BIN:-default}
convention already used by clean_failover_preservation.sh/run-ha-soak.sh.
Skip messages in both live E2E test files now mention the override.
TDD: 4 new unit tests in test_live_e2e_common.py (no override, empty
override, override-becomes-sole-candidate, and an end-to-end reload proving
the real module constants respect their env vars). Full xceadmin suite
(288 tests) confirmed unaffected: 287 pass, 1 pre-existing unrelated
failure (a stale redirect-gap assertion, unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Now that xce_test auto-loads a personal rc file at startup (this spec),
harden the three existing non-interactive xce_test callers so a future
developer's ~/.config/xce_test/rc can never silently affect a CI/soak run:
run-ha-soak.sh, run-resource-verify.sh, and clean_failover_preservation.sh
all now pass --norc to their xtest() invocation. No behavior change today
(no rc file exists in this environment yet, per xce-test-rc-file task 3.3's
own verification) -- this is forward-looking reproducibility hardening.
Independently re-verified end-to-end: run-ha-soak.sh (--sequential, 2
rounds, zero job loss) and run-resource-verify.sh (1 resource, 3 layers,
zero mismatches) both PASS unchanged with --norc added.
clean_failover_preservation.sh could not be run in this sandbox -- its
DATADIR precondition (<repo>/ha/data/cluster.bin + certs) was deleted by an
unrelated earlier commit (ee8473bd, admin-config-tool) and never restored;
confirmed via git history this is pre-existing and unrelated to this edit
(the precondition check runs before xtest() and fails identically on the
unmodified script). Logged as bug-1721, not fixed here -- out of this
spec's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
No product/spec-test code changed. Ran the real xce-test-ha soak and
xce-test-resources verification drivers with no rc file present anywhere
in the environment they run under -- both pass their own documented
success criteria unchanged (zero job loss across rolling failover; zero
resource-value mismatches across controller/scheduler/apiserver), proving
Requirement 1.3's "no rc file -> unchanged behavior" guarantee holds for
real, pre-existing automation that has no awareness this feature exists.
Marks all tasks in the xce-test-rc-file spec complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Extends test_rcfile_integration.sh with 2 scenarios proving Requirements
3.1-3.4: a failing rc-file command in piped/scripted mode aborts before
any of the piped script's own commands run, exiting nonzero; the same
failing command in an interactive session still reports the failure,
continues executing the rc file's remaining lines, and reaches the normal
prompt -- while still carrying the session's sticky nonzero exit-code
verdict (g_session_failed is never cleared, matching how any other
failing-command-then-quit interactive session already behaves).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_rcfile_integration.sh, mirroring clean_failover_preservation.sh's
shell-driven black-box pattern against the real built xce_test binary:
- a valid rc file's set masters override takes effect in both a piped run
and a genuine pty-driven interactive run, before the normal session begins
- --norc leaves the default masters value untouched with the same rc file
present
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds the integration call site in main(), between g_continue_on_error's
computation and the read loop: unless --norc is given, resolve the rc-file
path and load it through preprocess_and_run (the same command-execution
path used for typed/piped input). A NULL path (no XDG/HOME) or missing
file falls through silently to the read loop. A failing rc-file command
under scripted/piped abort-on-first-failure mode short-circuits straight
to the existing g_session_failed exit-code convention, never entering the
read loop -- so a piped script's own commands never run after an rc-file
failure. Verified end-to-end against the built binary, including a real
pty for the interactive path, in both the failure and success cases.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds struct arguments.norc, a --norc argp option, OPT_NORC (matching
main.c's OPT_PEERS/OPT_CONF_KEEP_N idiom), and its parse_opt case to
test.c. Parsing/recording only -- no wiring into main()'s control flow
or rcfile.c yet (task 2.2).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds rcfile_load(path, runner) to rcfile.c/rcfile.h: reads lines with
getline() (no fixed-size buffer, so a long line like a multi-peer
`set masters ...` can never be silently truncated or split), strips the
trailing newline, invokes the caller-supplied runner once per line, and
stops at the first nonzero runner return, propagating that value. A missing
rc file is a silent no-op; any other open failure prints one stderr warning
and returns 0.
Found during GREEN: fopen() succeeds on a directory path on this platform,
so the EISDIR failure only surfaces on the first getline() call (ferror(f)
set) -- added a post-loop ferror() check to catch it (bug-1716).
5 new unit tests (11 total with task 1.1): ENOENT-silent, per-line-in-order
with newline stripped, stop-on-nonzero with exact propagation, non-ENOENT
open-failure warning, and a >4KB single line proving no truncation/split.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds rcfile_resolve_path() (rcfile.c/rcfile.h): resolves $XDG_CONFIG_HOME/xce_test/rc
when set and non-empty, else $HOME/.config/xce_test/rc, else NULL. Standalone
libc-only module mirroring the configcompare.c pattern; wired into xce_test_src
and controller_unit_test_src, registered in controller_test.c's Unity runner.
6 unit tests in test_rcfile.c cover all XDG/HOME combinations plus a dedicated
env-var save/restore isolation test, addressing the cross-test leak risk found
during design review (controller_unit_test runs 80+ test files in one process).
Adds the xce-test-rc-file spec (requirements/design/tasks/research), including
design fixes from validation: getline()-based line reading (task 1.2, avoids
truncating long rc-file lines) and the env-isolation requirement above.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
