modelJobToSubmit/freeJobSubmit (submit direction) and
cJobToModel/cJobArrayToModels (receive direction, both single-job and
array-job paths) now carry from_host across the CGO boundary.
rawSubmitData/rawJobToModel (the pure-Go, CGO-free mapping layer) gain
the field too, completing the round-trip so JobInfo.FromHost reflects
whatever the controller actually has -- which, as of the prior two
commits, is now always the client-asserted hostname. backend/handler.go
needs no change: its existing blind overwrite in OnJobUpdated is
correct by construction once every controller push genuinely carries
the right value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
- src/schema/controller.capnp: JobSubmitData gains
fromHost @10: Text $C.mapname("from_host") -- auto-regenerates
controller.capnp.c/h via the existing meson capnpc-c custom_target.
- src/include/xce/types.h: job_submit_data_t gains char* from_host.
- src/lib/jobj.c: jobsub_init/jobsub_destroy (the hand-written,
ref-counted, content-deduped job-submit interning object -- not
auto-generated) thread from_host through the deep-copy, the dedup
identity comparison, and the destroy path.
Closes the wire-format half of the round-trip: from_host now survives
from the apiserver's submission all the way through the controller and
back on every subsequent job-state push, and rides along for free in
the existing WAL_JOB_SUBMIT persistence across a controller restart.
Did not need to touch job.c's build_job_submit_payloads() (bare
pointer borrow of the already-decoded submit struct) or any WAL
apply-path code (capnp_encode_job/decode_Job already generically
handle the whole xce_job_t including its full .submit sub-struct).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
CreateJob now does FromHost: req.FromHost, a straight passthrough of
whatever xcecli (or any other client) sends in JobSubmitRequest.FromHost,
instead of deriving it server-side via ctx.ClientIP().
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
JobSubmitRequest gains a FromHost field, populated in BuildJobRequest()
via os.Hostname(). Matches how traditional batch schedulers report the
submitting host, and sidesteps having the server derive it from the
connection (which apiserver was doing via ctx.ClientIP() -- an IP not
a hostname, and spoofable via X-Forwarded-For since this apiserver
never configures gin's TrustedProxies).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
OnHostConfigUpdated only applied config-sourced fields (MaxSlot,
HostModel, HostType, DisptWindow) onto a host whose name exact-matched
an entry already in the store. hosts.yaml commonly defines only a
wildcard "default" entry, so a dynamically-joined worker never
received its own config push and MaxSlot stayed at Go's zero-value
forever -- decoupled from the C-side scheduling path, which already
resolves the real value via worker.c's wildcard fallback.
The "default" pseudo-host's config is now cached and retroactively
backfilled onto any registered host with no explicit config of its
own, in either arrival order (config-before-worker or
worker-before-config). The -1 HOSTCONFIG_MXJ_AUTO_DETECT sentinel
(bug-1556) is resolved to the host's real NumCPUs when applied via
fallback, mirroring worker.c's own resolution, so the display shows
usable capacity rather than a bare -1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
GetJobsInfo/GetJobByIDs validated the -o/-l field list against
models.JobSubmitRequest{} -- the job-submission payload -- instead of
models.JobInfo{}, the query/response model (the already-correct
pattern used elsewhere in this file at line 1131). Response-only
fields like pending_reason don't exist on JobSubmitRequest, so any
such field made the whole listing fail outright with "unsupported
query field", not just omit that column.
core_limit/mem_limit exist on JobSubmitRequest but not on JobInfo --
switching wholesale would have newly broken those two previously-valid
fields. Added CoreLimit/MemLimit to JobAdditionalFields, the existing
merge-in bucket already used for pids/pgids/interactive_job_url.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
ExtractErrorMessage()'s connection-refused branch returned the bare
OS error string verbatim. A CLI call right after apiserver startup
(before its listener is up) surfaced a raw "get the request failed:
[connection refused]" with no indication it's a transient startup
race, not a real failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Fixes test_commit_barrier_reply.c's test_jobdispatch_receipt_ack_no_wal_append
(owned by ha-commit-barrier-rollout), the last remaining assertion this
feature's earlier tasks left stale:
- WAL-count delta corrected from "must be zero" to lsn_before + 1 -- the
batch's successful dispatch now appends one WAL_JOB_DISPATCH fact
(task 3.1); mark_job_fail() was traced and confirmed to append
nothing via any path.
- The batch's terminal-failure job was replaced with a genuinely
retryable one (WORKER_NULL site, unregistered host, new job id)
rather than degrading the notify-count assertion to zero, which
would also have silently killed the test's own reply-before-notify
ordering pin (nothing left to order against). This keeps the test
meaningfully exercising notify_dispatch()'s wiring through
handle_jobdispatch().
Full regression, independently re-verified by review:
- Scheduler suite at documented baseline (51/52, 1 pre-existing
unrelated failure). Investigated and logged bug-1632, a new sibling
of the already-known bug-1598 stack-lifetime pattern in a different
test file -- confirmed pre-existing, untouched by this feature, and
non-deterministic.
- Controller suite clean: 1133 tests, only the 2 pre-existing bug-1561
failures.
- Sequential HA soak: PASS (10 rounds, 36 jobs, zero loss, all 3 nodes
led).
- Chaotic HA soak: PASS (10 rounds, committed_divergence=0
committed_job_loss=0).
This completes the scheduler-dispatch-commit spec: bug-1624
(resource-in-use round-boundary loss) and bug-1625 (scheduler discards
dispatch-failure notify) are both fixed and regression-tested.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
Adds test_handle_jobdispatch_terminal_no_notify_no_leak to
test_jobdispatch_no_double_notify.c, closing a gap neither existing
test fully covered: test_dispatch_fail_terminal.c (task 3.2) bypasses
handle_jobdispatch() entirely via dispatch_a_job_test(), so it can't
exercise the double-notify race at all; test_jobdispatch_no_double_notify.c
(task 4.1) only proves an aggregate 3-job-batch notify count, not a
direct per-job zero-notify proof.
The new test drives a single terminal-classified job through the real
handle_jobdispatch() entry point and asserts, in one run: JOB_STAT_EXIT
with allocs left NULL, a direct (non-aggregate) zero-count read from
the notify-capture spy proving no jobDispatchFail was produced for this
job, and (via task 3.2's canary-reference technique, armed before the
call under test) no reference over-release.
Test-only; job.c/handler.c are untouched. RED phase verified via an
actual revert-rebuild-rerun of handle_jobdispatch() to its pre-task-4.1
shape. Independently reviewed and approved -- the reviewer notably
declined to trust a relayed build report and re-ran verification
first-hand before approving.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
Adds test_rpc_failure_bound_resets_after_intervening_success to
test_dispatch_fail_retry.c, proving Requirement 4.3's literal claim
end-to-end through the real dispatch_a_job_test() entry point: a fail,
then a real success, then another fail resets the consecutive-failure
counter so the second failure is evaluated as failure #1, not #2.
The first draft of this test also asserted job.state/job.allocs revert
to PEND/NULL on that post-success failure -- this genuinely fails
against current code (none of dispatch_a_job_()'s four retryable
branches revert state/allocs when failing from an already-RUN job,
since they were only ever designed for a job whose every attempt starts
from PEND). A debug investigation, independently re-verified even more
exhaustively during review (full trace of every pendJobs-reinsertion
and controller-side state-mutation site), confirmed this scenario is
unreachable via any real caller today -- the scheduler's picker
permanently removes a job from its PEND pool the instant it dispatches
it, and nothing already-implemented ever re-adds a RUN job to it. Only
the not-yet-implemented scheduler-preemption spec could someday create
that path. Logged as bug-1631, deliberately deferred rather than fixed
now (no reachable caller to regression-test against); the two
over-reaching assertions were trimmed, keeping only the counter proof
Requirement 4.3 actually asks for. Independently reviewed and approved.
Scenarios 1 (retryable -> PEND-eligible) and 2 (bound exceeded -> EXIT)
were already fully covered by tasks 2.2's and 3.3's existing tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
Extends task 2.1's test_scheduler_dispatch_commit.c with two tests
proving resource-in-use accounting survives more than a single round
boundary between dispatch and worker confirmation, per task 5.1's
"several rounds" / "matching a same-round confirmation" language:
- test_dispatch_commit_survives_several_rounds_before_confirmation:
loops 5 round-boundary cycles, asserting resInUse after EVERY round,
not just once.
- test_dispatch_commit_multi_round_gap_matches_same_round_confirmation:
compares resInUse after 1 round vs after 5 rounds and asserts
equality -- a same-round confirmation is modeled as nrounds==1 since
design.md documents re-applying JOB_STAT_RUN to an already-RUN job as
a confirmed no-op, so resInUse depends only on job->state, never on
confirmation timing.
Test-only; scheduler.c is untouched. RED phase independently verified
by both implementer and reviewer via an actual revert-rebuild-rerun
cycle (4/5 tests fail without the task 2.1 fix). Independently reviewed
and approved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
dispatch_a_job_() (tasks 3.1-3.3) now owns notification entirely for
every outcome -- notify_dispatch() on a retryable failure, mark_job_fail()'s
own hook chain on a terminal failure, nothing on success. handle_jobdispatch()'s
loop still unconditionally called notify_dispatch() itself for every job
regardless, producing two conflicting outcome reports for any failed
dispatch. Removed that caller-side call; the loop is now a thin
dispatch-and-free loop with no notification logic of its own.
New test_jobdispatch_no_double_notify.c drives a 3-job batch (success,
retryable, terminal) through handle_jobdispatch() and confirms exactly
one notification fires (from the retryable job only), not four.
Independently reviewed and approved. Confirmed handle_jobdispatch()'s
sole caller never depended on its notification behavior (always returns
0 unconditionally). Flagged that test_commit_barrier_reply.c's
test_jobdispatch_receipt_ack_no_wal_append is now stale for a second,
independent reason (notify count, not just WAL count) -- both left for
task 5.4 per Requirement 5.3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
The four retryable-classified failure sites in dispatch_a_job_() (worker
lookup, worker null, worker not ready, RPC failure) now resolve instead
of just returning an error code: lock(j, RTTI_LOCK_WRITE) -> genuinely
classify via xce_classify_dispatch_fail() -> increment dispatch_fail_count
-> within XCE_DISPATCH_FAIL_RETRY_BOUND (3) resolves retryable (nothing
further under the lock), past it resolves terminal via mark_job_fail()
exactly like task 3.2's sites -> unlock(j) -> dereference(j) (closing
bug-1627's 4-site reference leak) -> outside the lock, notify_dispatch()
fires only on the retryable outcome. Each site's natural error code is
captured into a stable local before any intervening call, since errno
isn't guaranteed stable once lock()/ERROR()/free_launchinfo() sit between
the original failure and the eventual return.
handle_jobdispatch() is intentionally untouched -- it still unconditionally
calls notify_dispatch() itself for every outcome, so the retryable path
now temporarily double-notifies until task 4.1 removes that caller-side
call. Expected, not a defect.
New test_dispatch_fail_retry.c (4 tests, including a bounded-escalation
test driving 4 consecutive RPC failures on the same job: 3rd retries,
4th resolves EXIT) plus a fixed stale assertion in test_dispatch_commit.c
and additive, read-only notify-capture accessors in
test_commit_barrier_reply.c (the binary's sole notify() stub host).
Independently reviewed and approved, including explicit scrutiny of the
cross-file test accessor addition and the stale-assertion fix -- both
confirmed legitimate, not scope creep or regression-hiding.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
The two terminal-classified failure sites in dispatch_a_job_() (invalid
allocation, queue-exec-info lookup failure) now actually resolve the job
instead of leaving it stuck at PEND forever with a dangling reference to
nowhere. Each site: lock(j, RTTI_LOCK_WRITE) -> genuinely calls
xce_classify_dispatch_fail(site) and branches on the result (not a
hardcoded assumption) -> mark_job_fail() under the lock (stage=0 routes
apply_job_fail_transition() to JOB_STAT_EXIT) -> unlock(j) -> the existing
dereference(j) (kept exactly where it was, not duplicated) -> the existing
return code, unchanged. No notify_dispatch() added -- mark_job_fail()'s
own hook chain is the sole notification for a terminal outcome per
design.md; handle_jobdispatch()'s still-unconditional notify_dispatch()
call is untouched (task 4.1's job).
New test_dispatch_fail_terminal.c proves both sites resolve to
JOB_STAT_EXIT with allocs left NULL, using a canary-reference technique
to prove no reference over-release (job_t's real refcount has no test
accessor). Independently reviewed and approved; applied the reviewer's
one non-blocking fast-follow (arm the canary before the call under test,
not after, so it's an independent proof rather than relying on other
assertions to crash first) and re-verified with a clean incremental build.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
dispatch_a_job_()'s previously-unconditional job.allocs commit is now
gated on xce_startjob_req() actually succeeding (rc >= 0). In that same
successful branch, under a newly-acquired lock(j, RTTI_LOCK_WRITE) (a
real pre-existing locking gap this call site had -- it took no lock on
the job object itself before this), the job transitions to JOB_STAT_RUN,
dispatch_fail_count resets to 0, and a WAL_JOB_DISPATCH fact is appended
for durability. The failure path is otherwise unchanged -- classification
and resolution of the six failure sites is tasks 3.2/3.3's job, not this
one's.
Adds an XCE_TESTING-only dispatch_a_job_test() wrapper (dispatch_a_job_()
is static) and test_dispatch_commit.c, proving: state->RUN, allocs
committed correctly, counter resets, and durability survives a real
persist_recover() replay with no WAL_JOB_START present.
Independently reviewed and approved. Two pre-existing, out-of-boundary
findings surfaced and logged for later tasks: test_commit_barrier_reply.c's
test_jobdispatch_receipt_ack_no_wal_append will need updating by task 5.4
(Req 5.3 anticipates this), and bug-1429's hang was reconfirmed a 3rd time
(unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
Adds a case for NotifyData_data_jobDispatchFail to sched_handle_notify()'s
switch, fixing bug-1625 -- the notify was previously silently discarded
(falling to default: -EINVAL), leaving job->accounts/job->allocs stale
forever on any dispatch failure. No classification logic here: the
controller has already decided PEND is correct before sending the notify;
this reuses the same XCE_SCHED_OP_JSTATE_CHANGE enqueue path the adjacent
schedExitJob/schedFailJob cases already use, and relies on core_events.c's
existing free-on-non-RUN branch to release the stale accounts/allocs.
Independently reviewed and approved: full scheduler suite passes (51/52,
one pre-existing unrelated flake -- bug-1560).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
sched_commit_decision()'s DISPATCH branch now sets job->state = JOB_STAT_RUN
in the same operation that already commits job->allocs/job->accounts, per
Requirement 1.1/1.2/1.3. sched_account_jobs()'s existing JOB_IS_RUN gate is
unmodified -- this closes bug-1624 by giving it the input it was missing: a
dispatched job now survives a round boundary in resource-in-use totals
regardless of how many rounds pass before worker confirmation arrives.
Regression test drives the real, unmodified sched_account_jobs() across a
simulated round boundary (resInUse reset, no confirmation event) and
independently reverting the one-line change reproduces bug-1624 exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add uint32_t dispatch_fail_count to the job_t wrapper struct (xce/types.h,
alongside submit_lsn, same in-memory-only/reset-on-failover precedent),
per Requirement 4.3. Data model only -- xce_job_t (wire type) untouched,
job.c untouched; increment/reset wiring and the bounded-retry threshold
check are tasks 3.1/3.3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New wal_entry_type WAL_JOB_DISPATCH=12, reusing wal_jobstate_t verbatim
(_pad=0, absolute-value semantics), wired into both apply_wal_entry()'s
and apply_committed_entry()'s existing job-state fall-through groups per
Requirement 2.4. Restart-replay test writes only WAL_JOB_SUBMIT+
WAL_JOB_DISPATCH to a real on-disk WAL and confirms job.state==RUN after
a fresh persist_recover(), independent of WAL_JOB_START.
Review found apply_wal_entry() has zero production call sites today
(persist_recover() and standby catch-up both route through
apply_committed_entry() post the 2026-07-10 ha_apply_single_entry
refactor) -- design.md's stale characterization corrected; the dual
case-label registration itself is unaffected and correct.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Define xce_dispatch_fail_site_t/xce_dispatch_fail_class_t and
xce_classify_dispatch_fail() in job.c, mapping each of dispatch_a_job_()'s
six post-lookup failure points to retryable or terminal per Requirement
4.1/4.2. Standalone, not yet wired into dispatch_a_job_() (deferred to
tasks 3.2/3.3). Also adds this spec's requirements/design/research/tasks
docs (first commit touching this spec).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The original motivating recording (scheduler-replay-20260820-070103.replay,
named in bug-1611) no longer exists anywhere on disk -- confirmed via
exhaustive filesystem search. Substituted a real, driven 24-round/12-job
resource-constrained recording using actual driver.c/builtin_fcfs/resource.c
code, replayed through the real, unmodified interactive REPL.
This also surfaced and honestly documents a real structural limit:
Recorded-Decision Reconciliation only succeeds when a job's recorded
DISPATCH decision and its JOB_STATE_CHANGE-to-RUN confirmation land in the
SAME replay round (reconciliation searches only that round's own decoded
data). Constructing that exact coincidence from organic, continuous
scheduling is structurally impossible -- a job can't be both "freshly PEND
for this round's picker" and "already RUN before this round's picker runs"
at once. One job (ANCHOR_ID) engineers this coincidence using entirely
real constructors (matching the same precedent tasks 3 and 4.1 already
established); the other 11 jobs arrive/contend/dispatch/finish with zero
special-casing across all 24 real rounds. Recorded in tasks.md's
Implementation Notes as a scope finding for follow-up: in a real deployment
with genuine dispatch-to-confirmation latency, this narrows the feature's
real-world coverage of bug-1611/1613's dominant manifestation.
RED phase: reverting task 3's reconciliation call site makes this test
fail with the exact bug-1611 symptom in its own assertion message, plus 3
sibling tests break too; restored and green. No production code touched
(final task in the feature -- all production work landed in task 3).
50/51 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Four checks in one run: (a) REPL-vs-batch decision equivalence unmodified
and still passing; (b) --sim-verify reports no false mismatch for a
recording with multiple already-resolved jobs across rounds (non-adjacent
memmove compaction, no cross-round bleed); (c) a replay starting from a
snapshot with a job already RUN-state and no in-window recorded decision
degrades gracefully (no crash, no fabricated value, a second ordinary job
unaffected) -- the first test to exercise reconciliation's "not found"
path through the real sched_sim_driver_run()/xce_sched_core_load_snapshot()
path rather than a hand-built core; (d) normal-mode behavior is byte-for-
byte unchanged (zero production diff, the strongest possible evidence).
Found and fixed a real test-authoring gap while building (b): builtin_fcfs
re-offers a still-PEND job as a fresh PEND every round (confirmed
pre-existing, unrelated, also visible in test_repl_equivalence.c's own
fixture) -- the first draft's round-2 fixture omitted this recurrence.
No production code touched (task 3 already completed the feature) --
git diff on sim_loop.c/resource.c/scheduler.c/driver.c/sim_verify.c is
empty. 49/50 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an integration test comparing round-by-round resInUse and scheduling
outcomes between a real driven normal-mode recording and a simulation-mode
replay of the identical contention scenario (worker capacity 10.0, jobs
wanting 8.0/5.0/2.0, plus a 4th job proving cross-round persistence).
Confirmed (by direct read of sim_picker_thread_fn(), independently
re-verified in review) that sim's live picker is structurally PEND-only --
it has no capacity-check/dispatch path for a freshly-evaluated job. This
means task 1.2's single-round fixture cannot be literally replayed-and-
diffed for full parity: that comparison would be tautological for
freshly-arriving jobs (sim always PENDs them regardless of accounting
correctness) and can't prove multi-round persistence anyway. Built instead
as a real recording plus a hand-crafted sim fixture modeling the identical
facts, including an already-resolved-before-replay job to exercise
reconciliation -- the same documented-carve-out approach test_sim_equivalence.c
already established for PEND-only proofs, applied here to reconciled jobs too.
No production code touched (task 3 already completed the feature). RED
phase proven by temporarily reverting task 3's reconciliation call site:
new test failed with "Expected 10 Was 0", restored and green.
48/49 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This is the integration point that makes the whole feature real
end-to-end. Calls sim_reconcile_unaccounted_jobs() (tasks 2.2-2.4) at
exactly the position design.md's mermaid diagram requires: after
xce_core_begin_session()'s reset, strictly before sched_account_jobs()
(so the same round's re-charge picks up every newly-reconstructed job,
Req 1.3/1.4) and before xce_core_signal_go() unblocks the live picker (so
still-pending jobs' capacity checks this round see correct resInUse, Req
2.1/2.2). decoded_session and record_ctx are the same per-round locals
sched_sim_driver_run() already declares -- no new plumbing.
sim_verify.c needs zero changes: reconciliation consumes/compacts
decoded_session->decisions before the verify drain loop runs each round,
which is what keeps the existing positional comparison correct (Req 5.2)
-- confirmed by extending test_sim_verify.c with a no-false-mismatch test
against a recording containing already-resolved jobs.
New end-to-end test proves the ordering matters for real: a reconciled
job's resInUse is charged before a live sched_check_host_satisfy() call
for a different job in the same round correctly denies it (5.0 wanted vs
2.0 remaining) and grants an exact-fit request (2.0) -- using the real
scheduler.c/resource.c accounting and capacity-check code, not stubs.
Found and fixed a test-authoring bug (get_resvals() returns nres on
success, not 0) during GREEN verification -- logged as bug-1622, a test
bug, not production.
47/48 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Feeds a reconciled job's decision (task 2.3) through the exact same
recording/print/REPL-check calls the existing decision-drain loop uses for
a freshly-drained decision, via make_dispatch_decision() -- the same
DISPATCH-construction idiom the live allocator path already uses.
Dedup (Requirement 3.3) needs no new mechanism: the job->accounts gate
from tasks 2.2/2.3 already prevents double-reporting, now proven at the
reporting layer too (zero recorded decisions / no console output for an
already-accounted job).
Required extending sim_reconcile_unaccounted_jobs()'s signature with a
third parameter, a nullable recording-context handle -- design.md's
Service Interface updated to match, with the confirmed reasoning: the
handle only exists as a stack-local inside the round loop, and task 3's
still-unwired call site sits in that same scope, so this is necessary
plumbing, not scope creep.
46/47 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends sim_reconcile_unaccounted_jobs()'s FOUND branch: takes ownership of
a matched recorded DISPATCH decision's allocation (job->allocs = entry's
alloc, not a copy -- mirrors scheduler.c's sched_commit_decision()
precedent exactly), builds job->accounts via make_job_account(), then
safely consumes the recorded entry -- nulling the source alloc pointer
BEFORE freeing the entry (preventing a double-free through job->allocs),
then compacting decoded_session->decisions via shift-down (memmove, never
swap-with-last, which would reorder the remaining entries and break the
positional --sim-verify comparison a later task depends on).
Fixed a real correctness bug found while implementing: the "not found"
check (i == n_decisions) goes stale once n_decisions can be decremented
mid-loop by compaction -- replaced with an explicit found flag.
Primary correctness bar: reconcile-then-release-through-normal-job-cleanup
is ASAN-clean. Verified with ASAN_OPTIONS=detect_leaks=1 (beyond this
suite's registered detect_leaks=0): zero double-free/UAF/overflow. Found
and logged (not fixed, out of boundary) a real pre-existing leak in
src/lib/misc.c's free_host_account() -- equally present on the live
dispatch path, bug-1621.
46/47 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t*, replay_session_t*)
to sim_loop.c -- not yet wired into the round boundary (task 3). Once per
call, finds every RUN-state job with job->accounts == NULL (the signature of
a job already resolved before this replay's own live decision-making ever
considered it) and searches this round's decoded recorded-decision stream
for a matching DISPATCH entry by jobid.
This task implements only the trigger + lookup + not-found path: when no
match exists, the job's allocation is left unknown (not fabricated), no
error is raised, and the job is safe to re-check next round (Requirement
4.1, 4.2). Match-found reconstruction (ownership transfer, freeing/
compacting the consumed entry, reporting) is tasks 2.3/2.4, not implemented
here -- decoded_session's decisions are read but not yet mutated.
6 new tests cover: not-found leaves job unaccounted, cross-round safety,
already-accounted jobs are skipped, a same-jobid PEND entry is correctly
not mistaken for a DISPATCH match, non-RUN-state jobs are gated out, and
NULL-argument safety. RED phase: compile-error-before/link-succeeds-after,
the correct RED variant for genuinely new code with no prior behavior.
46/47 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces sim_loop.c's monolithic xce_core_go() with the same
xce_core_begin_session() -> sched_account_jobs() -> xce_core_signal_go()
sequence driver.c already uses for normal-mode rounds, opening the same
interleaving window and calling the now-core-level accounting service
(task 1.1) at the same relative position. Error-handling for a failed
session start is preserved byte-for-byte (same cleanup, same rc source).
Both interactive and batch simulation runs go through this one call site.
This is bug-1613's wiring half: sim_loop.c now calls sched_account_jobs()
at all (it never did before). resInUse is still zero end-to-end after this
task alone, because job->accounts is never populated by any live path in
sim mode yet -- that's Recorded-Decision Reconciliation (tasks 2.2-2.4),
not yet implemented; a comment marks where it slots in.
Adds a call-count/session-window regression test proving the wiring itself
(RED-phase reproduced live via git stash + rebuild, matching bug-1613's
symptom exactly). 8 sibling white-box test files needed a link-satisfying
sched_account_jobs() stub, discovered via actual linker errors.
45/46 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a dual-use .replay fixture (3 jobs competing for insufficient worker
capacity) that later tasks need to prove capacity-aware parity under
contention: a real recorded normal-mode session (built by driving the
genuine sched_check_host_satisfy()/sched_commit_job_alloc() capacity/
accounting chain, not mocked) that is also directly replayable through the
real simulation driver. Two tests: fixture shape (job B genuinely denied
while A+C hold the full 10.0-unit capacity, verified against the decoded
recorded decision stream) and sim-driver replayability.
RED-phase proven: raising WORKER_CAPACITY to eliminate contention makes the
shape assertion fail as expected before reverting to the genuine-contention
value. 44/45 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Relocates sched_account_jobs()/sched_charge_account()/sched_uncharge_account()/
sched_update_account()/sched_update_res_accounts()/sched_commit_job_alloc()
from xce_scheduler_t* to xce_sched_core_t*, mirroring the sched_resolve_job_picker()
precedent, so the simulation driver can call the same accounting service the
normal-mode driver already uses. Pure signature relocation -- every body
reference was already sched->core.*, no logic change. Updates the 3 real call
sites (driver.c's normal-mode round, scheduler.c's sched_begin_session and
sched_commit_decision) and 7 existing tests' link-satisfying stub signatures.
Adds test_resource_accounting.c proving Requirement 1.4: the relocated
accounting produces identical resInUse/sharedInUse output whether called
against a core embedded in xce_scheduler_t or xce_sched_sim_driver_t.
43/44 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated recording.c sandbox-permission ordering issue).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Elevates the normal-mode/simulation-mode driver equivalence guarantee
from a one-line architectural description in CLAUDE.md to an enforced
steering constraint (scheduler-driver-equivalence.md), backed by
bug-1611/bug-1613 as concrete evidence of violation. Scopes the fix as
its own spec (scheduler-sim-decision-parity, requirements + design)
rather than reopening the closed scheduler-core spec or folding it
into scheduler-sim-repl-verbosity's explicitly narrower boundary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends bug-1588's job.c fix to the remaining raw-%lu jobid logging sites:
8 worker daemon files (xcecd_worker.c, controller_connector.c, uv_ipc.c,
xcecd_worker_utils.c, xcecd_dbus_manager.c, xcecd.c, main.c, xcecd_cgroups.c)
and src/controller/testsuite/test.c (xce_test, heavily used for controller
testing). 99 call sites converted to fmt_jobid()+%s, matching the existing
convention. Non-jobid %lu values (lsn, client id, byte offset) and raw-numeric
identifier/path construction (checkpoint dir, systemd unit name, XCE_JOBID
env var) are deliberately left untouched.
Verified via /xce-build worker and controller (asan, --incremental): both
compile clean with zero warnings tied to the change; worker suite 4/4 OK,
controller suite 4/5 OK (the one failure is the pre-existing bug-1561
overlayfs fdatasync flake, unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-dirty-fingerprint-nix-production-path.sh, reusing task
6.1's isolated-git-worktree technique. Runs nix build ./src#xcecore_debug
(no --impure) twice with different tracked-file edits to flake.nix,
cross-checking nix eval's eval-time myDirtyFingerprint value against the
real build log's mesonConfigurePhase flags: line for each.
This sandbox's nix build dispatches by default to an unreachable-in-
practice remote builder, making a directly-runnable binary impractical to
retrieve -- confirmed genuinely expected sandbox behavior. The eval+build-
log cross-check is not a weaker substitute: self.narHash's value is baked
as a literal string into the derivation at eval time, before any build
phase runs, so eval-time and build-time values cannot diverge by
construction. Forcing --builders '' (local-only) was investigated and
reaches real local compilation with a direct log match, but still can't
produce a fully linked binary here due to a second, unrelated network
dependency -- confirming no stronger approach is currently available.
Verified independently by review with real nix build invocations,
including a live-witnessed remote-builder connection attempt and a
genuine local-only run: two builds with different edits produce different
fingerprints in both eval and build-log evidence; reverting reproduces the
original value exactly; --impure never used; no git dependency in the
build sandbox; worktree list and flake.nix/nixSupport/ confirmed clean
before and after every run.
This is the final task of the dirty-content-fingerprint spec -- all 16
sub-tasks are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-dirty-fingerprint-startup-log-unchanged.sh, reusing
task 6.1's already-approved isolated-git-worktree technique. Builds
scheduler and apiserver for real on a genuinely dirty tree, starts both
daemons, and proves --version/-version show a real fingerprint while
their startup logs show the unchanged, fingerprint-free original format.
Apiserver's injected fingerprint reuses scheduler's own extracted value
(same dirty tree) rather than an independent recomputation -- verified
independently by review as legitimate, not tautological: apiserver has no
fingerprint-computation code of its own (only ever receives an externally
injected value via -ldflags), and the test still exercises apiserver's
own real composeVersionDisplay()/logger.Info code on a genuinely running
binary, with a round-trip check that would catch a real strip-and-append
bug.
Verified independently by review with a fresh full run plus a separately-
authored 4-case discrimination probe outside the script's own demo block.
No leftover processes, worktrees, or build dirs after either run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-dirty-fingerprint-distinguishability.sh, driving a real
xce-build.sh scheduler cycle through the 4 required --version fingerprint
states (clean/dirty-A/dirty-B-differs/reverted-A-matches). Uses a genuine
isolated git worktree (git worktree add --detach at a unique mktemp -u
path) to sidestep this actively-shared repo's ambient concurrent-editing
noise -- all file edits confined strictly inside the isolated worktree,
cleaned up via a trap registered before creation so it safely no-ops if
creation never happened.
Verified independently by review across two full live runs, including a
deliberately sabotaged copy forcing edit-B to reproduce edit-A's content:
the sabotaged run correctly failed with a clear message and exit 1,
proving genuine discriminating power. git worktree list and the real
repo's working tree confirmed byte-identical to baseline before and after
every run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds DIRTY_FINGERPRINT = "" to _build_info.py's committed fallback stub
and cli.py's separate ImportError fallback class. New
_compose_version_display() applies the strip-and-append rule via a
genuine anchored removesuffix("-dirty"), mirroring the C/Go
implementations (tasks 3.1/4.4/4.5).
Updates one pre-existing test whose hard-coded literal-GIT_COMMIT
assertion no longer holds once a fingerprint is present, to compute its
expected value via the production function instead -- independently
assessed by review as a legitimate strengthening, not a weakened check.
4 new dedicated unit tests independently guard the transform itself with
hard-coded literal assertions, including an anchored-suffix-not-blind-
substring regression case.
Verified independently by review (including a second, differently-
constructed mutation test) across both install paths via a full real
/xce-build xceadmin cycle: editable install shows "unknown, unknown" with
no import error; wheel built from the dirty tree shows <hash>-<fingerprint>,
never <hash>-dirty-<fingerprint>. 267 passed/3 skipped.
This completes all xceadmin work (tasks 5.1-5.2) for this spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Threads dirty_fingerprint through as an 8th positional arg from
xceadmin/meson.build's custom_target into meson-build.sh, which adds
DIRTY_FINGERPRINT to the generated _build_info.py with an independent
empty-string fallback (not "unknown", matching this feature's convention
everywhere else). GIT_COMMIT/BUILD_TIME's existing generation and
"unknown" fallback are completely untouched.
Verified independently by review across three full real build+test+wheel
cycles: dirty tree produces a real, non-empty fingerprint present
identically in the mirrored tree, build/lib copy, an installed venv, and
the packaged wheel, cross-checked byte-for-byte against that same build's
compile_commands.json; explicit-empty fingerprint with real commit/time
resolves DIRTY_FINGERPRINT to genuinely empty; all-empty scenario confirms
the two fallback conventions ("unknown" vs empty) coexist without
interference. 261 passed/3 skipped throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds DirtyFingerprint var and composeVersionDisplay() helper to
xcecli/cmd/main.go, a logical duplicate of apiserver's just-approved
implementation (task 4.4, same anchored strings.TrimSuffix strip-and-
append rule). Applied at the sole consumption point: the version string
fed into router.Execute(cm, version), which sets Cobra's rootCmd.Version.
router.go is untouched -- xcecli has no startup log to protect, already
established out of scope by the original build-time-versioning feature.
New main_test.go covers the same 6 cases as apiserver's tests. Verified
independently by review (including a second, independent mutation test)
with real builds across all 3 required scenarios and a full xcecli module
test suite run showing no regressions.
This completes all Go binary work (tasks 4.1-4.5) for this spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds DirtyFingerprint package var (default "") and a composeVersionDisplay()
helper implementing the same strip-and-append rule as the C daemons: strips
a trailing -dirty from GitCommit and appends -<fingerprint> when available,
via a genuine anchored strings.TrimSuffix, not a blind substring replace.
GitCommit's declaration and the logger.Info startup-log call site (with its
zap fields) are completely untouched.
New main_test.go covers 6 cases via table-driven tests. Verified independently
by review, including a self-constructed mutation test (distinct from the
implementer's own) proving the anchored-suffix regression test is genuinely
meaningful, a real build across all 5 ldflags scenarios, and a live daemon
startup-log capture confirming no leak into the log even on the same binary
whose -version output shows the fingerprint.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an independent DIRTY_FINGERPRINT computation to src/frontend/Makefile,
mirroring xce-build.sh's task 2.1 algorithm exactly, wired into
VERSION_LDFLAGS as -X main.DirtyFingerprint=. GIT_COMMIT/GIT_STATUS/
BUILD_TIME (including GIT_COMMIT's own -dirty suffix) are completely
untouched.
This took two rejected review rounds to get the failure-detection right:
round 1 found a `{ git diff HEAD; ...; } >file` grouped command whose exit
status only reflected the last command, silently swallowing a git diff
failure and fabricating a hash from incomplete data. Round 2 found the
identical bug class one step further down (git ls-files | sort | xargs
cat's pipe swallowing an ls-files failure). Fixed by pulling git ls-files
out of the pipe into its own directly-redirected, individually-checked
step, deliberately chosen over `set -o pipefail;` (which also worked)
because this repo's own build-deployment.md documents a past incident of
code silently relying on /bin/sh happening to be bash with no SHELL:=bash
pin.
One narrow, explicitly-disclosed residual gap remains (a sort-only
failure, not cat, in the trailing pipe) and was assessed as acceptable
low risk after independent verification across three review rounds.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both apiserver/meson.build and xcecli/meson.build add
XCE_DIRTY_FINGERPRINT=get_option('dirty_fingerprint'), unquoted,
identically positioned to the existing XCE_GIT_COMMIT/XCE_BUILD_TIME
entries.
Verified independently by review with a distinct test value: fresh
configure + real ninja builds of both binaries confirm the value reaches
go build's actual invocation (visible in go version -m's recorded -ldflags
build setting), matches the C-side -DVERSION_DIRTY_FINGERPRINT= from the
same configure, and degrades to empty identically to the two pre-existing
vars when omitted.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
go_build now reads XCE_DIRTY_FINGERPRINT (empty-safe) and appends
-X main.DirtyFingerprint=... to its existing -ldflags construction,
falling back to plain empty string when unset -- deliberately NOT the
"unknown" fallback GitCommit/BuildTime use, matching dirty_fingerprint's
empty-means-nothing-to-show convention everywhere else in this feature.
Existing GitCommit/BuildTime fallback logic is completely untouched.
Verified independently by review with real scratch Go builds traced via
bash -x: unset -> empty flag value (not fabricated); set -> exact value
carried through; all-unset -> GitCommit/BuildTime still fall back to
"unknown" while DirtyFingerprint stays empty, confirming the two
conventions coexist without interference.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Worker's hand-rolled OPT_VERSION case now composes via a genuinely
separate version_display_buf/xce_compose_version_ex() call, mirroring
task 3.2's already-approved controller/scheduler pattern exactly. Both
startup-log XCECD_LOG call sites (--peers and no-peers) are left
completely untouched, same xce_compose_version() call, same version_buf.
Verified independently by review with a real build and live daemon runs
in both branches: --version shows <commit>-<fingerprint> matching
compile_commands.json, -dirty correctly stripped not doubled; both
startup log branches show the old fingerprint-less format unchanged.
All three C daemons (controller, scheduler, worker) are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
controller/main.c and scheduler/src/main.c now compose argp_program_version
via a genuinely separate buffer/call to xce_compose_version_ex() (reading
STR(VERSION_DIRTY_FINGERPRINT), matching the existing commit_id/build_time
macro pattern). The startup-log composition call site -- same function,
same arguments, same buffer -- is left completely unchanged in both files.
controller/testsuite/test.c (xce_test, confirmed to have no startup log)
gets a direct in-place swap. All three files carry explicit comments
marking the --version/startup-log divergence as deliberate.
Verified independently by review with real builds and live daemon starts:
--version shows <commit>-<fingerprint> for all three binaries, matching
compile_commands.json; live startup logs for both daemons show the
unchanged old <commit>-dirty format with the fingerprint never present.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds xce_compose_version_ex() as a sibling of the existing, completely
untouched xce_compose_version(): when dirty_fingerprint is non-empty, it
strips a trailing -dirty from the commit field (anchored tail-slice
strcmp, not a substring search) and appends -<dirty_fingerprint> instead;
when empty/NULL, it delegates directly to the original function, so
output is structurally guaranteed byte-identical -- including the
Requirement 5.2 fallback case (dirty tree, fingerprint computation
failed) where commit_id's own pre-existing -dirty suffix already signals
"modified" with no new sentinel needed.
No main.c call sites touched -- consuming daemons (tasks 3.2/3.3) still
call the original xce_compose_version() and will switch to the new
function themselves.
Verified independently by review, full lib suite on both debug and ASAN
profiles: 301 tests, 0 failures, 0 ignored; suffix-anchoring, buffer
safety, and the empty-fingerprint delegation path all confirmed correct
by hand-tracing the implementation, not just trusting the test results.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds myDirtyFingerprint to flake.nix's existing eval-time let block,
computed as builtins.hashString "sha256" self.narHash truncated to 8 hex
chars (self.narHash is SRI base64, not hex -- rehashing is necessary),
gated on the same clean/dirty branch myCommitId already uses. No
preConfigure build-phase hook needed -- self.narHash has no pure-eval trap
unlike build_time's builtins.currentTime. Threaded through
nixSupport/default.nix into xce.nix/xcecli.nix's eval-time mesonFlags as
-Ddirty_fingerprint=, alongside the completely untouched -Dcommit_id=/
myCommitId and build_time's separate preConfigure mechanism.
Verified independently by review with real nix build/nix log inspection
on both xcecore_debug and xcecli: real, non-empty, content-sensitive
values reach mesonConfigurePhase, cross-checked deterministic via an
independent Python recomputation, with commit_id fully unaffected. Full
artifact completion is blocked by a pre-existing, unrelated sandbox
network limitation (confirmed recurring on two disjoint Go module sets),
not caused by this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an independent dirty_fingerprint computation to xce-build.sh: sha256
over git diff HEAD's output plus the sorted content of untracked,
non-ignored files (git ls-files --others --exclude-standard), truncated to
8 hex chars. Resolves to empty on a clean tree or if the hash computation
itself fails -- commit_id's own computation and -dirty suffix are left
completely untouched. Zero working-tree/index mutation.
Verified independently by review with real builds (HEAD pinned to control
for concurrent background commits): distinguishes different uncommitted
content, reproduces identical fingerprints for identical content, excludes
.gitignore'd files, degrades gracefully under 3 independently-injected
failure modes without aborting the build, applies to every profile, and
leaves the original feature's regression script passing unmodified.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a dirty_fingerprint string meson option (default '') alongside the
already-shipped build-time-versioning feature's commit_id/build_time
options, and threads it through add_project_arguments() as
VERSION_DIRTY_FINGERPRINT for every C component. Purely additive,
non-behavioral plumbing -- nothing consumes the new define yet.
Verified independently by review: a bare meson setup with zero -D
options configures cleanly, and all 542 C compile commands carry the new
(empty) define, matching the existing commit_id/build_time defines'
counts exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-xceadmin-build-install-version.sh (same house style as
tasks 6.1/6.2), which runs xceadmin's full real meson-driven build cycle
(schema sync, venv install, pytest, wheel build), installs the resulting
wheel into a fresh scratch venv, and separately does an editable install
into a second fresh scratch venv. Asserts the wheel install carries real,
non-"unknown" commit id and build time while the editable install carries
the committed "unknown"/"unknown" fallback stub, and explicitly asserts
the two differ -- proving the generated build-info module genuinely
replaces the fallback rather than both installs coincidentally agreeing.
Verified independently by review with a full from-scratch re-run: 261
passed/3 skipped (matching tasks 5.1/5.2's established baseline), both
install paths correct, and the demonstration block's crux case (identical
build-info values must be rejected by the differ-check) confirmed correct.
This is the final task of the build-time-versioning spec; all 13 sub-tasks
are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
