Four checks in one run: (a) REPL-vs-batch decision equivalence unmodified
and still passing; (b) --sim-verify reports no false mismatch for a
recording with multiple already-resolved jobs across rounds (non-adjacent
memmove compaction, no cross-round bleed); (c) a replay starting from a
snapshot with a job already RUN-state and no in-window recorded decision
degrades gracefully (no crash, no fabricated value, a second ordinary job
unaffected) -- the first test to exercise reconciliation's "not found"
path through the real sched_sim_driver_run()/xce_sched_core_load_snapshot()
path rather than a hand-built core; (d) normal-mode behavior is byte-for-
byte unchanged (zero production diff, the strongest possible evidence).
Found and fixed a real test-authoring gap while building (b): builtin_fcfs
re-offers a still-PEND job as a fresh PEND every round (confirmed
pre-existing, unrelated, also visible in test_repl_equivalence.c's own
fixture) -- the first draft's round-2 fixture omitted this recurrence.
No production code touched (task 3 already completed the feature) --
git diff on sim_loop.c/resource.c/scheduler.c/driver.c/sim_verify.c is
empty. 49/50 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an integration test comparing round-by-round resInUse and scheduling
outcomes between a real driven normal-mode recording and a simulation-mode
replay of the identical contention scenario (worker capacity 10.0, jobs
wanting 8.0/5.0/2.0, plus a 4th job proving cross-round persistence).
Confirmed (by direct read of sim_picker_thread_fn(), independently
re-verified in review) that sim's live picker is structurally PEND-only --
it has no capacity-check/dispatch path for a freshly-evaluated job. This
means task 1.2's single-round fixture cannot be literally replayed-and-
diffed for full parity: that comparison would be tautological for
freshly-arriving jobs (sim always PENDs them regardless of accounting
correctness) and can't prove multi-round persistence anyway. Built instead
as a real recording plus a hand-crafted sim fixture modeling the identical
facts, including an already-resolved-before-replay job to exercise
reconciliation -- the same documented-carve-out approach test_sim_equivalence.c
already established for PEND-only proofs, applied here to reconciled jobs too.
No production code touched (task 3 already completed the feature). RED
phase proven by temporarily reverting task 3's reconciliation call site:
new test failed with "Expected 10 Was 0", restored and green.
48/49 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This is the integration point that makes the whole feature real
end-to-end. Calls sim_reconcile_unaccounted_jobs() (tasks 2.2-2.4) at
exactly the position design.md's mermaid diagram requires: after
xce_core_begin_session()'s reset, strictly before sched_account_jobs()
(so the same round's re-charge picks up every newly-reconstructed job,
Req 1.3/1.4) and before xce_core_signal_go() unblocks the live picker (so
still-pending jobs' capacity checks this round see correct resInUse, Req
2.1/2.2). decoded_session and record_ctx are the same per-round locals
sched_sim_driver_run() already declares -- no new plumbing.
sim_verify.c needs zero changes: reconciliation consumes/compacts
decoded_session->decisions before the verify drain loop runs each round,
which is what keeps the existing positional comparison correct (Req 5.2)
-- confirmed by extending test_sim_verify.c with a no-false-mismatch test
against a recording containing already-resolved jobs.
New end-to-end test proves the ordering matters for real: a reconciled
job's resInUse is charged before a live sched_check_host_satisfy() call
for a different job in the same round correctly denies it (5.0 wanted vs
2.0 remaining) and grants an exact-fit request (2.0) -- using the real
scheduler.c/resource.c accounting and capacity-check code, not stubs.
Found and fixed a test-authoring bug (get_resvals() returns nres on
success, not 0) during GREEN verification -- logged as bug-1622, a test
bug, not production.
47/48 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Feeds a reconciled job's decision (task 2.3) through the exact same
recording/print/REPL-check calls the existing decision-drain loop uses for
a freshly-drained decision, via make_dispatch_decision() -- the same
DISPATCH-construction idiom the live allocator path already uses.
Dedup (Requirement 3.3) needs no new mechanism: the job->accounts gate
from tasks 2.2/2.3 already prevents double-reporting, now proven at the
reporting layer too (zero recorded decisions / no console output for an
already-accounted job).
Required extending sim_reconcile_unaccounted_jobs()'s signature with a
third parameter, a nullable recording-context handle -- design.md's
Service Interface updated to match, with the confirmed reasoning: the
handle only exists as a stack-local inside the round loop, and task 3's
still-unwired call site sits in that same scope, so this is necessary
plumbing, not scope creep.
46/47 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends sim_reconcile_unaccounted_jobs()'s FOUND branch: takes ownership of
a matched recorded DISPATCH decision's allocation (job->allocs = entry's
alloc, not a copy -- mirrors scheduler.c's sched_commit_decision()
precedent exactly), builds job->accounts via make_job_account(), then
safely consumes the recorded entry -- nulling the source alloc pointer
BEFORE freeing the entry (preventing a double-free through job->allocs),
then compacting decoded_session->decisions via shift-down (memmove, never
swap-with-last, which would reorder the remaining entries and break the
positional --sim-verify comparison a later task depends on).
Fixed a real correctness bug found while implementing: the "not found"
check (i == n_decisions) goes stale once n_decisions can be decremented
mid-loop by compaction -- replaced with an explicit found flag.
Primary correctness bar: reconcile-then-release-through-normal-job-cleanup
is ASAN-clean. Verified with ASAN_OPTIONS=detect_leaks=1 (beyond this
suite's registered detect_leaks=0): zero double-free/UAF/overflow. Found
and logged (not fixed, out of boundary) a real pre-existing leak in
src/lib/misc.c's free_host_account() -- equally present on the live
dispatch path, bug-1621.
46/47 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t*, replay_session_t*)
to sim_loop.c -- not yet wired into the round boundary (task 3). Once per
call, finds every RUN-state job with job->accounts == NULL (the signature of
a job already resolved before this replay's own live decision-making ever
considered it) and searches this round's decoded recorded-decision stream
for a matching DISPATCH entry by jobid.
This task implements only the trigger + lookup + not-found path: when no
match exists, the job's allocation is left unknown (not fabricated), no
error is raised, and the job is safe to re-check next round (Requirement
4.1, 4.2). Match-found reconstruction (ownership transfer, freeing/
compacting the consumed entry, reporting) is tasks 2.3/2.4, not implemented
here -- decoded_session's decisions are read but not yet mutated.
6 new tests cover: not-found leaves job unaccounted, cross-round safety,
already-accounted jobs are skipped, a same-jobid PEND entry is correctly
not mistaken for a DISPATCH match, non-RUN-state jobs are gated out, and
NULL-argument safety. RED phase: compile-error-before/link-succeeds-after,
the correct RED variant for genuinely new code with no prior behavior.
46/47 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces sim_loop.c's monolithic xce_core_go() with the same
xce_core_begin_session() -> sched_account_jobs() -> xce_core_signal_go()
sequence driver.c already uses for normal-mode rounds, opening the same
interleaving window and calling the now-core-level accounting service
(task 1.1) at the same relative position. Error-handling for a failed
session start is preserved byte-for-byte (same cleanup, same rc source).
Both interactive and batch simulation runs go through this one call site.
This is bug-1613's wiring half: sim_loop.c now calls sched_account_jobs()
at all (it never did before). resInUse is still zero end-to-end after this
task alone, because job->accounts is never populated by any live path in
sim mode yet -- that's Recorded-Decision Reconciliation (tasks 2.2-2.4),
not yet implemented; a comment marks where it slots in.
Adds a call-count/session-window regression test proving the wiring itself
(RED-phase reproduced live via git stash + rebuild, matching bug-1613's
symptom exactly). 8 sibling white-box test files needed a link-satisfying
sched_account_jobs() stub, discovered via actual linker errors.
45/46 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a dual-use .replay fixture (3 jobs competing for insufficient worker
capacity) that later tasks need to prove capacity-aware parity under
contention: a real recorded normal-mode session (built by driving the
genuine sched_check_host_satisfy()/sched_commit_job_alloc() capacity/
accounting chain, not mocked) that is also directly replayable through the
real simulation driver. Two tests: fixture shape (job B genuinely denied
while A+C hold the full 10.0-unit capacity, verified against the decoded
recorded decision stream) and sim-driver replayability.
RED-phase proven: raising WORKER_CAPACITY to eliminate contention makes the
shape assertion fail as expected before reverting to the genuine-contention
value. 44/45 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Relocates sched_account_jobs()/sched_charge_account()/sched_uncharge_account()/
sched_update_account()/sched_update_res_accounts()/sched_commit_job_alloc()
from xce_scheduler_t* to xce_sched_core_t*, mirroring the sched_resolve_job_picker()
precedent, so the simulation driver can call the same accounting service the
normal-mode driver already uses. Pure signature relocation -- every body
reference was already sched->core.*, no logic change. Updates the 3 real call
sites (driver.c's normal-mode round, scheduler.c's sched_begin_session and
sched_commit_decision) and 7 existing tests' link-satisfying stub signatures.
Adds test_resource_accounting.c proving Requirement 1.4: the relocated
accounting produces identical resInUse/sharedInUse output whether called
against a core embedded in xce_scheduler_t or xce_sched_sim_driver_t.
43/44 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated recording.c sandbox-permission ordering issue).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Elevates the normal-mode/simulation-mode driver equivalence guarantee
from a one-line architectural description in CLAUDE.md to an enforced
steering constraint (scheduler-driver-equivalence.md), backed by
bug-1611/bug-1613 as concrete evidence of violation. Scopes the fix as
its own spec (scheduler-sim-decision-parity, requirements + design)
rather than reopening the closed scheduler-core spec or folding it
into scheduler-sim-repl-verbosity's explicitly narrower boundary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends bug-1588's job.c fix to the remaining raw-%lu jobid logging sites:
8 worker daemon files (xcecd_worker.c, controller_connector.c, uv_ipc.c,
xcecd_worker_utils.c, xcecd_dbus_manager.c, xcecd.c, main.c, xcecd_cgroups.c)
and src/controller/testsuite/test.c (xce_test, heavily used for controller
testing). 99 call sites converted to fmt_jobid()+%s, matching the existing
convention. Non-jobid %lu values (lsn, client id, byte offset) and raw-numeric
identifier/path construction (checkpoint dir, systemd unit name, XCE_JOBID
env var) are deliberately left untouched.
Verified via /xce-build worker and controller (asan, --incremental): both
compile clean with zero warnings tied to the change; worker suite 4/4 OK,
controller suite 4/5 OK (the one failure is the pre-existing bug-1561
overlayfs fdatasync flake, unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-dirty-fingerprint-nix-production-path.sh, reusing task
6.1's isolated-git-worktree technique. Runs nix build ./src#xcecore_debug
(no --impure) twice with different tracked-file edits to flake.nix,
cross-checking nix eval's eval-time myDirtyFingerprint value against the
real build log's mesonConfigurePhase flags: line for each.
This sandbox's nix build dispatches by default to an unreachable-in-
practice remote builder, making a directly-runnable binary impractical to
retrieve -- confirmed genuinely expected sandbox behavior. The eval+build-
log cross-check is not a weaker substitute: self.narHash's value is baked
as a literal string into the derivation at eval time, before any build
phase runs, so eval-time and build-time values cannot diverge by
construction. Forcing --builders '' (local-only) was investigated and
reaches real local compilation with a direct log match, but still can't
produce a fully linked binary here due to a second, unrelated network
dependency -- confirming no stronger approach is currently available.
Verified independently by review with real nix build invocations,
including a live-witnessed remote-builder connection attempt and a
genuine local-only run: two builds with different edits produce different
fingerprints in both eval and build-log evidence; reverting reproduces the
original value exactly; --impure never used; no git dependency in the
build sandbox; worktree list and flake.nix/nixSupport/ confirmed clean
before and after every run.
This is the final task of the dirty-content-fingerprint spec -- all 16
sub-tasks are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-dirty-fingerprint-startup-log-unchanged.sh, reusing
task 6.1's already-approved isolated-git-worktree technique. Builds
scheduler and apiserver for real on a genuinely dirty tree, starts both
daemons, and proves --version/-version show a real fingerprint while
their startup logs show the unchanged, fingerprint-free original format.
Apiserver's injected fingerprint reuses scheduler's own extracted value
(same dirty tree) rather than an independent recomputation -- verified
independently by review as legitimate, not tautological: apiserver has no
fingerprint-computation code of its own (only ever receives an externally
injected value via -ldflags), and the test still exercises apiserver's
own real composeVersionDisplay()/logger.Info code on a genuinely running
binary, with a round-trip check that would catch a real strip-and-append
bug.
Verified independently by review with a fresh full run plus a separately-
authored 4-case discrimination probe outside the script's own demo block.
No leftover processes, worktrees, or build dirs after either run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-dirty-fingerprint-distinguishability.sh, driving a real
xce-build.sh scheduler cycle through the 4 required --version fingerprint
states (clean/dirty-A/dirty-B-differs/reverted-A-matches). Uses a genuine
isolated git worktree (git worktree add --detach at a unique mktemp -u
path) to sidestep this actively-shared repo's ambient concurrent-editing
noise -- all file edits confined strictly inside the isolated worktree,
cleaned up via a trap registered before creation so it safely no-ops if
creation never happened.
Verified independently by review across two full live runs, including a
deliberately sabotaged copy forcing edit-B to reproduce edit-A's content:
the sabotaged run correctly failed with a clear message and exit 1,
proving genuine discriminating power. git worktree list and the real
repo's working tree confirmed byte-identical to baseline before and after
every run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds DIRTY_FINGERPRINT = "" to _build_info.py's committed fallback stub
and cli.py's separate ImportError fallback class. New
_compose_version_display() applies the strip-and-append rule via a
genuine anchored removesuffix("-dirty"), mirroring the C/Go
implementations (tasks 3.1/4.4/4.5).
Updates one pre-existing test whose hard-coded literal-GIT_COMMIT
assertion no longer holds once a fingerprint is present, to compute its
expected value via the production function instead -- independently
assessed by review as a legitimate strengthening, not a weakened check.
4 new dedicated unit tests independently guard the transform itself with
hard-coded literal assertions, including an anchored-suffix-not-blind-
substring regression case.
Verified independently by review (including a second, differently-
constructed mutation test) across both install paths via a full real
/xce-build xceadmin cycle: editable install shows "unknown, unknown" with
no import error; wheel built from the dirty tree shows <hash>-<fingerprint>,
never <hash>-dirty-<fingerprint>. 267 passed/3 skipped.
This completes all xceadmin work (tasks 5.1-5.2) for this spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Threads dirty_fingerprint through as an 8th positional arg from
xceadmin/meson.build's custom_target into meson-build.sh, which adds
DIRTY_FINGERPRINT to the generated _build_info.py with an independent
empty-string fallback (not "unknown", matching this feature's convention
everywhere else). GIT_COMMIT/BUILD_TIME's existing generation and
"unknown" fallback are completely untouched.
Verified independently by review across three full real build+test+wheel
cycles: dirty tree produces a real, non-empty fingerprint present
identically in the mirrored tree, build/lib copy, an installed venv, and
the packaged wheel, cross-checked byte-for-byte against that same build's
compile_commands.json; explicit-empty fingerprint with real commit/time
resolves DIRTY_FINGERPRINT to genuinely empty; all-empty scenario confirms
the two fallback conventions ("unknown" vs empty) coexist without
interference. 261 passed/3 skipped throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds DirtyFingerprint var and composeVersionDisplay() helper to
xcecli/cmd/main.go, a logical duplicate of apiserver's just-approved
implementation (task 4.4, same anchored strings.TrimSuffix strip-and-
append rule). Applied at the sole consumption point: the version string
fed into router.Execute(cm, version), which sets Cobra's rootCmd.Version.
router.go is untouched -- xcecli has no startup log to protect, already
established out of scope by the original build-time-versioning feature.
New main_test.go covers the same 6 cases as apiserver's tests. Verified
independently by review (including a second, independent mutation test)
with real builds across all 3 required scenarios and a full xcecli module
test suite run showing no regressions.
This completes all Go binary work (tasks 4.1-4.5) for this spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds DirtyFingerprint package var (default "") and a composeVersionDisplay()
helper implementing the same strip-and-append rule as the C daemons: strips
a trailing -dirty from GitCommit and appends -<fingerprint> when available,
via a genuine anchored strings.TrimSuffix, not a blind substring replace.
GitCommit's declaration and the logger.Info startup-log call site (with its
zap fields) are completely untouched.
New main_test.go covers 6 cases via table-driven tests. Verified independently
by review, including a self-constructed mutation test (distinct from the
implementer's own) proving the anchored-suffix regression test is genuinely
meaningful, a real build across all 5 ldflags scenarios, and a live daemon
startup-log capture confirming no leak into the log even on the same binary
whose -version output shows the fingerprint.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an independent DIRTY_FINGERPRINT computation to src/frontend/Makefile,
mirroring xce-build.sh's task 2.1 algorithm exactly, wired into
VERSION_LDFLAGS as -X main.DirtyFingerprint=. GIT_COMMIT/GIT_STATUS/
BUILD_TIME (including GIT_COMMIT's own -dirty suffix) are completely
untouched.
This took two rejected review rounds to get the failure-detection right:
round 1 found a `{ git diff HEAD; ...; } >file` grouped command whose exit
status only reflected the last command, silently swallowing a git diff
failure and fabricating a hash from incomplete data. Round 2 found the
identical bug class one step further down (git ls-files | sort | xargs
cat's pipe swallowing an ls-files failure). Fixed by pulling git ls-files
out of the pipe into its own directly-redirected, individually-checked
step, deliberately chosen over `set -o pipefail;` (which also worked)
because this repo's own build-deployment.md documents a past incident of
code silently relying on /bin/sh happening to be bash with no SHELL:=bash
pin.
One narrow, explicitly-disclosed residual gap remains (a sort-only
failure, not cat, in the trailing pipe) and was assessed as acceptable
low risk after independent verification across three review rounds.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both apiserver/meson.build and xcecli/meson.build add
XCE_DIRTY_FINGERPRINT=get_option('dirty_fingerprint'), unquoted,
identically positioned to the existing XCE_GIT_COMMIT/XCE_BUILD_TIME
entries.
Verified independently by review with a distinct test value: fresh
configure + real ninja builds of both binaries confirm the value reaches
go build's actual invocation (visible in go version -m's recorded -ldflags
build setting), matches the C-side -DVERSION_DIRTY_FINGERPRINT= from the
same configure, and degrades to empty identically to the two pre-existing
vars when omitted.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
go_build now reads XCE_DIRTY_FINGERPRINT (empty-safe) and appends
-X main.DirtyFingerprint=... to its existing -ldflags construction,
falling back to plain empty string when unset -- deliberately NOT the
"unknown" fallback GitCommit/BuildTime use, matching dirty_fingerprint's
empty-means-nothing-to-show convention everywhere else in this feature.
Existing GitCommit/BuildTime fallback logic is completely untouched.
Verified independently by review with real scratch Go builds traced via
bash -x: unset -> empty flag value (not fabricated); set -> exact value
carried through; all-unset -> GitCommit/BuildTime still fall back to
"unknown" while DirtyFingerprint stays empty, confirming the two
conventions coexist without interference.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Worker's hand-rolled OPT_VERSION case now composes via a genuinely
separate version_display_buf/xce_compose_version_ex() call, mirroring
task 3.2's already-approved controller/scheduler pattern exactly. Both
startup-log XCECD_LOG call sites (--peers and no-peers) are left
completely untouched, same xce_compose_version() call, same version_buf.
Verified independently by review with a real build and live daemon runs
in both branches: --version shows <commit>-<fingerprint> matching
compile_commands.json, -dirty correctly stripped not doubled; both
startup log branches show the old fingerprint-less format unchanged.
All three C daemons (controller, scheduler, worker) are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
controller/main.c and scheduler/src/main.c now compose argp_program_version
via a genuinely separate buffer/call to xce_compose_version_ex() (reading
STR(VERSION_DIRTY_FINGERPRINT), matching the existing commit_id/build_time
macro pattern). The startup-log composition call site -- same function,
same arguments, same buffer -- is left completely unchanged in both files.
controller/testsuite/test.c (xce_test, confirmed to have no startup log)
gets a direct in-place swap. All three files carry explicit comments
marking the --version/startup-log divergence as deliberate.
Verified independently by review with real builds and live daemon starts:
--version shows <commit>-<fingerprint> for all three binaries, matching
compile_commands.json; live startup logs for both daemons show the
unchanged old <commit>-dirty format with the fingerprint never present.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds xce_compose_version_ex() as a sibling of the existing, completely
untouched xce_compose_version(): when dirty_fingerprint is non-empty, it
strips a trailing -dirty from the commit field (anchored tail-slice
strcmp, not a substring search) and appends -<dirty_fingerprint> instead;
when empty/NULL, it delegates directly to the original function, so
output is structurally guaranteed byte-identical -- including the
Requirement 5.2 fallback case (dirty tree, fingerprint computation
failed) where commit_id's own pre-existing -dirty suffix already signals
"modified" with no new sentinel needed.
No main.c call sites touched -- consuming daemons (tasks 3.2/3.3) still
call the original xce_compose_version() and will switch to the new
function themselves.
Verified independently by review, full lib suite on both debug and ASAN
profiles: 301 tests, 0 failures, 0 ignored; suffix-anchoring, buffer
safety, and the empty-fingerprint delegation path all confirmed correct
by hand-tracing the implementation, not just trusting the test results.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds myDirtyFingerprint to flake.nix's existing eval-time let block,
computed as builtins.hashString "sha256" self.narHash truncated to 8 hex
chars (self.narHash is SRI base64, not hex -- rehashing is necessary),
gated on the same clean/dirty branch myCommitId already uses. No
preConfigure build-phase hook needed -- self.narHash has no pure-eval trap
unlike build_time's builtins.currentTime. Threaded through
nixSupport/default.nix into xce.nix/xcecli.nix's eval-time mesonFlags as
-Ddirty_fingerprint=, alongside the completely untouched -Dcommit_id=/
myCommitId and build_time's separate preConfigure mechanism.
Verified independently by review with real nix build/nix log inspection
on both xcecore_debug and xcecli: real, non-empty, content-sensitive
values reach mesonConfigurePhase, cross-checked deterministic via an
independent Python recomputation, with commit_id fully unaffected. Full
artifact completion is blocked by a pre-existing, unrelated sandbox
network limitation (confirmed recurring on two disjoint Go module sets),
not caused by this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an independent dirty_fingerprint computation to xce-build.sh: sha256
over git diff HEAD's output plus the sorted content of untracked,
non-ignored files (git ls-files --others --exclude-standard), truncated to
8 hex chars. Resolves to empty on a clean tree or if the hash computation
itself fails -- commit_id's own computation and -dirty suffix are left
completely untouched. Zero working-tree/index mutation.
Verified independently by review with real builds (HEAD pinned to control
for concurrent background commits): distinguishes different uncommitted
content, reproduces identical fingerprints for identical content, excludes
.gitignore'd files, degrades gracefully under 3 independently-injected
failure modes without aborting the build, applies to every profile, and
leaves the original feature's regression script passing unmodified.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a dirty_fingerprint string meson option (default '') alongside the
already-shipped build-time-versioning feature's commit_id/build_time
options, and threads it through add_project_arguments() as
VERSION_DIRTY_FINGERPRINT for every C component. Purely additive,
non-behavioral plumbing -- nothing consumes the new define yet.
Verified independently by review: a bare meson setup with zero -D
options configures cleanly, and all 542 C compile commands carry the new
(empty) define, matching the existing commit_id/build_time defines'
counts exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-xceadmin-build-install-version.sh (same house style as
tasks 6.1/6.2), which runs xceadmin's full real meson-driven build cycle
(schema sync, venv install, pytest, wheel build), installs the resulting
wheel into a fresh scratch venv, and separately does an editable install
into a second fresh scratch venv. Asserts the wheel install carries real,
non-"unknown" commit id and build time while the editable install carries
the committed "unknown"/"unknown" fallback stub, and explicitly asserts
the two differ -- proving the generated build-info module genuinely
replaces the fallback rather than both installs coincidentally agreeing.
Verified independently by review with a full from-scratch re-run: 261
passed/3 skipped (matching tasks 5.1/5.2's established baseline), both
install paths correct, and the demonstration block's crux case (identical
build-info values must be rejected by the differ-check) confirmed correct.
This is the final task of the build-time-versioning spec; all 13 sub-tasks
are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-version-startup-consistency.sh (same house style as
task 6.1's test-build-version-regression.sh), which builds xce_scheduler
and xce_apiserver for real, starts each daemon, and asserts the commit id
and build time reported by --version/-version are identical to the values
in the daemon's own startup log -- proving Requirement 2.2's guarantee
that the two can never disagree for the same artifact.
apiserver's build routes around two pre-existing, unrelated sandbox
blockers (bug-1597's missing CGO env in `make build-server`, and
go_build's hardcoded GOPROXY hanging in this sandbox) via a direct `go
build -mod=readonly -ldflags` invocation that computes commit_id/build_time
identically to go_build and injects them the same way; only the
orchestration layer differs, not what main.go (task 4.4, unmodified)
consumes. Verified independently by review, including live reproduction
of both build-path blockers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-build-version-regression.sh, a standalone script (same
convention as test-commit-durability.sh/test-heartbeat-tolerance.sh) that
runs real xce-build.sh scheduler builds in both asan and debug profiles,
extracts the commit id from xce_scheduler --version, and fails loudly if it
doesn't match a live git rev-parse --short HEAD (+ -dirty) -- the exact
staleness this feature was built to make visible without forensic git
commands.
Tolerates xce-build.sh's own known-unrelated flaky test exit (bug-1560) via
captured exit status rather than a blanket set -e bypass, so a genuine
compile failure still fails the check. A built-in demonstration proves the
comparison logic rejects a wrong hash, an empty commit id, and the literal
"unknown" fallback. Verified independently by review with a full from-
scratch re-run of both profiles.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Commits a fallback src/xceadmin/src/xceadmin/_build_info.py (GIT_COMMIT/
BUILD_TIME both "unknown") for install paths that never run meson-build.sh
(e.g. `pip install -e .`), and wires click.version_option() on cli.py's
top-level group. cli.py imports _build_info defensively (try/except
ImportError with an inline fallback) and composes the version string from
importlib.metadata.version("xceadmin") plus both build-info fields.
Verified independently by review: editable install shows "unknown" for
both fields with no import error; the meson-built wheel shows real,
matching commit id + build time; full xceadmin suite 261 passed/3 skipped
(258 baseline + 3 new tests); ruff clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
xceadmin/meson.build passes the resolved commit_id/build_time meson
options as two new positional args (matching this custom_target's
existing all-positional style) to meson-build.sh, which writes a small
_build_info.py (GIT_COMMIT/BUILD_TIME, independent per-field "unknown"
fallback) into the mirrored source tree's xceadmin package directory,
before the pip install step -- same before-install ordering as the
existing schema-sync step. A file-existence guard checks the exact path
just written and fails loudly rather than silently proceeding with a
missing/wrong module.
Live-verified via a full real build+test+wheel cycle (run independently
twice, once by implementation and once by review): the mirrored tree's
_build_info.py and the packaged wheel both contain real, matching,
non-"unknown" values (commit id matching git rev-parse --short HEAD plus
dirty suffix, real UTC timestamp); full pytest suite passes (258 passed,
3 skipped -- documented live-HA e2e tests, skip-by-design).
Task 5.1/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds GitCommit/BuildTime vars to xcecli's main.go. Cobra's rootCmd is
declared unexported in router.go (a different file than the task's
original main.go-only boundary), so main.go can't reach it directly --
resolved by extending router.go's sole exported entry point,
Execute(cm, version), to accept the composed version string and set
rootCmd.Version before rootCmd.Execute() runs. Cobra then auto-registers
--version/-v natively once rootCmd.Version is non-empty (confirmed via
reading Cobra v1.9.1 source directly) -- zero additional flag-parsing
code, as designed.
Live-verified via both build paths (meson xce-build.sh and make
build-cli): real commit id (matching git rev-parse --short HEAD plus
dirty suffix) and build time, exit 0.
xcecli's startup log deliberately not extended -- Requirement 3 scopes
that to the four daemons only; xcecli is a one-shot CLI. Corrects a
stale design.md line that had assumed otherwise.
Task 4.5/13 -- closes out the Go binary version display phase (tasks
4.1-4.5).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Declares GitCommit/BuildTime package vars (default "unknown", target of
task 4.1's -ldflags -X injection), a -version flag checked before
utils.PromptForTopDir()/InitLogger() (zero filesystem side effects), and
extends the existing "Initialize Logger done!" startup log line with
zap.String structured fields carrying both values, per design.md's
explicit mandate.
Live-verified end to end: -version output and the startup log's first
line are byte-identical, with a real commit id (matching
git rev-parse --short HEAD plus dirty suffix) and UTC timestamp -- the
first full round-trip proof in this spec (earlier Go tasks could only
verify via linked build-info metadata, since no consuming vars existed
yet).
Task 4.4/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dev-loop Go build path (no meson involved) now computes commit id (git,
-dirty suffixed, "unknown" on git failure) and build time (UTC date) via
Make := (computed once per invocation), passing the identical -ldflags
shape go_build (task 4.1) already establishes.
Live-verified: make build-cli embeds real, git-matching values (go
version -m). make build-server hits a pre-existing, unrelated CGO
failure (bug-1597, missing CGO_CFLAGS/CGO_LDFLAGS) -- confirmed via
stash A/B that this reproduces identically on the unmodified Makefile,
not a regression from this change. Reproduced apiserver's build
invocation manually with bug-1597's own documented workaround to prove
the ldflags mechanism itself works correctly.
Task 4.3/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apiserver/meson.build and xcecli/meson.build now pass
XCE_GIT_COMMIT/XCE_BUILD_TIME (from the project's commit_id/build_time
meson options) into go_build's environment, mirroring the existing
CURRENT_SOURCE_DIR/BUILD_ROOT env-passing mechanism. Deliberately
unquoted: go_build's fix_input_envs() only quote-strips
CURRENT_SOURCE_DIR/BUILD_ROOT, not these two -- quoting them would embed
literal quote characters and corrupt the -ldflags argument.
Live-verified: real values (matching git rev-parse --short HEAD plus
dirty suffix, and a real UTC timestamp) reach go build's actual
invocation and are embedded in both binaries' Go build-info metadata.
No main.GitCommit/main.BuildTime symbols exist in either binary yet
(confirmed via go tool nm) since apiserver/xcecli's main packages don't
declare those vars until tasks 4.4/4.5 -- expected, this task's scope is
the env-var plumbing only.
Task 4.2/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reads XCE_GIT_COMMIT/XCE_BUILD_TIME from the environment (empty-safe via
[ -z ... ], not just unset-safe -- a classic ${VAR:-default} gotcha
avoided), falls back to "unknown", and appends
-ldflags "-X main.GitCommit=... -X main.BuildTime=..." to the go build
invocation alongside the existing -gcflags.
Confirmed via live testing: -X on a main package var that doesn't exist
yet is silently ignored by the Go linker (not a build error), so this
task has no real ordering dependency on tasks 4.4/4.5 (which add the
consuming vars to apiserver/xcecli).
Task 4.1/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Worker uses plain getopt_long(), not argp, so it has no free --version
handling like controller/scheduler. Adds a long-only OPT_VERSION=257
option and a handling case that computes xce_compose_version() into a
static buffer and returns before any startup work (check_user()/
xcecd_init/xcecd_start). Both startup INFO log branches (--peers and
no-peers) extended to log the identical buffer.
Live-verified: --version and both startup-log branches are
byte-identical (captured via a real daemon run under sudo).
Also updates bug-1519 (pre-existing, unrelated ASAN "Joining already
joined thread" abort during worker shutdown): independently reproduced
under real root during this task's review, refining its prior "never
reproduced under real root" note -- not a userns/unshare artifact, still
unfixed, does not affect this task's code path.
Task 3.3/13 of build-time-versioning. This closes out the "Core: C
daemon version display" phase (tasks 3.1-3.3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Migrates argp_program_version in controller/main.c, scheduler/main.c,
and xce_test's test.c from the compile-time MK_VERSION/STR construction
to xce_compose_version(), computed into a static buffer and assigned
before argp_parse() runs (argp_program_version is a file-scope pointer,
not a constant expression it can be initialized from at compile time).
Each daemon's startup log now carries the identical version string
(never recomputed) on its first INFO line -- scheduler's BOTH
mutually-exclusive branches (--peers and no-peers), not just the one
literally named in the task text, since "when the daemon starts" is
unconditional.
Live-verified: --version and startup-log output are byte-identical for
both daemons across all branches. No remaining MK_VERSION callers
repo-wide.
Task 3.2/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds xce_version_field()/xce_compose_version() (utils.h/utils.c) to
compose a version string carrying commit id + build time, rendering
"unknown" for either field when its underlying meson-supplied define is
empty -- ready for controller/scheduler/worker to consume.
MK_VERSION/STR (the pre-existing, compile-time-only macro pair) can't
express this empty-to-"unknown" substitution at the preprocessor level,
so this is a runtime mechanism instead. Round 1 review REJECTED an
earlier version of this task for silently orphaning MK_VERSION/STR
(the task explicitly asked to "extend" them) and for leaving an
undocumented architectural trap: argp_program_version is a file-scope
const char* initialized at compile time, which cannot call a runtime
function -- consuming code must instead compute into a static buffer
inside main() and assign the pointer before argp_parse() runs. Round 2
adds an explicit SUPERSEDED doc comment on MK_VERSION/STR plus NOTE FOR
CALLERS guidance covering both points, and was APPROVED.
Also expands task 3.2's boundary to include controller/testsuite/test.c
(a third MK_VERSION/STR call site discovered during review) so no
caller of the superseded macro survives implementation.
5 new unit tests in test_utils.c (RED->GREEN, TDD).
Task 3.1/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
First attempt used builtins.currentTime at flake-evaluation time -- this
broke the plain, documented `nix build .#xcecore_debug` command entirely
(pure-eval strips builtins.currentTime, requiring --impure) and produced
a raw epoch-seconds string instead of UTC ISO-8601. Caught by task-local
review (REJECTED, round 1) before merge.
Fixed by moving the computation into each derivation's own build phase:
a preConfigure hook in xce.nix/xcecli.nix runs `date -u` (stdenv's
bundled coreutils, unaffected by pure-eval) and appends -Dbuild_time= to
mesonFlagsArray, with a fail-loud guard against empty output. commit_id's
existing self.shortRev/self.dirtyShortRev eval-time mechanism is
untouched. Verified via nix eval (no --impure) and nix build reaching
configure with real values on both xcecore_debug and xcecli (round 2,
APPROVED).
Task 2.2/13 of build-time-versioning. Also corrects design.md/research.md,
which described the now-superseded builtins.currentTime approach.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The dev-loop build path (/xce-build, the one behind the motivating
incident) never set -Dcommit_id= at all, so every locally-built binary
carried an empty commit id. xce-build.sh now computes commit_id (git
rev-parse --short HEAD, -dirty suffixed on uncommitted changes, unknown
fallback on a genuine git failure) and build_time (UTC date), appends
both to MESON_OPTS for every profile before meson setup, and warns
loudly to stderr if either ever resolves empty anyway.
Task 2.1/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add a build_time meson option (string, default '') sibling to the
existing commit_id option, threaded project-wide via the same
add_project_arguments() call that already defines VERSION_COMMIT_ID --
one call, one mechanism, no new plumbing. No consumer wired yet (later
tasks); a plain meson setup with neither option set still configures
cleanly with both defines resolving to empty string.
Task 1/13 of build-time-versioning (spec: .kiro/specs/build-time-versioning/).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
repl_print_event_summary() (interactive REPL event echo) now prefixes
each echoed event with "[sim] t=<ns> (<human>) session=<N>", using the
same xce_core_now()/repl_format_ts_human()/session_count sources "info
session" already reports. Uses session_count+1, not the raw count:
events stream in and are echoed individually before the ReplaySession/
begin_round message that increments session_count is even read (file
order is event, event, ..., ReplaySession), so an echoed event always
belongs to the session about to begin.
Add sim_prescan_session_count() (sim.c): scans the replay file from the
current position via shape-only classification (mirroring sim_loop.c's
own ReplayEvent/ReplaySession/SnapshotEntry dispatch), counts
ReplaySession messages, then rewinds via fseek back to the exact ftell()
position -- same rewind technique sim_locate_and_load_snapshot() already
uses. Wired into sched_sim_driver_start(): prints "[sim] replay file
contains N sessions" once at load, non-fatal on scan failure.
Add test_replay_load_reports_total_session_count; extend
test_step_echoes_events_before_session_decisions with t=/session=1
assertions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
repl_info_worker_one() ("info worker <name>") unconditionally showed a
total=/available= pair for every numeric per-worker resource, regardless
of whether it was actually consumable -- misleading for a static
per-worker attribute nothing ever allocates against.
repl_info_resource_one() ("info resource <name|all>") never reported a
consumable GLOBAL/SHARED resource's in-use amount at all, even though
core->sharedInUse (the cluster-wide mirror of a worker's resInUse)
already exists and is populated by the real resource-accounting code.
Both now gate on XCE_RESF_CONSUMABLE: consumable resources report
value=<capacity> used=<in-use>, non-consumable resources keep a single
value=<current> line. Also added the resource definition's missing
ordtype= field.
Add test_info_resource_reports_consumable_value_and_used and
test_info_worker_reports_consumable_value_and_used; update 3 pre-existing
tests whose asserted output shape changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apply_committed_entry()'s shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/
CKPT/PREASON dispatch case only ever called the generic, state-only
apply_job_update_transition(j, state) -- start_time/finish_time/run_time
were never touched, so `xcecli joblist` always showed StartTime=0 for
RUN/DONE/EXIT jobs and FinishTime=0 for DONE jobs (EXIT jobs got a
finish_time only via the separate, richer WAL_JOB_FAIL dispatch path).
Fix is additive: stamp time(NULL) keyed on wal_type right after
apply_job_update_transition() succeeds, still under the same
lock(j, RTTI_LOCK_WRITE) -- WAL_JOB_START sets start_time, WAL_JOB_FINISH
sets finish_time and derives run_time. No change to
apply_job_update_transition()'s signature or the dispatch structure.
Add test_ace_job_start_sets_start_time and
test_ace_job_finish_sets_finish_time_and_run_time to
test_apply_committed_entry.c, following the existing tace_seed_job
fixture pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
makeGroup()'s existing-group update path (hit whenever a config-push
re-applies a group name that already has members, e.g. a retried config
push during controller failover) released xce_g_member_t.u.name via
FREEUP() (plain free()). u.name is allocated by makeString(), an
rtti-managed, refcounted pointer that is not a malloc()-base address --
freeing it directly corrupts the heap. group_killer() in the same file
already releases the identical field correctly via dereference().
Switch both FREEUP(m->u.name) call sites to dereference(m->u.name) to
match. Add a regression test that re-applies config for the same group
name twice, reproducing the exact ASAN bad-free at group.c:234 pre-fix.
This is the root cause of the reproducible xce_scheduler
"free(): invalid pointer" SIGABRT crashes seen during 3-attempt
controller-failover config-push retries (8 confirmed core dumps).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Correct filtering (bug-1593: only XCE_RESF_GLOBAL/XCE_RESF_SHARED
resources list under "all") but confusing UX: a session with only
host-based resources (e.g. just "slots") produced zero output lines,
indistinguishable from a REPL/dispatch bug. Now prints an explicit
"no global/shared resources defined" message when the count is zero.
xce_scheduler link-wraps printf (-Wl,--wrap=printf, src/meson.build's
wrap_link_args) to route every plain printf() call through tlog, stamping
the full "[timestamp] [pid] [LEVEL] [xce_vprintf]" prefix. sim_print_decision()
(sim_loop.c) used plain printf() for its DISPATCH/PEND/PREEMPT lines, so
those lines carried that prefix while every sibling "[sim] ..." line in
the same REPL/sim output (event echoes, breakpoint hits, banners, info
command output -- all fprintf(stdout, ...)) stayed prefix-free. Found via
a live deployment log. Switched to fprintf(stdout, ...), matching the
established convention everywhere else in this codebase.
Invisible to the entire prior test suite for a structural reason: only
the real xce_scheduler executable links with wrap_link_args, not the
white-box unit-test binaries, so printf() and fprintf(stdout, ...) were
indistinguishable inside any unit test. Added a mechanical regression
guard instead of a runtime test: test_no_bare_printf.c source-scans every
src/scheduler/src/*.c and src/scheduler/src/plugins/*.c file for a bare
printf( call (same source-text-scan genre as test_core_isolation.c).
