Adds src/tools/test-dirty-fingerprint-distinguishability.sh, driving a real
xce-build.sh scheduler cycle through the 4 required --version fingerprint
states (clean/dirty-A/dirty-B-differs/reverted-A-matches). Uses a genuine
isolated git worktree (git worktree add --detach at a unique mktemp -u
path) to sidestep this actively-shared repo's ambient concurrent-editing
noise -- all file edits confined strictly inside the isolated worktree,
cleaned up via a trap registered before creation so it safely no-ops if
creation never happened.
Verified independently by review across two full live runs, including a
deliberately sabotaged copy forcing edit-B to reproduce edit-A's content:
the sabotaged run correctly failed with a clear message and exit 1,
proving genuine discriminating power. git worktree list and the real
repo's working tree confirmed byte-identical to baseline before and after
every run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds DIRTY_FINGERPRINT = "" to _build_info.py's committed fallback stub
and cli.py's separate ImportError fallback class. New
_compose_version_display() applies the strip-and-append rule via a
genuine anchored removesuffix("-dirty"), mirroring the C/Go
implementations (tasks 3.1/4.4/4.5).
Updates one pre-existing test whose hard-coded literal-GIT_COMMIT
assertion no longer holds once a fingerprint is present, to compute its
expected value via the production function instead -- independently
assessed by review as a legitimate strengthening, not a weakened check.
4 new dedicated unit tests independently guard the transform itself with
hard-coded literal assertions, including an anchored-suffix-not-blind-
substring regression case.
Verified independently by review (including a second, differently-
constructed mutation test) across both install paths via a full real
/xce-build xceadmin cycle: editable install shows "unknown, unknown" with
no import error; wheel built from the dirty tree shows <hash>-<fingerprint>,
never <hash>-dirty-<fingerprint>. 267 passed/3 skipped.
This completes all xceadmin work (tasks 5.1-5.2) for this spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Threads dirty_fingerprint through as an 8th positional arg from
xceadmin/meson.build's custom_target into meson-build.sh, which adds
DIRTY_FINGERPRINT to the generated _build_info.py with an independent
empty-string fallback (not "unknown", matching this feature's convention
everywhere else). GIT_COMMIT/BUILD_TIME's existing generation and
"unknown" fallback are completely untouched.
Verified independently by review across three full real build+test+wheel
cycles: dirty tree produces a real, non-empty fingerprint present
identically in the mirrored tree, build/lib copy, an installed venv, and
the packaged wheel, cross-checked byte-for-byte against that same build's
compile_commands.json; explicit-empty fingerprint with real commit/time
resolves DIRTY_FINGERPRINT to genuinely empty; all-empty scenario confirms
the two fallback conventions ("unknown" vs empty) coexist without
interference. 261 passed/3 skipped throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds DirtyFingerprint var and composeVersionDisplay() helper to
xcecli/cmd/main.go, a logical duplicate of apiserver's just-approved
implementation (task 4.4, same anchored strings.TrimSuffix strip-and-
append rule). Applied at the sole consumption point: the version string
fed into router.Execute(cm, version), which sets Cobra's rootCmd.Version.
router.go is untouched -- xcecli has no startup log to protect, already
established out of scope by the original build-time-versioning feature.
New main_test.go covers the same 6 cases as apiserver's tests. Verified
independently by review (including a second, independent mutation test)
with real builds across all 3 required scenarios and a full xcecli module
test suite run showing no regressions.
This completes all Go binary work (tasks 4.1-4.5) for this spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds DirtyFingerprint package var (default "") and a composeVersionDisplay()
helper implementing the same strip-and-append rule as the C daemons: strips
a trailing -dirty from GitCommit and appends -<fingerprint> when available,
via a genuine anchored strings.TrimSuffix, not a blind substring replace.
GitCommit's declaration and the logger.Info startup-log call site (with its
zap fields) are completely untouched.
New main_test.go covers 6 cases via table-driven tests. Verified independently
by review, including a self-constructed mutation test (distinct from the
implementer's own) proving the anchored-suffix regression test is genuinely
meaningful, a real build across all 5 ldflags scenarios, and a live daemon
startup-log capture confirming no leak into the log even on the same binary
whose -version output shows the fingerprint.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an independent DIRTY_FINGERPRINT computation to src/frontend/Makefile,
mirroring xce-build.sh's task 2.1 algorithm exactly, wired into
VERSION_LDFLAGS as -X main.DirtyFingerprint=. GIT_COMMIT/GIT_STATUS/
BUILD_TIME (including GIT_COMMIT's own -dirty suffix) are completely
untouched.
This took two rejected review rounds to get the failure-detection right:
round 1 found a `{ git diff HEAD; ...; } >file` grouped command whose exit
status only reflected the last command, silently swallowing a git diff
failure and fabricating a hash from incomplete data. Round 2 found the
identical bug class one step further down (git ls-files | sort | xargs
cat's pipe swallowing an ls-files failure). Fixed by pulling git ls-files
out of the pipe into its own directly-redirected, individually-checked
step, deliberately chosen over `set -o pipefail;` (which also worked)
because this repo's own build-deployment.md documents a past incident of
code silently relying on /bin/sh happening to be bash with no SHELL:=bash
pin.
One narrow, explicitly-disclosed residual gap remains (a sort-only
failure, not cat, in the trailing pipe) and was assessed as acceptable
low risk after independent verification across three review rounds.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both apiserver/meson.build and xcecli/meson.build add
XCE_DIRTY_FINGERPRINT=get_option('dirty_fingerprint'), unquoted,
identically positioned to the existing XCE_GIT_COMMIT/XCE_BUILD_TIME
entries.
Verified independently by review with a distinct test value: fresh
configure + real ninja builds of both binaries confirm the value reaches
go build's actual invocation (visible in go version -m's recorded -ldflags
build setting), matches the C-side -DVERSION_DIRTY_FINGERPRINT= from the
same configure, and degrades to empty identically to the two pre-existing
vars when omitted.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
go_build now reads XCE_DIRTY_FINGERPRINT (empty-safe) and appends
-X main.DirtyFingerprint=... to its existing -ldflags construction,
falling back to plain empty string when unset -- deliberately NOT the
"unknown" fallback GitCommit/BuildTime use, matching dirty_fingerprint's
empty-means-nothing-to-show convention everywhere else in this feature.
Existing GitCommit/BuildTime fallback logic is completely untouched.
Verified independently by review with real scratch Go builds traced via
bash -x: unset -> empty flag value (not fabricated); set -> exact value
carried through; all-unset -> GitCommit/BuildTime still fall back to
"unknown" while DirtyFingerprint stays empty, confirming the two
conventions coexist without interference.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Worker's hand-rolled OPT_VERSION case now composes via a genuinely
separate version_display_buf/xce_compose_version_ex() call, mirroring
task 3.2's already-approved controller/scheduler pattern exactly. Both
startup-log XCECD_LOG call sites (--peers and no-peers) are left
completely untouched, same xce_compose_version() call, same version_buf.
Verified independently by review with a real build and live daemon runs
in both branches: --version shows <commit>-<fingerprint> matching
compile_commands.json, -dirty correctly stripped not doubled; both
startup log branches show the old fingerprint-less format unchanged.
All three C daemons (controller, scheduler, worker) are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
controller/main.c and scheduler/src/main.c now compose argp_program_version
via a genuinely separate buffer/call to xce_compose_version_ex() (reading
STR(VERSION_DIRTY_FINGERPRINT), matching the existing commit_id/build_time
macro pattern). The startup-log composition call site -- same function,
same arguments, same buffer -- is left completely unchanged in both files.
controller/testsuite/test.c (xce_test, confirmed to have no startup log)
gets a direct in-place swap. All three files carry explicit comments
marking the --version/startup-log divergence as deliberate.
Verified independently by review with real builds and live daemon starts:
--version shows <commit>-<fingerprint> for all three binaries, matching
compile_commands.json; live startup logs for both daemons show the
unchanged old <commit>-dirty format with the fingerprint never present.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds xce_compose_version_ex() as a sibling of the existing, completely
untouched xce_compose_version(): when dirty_fingerprint is non-empty, it
strips a trailing -dirty from the commit field (anchored tail-slice
strcmp, not a substring search) and appends -<dirty_fingerprint> instead;
when empty/NULL, it delegates directly to the original function, so
output is structurally guaranteed byte-identical -- including the
Requirement 5.2 fallback case (dirty tree, fingerprint computation
failed) where commit_id's own pre-existing -dirty suffix already signals
"modified" with no new sentinel needed.
No main.c call sites touched -- consuming daemons (tasks 3.2/3.3) still
call the original xce_compose_version() and will switch to the new
function themselves.
Verified independently by review, full lib suite on both debug and ASAN
profiles: 301 tests, 0 failures, 0 ignored; suffix-anchoring, buffer
safety, and the empty-fingerprint delegation path all confirmed correct
by hand-tracing the implementation, not just trusting the test results.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds myDirtyFingerprint to flake.nix's existing eval-time let block,
computed as builtins.hashString "sha256" self.narHash truncated to 8 hex
chars (self.narHash is SRI base64, not hex -- rehashing is necessary),
gated on the same clean/dirty branch myCommitId already uses. No
preConfigure build-phase hook needed -- self.narHash has no pure-eval trap
unlike build_time's builtins.currentTime. Threaded through
nixSupport/default.nix into xce.nix/xcecli.nix's eval-time mesonFlags as
-Ddirty_fingerprint=, alongside the completely untouched -Dcommit_id=/
myCommitId and build_time's separate preConfigure mechanism.
Verified independently by review with real nix build/nix log inspection
on both xcecore_debug and xcecli: real, non-empty, content-sensitive
values reach mesonConfigurePhase, cross-checked deterministic via an
independent Python recomputation, with commit_id fully unaffected. Full
artifact completion is blocked by a pre-existing, unrelated sandbox
network limitation (confirmed recurring on two disjoint Go module sets),
not caused by this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an independent dirty_fingerprint computation to xce-build.sh: sha256
over git diff HEAD's output plus the sorted content of untracked,
non-ignored files (git ls-files --others --exclude-standard), truncated to
8 hex chars. Resolves to empty on a clean tree or if the hash computation
itself fails -- commit_id's own computation and -dirty suffix are left
completely untouched. Zero working-tree/index mutation.
Verified independently by review with real builds (HEAD pinned to control
for concurrent background commits): distinguishes different uncommitted
content, reproduces identical fingerprints for identical content, excludes
.gitignore'd files, degrades gracefully under 3 independently-injected
failure modes without aborting the build, applies to every profile, and
leaves the original feature's regression script passing unmodified.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a dirty_fingerprint string meson option (default '') alongside the
already-shipped build-time-versioning feature's commit_id/build_time
options, and threads it through add_project_arguments() as
VERSION_DIRTY_FINGERPRINT for every C component. Purely additive,
non-behavioral plumbing -- nothing consumes the new define yet.
Verified independently by review: a bare meson setup with zero -D
options configures cleanly, and all 542 C compile commands carry the new
(empty) define, matching the existing commit_id/build_time defines'
counts exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends the already-shipped build-time-versioning feature so --version
output can distinguish different uncommitted-change states via a short
content-based fingerprint, instead of just a boolean -dirty flag.
Empirically confirmed during design (live probe against src/flake.nix,
reverted) that Nix's self.narHash is reachable at eval-time and content-
sensitive, resolving the spec's one open technical question for the
production build path without requiring git inside the sandbox.
Caught and corrected a design flaw before finalizing: an earlier approach
would have retired commit_id's -dirty suffix at its computation site,
which traces through to silently break the startup log's existing
modified-tree signal (same underlying value). Resolved by leaving
commit_id completely untouched everywhere and doing the -dirty-stripping
only inside --version's composition, only when a fingerprint is available
-- which also satisfies Requirement 5.2's fallback for free.
Independent task-graph sanity review (kiro-spec-tasks Step 3.5) returned
NEEDS_FIXES on 3 minor items; all fixed and self-verified before approval.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-xceadmin-build-install-version.sh (same house style as
tasks 6.1/6.2), which runs xceadmin's full real meson-driven build cycle
(schema sync, venv install, pytest, wheel build), installs the resulting
wheel into a fresh scratch venv, and separately does an editable install
into a second fresh scratch venv. Asserts the wheel install carries real,
non-"unknown" commit id and build time while the editable install carries
the committed "unknown"/"unknown" fallback stub, and explicitly asserts
the two differ -- proving the generated build-info module genuinely
replaces the fallback rather than both installs coincidentally agreeing.
Verified independently by review with a full from-scratch re-run: 261
passed/3 skipped (matching tasks 5.1/5.2's established baseline), both
install paths correct, and the demonstration block's crux case (identical
build-info values must be rejected by the differ-check) confirmed correct.
This is the final task of the build-time-versioning spec; all 13 sub-tasks
are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-version-startup-consistency.sh (same house style as
task 6.1's test-build-version-regression.sh), which builds xce_scheduler
and xce_apiserver for real, starts each daemon, and asserts the commit id
and build time reported by --version/-version are identical to the values
in the daemon's own startup log -- proving Requirement 2.2's guarantee
that the two can never disagree for the same artifact.
apiserver's build routes around two pre-existing, unrelated sandbox
blockers (bug-1597's missing CGO env in `make build-server`, and
go_build's hardcoded GOPROXY hanging in this sandbox) via a direct `go
build -mod=readonly -ldflags` invocation that computes commit_id/build_time
identically to go_build and injects them the same way; only the
orchestration layer differs, not what main.go (task 4.4, unmodified)
consumes. Verified independently by review, including live reproduction
of both build-path blockers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds src/tools/test-build-version-regression.sh, a standalone script (same
convention as test-commit-durability.sh/test-heartbeat-tolerance.sh) that
runs real xce-build.sh scheduler builds in both asan and debug profiles,
extracts the commit id from xce_scheduler --version, and fails loudly if it
doesn't match a live git rev-parse --short HEAD (+ -dirty) -- the exact
staleness this feature was built to make visible without forensic git
commands.
Tolerates xce-build.sh's own known-unrelated flaky test exit (bug-1560) via
captured exit status rather than a blanket set -e bypass, so a genuine
compile failure still fails the check. A built-in demonstration proves the
comparison logic rejects a wrong hash, an empty commit id, and the literal
"unknown" fallback. Verified independently by review with a full from-
scratch re-run of both profiles.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Commits a fallback src/xceadmin/src/xceadmin/_build_info.py (GIT_COMMIT/
BUILD_TIME both "unknown") for install paths that never run meson-build.sh
(e.g. `pip install -e .`), and wires click.version_option() on cli.py's
top-level group. cli.py imports _build_info defensively (try/except
ImportError with an inline fallback) and composes the version string from
importlib.metadata.version("xceadmin") plus both build-info fields.
Verified independently by review: editable install shows "unknown" for
both fields with no import error; the meson-built wheel shows real,
matching commit id + build time; full xceadmin suite 261 passed/3 skipped
(258 baseline + 3 new tests); ruff clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
xceadmin/meson.build passes the resolved commit_id/build_time meson
options as two new positional args (matching this custom_target's
existing all-positional style) to meson-build.sh, which writes a small
_build_info.py (GIT_COMMIT/BUILD_TIME, independent per-field "unknown"
fallback) into the mirrored source tree's xceadmin package directory,
before the pip install step -- same before-install ordering as the
existing schema-sync step. A file-existence guard checks the exact path
just written and fails loudly rather than silently proceeding with a
missing/wrong module.
Live-verified via a full real build+test+wheel cycle (run independently
twice, once by implementation and once by review): the mirrored tree's
_build_info.py and the packaged wheel both contain real, matching,
non-"unknown" values (commit id matching git rev-parse --short HEAD plus
dirty suffix, real UTC timestamp); full pytest suite passes (258 passed,
3 skipped -- documented live-HA e2e tests, skip-by-design).
Task 5.1/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds GitCommit/BuildTime vars to xcecli's main.go. Cobra's rootCmd is
declared unexported in router.go (a different file than the task's
original main.go-only boundary), so main.go can't reach it directly --
resolved by extending router.go's sole exported entry point,
Execute(cm, version), to accept the composed version string and set
rootCmd.Version before rootCmd.Execute() runs. Cobra then auto-registers
--version/-v natively once rootCmd.Version is non-empty (confirmed via
reading Cobra v1.9.1 source directly) -- zero additional flag-parsing
code, as designed.
Live-verified via both build paths (meson xce-build.sh and make
build-cli): real commit id (matching git rev-parse --short HEAD plus
dirty suffix) and build time, exit 0.
xcecli's startup log deliberately not extended -- Requirement 3 scopes
that to the four daemons only; xcecli is a one-shot CLI. Corrects a
stale design.md line that had assumed otherwise.
Task 4.5/13 -- closes out the Go binary version display phase (tasks
4.1-4.5).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Declares GitCommit/BuildTime package vars (default "unknown", target of
task 4.1's -ldflags -X injection), a -version flag checked before
utils.PromptForTopDir()/InitLogger() (zero filesystem side effects), and
extends the existing "Initialize Logger done!" startup log line with
zap.String structured fields carrying both values, per design.md's
explicit mandate.
Live-verified end to end: -version output and the startup log's first
line are byte-identical, with a real commit id (matching
git rev-parse --short HEAD plus dirty suffix) and UTC timestamp -- the
first full round-trip proof in this spec (earlier Go tasks could only
verify via linked build-info metadata, since no consuming vars existed
yet).
Task 4.4/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dev-loop Go build path (no meson involved) now computes commit id (git,
-dirty suffixed, "unknown" on git failure) and build time (UTC date) via
Make := (computed once per invocation), passing the identical -ldflags
shape go_build (task 4.1) already establishes.
Live-verified: make build-cli embeds real, git-matching values (go
version -m). make build-server hits a pre-existing, unrelated CGO
failure (bug-1597, missing CGO_CFLAGS/CGO_LDFLAGS) -- confirmed via
stash A/B that this reproduces identically on the unmodified Makefile,
not a regression from this change. Reproduced apiserver's build
invocation manually with bug-1597's own documented workaround to prove
the ldflags mechanism itself works correctly.
Task 4.3/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apiserver/meson.build and xcecli/meson.build now pass
XCE_GIT_COMMIT/XCE_BUILD_TIME (from the project's commit_id/build_time
meson options) into go_build's environment, mirroring the existing
CURRENT_SOURCE_DIR/BUILD_ROOT env-passing mechanism. Deliberately
unquoted: go_build's fix_input_envs() only quote-strips
CURRENT_SOURCE_DIR/BUILD_ROOT, not these two -- quoting them would embed
literal quote characters and corrupt the -ldflags argument.
Live-verified: real values (matching git rev-parse --short HEAD plus
dirty suffix, and a real UTC timestamp) reach go build's actual
invocation and are embedded in both binaries' Go build-info metadata.
No main.GitCommit/main.BuildTime symbols exist in either binary yet
(confirmed via go tool nm) since apiserver/xcecli's main packages don't
declare those vars until tasks 4.4/4.5 -- expected, this task's scope is
the env-var plumbing only.
Task 4.2/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reads XCE_GIT_COMMIT/XCE_BUILD_TIME from the environment (empty-safe via
[ -z ... ], not just unset-safe -- a classic ${VAR:-default} gotcha
avoided), falls back to "unknown", and appends
-ldflags "-X main.GitCommit=... -X main.BuildTime=..." to the go build
invocation alongside the existing -gcflags.
Confirmed via live testing: -X on a main package var that doesn't exist
yet is silently ignored by the Go linker (not a build error), so this
task has no real ordering dependency on tasks 4.4/4.5 (which add the
consuming vars to apiserver/xcecli).
Task 4.1/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Worker uses plain getopt_long(), not argp, so it has no free --version
handling like controller/scheduler. Adds a long-only OPT_VERSION=257
option and a handling case that computes xce_compose_version() into a
static buffer and returns before any startup work (check_user()/
xcecd_init/xcecd_start). Both startup INFO log branches (--peers and
no-peers) extended to log the identical buffer.
Live-verified: --version and both startup-log branches are
byte-identical (captured via a real daemon run under sudo).
Also updates bug-1519 (pre-existing, unrelated ASAN "Joining already
joined thread" abort during worker shutdown): independently reproduced
under real root during this task's review, refining its prior "never
reproduced under real root" note -- not a userns/unshare artifact, still
unfixed, does not affect this task's code path.
Task 3.3/13 of build-time-versioning. This closes out the "Core: C
daemon version display" phase (tasks 3.1-3.3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Migrates argp_program_version in controller/main.c, scheduler/main.c,
and xce_test's test.c from the compile-time MK_VERSION/STR construction
to xce_compose_version(), computed into a static buffer and assigned
before argp_parse() runs (argp_program_version is a file-scope pointer,
not a constant expression it can be initialized from at compile time).
Each daemon's startup log now carries the identical version string
(never recomputed) on its first INFO line -- scheduler's BOTH
mutually-exclusive branches (--peers and no-peers), not just the one
literally named in the task text, since "when the daemon starts" is
unconditional.
Live-verified: --version and startup-log output are byte-identical for
both daemons across all branches. No remaining MK_VERSION callers
repo-wide.
Task 3.2/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds xce_version_field()/xce_compose_version() (utils.h/utils.c) to
compose a version string carrying commit id + build time, rendering
"unknown" for either field when its underlying meson-supplied define is
empty -- ready for controller/scheduler/worker to consume.
MK_VERSION/STR (the pre-existing, compile-time-only macro pair) can't
express this empty-to-"unknown" substitution at the preprocessor level,
so this is a runtime mechanism instead. Round 1 review REJECTED an
earlier version of this task for silently orphaning MK_VERSION/STR
(the task explicitly asked to "extend" them) and for leaving an
undocumented architectural trap: argp_program_version is a file-scope
const char* initialized at compile time, which cannot call a runtime
function -- consuming code must instead compute into a static buffer
inside main() and assign the pointer before argp_parse() runs. Round 2
adds an explicit SUPERSEDED doc comment on MK_VERSION/STR plus NOTE FOR
CALLERS guidance covering both points, and was APPROVED.
Also expands task 3.2's boundary to include controller/testsuite/test.c
(a third MK_VERSION/STR call site discovered during review) so no
caller of the superseded macro survives implementation.
5 new unit tests in test_utils.c (RED->GREEN, TDD).
Task 3.1/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
First attempt used builtins.currentTime at flake-evaluation time -- this
broke the plain, documented `nix build .#xcecore_debug` command entirely
(pure-eval strips builtins.currentTime, requiring --impure) and produced
a raw epoch-seconds string instead of UTC ISO-8601. Caught by task-local
review (REJECTED, round 1) before merge.
Fixed by moving the computation into each derivation's own build phase:
a preConfigure hook in xce.nix/xcecli.nix runs `date -u` (stdenv's
bundled coreutils, unaffected by pure-eval) and appends -Dbuild_time= to
mesonFlagsArray, with a fail-loud guard against empty output. commit_id's
existing self.shortRev/self.dirtyShortRev eval-time mechanism is
untouched. Verified via nix eval (no --impure) and nix build reaching
configure with real values on both xcecore_debug and xcecli (round 2,
APPROVED).
Task 2.2/13 of build-time-versioning. Also corrects design.md/research.md,
which described the now-superseded builtins.currentTime approach.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The dev-loop build path (/xce-build, the one behind the motivating
incident) never set -Dcommit_id= at all, so every locally-built binary
carried an empty commit id. xce-build.sh now computes commit_id (git
rev-parse --short HEAD, -dirty suffixed on uncommitted changes, unknown
fallback on a genuine git failure) and build_time (UTC date), appends
both to MESON_OPTS for every profile before meson setup, and warns
loudly to stderr if either ever resolves empty anyway.
Task 2.1/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Kiro spec-quick output for the build-time versioning feature: every
component embeds a git commit id + build timestamp, shown via --version
and (for the four daemons) a startup log line. Generated via
/kiro-spec-quick, sanity-reviewed, auto-approved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add a build_time meson option (string, default '') sibling to the
existing commit_id option, threaded project-wide via the same
add_project_arguments() call that already defines VERSION_COMMIT_ID --
one call, one mechanism, no new plumbing. No consumer wired yet (later
tasks); a plain meson setup with neither option set still configures
cleanly with both defines resolving to empty string.
Task 1/13 of build-time-versioning (spec: .kiro/specs/build-time-versioning/).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Surfaced while investigating a user request to show total/current values
for GLOBAL/SHARED resources in the scheduler REPL's "info resource"
command -- researched and confirmed that concept doesn't exist anywhere
in this codebase (resourcemap config field dropped at build time, capnp
fields commented out, worker.c actively rejects XCE_RESF_SHARED resources
today). This spec already owns closing that gap; it just never made it
into the roadmap's backlog list.
- xce_test shell command support: was already implemented (xce-test-e2e-
primitives task 2.1, closed 2026-06-19) -- stale "not started" entry
corrected.
- scheduler-core tasks 2.1-2.6 _Verified: backfill: was already done
(commit d7b49690, 2026-08-08) -- corrects a status line I myself wrote
incorrectly earlier today by trusting the original 2026-08-07 finding's
wording without re-checking the file.
- job_ctrl direct-mutate-path investigation: completed. Real risk
confirmed (pre-commit state pushed live to scheduler + apiserver/clients
with no rollback on commit failure), scope wider than originally framed
(on_job_stop/on_job_resume/on_job_ckpt share the same pattern, fed by
ordinary traffic not just rare admin commands), plus a separate
read-lock-during-mutation bug found along the way. Not promoted to a
spec yet -- scope/framing corrected first per this session's decision.
- OP_upgradeConfig standby redirect verification: investigated. The
item's own premise doesn't match production (a standby never actually
returns a redirect for this opcode -- a documented, deliberate feature
gap, not a missing test); coverage confirmed already complete on both
the client mock side and a live 3-node regression test.
design.md's own "Open Questions/Risks" list (items 5 and 6) still described
the OPCode_schedAdmin admission gap and the tlog activation gap as open/
deferred, even though both were fixed earlier in this session -- marked
FIXED with pointers to the actual change. research.md's schedAdmin
decision record and its Risks & Mitigations bullet got the same treatment
(SUPERSEDES note, historical record preserved rather than rewritten).
tasks.md's 2026-08-07 GO/NO-GO follow-up list gained a status-update line:
follow-up 1 (tlog) is now fixed, follow-up 2 (fcfs.c/builtin_fcfs.c
traceability) is only partially closed (design.md text was already
correct; tasks 2.1-2.6 still lack _Verified: lines -- flagged, not
fixed here since I have no fresh verification evidence for those tasks).
roadmap.md's scheduler-core summary line corrected to attribute each gap
to the right source (2026-08-07 validate-impl vs today's separate
design-review pass) instead of conflating them.
Task 12.5 (REPL live-operator-testing fixes) was implemented and its bugs
fixed in an earlier commit, but the tasks.md checkbox was never flipped and
carried no _Verified: line, unlike every other completed task in this spec.
Ran /kiro-verify-completion against it with fresh evidence rather than
trusting the earlier self-report: fresh build+full suite (41/42, only the
pre-existing unrelated bug-1560 failing), a live subprocess smoke test of
the real xce_scheduler binary for all five amendment gaps (including a
hand-built .replay fixture with resource data to close the one gap the
first pass left unit-tested-only), and direct source inspection of each
fix. All five confirmed correct; checkbox flipped, _Verified: line added.
Two retrospective design-review findings:
- design.md's System Flows sequence diagram showed the controller-layer
resval check as an independently-launched client session, implying no
dependency on the injecting worker staying connected. As-built (bug-1579),
host-attrs resources reset to baseline the instant the worker disconnects,
so the check actually runs nested inside the still-open worker session via
xce_test's `!` shell-escape. Updated the diagram and prose to show this,
and documented the still-open residual risk for custom host-attrs
resource names in Error Handling.
- Added explicit "do not change this format" warning comments at both
trace-log call sites (core_events.c's RESVALS_DELTA case, apiserver's
OnResourceLocationUpdated) that xce-test-resources' log-grep depends on --
an unrelated logging cleanup at either site would otherwise silently turn
every future verification run into a false UNREACHABLE/timeout.
No behavior change in either source file -- comments only.
The schedAdmin dispatch case blocks its vtask handler thread for up to 5s
waiting on a cross-process reply while holding c->sched.lock -- bounded
blocking on a vtask thread, which ha-async-io.md permits only under an
admission reservation. The real risk isn't just the wait itself: since
c->sched.lock serializes concurrent requests one at a time, a burst of N
concurrent schedAdmin calls could already occupy N vtask threads for up to
N*5s each before this fix.
Now acquires vtaskmgr_blocking_acquire(&c->vtaskmgr) before ever locking
c->sched.lock, releasing on every exit path -- the same shared admission
budget commit.c's commit-wait barrier already gates on (wired via
commit_set_pool). Admission-exhausted requests fast-reject with
{ok=false, error="server busy, try again"} rather than queuing. New test:
test_schedadmin_fast_rejects_when_admission_exhausted, using a poison
c->sched.ep sentinel to prove the fast-reject happens before any
lock/forward attempt.
Bugs fixed, discovered across a day of live-operator testing against a
real deployed cluster:
- bug-1584: clock anchors never initialized to real wall-clock values,
timestamps showed system uptime instead of Unix epoch.
- bug-1585: heap-use-after-free in the normal-mode driver's periodic
resval update path -- core_events.c's RESVALS_FULL/_DELTA handlers do a
shallow zarray_put() of the wire-decoded xce_val_t*, but driver.c's
release_translated_payload() unconditionally freed the same decoded
array right after, so the next periodic report for the same
worker+resource triggered a UAF read via kill_sched_resval(). New
regression test: test_resval_ownership.c.
- bug-1587/1588/1591: raw-ns/raw-uint64 jobid/timestamp prints across the
REPL/SIM family converted to the same human-readable formatting already
used elsewhere (fmt_jobid(), timestamp formatting).
- bug-1590: repl_info.c's XCE_VAL_NUMERIC checks corrected to
Value_u_fvalue, fixing `info worker`/`info resource` falling back to
0.00 for real numeric resvals.
task 12.5 (Req 5.22 amendment) -- five REPL gaps surfaced by a live
operator session, each with its own bug id:
- bug-1594: `continue`/`step` reaching EOF with no breakpoint hit now
re-enters the REPL via xce_repl_on_replay_complete() instead of exiting
the process; step/continue issued after replay is already complete
re-prompt instead of silently exiting.
- bug-1595: `step`/`continue` now echo each input event (not just
decisions) via repl_print_event_summary(), in real event/decision
order.
- bug-1593: `info resource <name>|all` now reports resource definition
metadata restricted to XCE_RESF_GLOBAL/XCE_RESF_SHARED resources, read
from core->sharedInst, instead of aggregating host-based resources
across every worker.
- bug-1596: repl_split_tokens() strips one matching pair of surrounding
quote characters from each token.
See requirements.md's Req 5 AC 22 "Amendment (2026-08-19)" for the full
rationale on each of the five task 12.5 gaps.
Full scheduler suite green (41/42; the one remaining failure,
test_recordstart_symlink_escape_replies_error, is the pre-existing
unrelated bug-1560).
Every early-exit branch in the worker_list scan either re-tested the same
node forever (bare `continue`) or returned out of the whole function on the
first DONE/EXITED worker, so job array elements past the head of the list
never got their on_job_start report. Capture `next` before touching each
node and always advance to it. Affects xcecd_virtual and real xcecd alike
(same DBUS_MANAGER-gated function).
Also records a roadmap follow-up (xcecd_virtual concurrent-job support,
MXJ > 1) discovered while investigating this bug.
requirements.md, design.md, research.md, and spec.json were never committed
during implementation (only tasks.md was, per the per-task checkpoint
convention). Committing them now that all 15 tasks are implemented,
reviewed, and validated GO by /kiro-validate-impl.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
verdict-logic-selftest.sh sources the real aggregate_verdict()/
build_verdict_record() (via run-resource-verify.sh's BASH_SOURCE guard, no
live deployment triggered) and exercises them with synthetic per-layer
records: all-match/PASS positive control, a single-resource single-layer
MISMATCH within a multi-resource batch (asserting both correct attribution
of the failure AND that the other resources independently still report
PASS), UNREACHABLE distinct from MISMATCH, REJECTED, and a zero-records
degenerate case. 53 assertions total.
Proven non-tautological: breaking aggregate_verdict()'s comparison logic
(two different ways, independently reproduced by review) makes the selftest
genuinely fail; restoring makes it pass again.
This completes the resource-value-verification spec - all 15 tasks (1.1
through 8.2) implemented, reviewed, and committed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Validation-only, no code changes. Confirmed the complete spec end-to-end
against a freshly built controller+scheduler+apiserver: default 9-resource
set + one deliberately-unregistered name -> VERDICT resources=10
rejected=1 result=FAIL (exit 1), attributable specifically to the rejected
name, not a false PASS; positive control (defaults only) -> VERDICT
resources=9 rejected=0 result=PASS (exit 0). All 27 individual layer-checks
(9 resources x 3 layers) MATCH; raw scheduler.log/apiserver.log trace lines
independently verified against tasks 4.1/4.2's documented format; rejection
confirmed non-silent across 4 independent log sources; cleanup and log
preservation confirmed.
Surfaced (but correctly did not fix, out of boundary) bug-1583: the
apiserver stores resource-location updates with zero name validation, even
for names the controller/scheduler both reject. Confirmed this does not
affect this feature's own correctness - accept/reject is determined
entirely by cmd_resupdate()'s controller-side follow-up query, before any
layer is checked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
aggregate_verdict()/build_verdict_record() combine the three per-layer
check results into a single per-resource comparison, self-contained and
invokable with synthetic records (task 8.2). Uses ASCII Unit Separator
(\x1f) as the record delimiter after discovering bash's IFS tab-splitting
silently collapses adjacent empty fields even with IFS set to only a tab
(bug-1581), which corrupted UNREACHABLE-record field alignment.
Emits one machine-parseable VERDICT line (resources/layers_checked/
mismatches/unreachable/rejected/result) plus the terminal PASS/FAIL oracle
line and matching exit code. Cleanup is the same trap-based mechanism task
5.1 registered for the abort path - logs always preserved regardless of
--keep-running.
Round-2 fix: main()'s live-data wiring (build_verdict_records_from_tsv())
derived a layer's status from array-key existence rather than the actual
recorded outcome - since tasks 6.3/6.4 always set the key (even on a
genuine timeout, just with an empty value), every real UNREACHABLE was
silently misclassified as MISMATCH. Fixed to check SCHEDULER_CHECK_RESULTS/
APISERVER_CHECK_RESULTS string content instead (bug-1582). Live-confirmed
by killing the scheduler mid-run: VERDICT now correctly reports
unreachable=9 mismatches=0, not the reverse.
SKILL.md rewritten to describe the now-complete, functional tool.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
poll_apiserver_log()/query_apiserver_resvals() mirror task 6.3's
scheduler-check pattern: bounded-wait polling (batched across all pending
resources per iteration) of apiserver.log for task 4.2's "resource-location
stored" trace line, parsing the trailing zap JSON block. Reports MATCH /
MISMATCH (both values) / UNREACHABLE (timeout, distinct from mismatch),
tracked in APISERVER_OBSERVED/APISERVER_CHECK_RESULTS +
apiserver-check-results.tsv.
This completes all three per-layer checks. Live-confirmed: the full default
resource set (including cpuf, post-bug-1579-fix) MATCHes simultaneously at
the controller, scheduler, and apiserver layers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously worked around (task 6.2) by removing the one host-attrs resource
(cpuf) from the default set - a custom resource=value arg using any
host-attrs name would still false-mismatch at the controller layer, since
inject_resources() disconnected the worker before the separate controller-
check session ran, and the controller resets host-attrs resources on
disconnect.
Real fix: xce_test's `!<shell-command>` escape is handled in the single
top-level REPL loop before context-specific dispatch, so it works in any
context - including worker mode - without calling leave_ctx()/disconnecting.
inject_resources() now shells out via `!` to run the controller check
in-line, between resupdate and quit, while the worker's connection is still
open. query_controller_resvals()'s comparison logic is unchanged - this is
a pure sequencing fix. cpuf restored to DEFAULT_RESOURCE_ARGS as live proof.
Live-verified: cpuf now MATCHes at the controller layer (was always a false
MISMATCH before); adversarial shell-injection attempt via a malicious
resource name confirmed safe (printf %q quoting); SIGTERM delivered exactly
during the nested check confirmed zero orphaned/zombie processes across 4
independent runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
poll_scheduler_log()/query_scheduler_resvals() tail the live scheduler
daemon's log (task 4.1's trace line) for each accepted injected resource,
with a bounded ~15s wait batched across all pending resources per poll
iteration (not N times the timeout). Reports MATCH / MISMATCH (both
values) / UNREACHABLE (timeout, distinct from mismatch), tracked in
SCHEDULER_OBSERVED/SCHEDULER_CHECK_RESULTS + scheduler-check-results.tsv.
Live-proven immune to bug-1579 (unlike the controller check): injecting a
host-attrs resource (cpuf) simultaneously produces a controller MISMATCH
and a scheduler MATCH, since this check reads task 4.1's trace log (written
once, at apply-time) rather than re-querying live state after the worker
disconnects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
query_controller_resvals()/controller_values_match() query the controller
via resval for every accepted (not rejected) injected resource/worker pair
and report MATCH/MISMATCH with both values on a mismatch, tracked in
CONTROLLER_OBSERVED/CONTROLLER_CHECK_RESULTS + controller-check-results.tsv
for tasks 6.3/6.4/7.1.
Also fixes bug-1579 (round-2, in default set only): cpuf, a host-attrs
resource the controller resets to baseline on worker disconnect, always
false-mismatched since inject_resources() disconnects before this check
runs. Swapped for r1m (periodic/dynamic, disconnect-safe) in
DEFAULT_RESOURCE_ARGS. Custom host-attrs resource=value args still hit this
- documented as a known follow-up, not fixed here (would need a C-level
xce_test change or a coprocess restructuring).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
inject_resources() joins a synthetic worker through xce_test's worker
context and runs resupdate for the requested resource=value pairs
(default: Requirement 1's numeric gap set + mem/cpuf regression names,
deliberately excluding type/model due to bug-1574's live segfault risk).
Per-name accept/reject results captured into INJECTED_VALUES/
INJECTED_RESULTS plus injection-results.tsv for tasks 6.2-6.4.
Also suppresses the real xcecd_virtual worker daemon that deploy-ha.sh's
pre-existing worker branch would otherwise silently start whenever the
build dir contains one (XCE_WORKER_BIN=/bin/true), which would have
violated Requirement 2.1 - bug-1577.
Blocked pending a separate prerequisite fix (bug-1578, commit b406dd7):
live testing surfaced a pre-existing controller crash on worker disconnect,
unrelated to this feature, now fixed with regression coverage. Live-
reconfirmed post-fix: controller survives the full join+resupdate+quit flow
indefinitely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
run-resource-verify.sh deploys controller+scheduler(-vvv)+apiserver(debug
logging) via xce-deploy-ha's deploy-ha.sh -> start-all.sh into a fresh
scratch dir, confirms all three are genuinely up, and cleans up via a trap
registered before any process starts - covering both normal completion and
a mid-deployment abort. Logs always preserved regardless of --keep-running.
Independently stress-tested with 8+ SIGTERM interrupts at varying
deployment points and a genuine crashing-apiserver scenario - zero leftover
processes in every case. Single-node by default (propagation testing, not
HA failover), overridable via $XCE_RESVERIFY_PEERS.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
