myxce/.kiro/specs/dirty-content-fingerprint/tasks.md

Implementation Plan

• 1. Foundation: dirty_fingerprint meson option and C-wide define

  • dirty_fingerprint meson option (string, default '') added in src/meson.options, byte-for-byte matching commit_id/build_time's style/type/yield; add_project_arguments() extended with -DVERSION_DIRTY_FINGERPRINT=
  • Observable confirmed live (independently re-verified by review, fresh scratch build): a bare meson setup with zero -D options configures cleanly (73 targets); all 542 C compile commands contain the new define (empty string), matching the existing two defines' counts exactly; no consumption code touched yet
  • Requirements: 1.1, 5.1

• 2. Foundation: compute real fingerprint values at both build entry points

• 2.1 (P) xce-build.sh computes the dev-loop fingerprint, independently of commit_id

  • dirty_fingerprint computed via git diff HEAD + git ls-files -z --others --exclude-standard | LC_ALL=C sort -z content, sha256sum truncated to 8 hex chars; empty on a clean tree or on computation failure (3 independently-verified graceful-degradation paths: mktemp failure, sha256sum failure, git diff failure -- none abort the script despite set -euo pipefail); zero working-tree/index mutation (read-only git invocations only); commit_id's own computation left byte-for-byte untouched
  • Observable confirmed live (independently re-verified by review, real builds with HEAD pinned to control for concurrent background commits): editing a tracked file and rebuilding produces a fingerprint; different content produces a different fingerprint; reverting to the first edit's exact bytes reproduces the first fingerprint exactly; a .gitignore-matched *.bak file leaves the fingerprint unchanged; commit_id's -dirty suffix stayed intact across every build; every profile (asan/debug/release) carries the option; the original feature's unmodified test-build-version-regression.sh passes twice in full (asan+debug)
  • Requirements: 1.1, 1.2, 1.3, 1.4, 5.1
  • Boundary: xce-build.sh

• 2.2 (P) Nix flake computes the fingerprint via self.narHash at eval time, independently of myCommitId

  • myDirtyFingerprint computed as builtins.hashString "sha256" self.narHash truncated to 8 hex chars (self.narHash is Nix's SRI base64 format, not hex -- rehashing was confirmed necessary, not optional), gated on the same hasAttr "shortRev"-style clean/dirty branch myCommitId already uses; threaded through nixSupport/default.nix into xce.nix/xcecli.nix's eval-time mesonFlags list as -Ddirty_fingerprint=, alongside the untouched -Dcommit_id=; build_time's separate preConfigure/mesonFlagsArray mechanism left completely undisturbed
  • Observable confirmed live (independently re-verified by review, including a full re-run of both nix build ./src#xcecore_debug and ./src#xcecli, no --impure): nix eval resolves a real, non-empty, content-sensitive -Ddirty_fingerprint= value matching an independent Python cross-check; editing a tracked file changes the resolved value; -Dcommit_id= unaffected throughout; nix log confirms both values reach the real mesonConfigurePhase invocation. Full artifact completion is blocked in this sandbox by a pre-existing, unrelated network limitation (an autoscaler Go module fetch needing goproxy.cn, confirmed recurring identically on two disjoint Go module sets) -- not a regression from this task, and orthogonal to the eval/meson-configuration behavior this task's Observable actually requires
  • Requirements: 1.1, 1.2, 1.3, 5.1
  • Boundary: Nix packaging

• 3. Core: C daemon --version fingerprint display

• 3.1 Extend the shared version-composition mechanism with an optional fingerprint field, without touching commit_id

  • New sibling function xce_compose_version_ex() added alongside the completely untouched xce_compose_version() (zero diff lines in its body or its 3 pre-existing tests); anchors the -dirty strip via a true tail-slice strcmp (not strstr), so it cannot false-match a -dirty substring elsewhere; correctly appends -<fingerprint> with exactly one dash even when the commit field has no -dirty to strip; delegates via a genuine return xce_compose_version(...) call (not a re-implementation) when the fingerprint is empty/NULL, structurally guaranteeing byte-identical output, proven by a dedicated equivalence regression test
  • No main.c call sites were touched by this task (grep-confirmed all still call the original, untouched xce_compose_version()) -- that's tasks 3.2/3.3's job
  • Observable confirmed live (independently re-verified by review, full lib suite on both debug and ASAN profiles): 301 tests, 0 failures, 0 ignored on both profiles; all 3 required cases present with byte-exact assertions, plus 2 meaningful additional regression/defensive tests; buffer handling confirmed snprintf-bounded with no overflow risk
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 5.2
  • Boundary: utils.h, utils.c

• 3.2 (P) Controller and scheduler: fingerprint in --version, startup log unaffected

  • argp_program_version for controller/scheduler now composed via a genuinely separate buffer/call to xce_compose_version_ex() (reading STR(VERSION_DIRTY_FINGERPRINT), matching the existing commit_id/build_time macro pattern); the startup-log call site's xce_compose_version() call, arguments, and buffer are byte-for-byte unchanged (independently verified by review via diff + grep); controller/testsuite/test.c (no startup log, confirmed via grep) got a direct in-place swap; all three files carry explicit code comments marking the divergence as deliberate
  • Observable confirmed live (independently re-verified by review, real builds + live daemon starts): xce_controller --version/xce_test --version/xce_scheduler --version all show <commit>-<fingerprint>, cross-checked against compile_commands.json; live controller and scheduler startup logs show the old <commit>-dirty format unchanged, fingerprint never present, build_time matching --version exactly
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 4.1
  • Boundary: controller/main.c, scheduler/main.c, controller/testsuite/test.c
  • Depends: 3.1

• 3.3 (P) Worker: fingerprint in --version, startup log unaffected

  • OPT_VERSION's case now composes via a genuinely separate version_display_buf/xce_compose_version_ex() call, byte-for-byte mirroring task 3.2's already-approved controller/scheduler pattern (independently confirmed by review via direct diff comparison against commit fd62e26); both XCECD_LOG startup-log call sites (--peers and no---peers) left completely untouched, same xce_compose_version() call, same version_buf
  • Observable confirmed live (independently re-verified by review, real build + live daemon runs in both branches): xcecd --version shows <commit>-<fingerprint> matching compile_commands.json exactly, -dirty correctly stripped not doubled; both startup log branches show the old fingerprint-less <commit>-dirty format unchanged
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 4.1
  • Boundary: worker/main.c
  • Depends: 3.1

• 4. Core: Go binary --version/-version fingerprint display

• 4.1 go_build injects the fingerprint via ldflags

  • go_build reads XCE_DIRTY_FINGERPRINT (empty-safe) and appends -X main.DirtyFingerprint=... to the existing -ldflags construction; falls back to plain empty string (deliberately NOT "unknown", unlike GitCommit/BuildTime), matching dirty_fingerprint's convention everywhere else in this feature; the existing GitCommit/BuildTime unknown-fallback logic is completely untouched
  • Observable confirmed live (independently re-verified by review, real scratch Go builds traced via bash -x): fingerprint unset -> -X main.DirtyFingerprint= empty, binary confirms empty (not fabricated); fingerprint set -> flag carries the exact value; all three vars unset -> GitCommit/BuildTime still fall back to unknown, DirtyFingerprint stays empty, the two fallback conventions coexist without interference
  • Requirements: 1.1, 5.1
  • Boundary: go_build script

• 4.2 (P) apiserver and xcecli meson builds pass the resolved value through

  • Both apiserver/meson.build and xcecli/meson.build add 'XCE_DIRTY_FINGERPRINT=' + get_option('dirty_fingerprint'), unquoted, identically positioned to the existing XCE_GIT_COMMIT/XCE_BUILD_TIME entries
  • Observable confirmed live (independently re-verified by review with a distinct test value, fresh configure, and real ninja builds of both binaries): the value reaches go build's real invocation and shows up in go version -m's recorded -ldflags build setting for both apiserver and xcecli; matches the C-side -DVERSION_DIRTY_FINGERPRINT= byte-for-byte from the same configure; degrades to empty identically to the two pre-existing vars when omitted
  • Requirements: 1.1, 4.1
  • Boundary: apiserver/meson.build, xcecli/meson.build
  • Depends: 4.1

• 4.3 (P) Makefile dev-loop path computes the same fingerprint, independently of GIT_COMMIT

  • GIT_COMMIT/GIT_STATUS/BUILD_TIME (and GIT_COMMIT's own -dirty suffix) confirmed byte-for-byte untouched throughout two rounds of adversarial remediation
  • DIRTY_FINGERPRINT computed independently, mirroring task 2.1's algorithm exactly (cross-checked byte-identical on the real dirty tree); failure-detection required two remediation rounds to get right -- round 1 fixed a { git diff HEAD; ...; } >file grouped command silently swallowing a git diff HEAD failure (only the last command's exit status was checked); round 2 found the SAME class of bug one step further down (git ls-files | sort | xargs cat's pipe also swallowing an ls-files failure) and fixed it by pulling git ls-files out of the pipe entirely into its own directly-redirected, individually-checked step -- deliberately choosing this over a set -o pipefail; prefix (which empirically also worked) because this repo's own build-deployment.md documents a past incident of code silently relying on /bin/sh happening to be bash with no SHELL:=bash pin
  • One narrow, explicitly-disclosed residual gap remains and was assessed as acceptable: a sort-only failure (not cat) in the trailing sort -z | xargs -0 -r cat -- pipe can still be masked -- independently confirmed present, low-risk, and accurately scoped, not grounds for further rejection
  • Observable confirmed live (independently re-verified across 3 review rounds, real builds + two independently-reproduced failure-injection wrappers with control tests proving each wrapper genuinely triggers the old bug): make -n build-cli's fingerprint matches a standalone reference computation byte-for-byte; both the git diff-fails and git ls-files-fails scenarios correctly degrade to empty, not a fabricated hash; GIT_COMMIT's -dirty behavior unaffected throughout
  • Requirements: 1.1, 1.2, 1.3, 1.4, 5.1
  • Boundary: Makefile
  • Depends: 4.1

• 4.4 (P) apiserver: fingerprint in -version, startup log unaffected

  • DirtyFingerprint package var (default "") + composeVersionDisplay() helper doing the strip-and-append via a genuine anchored strings.TrimSuffix (not a blind substring replace, independently confirmed via a self-constructed mutation test); GitCommit's declaration and the logger.Info startup-log call site (including its zap fields) confirmed byte-for-byte untouched
  • Observable confirmed live (independently re-verified by review, real build across all 5 scenarios + a live daemon-start log capture): -version shows <hash>-<fingerprint> never <hash>-dirty-<fingerprint>; the startup log shows the raw, un-fingerprinted GitCommit (still -dirty-suffixed when applicable) on the same binary whose -version output differs; go.sum/go.mod untouched
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 4.1
  • Boundary: apiserver/cmd/main.go
  • Depends: 4.1

• 4.5 (P) xcecli: fingerprint in --version

  • DirtyFingerprint var + composeVersionDisplay() helper, a verbatim logical duplicate of apiserver's just-approved implementation (task 4.4); applied at the sole consumption point (the version string fed into router.Execute(cm, version)); router.go confirmed untouched (independently re-verified by the parent controller after the review, not just the reviewer, given a mid-review incident where the reviewer's own tooling briefly discarded the uncommitted diff before catching and byte-for-byte restoring it)
  • Observable confirmed live (independently re-verified by review, real builds across all 3 scenarios + full xcecli module test suite): --version/-v shows <hash>-<fingerprint> on dirty+fingerprint, <hash> alone on clean, <hash>-dirty unchanged on dirty+degraded-fingerprint; all exit 0; no regressions in any other xcecli package
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 4.1
  • Boundary: xcecli/cmd/main.go
  • Depends: 4.1

• 5. Core: xceadmin fingerprint display

• 5.1 (P) meson-build.sh generates the fingerprint field in the build-info module

  • dirty_fingerprint threaded through as an 8th positional arg ($8), correctly numbered and consistent between meson.build and meson-build.sh; _build_info.py gains DIRTY_FINGERPRINT with an independent empty-string fallback (not "unknown"); GIT_COMMIT/BUILD_TIME's existing generation and fallback confirmed byte-for-byte untouched
  • Observable confirmed live (independently re-verified by review, full real build+test+wheel cycles across 3 scenarios): dirty tree -> real, non-empty DIRTY_FINGERPRINT present identically in the mirrored tree, the build/lib copy, an installed venv, and the packaged wheel, cross-checked byte-for-byte against that same build's compile_commands.json; explicit-empty fingerprint -> genuinely "" while GIT_COMMIT/BUILD_TIME pass through real values; all-empty -> GIT_COMMIT/BUILD_TIME still fall back to "unknown" while DIRTY_FINGERPRINT stays ""; 261 passed/3 skipped in all three scenarios
  • Requirements: 1.1, 1.2, 1.3, 1.4, 5.1
  • Boundary: xceadmin/meson.build, xceadmin/meson-build.sh
  • Depends: 1

• 5.2 (P) Committed fallback stub and CLI fingerprint wiring

  • _build_info.py's fallback stub gains DIRTY_FINGERPRINT = ""; cli.py's new _compose_version_display() applies the strip-and-append rule via a genuine anchored removesuffix("-dirty") (independently confirmed via a second, differently-constructed mutation test); both fallback paths (_build_info.py's module stub and cli.py's separate ImportError fallback class) correctly carry the field
  • A pre-existing test's hard-coded literal-GIT_COMMIT assertion was updated to compute its expected value via the production _compose_version_display() function instead -- independently assessed by review as a legitimate strengthening (the old assertion's premise no longer holds once a fingerprint is present), not a weakened/gutted check; the 4 new dedicated unit tests independently guard the transform itself with hard-coded literal assertions
  • Observable confirmed live (independently re-verified by review, both install paths via a full real /xce-build xceadmin cycle): editable install shows 0.1.0 (unknown, unknown), no import error; wheel built from the dirty tree shows <hash>-<fingerprint>, never <hash>-dirty-<fingerprint>; full suite 267 passed/3 skipped (261 baseline + 6 new)
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 4.1
  • Boundary: xceadmin/src/xceadmin
  • Depends: 5.1

• 6. Integration: cross-toolchain consistency and regression verification

• 6.1 Dev-loop distinguishability and clean-tree regression test

  • New script src/tools/test-dirty-fingerprint-distinguishability.sh uses a genuine isolated git worktree add --detach (unique mktemp -u path, trap cleanup EXIT registered before creation, safe no-op if creation never happened, rm -rf+prune fallback) to sidestep this actively-shared repo's ambient concurrent-editing noise -- all file edits confined strictly inside the isolated worktree, the real repo's own working tree never touched
  • Drives a real xce-build.sh scheduler cycle through the 4 required states via genuine builds; fails loudly with a clear message on any mismatch
  • Observable confirmed live (independently re-verified by review across two full live runs, including a deliberately sabotaged copy forcing edit-B to reproduce edit-A's content): clean/no-suffix, dirty-A, dirty-B-differs-from-A, reverted-to-A-matches-A all held with real fingerprints; sabotaged run correctly failed with a clear message and exit 1, proving genuine (not just synthetic) discriminating power; git worktree list and the real repo's working tree confirmed byte-identical to baseline before and after every run, both by implementer and reviewer
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.1
  • Depends: 2.1, 3.2

• 6.2 Startup-log-unchanged consistency test

  • New script src/tools/test-dirty-fingerprint-startup-log-unchanged.sh reuses task 6.1's already-approved isolated-git-worktree technique; builds scheduler and apiserver for real on a genuinely dirty tree, starts both daemons, captures --version/-version (expect fingerprint) and each startup log (expect old fingerprint-free format)
  • Apiserver's injected fingerprint value is reused from scheduler's own extracted value (same dirty tree, same content) rather than independently recomputed -- independently assessed by review as legitimate, not tautological: apiserver has no independent fingerprint-computation code of its own in production (only ever receives an externally-injected value via -ldflags), the script still exercises apiserver's own real, unmocked composeVersionDisplay()/logger.Info code paths on a genuinely running compiled binary, and a round-trip api_fp == sched_fp check would catch a real strip-and-append bug
  • Observable confirmed live (independently re-verified by review, fresh full run + a separately-authored 4-case discrimination probe outside the script's own demo block): both daemons' --version/-version show real fingerprints while their startup logs show the unchanged, fingerprint-free old format; no leftover processes, worktrees, or build dirs after either run
  • Requirements: 3.1, 3.2
  • Depends: 3.2, 4.4

• 6.3 Nix production path end-to-end fingerprint verification

  • New script src/tools/test-dirty-fingerprint-nix-production-path.sh (reusing task 6.1's isolated-worktree technique) edits flake.nix, runs nix build ./src#xcecore_debug -L --keep-going (no --impure) twice with different content, and cross-checks nix eval's eval-time myDirtyFingerprint value against the real build log's mesonConfigurePhase flags: line for each
  • This sandbox's nix build dispatches by default to an unreachable-in-practice remote builder (ssh://rshen@172.31.109.18, confirmed genuinely expected behavior, not a misconfiguration) making a directly-runnable binary impractical to retrieve; the eval+build-log cross-check is not a weaker substitute -- independently confirmed structurally sound by review, since self.narHash's value is computed at flake-evaluation time and baked as a literal string into the derivation before any build phase runs, so eval-time and build-time values cannot diverge by construction, not just by empirical coincidence. Forcing --builders '' (local-only) was investigated per a live suggestion and does reach real local .c compilation with a direct log match, but still can't produce a fully linked binary in this sandbox (a second, unrelated network dependency), confirming no stronger approach is currently available; recorded as a learning for future sessions
  • Observable confirmed live (independently re-verified by review, real nix build invocations including a live-witnessed remote-builder connection attempt and a genuine local-only run): two builds with different tracked-file edits produce different fingerprints in both eval and build-log evidence; reverting to the first edit's exact content reproduces the first fingerprint exactly; --impure never used; no git dependency in the build sandbox; flake.nix/nixSupport/ and the worktree list confirmed byte-identical to baseline after every run
  • Requirements: 1.1, 1.2, 1.3, 5.1
  • Depends: 2.2

Implementation Notes

• Task 4.3: a { cmd1; cmd2; } >file or cmd1 | cmd2 | cmd3 shell construct's exit status (as checked by if ! ... ; then ...; fi) only reflects the LAST command's exit status, not any earlier command's -- this silently swallows an earlier failure and can produce fabricated output from partial/incomplete data instead of the intended graceful-degradation path. Took two rejected review rounds to fully generalize (first fixed for git diff HEAD, then found again one step later for git ls-files | sort | xargs). When writing any multi-stage shell pipeline whose failure must be individually detectable, either enable pipefail explicitly (verify the actual invoking shell first -- $(shell ...) in Make does NOT inherit a Makefile's SHELL/.SHELLFLAGS settings) or restructure each stage to its own directly-redirected, individually-$?-checked step (the safer default absent a SHELL:=bash-style pin, per this repo's own documented /bin/sh-is-not-guaranteed-bash incident in build-deployment.md).