myxce/.kiro/specs/dirty-content-fingerprint/design.md

Design Document

Overview

Purpose: This feature gives every XCE component's --version output the ability to distinguish two builds made from the same commit but with different uncommitted changes — replacing the existing plain -dirty boolean suffix (which cannot tell two dirty builds apart) with a short, content-derived fingerprint.

Users: Developers and operators iterating on a dev-loop build (xce-build.sh) who rebuild repeatedly while editing, and need to tell which specific edit produced the binary in front of them, without falling back to manual file-timestamp or diff inspection.

Impact: Extends the already-shipped build-time-versioning feature's commit_id/build_time mechanism with a third, optional value. commit_id, build_time, and the startup-log output are all left completely untouched — same computation, same value, same call sites. Only the --version-composition call site changes: when a fingerprint is available, it strips commit_id's existing -dirty suffix (if present) and appends the fingerprint instead, producing a clean <commit_id>-<hash> shape without redundant markers; when no fingerprint is available, --version falls back to showing commit_id exactly as computed today (see research.md's "Leave commit_id's computation fully untouched" decision).

Goals

• A dirty build's --version output distinguishes itself from a different dirty build of the same commit, whenever their uncommitted content differs.
• Two builds with identical uncommitted content produce the identical fingerprint, reproducibly.
• A clean build's --version output is completely unchanged (just the commit identifier, as today).
• All six components (controller, scheduler, worker, apiserver, xcecli, xceadmin) get this capability.
• The daemon startup log line is untouched by this feature.

Non-Goals

• Showing which files changed or their diff content anywhere in --version output (fingerprint only, not a diff viewer).
• Extending the startup log line to include the fingerprint (explicitly out of scope per Requirement 3).
• Closing the Nix production path's untracked-file blind spot (documented, accepted limitation — see research.md).
• Any change to commit_id's computation, value, or its existing -dirty suffix, or to build_time's computation — genuinely unaffected by this feature everywhere. The -dirty-stripping needed for a clean --version format is display-only logic inside the --version-composition call, not a change to commit_id itself.

Boundary Commitments

This Spec Owns

• Computation of a new dirty_fingerprint value at both existing computation sites (xce-build.sh, flake.nix).
• Threading that value through the existing single-resolved-value-pair meson-option mechanism as a third option.
• The --version-only composition logic that combines commit_id and dirty_fingerprint for display (including the -dirty-stripping rule, applied only at this one call site — see research.md).

Out of Boundary

• The startup log line's content or format (unchanged; still shows only commit_id/build_time, per the original feature, via its completely untouched composition call site).
• The original feature's commit_id/build_time computation, values, or the -dirty suffix commit_id itself carries — genuinely unmodified everywhere, per the corrected design decision in research.md.
• Any mechanism for an operator to retrieve or view the actual uncommitted diff/file list (out of scope per requirements' Boundary Context).
• Making the Nix path's fingerprint reflect unstaged untracked files (documented limitation, not solved here).

Allowed Dependencies

• The original build-time-versioning feature's meson option mechanism (src/meson.options, src/meson.build's add_project_arguments()), extended with one new option, not replaced.
• xce-build.sh's existing commit_id/build_time computation block and MESON_OPTS construction (task 2.1 of the original feature).
• xce.nix/xcecli.nix's existing preConfigure build-phase hook (task 2.2 of the original feature).
• Each component's existing version-composition function/mechanism (xce_compose_version() for C, the Go ldflags vars, _build_info/cli.py for Python) — extended with one new optional field, not replaced.

Revalidation Triggers

• xce_compose_version()'s signature or call sites change shape (would affect how the new field is threaded through).
• The original feature's commit_id/build_time meson options are renamed or restructured.
• xce-build.sh's MESON_OPTS construction pattern changes.
• self.narHash's semantics change in a future Nix version (unlikely, but the design's Nix-path assumption depends on it).

Architecture

Existing Architecture Analysis

• The original feature established: two computation sites (xce-build.sh for dev-loop, xce.nix/xcecli.nix for production) each resolve commit_id/build_time into meson options; six consumption sites each read those options once and never recompute. This feature adds a third value to that exact same shape.
• Each of the six components already has exactly one place where its version string is composed for --version and, separately (for the four daemons), one place where the startup log's version fields are logged — both currently reading the identical commit_id/build_time values. This feature's --version-only scope means the --version composition call site gains the new field while the startup-log composition call site does not — the first deliberate divergence between those two call sites in this feature family.

Architecture Pattern & Boundary Map

Same two computation sites, one additional resolved value, fed only into each component's --version composition call — never into its startup-log call.

graph TB
    DevScript[xce-build.sh git diff plus untracked files sha256]
    NixPkg[flake.nix eval time self.narHash truncated]
    MesonOpt[meson dirty_fingerprint option]
    VersionCompose[per-component --version composition]
    LogCompose[per-component startup log composition unchanged]

    DevScript --> MesonOpt
    NixPkg --> MesonOpt
    MesonOpt --> VersionCompose
    MesonOpt -.not consumed.-> LogCompose

Architecture Integration:

• Selected pattern: extend the existing single-resolved-value-pair fan-out with a third, optional value that only one of each component's two existing consumption call sites reads.
• Domain/feature boundaries: computation (2 sites, unchanged locations) stays separated from consumption; the new divergence is which consumption call site (per component) reads the new value, not a new computation or plumbing layer.
• Existing patterns preserved: meson option → C define / Go ldflags / Python generated module, identical to the original feature.
• New components rationale: no new components — every change extends an existing computation site or an existing composition call site with one additional field/argument.
• Steering compliance: no new external dependency; TDD required for every new behavior (per testing.md).

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Dev-loop computation	git diff HEAD + git ls-files --others --exclude-standard, sha256sum (coreutils, already used by xce-build.sh)	Computes the fingerprint from tracked-modification diffs and untracked-file content	xce-build.sh
Nix production computation	self.narHash (Nix flake evaluator, eval-time)	Content hash of the dirty tree's tracked-file state, truncated the same way	xce.nix, xcecli.nix; empirically confirmed reachable and content-sensitive (see research.md)
C build	Meson preprocessor define (existing mechanism, one new define)	Carries dirty_fingerprint into xce_compose_version()'s --version-composition call only	src/meson.build
Go build	-ldflags -X (existing mechanism, one new var)	Injects main.DirtyFingerprint, read only by each binary's --version/-version handling	go_build, Makefile
Python build	Meson custom_target (existing mechanism, one new field)	_build_info.py gains a third field, read only by cli.py's click.version_option()	meson-build.sh

File Structure Plan

Modified Files

• src/meson.options — add a dirty_fingerprint string option (default ''), sibling to the existing commit_id/build_time.
• src/meson.build — add -DVERSION_DIRTY_FINGERPRINT= to the existing add_project_arguments() call.
• src/include/utils.h, src/lib/utils.c — extend xce_compose_version() (or add a sibling composition function used only by --version call sites) to optionally append -<dirty_fingerprint> when non-empty; leave the existing signature's startup-log call sites unaffected (they simply don't pass the new field, or call the original unchanged function).
• src/controller/main.c, src/scheduler/src/main.c, src/controller/testsuite/test.c, src/worker/src/main.c — --version/version-flag handling call sites gain the new field; startup-log call sites are explicitly left unchanged (one-line, clearly-commented divergence — see research.md's Design Decision).
• src/tools/xce-build.sh — compute dirty_fingerprint (git diff + untracked file content, sha256 truncated, empty when clean) and pass it via a new -Ddirty_fingerprint= entry in MESON_OPTS, alongside the existing commit_id/build_time entries.
• src/flake.nix — the existing eval-time let block gains myDirtyFingerprint (derived from self.narHash, truncated, empty on a clean tree), computed alongside myCommitId and passed into nixSupport/default.nix as a new argument, the same way myCommitId already is.
• src/nixSupport/default.nix — accepts and forwards myDirtyFingerprint into each callPackage ./xce.nix/./xcecli.nix call via inherit, alongside the existing myCommitId.
• src/nixSupport/xce.nix, src/nixSupport/xcecli.nix — accept the new myDirtyFingerprint argument and append "-Ddirty_fingerprint=" + myDirtyFingerprint to the existing mesonFlags list (eval-time, alongside -Dcommit_id=) — not through the preConfigure/mesonFlagsArray mechanism, which stays reserved for build_time.
• src/tools/go_build — read XCE_DIRTY_FINGERPRINT (empty-safe) and append it to the existing -ldflags -X construction as main.DirtyFingerprint.
• src/frontend/apiserver/meson.build, src/frontend/xcecli/meson.build — pass the new env var into go_build's env, same pattern as XCE_GIT_COMMIT/XCE_BUILD_TIME.
• src/frontend/Makefile — compute and pass the same -ldflags -X main.DirtyFingerprint=... for build-server/build-cli's dev-loop path.
• src/frontend/apiserver/cmd/main.go — declare var DirtyFingerprint = ""; -version flag's composed string appends it when non-empty; startup log's logger.Info(...) call is left unchanged (no new zap field).
• src/frontend/xcecli/cmd/main.go — declare the same var; the version string passed to router.Execute(cm, version) (which sets rootCmd.Version) appends it when non-empty.
• src/xceadmin/meson.build — pass one new positional arg (or env var) to meson-build.sh.
• src/xceadmin/meson-build.sh — _build_info.py gains a third field, DIRTY_FINGERPRINT (empty-string fallback, matching the existing per-field unknown-vs-empty convention used for the other two fields — see Implementation Notes).
• src/xceadmin/src/xceadmin/_build_info.py (committed fallback stub) — gains DIRTY_FINGERPRINT = "" alongside the existing two "unknown" fields.
• src/xceadmin/src/xceadmin/cli.py — _compose_version() appends the fingerprint when non-empty.

No new files. Every change extends an existing computation site, an existing meson option chain, or an existing composition call site.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1, 1.2, 1.3	Content-derived indicator present when dirty, deterministic, distinguishes different uncommitted content	xce-build.sh (git diff + untracked files), nix self.narHash	dirty_fingerprint meson option, per-component --version composition	Computation → meson → --version composition
1.4	Excludes ignored files	xce-build.sh's git ls-files --others --exclude-standard (respects .gitignore natively); nix's self.narHash (only ever sees files the flake's git input would include, which already excludes ignored files)	n/a	n/a
2.1	Clean tree shows commit id alone	All 6 --version composition call sites	dirty_fingerprint empty string → no suffix appended	n/a
3.1, 3.2	--version-only scope; startup log unchanged	Each daemon's two separate composition call sites	Startup-log call site never receives dirty_fingerprint	n/a
4.1	Applies to all 6 components	All 6 consumption sites	Same as original feature's Requirement 1.1/1.2 fan-out, one new field	n/a
5.1, 5.2	Graceful degradation when content can't be determined	Every --version composition call site (falls back to commit_id as-is when dirty_fingerprint is empty)	dirty_fingerprint empty on computation failure; commit_id's own pre-existing, untouched -dirty suffix supplies the "still indicate modified" signal for free	n/a

Components and Interfaces

C Daemons (controller, scheduler, worker)

Version composition, --version-only extension

Field	Detail
Intent	Append a content-derived fingerprint to --version's commit id when the tree is dirty, without touching the startup log's version fields
Requirements	1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 4.1

Responsibilities & Constraints

• commit_id and its existing -dirty suffix logic (from the original build-time-versioning feature) are not modified — identical computation, identical value, at both computation sites.
• xce_compose_version() (or a new sibling function reusing its internals) gains an optional dirty_fingerprint parameter, consumed only by the --version call path: when dirty_fingerprint is non-empty, the composition strips a trailing -dirty from commit_id (if present) and appends -<dirty_fingerprint> instead, producing <commit_id>-<hash>; when dirty_fingerprint is empty, the composition uses commit_id completely as-is — which already correctly shows either the bare hash (clean tree) or <hash>-dirty (tree is dirty but the fingerprint computation failed, Requirement 5.2's fallback, satisfied by the pre-existing mechanism with no new sentinel value needed).
• Each main.c's --version/argp_program_version call site passes the new field; the corresponding startup-log INFO call site is not touched at all — it keeps calling the original, unmodified composition path exactly as the original feature shipped it, so it never even references dirty_fingerprint.
• Worker's hand-added --version case (from the original feature) gets the same treatment as controller/scheduler.

Contracts: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Implementation Notes

• Integration: extends xce_compose_version() — check during implementation whether adding a parameter to the existing function is cleaner than adding a second function; either is acceptable as long as the startup-log call site's behavior is provably unchanged (byte-for-byte) by a regression test.
• Validation: (1) a build with dirty_fingerprint empty and a clean tree must produce --version output byte-identical to the original feature's already-shipped clean-tree behavior (<commit_id> alone); (2) a build with dirty_fingerprint empty and a dirty tree (fingerprint computation failed) must produce --version output byte-identical to the original feature's already-shipped dirty-tree behavior (<commit_id>-dirty) — Requirement 5.2, satisfied without new code beyond "don't touch commit_id when dirty_fingerprint is empty"; (3) a build with a real, non-empty dirty_fingerprint must produce <bare-commit-hash>-<dirty_fingerprint>, with the -dirty suffix stripped, not doubled.
• Risks: (1) accidentally threading the new field into the startup-log call site would silently violate Requirement 3.2 — must be caught by a test asserting the startup log line is unchanged from the original feature's already-shipped format; (2) the -dirty-stripping logic must anchor to the literal trailing suffix (not a blind substring removal) — covered by a dedicated unit test.

Go Binaries (apiserver, xcecli)

ldflags fingerprint injection, --version/-version-only

Field	Detail
Intent	Same --version-only extension for the Go toolchain
Requirements	1.1, 1.2, 1.3, 1.4, 2.1, 3.1, 3.2, 4.1

Responsibilities & Constraints

• go_build reads XCE_DIRTY_FINGERPRINT (empty-safe) and appends -X main.DirtyFingerprint=$v to its existing -ldflags construction. GitCommit itself (and whatever -dirty suffix it already carries) is completely untouched.
• apiserver/cmd/main.go: the -version flag's composition applies the same strip-and-append rule as the C daemons — if DirtyFingerprint is non-empty, strip a trailing -dirty from GitCommit and append -DirtyFingerprint; if empty, print GitCommit as-is (already correctly showing -dirty or not). The existing logger.Info("********** Initialize Logger done!") structured-field call is left completely unchanged (no new zap field, GitCommit passed exactly as before) — this is the explicit Requirement 3.2 boundary for this component.
• xcecli/cmd/main.go: the version string passed into router.Execute(cm, version) applies the same strip-and-append rule; xcecli has no startup log (already established as out of scope by the original feature), so there is no divergence to manage here.

Contracts: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Implementation Notes

• Integration: reuses go_build's existing env-var pattern, one more variable.
• Validation: apiserver's startup log must be provably unchanged (byte-for-byte against the original feature's shipped format) even when DirtyFingerprint is non-empty.
• Risks: none identified beyond the same one already flagged for C — accidental leakage of the new field into the startup-log call site.

Python (xceadmin)

Generated build-info module, third field

Field	Detail
Intent	Same --version-only extension for xceadmin, which has no startup log at all (one-shot CLI, already out of scope per the original feature)
Requirements	1.1, 1.2, 1.3, 1.4, 2.1, 4.1

Responsibilities & Constraints

• meson-build.sh writes DIRTY_FINGERPRINT (empty-string fallback) into _build_info.py alongside the existing GIT_COMMIT/BUILD_TIME fields. GIT_COMMIT itself (which already carries a -dirty suffix on a dirty tree, inherited from the shared commit_id meson option) is completely untouched.
• The committed fallback stub (src/xceadmin/src/xceadmin/_build_info.py) gains DIRTY_FINGERPRINT = "" — an editable install (pip install -e .) shows no fingerprint suffix, consistent with the stub's existing all-unknown/empty convention.
• cli.py's _compose_version() applies the same strip-and-append rule as the other components: if DIRTY_FINGERPRINT is non-empty, strip a trailing -dirty from GIT_COMMIT and append -DIRTY_FINGERPRINT; if empty, use GIT_COMMIT as-is.

Contracts: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Implementation Notes

• Integration: one more field in the same generated-module mechanism the original feature already built (task 5.1/5.2).
• Validation: editable install must still show no fingerprint (empty field, no import error); wheel install (built dirty) must show a real, non-empty fingerprint.
• Risks: none identified — xceadmin already has no startup log to accidentally leak into.

Build Orchestration

Dev-loop fingerprint computation

Field	Detail
Intent	Compute a content-based fingerprint of the uncommitted tree, with zero working-tree mutation
Requirements	1.1, 1.2, 1.3, 1.4, 5.1, 5.2

Responsibilities & Constraints

• commit_id and its existing -dirty suffix (xce-build.sh task 2.1 of the original feature) are not touched by this computation — this task only adds a new, independent dirty_fingerprint computation alongside it.
• Computes dirty_fingerprint as one of two states: empty when the tree is clean, or when the tree is dirty but the hash computation itself fails (in the latter case, commit_id's own pre-existing, untouched -dirty suffix already signals "modified" once --version falls back to using it as-is — Requirement 5.2, satisfied with no new sentinel value); a real hash (sha256sum truncated to a short hex prefix) over the concatenation of git diff HEAD's output plus the raw content of each file git ls-files --others --exclude-standard lists, in stable sorted order, when the tree is dirty and the computation succeeds.

Contracts: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Implementation Notes

• Integration: added to the same xce-build.sh block that already computes commit_id/build_time (original feature's task 2.1), no restructuring.
• Validation: the exact scenario this feature exists for — edit a file, build, note the fingerprint; edit further, rebuild, confirm a different fingerprint; revert to the first edit's exact content, rebuild, confirm the fingerprint matches the first build's again.
• Risks: git diff HEAD output size for a very large uncommitted change set is unbounded in principle — acceptable for this codebase's realistic dev-loop usage (source-code-sized diffs), not a concern requiring truncation of the input, only the output hash.

Nix production fingerprint computation

Field	Detail
Intent	Same capability for the Nix production path, within its sandboxed, no-git constraint
Requirements	1.1, 1.2, 1.3, 5.1

Responsibilities & Constraints

• myCommitId and its existing self.dirtyShortRev-derived -dirty suffix (original feature) are not touched by this computation.
• myDirtyFingerprint is derived from self.narHash at eval time, in flake.nix's existing let block, alongside myCommitId — not via a preConfigure build-phase hook. Unlike build_time (which needed a build-phase hook specifically because builtins.currentTime breaks pure-eval), self.narHash is a legitimate flake self-introspection field with no such trap — empirically confirmed reachable and stable under plain nix build/nix eval (no --impure) in research.md. It is passed into xce.nix/xcecli.nix (via nixSupport/default.nix) as a new function argument, mirroring myCommitId's existing parameter, and appended to each derivation's mesonFlags list directly ("-Ddirty_fingerprint=" + myDirtyFingerprint), the exact same way -Dcommit_id= already is — not through mesonFlagsArray/preConfigure, which stays reserved for genuinely build-phase-only values like build_time.
• Resolves to the same two states as the dev-loop path: empty when the tree is clean, checked the same way myCommitId already branches on hasAttr "shortRev" (a clean tree only has self.shortRev, not self.dirtyShortRev); otherwise a real hash, truncated to match the dev-loop path's format.
• Known, accepted limitation (see research.md): does not reflect brand-new untracked files unless staged. Must be documented as a code comment at this let-binding, not silently left as a surprise.

Contracts: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Implementation Notes

• Integration: extends flake.nix's existing eval-time myCommitId computation with one sibling value, passed through to xce.nix/xcecli.nix as one more mesonFlags entry — no new build-phase mechanism, no change to the existing build_time preConfigure hook.
• Validation: nix build .#xcecore_debug (no --impure, the documented canonical command) must still succeed and produce a binary whose --version fingerprint changes when a tracked file in the flake's source is edited and the flake is re-evaluated.
• Risks: low — research.md already empirically confirmed self.narHash is reachable and content-sensitive without --impure; the implementer should still re-confirm live as part of RED-phase validation rather than merely trusting this document, since the earlier build_time/builtins.currentTime mistake in the original feature came from exactly that kind of unverified assumption.

Testing Strategy

Unit Tests

• C: test_utils.c cases asserting xce_compose_version()'s --version-path composition: (1) dirty_fingerprint non-empty + commit_id ending in -dirty → strips the suffix, appends the fingerprint (<hash>-<fingerprint>, not <hash>-dirty-<fingerprint>); (2) dirty_fingerprint empty + clean commit_id → <hash> alone, unchanged; (3) dirty_fingerprint empty + commit_id ending in -dirty (fingerprint computation failed, tree still dirty) → <hash>-dirty, Requirement 5.2's fallback, produced by simply not touching commit_id — no special-casing needed in the composition function beyond the empty check.
• Shell: a unit-level check of xce-build.sh's fingerprint-computation logic confirms it emits a real hash when the tree is dirty and the computation succeeds, and confirms it emits empty (not a fabricated value) both when the tree is clean and when the hash-computation step is made to fail (simulated) on a dirty tree — the latter case relying on commit_id's own separate, untouched -dirty computation to still convey "modified" once composed.
• Go: a test asserting -version/--version output includes DirtyFingerprint when set, and that apiserver's startup-log-composition path is unaffected by its presence.
• Python: a pytest case asserting xceadmin --version includes the fingerprint field when _build_info.DIRTY_FINGERPRINT is non-empty, and omits any suffix when empty (both the committed-stub and generated-module cases, mirroring the original feature's task 5.2 test pattern).

Integration Tests

• xce-build.sh-driven dev-loop build (any C daemon): edit a tracked file, build, capture --version's fingerprint; edit again with different content, rebuild, confirm the fingerprint differs; revert to the first edit's exact content, rebuild, confirm the fingerprint matches the first capture exactly. This is the direct regression test for Requirement 1's core capability.
• Same daemon, confirm the startup log line is untouched (byte-identical to the original build-time-versioning feature's already-shipped format) regardless of whether the tree is dirty.
• A clean-tree build (git stash the dev tree, or run in a freshly-cloned worktree) confirms --version shows the commit id alone, no suffix — the direct regression test for Requirement 2.

E2E / Consistency

• For the Nix path: confirm nix build .#xcecore_debug (no --impure) succeeds and the resulting binary's --version fingerprint changes when a tracked source file is edited before evaluation — the direct regression test for the self.narHash-based mechanism actually working end-to-end, not just at the nix eval probe level this design's research was based on.