myxce/.kiro/specs/dirty-content-fingerprint/research.md

Research & Design Decisions

Summary

• Feature: dirty-content-fingerprint
• Discovery Scope: Extension (builds directly on the shipped build-time-versioning feature's commit_id/build_time mechanism)
• Key Findings:
  • Nix's flake evaluator exposes self.narHash at eval-time, reachable from inside xce.nix/xcecli.nix exactly like the already-used self.shortRev/self.dirtyShortRev — empirically confirmed content-sensitive (different tracked-file content produces a different hash; identical content reproduces the identical hash) and requires no git on the sandboxed build's PATH, respecting the same architectural constraint the original feature's research already established.
  • self.narHash has a real, empirically-confirmed blind spot: it reflects modifications to already-tracked files, but does not pick up brand-new untracked files unless they are git add-staged first. This differs from the dev-loop path's planned git diff HEAD + untracked-file-content approach, which naturally includes unstaged untracked files.
  • The dev-loop path (xce-build.sh) has real git available (established precedent from the original feature's task 2.1) and can compute an untracked-file-inclusive fingerprint directly via git diff HEAD plus git ls-files --others --exclude-standard (the exact enumeration git add -A would use, already respecting .gitignore), with zero working-tree mutation.
  • commit_id's existing computation (and its -dirty suffix) must stay completely untouched by this feature -- an earlier design draft that retired -dirty at the computation site was caught and corrected after tracing its blast radius onto the startup log, which reads the same commit_id value and would have silently lost its only modified-tree signal. The -dirty-stripping needed for a clean --version format instead happens only inside the --version-composition call, only when a fingerprint is actually available (see Design Decisions).

Research Log

Does Nix expose an eval-time content hash of the dirty tree, usable inside the sandboxed build without git on PATH?

• Context: The original build-time-versioning feature's nix path (xce.nix/xcecli.nix) gets its -dirty marker for free from self.dirtyShortRev, which is a boolean-ish suffix, not a content hash — it can't distinguish two different dirty states from each other, the same limitation this feature exists to fix for the dev-loop path. Needed to determine, empirically, whether an equivalent content-sensitive value exists for the nix path without shelling out to git inside the sandbox (which would violate the original feature's explicit "no git as a nix sandbox dependency" architectural decision).
• Sources Consulted: Live empirical testing against this repo's actual src/flake.nix, via nix flake metadata ./src --json and a temporary debugNarHash = self.narHash; output added to and then reverted from src/flake.nix.
• Findings:
  1. nix flake metadata ./src --json's locked.narHash field is present even for a dirty git-type flake input, and is reachable from inside the flake's own expressions as self.narHash (confirmed via nix eval ./src#debugNarHash --raw, temporarily added then reverted).
  2. Editing a tracked file in the flake's directory (src/flake.nix itself, via a temporary comment line) changed narHash from sha256-50twgW+... to sha256-FEEP3A...; a second, differently-worded edit changed it again to sha256-CdD8g1...; reverting the file to its exact original committed content (via git checkout --) reproduced the exact original narHash (modulo unrelated background tree changes between captures -- see caveat below). This confirms narHash is a genuine, deterministic content hash of tracked-file state, not merely a "some change exists" boolean.
  3. Creating a new, unstaged, untracked file (src/README.md) inside the flake's directory did not change narHash at all -- even after further editing that same untracked file's content. Only after git add-staging it (still uncommitted) did narHash change. This is a real, confirmed blind spot: unstaged untracked files are invisible to Nix's dirty-tree evaluation.
  4. Caveat on finding #2's revert check: the final "revert to baseline" capture did not exactly match the very first baseline capture, because unrelated .wolf/* bookkeeping files (tracked, modified by this session's OpenWolf hook automation between captures) also feed into the same repo-wide narHash. This is expected -- narHash reflects the entire dirty tree's tracked-file state, not just the file under test -- and does not undermine the core finding (content-determinism), it only means two captures separated by real intervening changes elsewhere in the tree will legitimately differ, exactly as intended.
• Implications: self.narHash is a viable, eval-time-safe basis for a nix-path content fingerprint, with one documented, accepted limitation: it will not reflect a build made dirty only by adding new untracked files that were never staged. Given the nix production path is not the path implicated in the original motivating incident (that was xce-build.sh scheduler debug), and given a workaround requiring git add or git on the sandbox's PATH was already ruled out by the original feature's architectural decision, this asymmetry is accepted and documented rather than engineered around.

Untracked-file enumeration for the dev-loop path, without mutating the working tree

• Context: The dev-loop path (xce-build.sh) needs to include untracked files' content in its fingerprint (unlike the nix path, which has the staging blind spot above) while making zero changes to the developer's working tree (no git add, no git stash).
• Findings: git ls-files --others --exclude-standard lists exactly the untracked files that are not excluded by .gitignore/.git/info/exclude -- the same set git add -A would pick up -- without touching the index or working tree. Combined with git diff HEAD (which already covers all tracked modifications, staged or not, relative to HEAD), concatenating git diff HEAD's output with the raw content of each file git ls-files --others --exclude-standard lists (in a stable, sorted order) and hashing the result gives a fingerprint matching Requirement 1's AC4 exactly ("only from files that would be included by committing all current changes... excluding ignored files").
• Implications: No new git subcommands beyond what's already used elsewhere in this codebase (git diff, git status --porcelain equivalents); no working-tree mutation; deterministic given a fixed input order.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Single shared meson option (matches original feature's pattern)	xce-build.sh/nix each compute their own fingerprint value, both feed a new dirty_fingerprint meson option, consumed once by C/Go/Python exactly like commit_id/build_time already are	Reuses the exact, already-proven single-resolved-value-pair architecture; zero new plumbing pattern to learn	Two computation sites, asymmetric capability (nix misses unstaged untracked files)	Selected -- see Decision below
Compute fingerprint independently per consumption site	Each of the 6 components computes its own fingerprint at its own build step	No new meson option	Directly reintroduces the exact multi-computation-site drift risk the original feature's architecture was designed to avoid	Rejected

Design Decisions

Decision: Extend the existing single-resolved-value-pair architecture with a third value

• Context: The original feature already established "one resolved value pair (commit_id, build_time), two computation sites, six consumption sites reading the same values, never recomputing." This feature adds a third value (the content fingerprint) that is conceptually parallel to commit_id (both describe "what source produced this build") but is only ever non-empty when the tree is dirty.
• Alternatives Considered:
  1. Fold the fingerprint directly into commit_id's own string (e.g., xce-build.sh composes <hash>-<content_hash> and passes that whole string as commit_id) -- rejected: conflates two independently-meaningful values into one opaque string, makes Requirement 2 (clean tree = commit id alone, byte-identical to today) awkward to guarantee at the composition site instead of trivially true by construction.
  2. A wholly separate new meson option (dirty_fingerprint), computed alongside commit_id/build_time by the same two computation sites, consumed only by the composition step that already builds the final version string for --version (not by the startup-log composition path) -- selected.
• Selected Approach: Add dirty_fingerprint as a new meson string option (default ''), computed by xce-build.sh (git diff + untracked file content, sha256 truncated) and by xce.nix/xcecli.nix (self.narHash, truncated the same way, empty when the tree is clean since there is nothing to distinguish). The existing C xce_compose_version()/Go ldflags/Python _build_info consumption sites gain a fourth field, used only by the --version-composition call site of each component, not by the startup-log composition call site -- this is the concrete mechanism satisfying Requirement 3's "--version only, startup log unchanged" boundary, and requires each of the 6 components' main/cli.py to call the composition function twice with different arguments (once for --version's output, once for the startup log's, as today) rather than sharing one fully-identical string as the original feature's Requirement 2.2 did for commit_id/build_time.
• Rationale: Keeps commit_id exactly as-is (Requirement 2 falls out for free -- a clean tree's dirty_fingerprint is simply empty, so nothing is appended), reuses the proven two-computation-sites/single-value-pair shape, and cleanly isolates the --version-only scope to a single new field rather than a startup-log/version-string fork.
• Trade-offs: The two composition call sites (startup log vs. --version) per component, previously guaranteed identical by construction (Requirement 2.2 of the original feature), now diverge by design for this one new field -- acceptable and explicitly scoped by this feature's Requirement 3, but must be clearly commented in each component's source so a future change doesn't "fix" the divergence by accident.
• Follow-up: Verify during implementation that omitting dirty_fingerprint from the startup-log composition call is a one-line, obviously-intentional diff (not something that looks like an oversight) in each of controller/scheduler/worker/apiserver's main.c/main.go.

Decision: Leave commit_id's computation fully untouched; strip its -dirty suffix only inside --version's composition, only when a fingerprint is available

• Context: The already-shipped build-time-versioning feature's commit_id computation unconditionally appends a literal -dirty suffix when the tree has uncommitted changes (xce-build.sh task 2.1, flake.nix's self.dirtyShortRev), and both the --version output and the startup log read that same commit_id value today. An earlier draft of this decision proposed retiring -dirty from commit_id's own computation entirely -- but tracing that change's full blast radius revealed a real regression: since the startup log reads the identical commit_id value, removing -dirty at the computation site would silently strip the startup log's only modified-tree signal too, even though this feature's Requirement 3 explicitly requires the startup log to be left unchanged. That draft under-scoped its own blast radius by only considering the --version surface.
• Alternatives Considered:
  1. Retire -dirty from commit_id's computation entirely (both computation sites) -- rejected after tracing the blast radius: silently regresses the startup log's existing modified-tree signal, a real behavior change to a surface this feature must leave untouched.
  2. Leave commit_id's -dirty suffix untouched everywhere, append dirty_fingerprint after it unconditionally when non-empty -- rejected: produces the redundant <hash>-dirty-<fingerprint> shape when a fingerprint is available, not the plain <commit_id>-<hash> shape the requirements were written around (the user's own stated preference during requirements gathering: "just commit id - hash of changed file contents, if no file has been changed just commit id is enough").
  3. Leave commit_id's computation completely unmodified everywhere (both computation sites, and therefore the startup log too); perform the -dirty-stripping only inside the --version-composition call, and only when dirty_fingerprint is non-empty -- selected.
• Selected Approach: commit_id is never touched by this feature -- identical computation, identical value, at both computation sites, for both consumption surfaces. dirty_fingerprint is computed independently (empty when clean or when the tree is dirty but the hash computation fails; a real content hash when the tree is dirty and computable). The --version-only composition call applies this rule: if dirty_fingerprint is non-empty, strip a trailing -dirty from commit_id (if present) and append -<dirty_fingerprint> instead; if dirty_fingerprint is empty, use commit_id completely as-is -- which, for a genuinely dirty tree where only the fingerprint computation failed, still correctly shows <hash>-dirty (Requirement 5.2's fallback, satisfied automatically by the pre-existing, unmodified -dirty mechanism, with zero new fallback marker/sentinel value needed).
• Rationale: Zero risk to the startup log (its composition call path never changes, doesn't even know dirty_fingerprint exists) and zero risk to any of the original feature's already-shipped tests that assert commit_id's -dirty behavior (none of that behavior changes) -- Requirement 5.2 falls out for free from a mechanism that already exists, rather than needing a new literal sentinel string. Strictly simpler than the rejected alternatives.
• Trade-offs: The stripping logic lives in exactly one place (the --version composition function), so it must be implemented carefully (string-suffix check, not a blind substring removal, in case a commit hash itself could theoretically end in characters resembling "dirty" -- extremely unlikely with hex hashes, but the implementation should anchor the check to the literal -dirty suffix position, not just presence anywhere in the string).
• Follow-up: None of the original feature's already-shipped tests need updating, since commit_id's behavior is provably unchanged -- implementation should add a regression assertion confirming this explicitly (e.g., test-build-version-regression.sh's existing -dirty assertions continue to pass completely unmodified).

Decision: Nix path accepts the untracked-file blind spot rather than working around it

• Context: Empirically confirmed (see Research Log) that self.narHash does not reflect new untracked files unless staged, while the dev-loop path's git diff + git ls-files --others --exclude-standard approach does.
• Alternatives Considered:
  1. Have the nix preConfigure build-phase hook shell out to git add -A --dry-run-style enumeration or otherwise touch git state to force-include untracked files -- rejected: git is deliberately not a nix devShell/build dependency in this repo (original feature's explicit architectural decision), and any workaround here would reintroduce exactly that dependency.
  2. Accept the asymmetry, document it clearly, and let the nix path's fingerprint simply not change for untracked-only dirty deltas -- selected.
• Selected Approach: flake.nix's existing eval-time let block (which already computes myCommitId via self.shortRev/self.dirtyShortRev) also computes myDirtyFingerprint from self.narHash (truncated), passed into xce.nix/xcecli.nix the same way myCommitId already is -- eval-time, via mesonFlags, not through the build-phase preConfigure/mesonFlagsArray mechanism reserved for build_time. No git-plumbing is added to the nix path.
• Rationale: The motivating incident for the entire build-time-versioning feature family was a dev-loop (xce-build.sh) build; the nix production path was never implicated and is not this feature's primary target. A silent, undocumented gap would be a defect; a clearly documented, deliberately scoped one is an acceptable trade-off consistent with the original feature's own precedent (research.md's "two independent computation sites... risk mitigated by identical shape, not identical computation").
• Trade-offs: An operator diagnosing two nix-built dirty artifacts that differ only by an unstaged untracked file addition will see identical dirty_fingerprint values despite the builds genuinely differing -- must be documented in design.md and in the nix path's own code comments, mirroring how the original feature documented its own builtins.currentTime pure-eval correction inline.
• Follow-up: If this gap ever proves operationally significant (e.g., a real incident traced to it), a future spec could revisit having xce-build.sh itself pre-stage untracked files in a way that doesn't require nix to shell out to git -- out of scope here.

Risks & Mitigations

• The --version-only -dirty-stripping logic (see Decision above) must anchor its check to the literal trailing -dirty suffix, not a blind substring removal, to avoid ever mis-stripping a commit hash that coincidentally contains similar characters -- mitigated by keeping the strip logic in exactly one place (the shared composition function) with a dedicated unit test for the boundary case.
• Two computation sites for the same new value (xce-build.sh git-based, nix self.narHash-based) could, in principle, produce different fingerprints for what a human would call "the same change" -- mitigated the same way the original feature's commit_id/build_time already are: identical shape (both are short hex digests, both empty on a clean tree), not identical computation, and each is only ever compared against a build from the same computation site in practice (an operator compares two dev-loop builds against each other, or two nix builds against each other, not across the two).
• The nix path's untracked-file blind spot (documented above) could mislead an operator into thinking two nix-built dirty artifacts are content-identical when they differ only by an added-but-unstaged file -- mitigated by explicit documentation in design.md, a code comment at the myDirtyFingerprint eval-time let-binding in flake.nix, and (per Requirement 5) the general graceful-degradation framing that this indicator is a best-effort distinguishing aid, not a cryptographic guarantee of the full working-tree state.
• Computing git diff HEAD plus untracked-file content on every dev-loop build adds a small amount of build-time cost -- mitigated by scope: this repo's working trees are source-code sized (not data-file sized), and the original feature's own xce-build.sh already runs git rev-parse/git status --porcelain on every invocation with no observed performance concern.

References

• src/flake.nix:12-32 -- existing myCommitId eval-time computation this feature's nix-path decision extends, and the empirical debugNarHash probe site (added and reverted during this research).
• src/flake.nix:12-32 -- existing eval-time myCommitId computation (original feature) that this feature's myDirtyFingerprint extends with the same mechanism, not the sibling preConfigure build-phase date -u hook in src/nixSupport/xce.nix/xcecli.nix (original feature's task 2.2), which stays reserved for build_time and is otherwise untouched.
• src/tools/xce-build.sh:80-119 -- existing commit_id/build_time computation block (original feature's task 2.1) that this feature's dev-loop fingerprint computation extends.
• .kiro/specs/build-time-versioning/research.md -- the original feature's own research log, in particular its "One resolved value pair, fed from two independent computation sites" decision, which this feature's architecture directly continues.