myxce/.kiro/specs/build-time-versioning/design.md

Design Document

Overview

Purpose: This feature gives every XCE component (controller, scheduler, worker, apiserver, xcecli, xceadmin) a build-time identity — the git commit it was built from and when it was built — visible from the running artifact itself, via a --version flag and (for the four long-running daemons) a startup log line.

Users: Operators and developers diagnosing a running deployment (the motivating incident: a stale scheduler binary kept running a known-fixed crash, and confirming that required manual ls -la mtime checks plus git merge-base --is-ancestor).

Impact: Extends the existing, partial C-side mechanism (controller/scheduler already have a commit-id-only --version via argp, but xce-build.sh never actually sets it, so it is empty on every local build today) to carry a build timestamp too, fills the worker/Go/Python gaps, and fixes the xce-build.sh gap that made the motivating incident possible in the first place.

Goals

• Every component's build artifact carries a git commit identifier (dirty-suffixed when applicable) and a build timestamp.
• --version on every component prints both and exits.
• Controller, scheduler, worker, and apiserver additionally log both at INFO level on startup.
• xce-build.sh (the dev-loop build path implicated in the motivating incident) actually sets these values — today it does not.
• The values are byte-for-byte identical between --version output and the startup log line for the same artifact.

Non-Goals

• Automatically detecting or alerting that a running binary is stale relative to current source (out of scope per requirements' Boundary Context — this feature only makes the comparison possible).
• Version compatibility negotiation between components running different versions.
• Retrofitting version info into already-built/deployed binaries.
• Migrating worker's getopt_long() argument parsing to argp (a much larger, unrelated change) — worker gets a hand-added --version case instead.

Boundary Commitments

This Spec Owns

• The two build-time values themselves (commit identifier, build timestamp) and how each of the six components' build steps receive them.
• --version output and (for the four daemons) the startup log line, for all six components.
• xce-build.sh's computation of both values for the dev-loop build path.
• The nix packages' (xce.nix, xcecli.nix) computation of both values for the production build path.

Out of Boundary

• Any runtime mechanism that compares a running binary's version against "current" source (no such comparison exists or is planned).
• xceadmin's own semantic package version (pyproject.toml's version = "0.1.0") — untouched; build metadata is additive, displayed alongside it, not a replacement for it.
• Worker's broader argument-parsing architecture.

Allowed Dependencies

• The existing commit_id meson option and VERSION_COMMIT_ID/MK_VERSION C mechanism (src/meson.build, src/meson.options, src/include/utils.h) — extended, not replaced.
• The existing Nix flake commit-id derivation (self.shortRev/self.dirtyShortRev, src/flake.nix) — extended with a new build-time value, same wiring style.
• src/tools/go_build's existing env-var-driven go build invocation pattern (already used for CGO flags).
• src/xceadmin/meson.build's existing custom_target → meson-build.sh argument-passing pattern.
• Go's standard -ldflags -X package-variable injection; Cobra's built-in Version field (xcecli already uses Cobra); Click's built-in version_option (xceadmin already uses Click).

Revalidation Triggers

• The commit_id meson option is renamed, removed, or its consumption in src/include/utils.h's MK_VERSION/STR macros changes shape.
• xce-build.sh's MESON_OPTS construction is refactored away from its current per-profile array pattern.
• src/tools/go_build's final go build invocation line changes.
• src/xceadmin/meson-build.sh's step ordering changes (the generated build-info module must land before the pip install step that packages it).

Architecture

Existing Architecture Analysis

• Controller and scheduler already parse --version/-V for free via GNU argp's argp_program_version mechanism (MK_VERSION("xce v0.0.1-", STR(VERSION_COMMIT_ID)), both main.cs) — this feature extends that same macro rather than adding a parallel one.
• VERSION_COMMIT_ID already flows into every C component uniformly via add_project_arguments() at the top-level src/meson.build — the new build-timestamp define follows the identical path, added once, inherited everywhere.
• Worker's getopt_long()-based parsing has no equivalent auto-version behavior and needs an explicit new case.
• git is deliberately absent from the nix devShell's declared buildInputs (src/nixSupport/xce.nix) — the sandboxed nix build path already avoids shelling out to git for exactly this reason (uses Nix's own self.shortRev introspection instead). The dev-loop xce-build.sh path runs inside nix develop, which layers onto the ambient host PATH rather than replacing it, so git IS usable there — and needs to be used there, since xce-build.sh today never sets commit_id at all (the actual gap behind the motivating incident).

Architecture Pattern & Boundary Map

Two independent computation sites (respecting the existing sandboxed-vs-ambient split) feed one resolved value pair into meson, which every language-specific consumption site reads from — never recomputes independently.

graph TB
    DevScript[xce-build.sh git plus date]
    NixPkg[xce.nix xcecli.nix self.shortRev plus preConfigure date]
    MesonOpts[meson commit_id and build_time options]
    CDefine[C VERSION_COMMIT_ID and VERSION_BUILD_TIME defines]
    GoLdflags[go_build ldflags -X injection]
    PyGen[meson-build.sh generated build info module]
    Controller[controller and scheduler argp version]
    Worker[worker getopt_long version case]
    Apiserver[apiserver flag version]
    Xcecli[xcecli cobra version]
    Xceadmin[xceadmin click version_option]

    DevScript --> MesonOpts
    NixPkg --> MesonOpts
    MesonOpts --> CDefine
    MesonOpts --> GoLdflags
    MesonOpts --> PyGen
    CDefine --> Controller
    CDefine --> Worker
    GoLdflags --> Apiserver
    GoLdflags --> Xcecli
    PyGen --> Xceadmin

Architecture Integration:

• Selected pattern: single resolved-value-pair fan-out, matching the existing commit_id mechanism's own shape.
• Domain/feature boundaries: computation (2 sites, environment-specific) is fully separated from consumption (6 sites, one per component) — a consumption site never knows or cares which computation site produced its value.
• Existing patterns preserved: argp version macro (C), Cobra Version field and Click version_option (already the idiomatic mechanism in each respective Go/Python tool), go_build's env-var plumbing, meson-build.sh's positional-arg plumbing.
• New components rationale: sim_prescan_session_count-style small, single-purpose additions only where a gap genuinely exists (worker's version case, Go's ldflags wiring, Python's generated module) — no new abstraction layer introduced.
• Steering compliance: no new external dependency (tech.md); wire/build-format changes are free pre-release (no back-compat burden); TDD required for every new behavior.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
C build (controller/scheduler/worker)	Meson preprocessor defines (existing mechanism)	Carries commit_id (existing) + new build_time into every C translation unit	src/meson.build:30
C CLI (controller/scheduler)	GNU argp (existing)	--version/-V auto-handling via argp_program_version	Extend MK_VERSION macro
C CLI (worker)	getopt_long() (existing)	New explicit --version case	No parser migration
Go build	-ldflags -X (standard Go idiom, new to this repo)	Injects main.GitCommit/main.BuildTime package vars	Via go_build script + Makefile
Go CLI (apiserver)	flag package (existing)	New -version flag	Matches existing single-flag style
Go CLI (xcecli)	Cobra (existing)	rootCmd.Version field (built-in --version)	Zero new parsing code
Python build	Meson custom_target (existing)	New generated build-info module written before pip install	meson-build.sh
Python CLI (xceadmin)	Click (existing)	click.version_option() (built-in --version)	Zero new parsing code
Nix packaging	Nix flake (existing)	self.shortRev/self.dirtyShortRev (existing, eval-time) + a preConfigure build-phase date -u hook for build_time (task 2.2 round 2: builtins.currentTime was tried first but breaks pure-eval nix build, i.e. requires --impure, and yields non-UTC epoch seconds — rejected in review, corrected to a build-phase computation)	xce.nix, xcecli.nix

File Structure Plan

Modified Files

• src/meson.options — add a build_time string option (default ''), alongside the existing commit_id.
• src/meson.build — add -DVERSION_BUILD_TIME= to the existing add_project_arguments() call (line ~30).
• src/include/utils.h — extend MK_VERSION/introduce a companion macro so the version string includes the build time, not just the commit id.
• src/controller/main.c, src/scheduler/src/main.c — update argp_program_version construction to include build time; add the startup INFO log line (each already has one startup INFO call site to extend/follow).
• src/worker/src/main.c — add a --version long option + case (prints the same shape, exits 0); add the startup INFO log line (follows its own existing startup INFO call site).
• src/tools/xce-build.sh — compute commit_id (git, -dirty suffixed) and build_time (UTC date) and pass both via new -Dcommit_id=/-Dbuild_time= entries in MESON_OPTS, falling back to "unknown" if git fails (e.g. not a git checkout).
• src/nixSupport/xce.nix, src/nixSupport/xcecli.nix — a preConfigure build-phase hook runs date -u +%Y-%m-%dT%H:%M:%SZ and appends -Dbuild_time= to mesonFlagsArray (build-time, not eval-time, unlike commit_id's self.shortRev wiring in src/flake.nix/src/nixSupport/default.nix, which is unchanged).
• src/tools/go_build — read XCE_GIT_COMMIT/XCE_BUILD_TIME env vars (empty-safe) and append -ldflags "-X main.GitCommit=... -X main.BuildTime=..." to the final go build invocation.
• src/frontend/apiserver/meson.build, src/frontend/xcecli/meson.build — pass the two new env vars into their custom_target() calls (mirrors existing env var passing).
• src/frontend/Makefile — add the same -ldflags construction (via git/date directly, dev-loop path) to build-server/build-cli targets.
• src/frontend/apiserver/cmd/main.go — declare var (GitCommit, BuildTime string); add a -version flag; add/extend the startup logger.Info(...) line.
• src/frontend/xcecli/cmd/main.go — declare the same two vars; set rootCmd.Version (Cobra); extend its own startup logger.Info(...) line.
• src/xceadmin/meson.build — pass two new positional args (or env vars) to meson-build.sh.
• src/xceadmin/meson-build.sh — write a small generated Python module (e.g. _build_info.py) into the mirrored source tree, before the pip install step, containing the two values (falls back to "unknown" if not provided).
• src/xceadmin/src/xceadmin/cli.py — read the generated module (import guarded for the case it's genuinely absent, e.g. a non-meson editable install) and wire click.version_option() on the top-level group.

New Files

• src/xceadmin/src/xceadmin/_build_info.py — generated at build time only (not checked in); a committed fallback stub (all fields "unknown") ships in-tree so pip install -e . (the documented fast-iteration path in build-deployment.md) still imports cleanly without having run the meson build.

No new C/Go/Python source files beyond the one generated-with-fallback-stub Python module above — every other change extends an existing file's existing responsibility (arg parsing gains a flag, a build script gains an option).

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1, 1.2	Every artifact carries commit id + build time	All 6 consumption sites + both computation sites	MK_VERSION macro, Go ldflags vars, Python _build_info module	Computation → meson → consumption
1.3	Dirty tree marked	xce-build.sh's git computation	commit_id string carries -dirty suffix	n/a
1.4	Unknown when undeterminable	xce-build.sh, meson-build.sh, go_build	fallback to literal "unknown"	n/a
2.1	--version prints and exits	argp (controller/scheduler), worker's new case, apiserver's -version, Cobra Version, Click version_option	six CLI entry points	n/a
2.2	--version and startup log agree	shared macro/var/module per component	same source value read twice, never recomputed	n/a
3.1, 3.2	Startup log line at INFO	controller/scheduler/worker/apiserver startup sequences	existing first-INFO-line call sites, extended	Startup → log
4.1, 4.2	Build tooling confirms embedding	xce-build.sh, nix packages	non-empty resolved value check before/after meson setup/build	Build → verify
4.3	Accurate under every C build profile	src/meson.build's project-wide add_project_arguments()	applies uniformly to asan/debug/release	n/a
5.1, 5.2	Values reflect actual tree/build event	xce-build.sh (git+date at invocation time), nix (self.shortRev at evaluation time + preConfigure date -u at build time)	no hardcoded/stale values	n/a

Components and Interfaces

C Daemons (controller, scheduler, worker)

Version macro

Field	Detail
Intent	Carry commit id + build time into every C daemon's --version output and startup log
Requirements	1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.3

Responsibilities & Constraints

• MK_VERSION (src/include/utils.h) extended to compose "xce v0.0.1-<commit_id> built <build_time>" (or equivalent) from the two new preprocessor defines — a single macro, both consumers (controller/scheduler's argp_program_version, worker's hand-added case) call it, guaranteeing --version and the startup log line render from the identical string.
• Controller/scheduler: their existing single startup INFO line (main.c:365/377+381 respectively, per discovery) gets this version string appended or logged as an immediately-preceding line — never a second, independently-formatted string.
• Worker: new {"version", no_argument, NULL, OPT_VERSION} long option in its existing long_opts[]/getopt_long() switch (main.c:358-364), printing the same macro's output and exiting 0 before any other startup work begins.

Contracts: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Implementation Notes

• Integration: extends src/include/utils.h's existing STR/MK_VERSION macros — no new header.
• Validation: a build with commit_id/build_time unset must still compile and print "unknown" fields, never a blank/garbled string.
• Risks: worker's getopt_long() short-option string ("n:l:vh") must gain a way to reach the new long-only option — use a long-option-only constant (no short equivalent), matching --peers's own existing precedent in the same long_opts[] array.

Go Binaries (apiserver, xcecli)

ldflags version injection

Field	Detail
Intent	Inject build-time values into each Go binary at link time, exposed via each CLI's native version mechanism
Requirements	1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2

Responsibilities & Constraints

• src/tools/go_build (consumed by both apiserver/meson.build and xcecli/meson.build's custom_target()) reads XCE_GIT_COMMIT/XCE_BUILD_TIME from its environment (empty-safe — falls back to "unknown" when unset, e.g. a bare go build invoked by hand) and appends -ldflags "-X main.GitCommit=$v1 -X main.BuildTime=$v2" to its existing go build invocation.
• src/frontend/Makefile's build-server/build-cli targets compute the same two values directly (git/date, dev-loop path, no meson involved) and pass the identical -ldflags shape — same output format as the meson path, different computation site (mirrors the C-side split).
• apiserver/cmd/main.go: package-level var (GitCommit = "unknown"; BuildTime = "unknown"), a new -version bool flag (checked first, before utils.InitLogger() — prints and exits, no logging side effects), and the existing logger.Info("********** Initialize Logger done!") call site extended to also log both values (same convention, "********** <action> done!" is untouched; the version fields ride as structured zap fields on that same line or an adjacent one).
• xcecli/cmd/main.go: same two vars; version string passed into router.Execute(cm, version), which sets rootCmd.Version before rootCmd.Execute() runs (Cobra then handles --version natively — zero new flag-parsing code). Correction (task 4.5): xcecli's startup log is deliberately NOT extended — Requirement 3 scopes the startup-log obligation to "the controller, scheduler, worker, or apiserver daemon" only; xcecli is a one-shot CLI, not a daemon, and was never in Requirement 3's scope.

Contracts: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Implementation Notes

• Integration: go_build's env-var pattern already exists for CGO flags — this reuses it, not a new plumbing mechanism.
• Validation: --version/-version output and the startup log's values must be the exact same two package vars, read twice, never two separate computations.
• Risks: none identified — -ldflags -X only accepts string vars, both fields are strings by construction.

Python (xceadmin)

Generated build-info module

Field	Detail
Intent	Package build-time values into the wheel so xceadmin --version reflects the actual build, not just pyproject.toml's static semantic version
Requirements	1.1, 1.2, 2.1, 4.1, 4.2

Responsibilities & Constraints

• meson-build.sh writes _build_info.py (containing GIT_COMMIT/BUILD_TIME string constants, sourced from two new args/env vars, "unknown" fallback) into the mirrored source tree's xceadmin/ package directory, before the pip install step (so it packages into both the dev venv install and the final wheel) — same ordering discipline the existing schema-sync step already follows (also runs before pip install).
• cli.py's top-level @click.group() gains @click.version_option(version=f"{pkg_version} ({_build_info.GIT_COMMIT}, {_build_info.BUILD_TIME})"), importing _build_info defensively (a committed fallback stub with "unknown" fields ships in-tree so a non-meson pip install -e . — the documented fast-iteration path — still imports cleanly).
• No startup log line (Requirement 3 scopes that to the four long-running daemons only; xceadmin is a one-shot CLI).

Contracts: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Implementation Notes

• Integration: extends meson-build.sh's existing custom_target sequence at exactly one new step, same file-write style already used for schema sync.
• Validation: pip install -e . (no meson) must still import cleanly against the committed fallback stub — this is the one file in this feature that must exist in two states (checked-in fallback vs. build-time-overwritten real values) and both must be valid Python.
• Risks: the generated module must not be checked in with real values (would go stale immediately) — only the "unknown"-only fallback stub is committed; .gitignore (or equivalent) must exclude the real generated file if meson-build.sh ever writes it into the actual source tree rather than only the mirrored build-tree copy (verify this during implementation — the mirroring step at meson-build.sh lines 32-46 should already keep it out of the real source tree, matching how the schema sync step works today).

Build Orchestration

xce-build.sh value computation

Field	Detail
Intent	Close the actual gap behind the motivating incident — the dev-loop build path must set real values, not leave them empty
Requirements	4.1, 4.2, 4.3, 5.1, 5.2

Responsibilities & Constraints

• Computes commit_id via git rev-parse --short HEAD, appending -dirty when git diff --quiet HEAD -- (or git status --porcelain) reports changes; falls back to "unknown" if git itself fails (e.g. run against an extracted source tarball with no .git).
• Computes build_time via date -u +%Y-%m-%dT%H:%M:%SZ.
• Appends -Dcommit_id="$commit_id" -Dbuild_time="$build_time" to MESON_OPTS (xce-build.sh:70-72) for every profile, before the existing meson setup/meson setup --reconfigure calls.

Contracts: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Implementation Notes

• Integration: a small addition right where MESON_OPTS is already constructed — no restructuring of the profile dispatch.
• Validation: the exact scenario from the motivating incident — build via /xce-build scheduler debug, run --version, confirm it shows a real (non-empty, non-"unknown") commit id matching git rev-parse --short HEAD at build time.
• Risks: none — purely additive to an existing array.

Testing Strategy

Unit Tests

• C: a new scheduler/controller/worker test (or a shared lib test if MK_VERSION's logic is factored into a testable helper) asserting the version string contains both the commit id and build time defines when set, and "unknown" fallbacks when unset.
• Go: a test in each cmd package (or a shared helper) asserting -version/--version output contains both GitCommit and BuildTime values.
• Python: a pytest case (xceadmin's existing suite, run by meson-build.sh) asserting xceadmin --version output contains the generated build-info fields, and that the committed fallback stub is itself valid, importable Python.

Integration Tests

• xce-build.sh-driven build (any profile) followed by running --version on the resulting controller/scheduler/worker binaries, asserting a real (non-"unknown") commit id is present — this is the regression test for the motivating incident itself.
• A go_build-driven meson build of apiserver/xcecli, followed by -version, asserting the same.
• xceadmin's existing meson-driven build+test+wheel cycle, followed by installing the wheel and running xceadmin --version.

E2E / Consistency

• For at least one C daemon and one Go binary: start it, capture its startup log line, separately run --version, and assert the two report byte-identical commit id and build time (Requirement 2.2).