lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(build): close capnp codegen race for controller/worker/scheduler too (bug-1798)

Browse source 
The previous fix (654c6e8c) only closed the missing-dependency-edge race
inside the libxce target itself. A live user report on a second
from-scratch `./tools/xce-build.sh all debug` run showed the same race
still hitting scheduler/src/sim_record.c and core_events.c, which reach
controller.capnp.h directly via a raw -Ilib include path with no
declared dependency on the codegen step at all.

Root cause: libxce_dep = declare_dependency(link_with: libxce) -- the
dependency object controller/worker/scheduler all consume uniformly --
only orders a consumer's LINK step after libxce is built; it never
propagated a build-order dependency on the capnp custom_target to a
consumer's own COMPILE steps.

Fix: add `sources: [gen_srcs]` to libxce_dep's declare_dependency() call.
This kwarg exists specifically for generated files that must be built
before any consumer of a dependency compiles, and since every component
already lists libxce_dep in its own dependencies, this one change
propagates the missing edge everywhere with no per-component changes.

Verified: two consecutive full from-scratch `./tools/xce-build.sh all
debug` builds (1173/1173 targets, the user's own exact repro command)
both succeeded with no retry; a subsequent incremental controller/ASAN
rebuild stayed clean (1331/1331 tests, 0 ASAN diagnostics).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
This commit is contained in:
Rongsong Shen 2026-09-10 07:31:06 +00:00
parent 654c6e8c75
commit c1c6c9a248
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
2 changed files with 14 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

6
.wolf/buglog.json
View file

@ -21646,10 +21646,10 @@
{
"id": "bug-1798",
"timestamp": "2026-09-10T06:57:00Z",
"error_message": "Fresh (from-scratch) build fails intermittently with `fatal error: controller.capnp.h: No such file or directory` for one or more consumers of utils.h (e.g. utils.c.o), reported live by the user running `./tools/xce-build.sh all debug`. This is the root-cause fix for a class of failures already recorded across many prior recurrences in this buglog (see related_bugs) as an unexplained/unfixed 'ninja-race' -- this entry identifies and fixes the actual missing-dependency-edge cause at the meson.build level.",
"error_message": "Fresh (from-scratch) build fails intermittently with `fatal error: controller.capnp.h: No such file or directory`, reported live by the user running `./tools/xce-build.sh all debug` -- first for a libxce-internal consumer (utils.c.o), then AGAIN after the first fix landed, this time for scheduler/src/sim_record.c.o and scheduler/src/core_events.c.o (files entirely outside libxce). This is the root-cause fix for a class of failures already recorded across many prior recurrences in this buglog (see related_bugs) as an unexplained/unfixed 'ninja-race' -- this entry identifies and fixes the actual missing-dependency-edge cause at the meson.build level, in two parts (see fix).",
"file": "src/lib/meson.build",
"root_cause": "The capnp code-generation custom_target (`gen_srcs`, line 6) produces TWO outputs, controller.capnp.c and controller.capnp.h, but `libxce_src` (line ~65) only listed `gen_srcs[0]` (the .c output) as a source of the `static_library('xce', ...)` target. controller.capnp.h is never itself listed as a source anywhere -- it is reached only via `#include \"controller.capnp.h\"` from utils.c and several other library files (hook.c, threshold.c, xlist.c, window.c, xceapi.c, wrap.c, zarray.c, htab.c, xstring.c, etc., all also part of libxce_src). Because indexing a multi-output custom_target (`gen_srcs[0]`) extracts only that one output for Meson's dependency-graph purposes, Ninja has no declared build edge requiring the codegen custom_target to finish before compiling those other, unrelated .c files. On an INCREMENTAL build this is invisible (the generated header from a prior build already exists on disk), which is why this session's own extensive controller-only `--incremental` rebuilds throughout the xce-test-cli-ux spec never hit it -- it only manifests on a genuinely from-scratch build (`rm -rf` + fresh `meson setup`, which `xce-build.sh <component> <profile>` does by default without `--incremental`), and even then only when ninja's scheduler happens to pick an unlucky order under parallelism -- explaining the long history of this exact symptom being reported as an intermittent, unfixed 'ninja-race' across at least 5 prior recorded recurrences (2026-07-16 through 2026-07-26) with no meson.build-level fix ever applied, only the workaround 'retry the build once'.",
"fix": "FIXED (2026-09-10). Changed `libxce_src`'s first element from `gen_srcs[0]` to the unindexed `gen_srcs` (src/lib/meson.build). Passing the whole custom_target object (both outputs) to a target's `sources:` list is Meson's documented pattern for exactly this case: Meson still compiles only the recognized-source (.c) output, but now correctly orders the ENTIRE `static_library('xce', ...)` target's build (every compile job in it) after BOTH of the custom_target's outputs are generated -- closing the missing-dependency-edge gap with no other code or behavior change. (Tried first: adding a `depends: [gen_srcs]` kwarg directly to the `static_library()` call -- rejected immediately by this project's pinned Meson 1.9.1 with 'Got unknown keyword arguments \"depends\"', that kwarg is not supported on `static_library()`/`build_target()` in this version; reverted before landing on the working fix.) Verified: two consecutive from-scratch `src/tools/xce-build.sh lib debug` builds (the exact race scenario -- clean `rm -rf` + fresh `meson setup`, no `--incremental`) both succeeded on the first invocation with no retry; the full `lib` unit test suite (`xcelib_test`) passed 313/313 with 0 failures; a subsequent incremental `src/tools/xce-build.sh controller asan xce.build --incremental` rebuild (this session's own established validation command) also passed clean, 1331 Tests 0 Failures 0 Ignored, 9/9 meson suites OK, confirming no regression to the controller/xce_test build this session's own xce-test-cli-ux spec work depends on. A single or even double clean-build success does not, by the race's own probabilistic nature, provide 100% certainty the underlying scheduling race can never recur in some other unlucky ninja ordering -- but the fix addresses the actual, previously-undiagnosed structural root cause (a genuinely missing build-graph edge), not just a timing workaround, so it is expected to eliminate the race class entirely rather than merely reduce its likelihood. Does not address the separate, distinct apiserver<-libxce.a CGO-link race noted in one prior recurrence (2026-07-26) -- that is a different meson target (frontend/apiserver's go_build custom_target referencing @BUILD_ROOT@/lib/libxce.a via an env-var string, not this custom_target/static_library relationship) and remains unfixed.",
"root_cause": "TWO separate but related missing-dependency-edge gaps around the same capnp code-generation custom_target (`gen_srcs`, line 6), which produces TWO outputs, controller.capnp.c and controller.capnp.h: (1) `libxce_src` (line ~65) only listed `gen_srcs[0]` (the .c output) as a source of the `static_library('xce', ...)` target, so Ninja had no build edge requiring codegen to finish before compiling libxce's OWN other .c files (utils.c, hook.c, threshold.c, xlist.c, window.c, xceapi.c, wrap.c, zarray.c, htab.c, xstring.c, etc.) that reach controller.capnp.h only via #include. (2) Separately and more broadly, `libxce_dep = declare_dependency(link_with: libxce)` (line 154) -- the dependency object controller/worker/scheduler ALL consume uniformly (`dependencies: [..., libxce_dep, ...]` in each of their own meson.build files) -- had no `sources:` kwarg at all, so `link_with` only ordered the LINK step of a consumer after libxce was fully built; it did NOT order a consumer's own COMPILE steps against controller.capnp.h, which scheduler (and controller/worker) reach directly via a raw `-Ilib`/`-I../lib` include path without ever formally depending on the codegen step. Fix (1) alone only closed the race for files inside libxce itself, which is why the user's SECOND from-scratch `all debug` run still failed, now in scheduler instead. On an INCREMENTAL build both gaps are invisible (the generated header from a prior build already exists on disk), which is why this session's own extensive controller-only `--incremental` rebuilds throughout the xce-test-cli-ux spec never hit either -- they only manifest on a genuinely from-scratch build (`rm -rf` + fresh `meson setup`, which `xce-build.sh <component> <profile>` does by default without `--incremental`), and even then only when ninja's scheduler happens to pick an unlucky order under parallelism -- explaining the long history of this exact symptom being reported as an intermittent, unfixed 'ninja-race' across at least 5 prior recorded recurrences (2026-07-16 through 2026-07-26) with no meson.build-level fix ever applied, only the workaround 'retry the build once'.",
"fix": "FIXED (2026-09-10), in two parts, both in src/lib/meson.build. Part 1: changed `libxce_src`'s first element from `gen_srcs[0]` to the unindexed `gen_srcs`, closing gap (1) -- Meson still compiles only the recognized-source (.c) output, but now orders the entire `static_library('xce', ...)` target's build after BOTH of the custom_target's outputs are generated. Part 2 (the fix that actually closes the general, project-wide race): added `sources: [gen_srcs]` to `libxce_dep`'s `declare_dependency()` call, closing gap (2) -- `declare_dependency()`'s `sources:` kwarg exists specifically for this case (generated files that must be built before ANY consumer of the dependency compiles), and since controller/worker/scheduler all already list `libxce_dep` in their own `dependencies:`, this single change propagates the missing build-order edge to every current and future consumer with no per-component changes needed. (Tried first, and reverted: a `depends: [gen_srcs]` kwarg directly on the `static_library()` call -- rejected immediately by this project's pinned Meson 1.9.1 with 'Got unknown keyword arguments \"depends\"', not supported on `static_library()`/`build_target()` in this version.) Verified: two consecutive from-scratch `src/tools/xce-build.sh lib debug` builds succeeded with Part 1 alone (313/313 lib tests passing) -- but this was INSUFFICIENT, confirmed live by the user's own second `./tools/xce-build.sh all debug` run still failing, now in scheduler/src/{sim_record,core_events}.c.o. After adding Part 2: two consecutive full from-scratch `./tools/xce-build.sh all debug` builds (1173/1173 targets each, the user's own exact repro command) both succeeded with zero failures and no retry; a subsequent incremental `src/tools/xce-build.sh controller asan xce.build --incremental` rebuild (this session's own established validation command) also passed clean, 1331 Tests 0 Failures 0 Ignored, 9/9 meson suites OK, confirming no regression to the controller/xce_test build this session's own xce-test-cli-ux spec work depends on. Two consecutive full-project clean-build successes do not, by the race's own probabilistic nature, provide absolute certainty against some other unlucky ninja ordering recurring -- but Part 2 addresses the actual general-case structural root cause (every consumer of libxce_dep now has a real build-graph edge on the codegen step, not just libxce itself), so it is expected to eliminate this race class project-wide rather than merely reduce its likelihood. Does not address the separate, distinct apiserver<-libxce.a CGO-link race noted in one prior recurrence (2026-07-26) -- that is a different meson target (frontend/apiserver's go_build custom_target referencing @BUILD_ROOT@/lib/libxce.a via an env-var string, not a `dependencies:`/`libxce_dep` relationship) and remains unfixed.",
"tags": [
"meson",
"build-system",

12
src/lib/meson.build
View file

@ -151,7 +151,17 @@ libxce = static_library('xce',
install: true)
libxce_dep = declare_dependency(link_with: libxce)
# bug-1798: `sources: [gen_srcs]` forces every consumer of libxce_dep
# (controller/worker/scheduler, none of which otherwise declare any
# dependency on this codegen step even though several of their own .c
# files #include "controller.capnp.h" directly via a raw -Ilib path) to
# build gen_srcs' outputs before compiling their own sources -- closing
# the same missing-dependency-edge race as the libxce_src fix above, but
# for consumers OUTSIDE the libxce target itself. link_with alone does not
# propagate this: it only orders the LINK step after libxce is built, not
# a consumer's own COMPILE steps against a header libxce never installs.
libxce_dep = declare_dependency(link_with: libxce, sources: [gen_srcs])
# libxce_shared_dep = declare_dependency(link_with: libxce_shared)
libxce_test_src = ['testsuite/test.c',