diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 60d83641..fb1e7a8e 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -21646,10 +21646,10 @@ { "id": "bug-1798", "timestamp": "2026-09-10T06:57:00Z", - "error_message": "Fresh (from-scratch) build fails intermittently with `fatal error: controller.capnp.h: No such file or directory`, reported live by the user running `./tools/xce-build.sh all debug` -- first for a libxce-internal consumer (utils.c.o), then AGAIN after the first fix landed, this time for scheduler/src/sim_record.c.o and scheduler/src/core_events.c.o (files entirely outside libxce). This is the root-cause fix for a class of failures already recorded across many prior recurrences in this buglog (see related_bugs) as an unexplained/unfixed 'ninja-race' -- this entry identifies and fixes the actual missing-dependency-edge cause at the meson.build level, in two parts (see fix).", + "error_message": "Fresh (from-scratch) build fails intermittently with `fatal error: controller.capnp.h: No such file or directory`, reported live by the user running `./tools/xce-build.sh all debug` -- first for a libxce-internal consumer (utils.c.o), then AGAIN after the first fix landed, this time for scheduler/src/sim_record.c.o and scheduler/src/core_events.c.o (files entirely outside libxce). This is the root-cause fix for a class of failures already recorded across many prior recurrences in this buglog (see related_bugs) as an unexplained/unfixed 'ninja-race' -- this entry identifies and fixes the actual missing-dependency-edge cause at the meson.build level, in three parts (see fix), and additionally audited the whole project for other instances of this same bug class.", "file": "src/lib/meson.build", - "root_cause": "TWO separate but related missing-dependency-edge gaps around the same capnp code-generation custom_target (`gen_srcs`, line 6), which produces TWO outputs, controller.capnp.c and controller.capnp.h: (1) `libxce_src` (line ~65) only listed `gen_srcs[0]` (the .c output) as a source of the `static_library('xce', ...)` target, so Ninja had no build edge requiring codegen to finish before compiling libxce's OWN other .c files (utils.c, hook.c, threshold.c, xlist.c, window.c, xceapi.c, wrap.c, zarray.c, htab.c, xstring.c, etc.) that reach controller.capnp.h only via #include. (2) Separately and more broadly, `libxce_dep = declare_dependency(link_with: libxce)` (line 154) -- the dependency object controller/worker/scheduler ALL consume uniformly (`dependencies: [..., libxce_dep, ...]` in each of their own meson.build files) -- had no `sources:` kwarg at all, so `link_with` only ordered the LINK step of a consumer after libxce was fully built; it did NOT order a consumer's own COMPILE steps against controller.capnp.h, which scheduler (and controller/worker) reach directly via a raw `-Ilib`/`-I../lib` include path without ever formally depending on the codegen step. Fix (1) alone only closed the race for files inside libxce itself, which is why the user's SECOND from-scratch `all debug` run still failed, now in scheduler instead. On an INCREMENTAL build both gaps are invisible (the generated header from a prior build already exists on disk), which is why this session's own extensive controller-only `--incremental` rebuilds throughout the xce-test-cli-ux spec never hit either -- they only manifest on a genuinely from-scratch build (`rm -rf` + fresh `meson setup`, which `xce-build.sh ` does by default without `--incremental`), and even then only when ninja's scheduler happens to pick an unlucky order under parallelism -- explaining the long history of this exact symptom being reported as an intermittent, unfixed 'ninja-race' across at least 5 prior recorded recurrences (2026-07-16 through 2026-07-26) with no meson.build-level fix ever applied, only the workaround 'retry the build once'.", - "fix": "FIXED (2026-09-10), in two parts, both in src/lib/meson.build. Part 1: changed `libxce_src`'s first element from `gen_srcs[0]` to the unindexed `gen_srcs`, closing gap (1) -- Meson still compiles only the recognized-source (.c) output, but now orders the entire `static_library('xce', ...)` target's build after BOTH of the custom_target's outputs are generated. Part 2 (the fix that actually closes the general, project-wide race): added `sources: [gen_srcs]` to `libxce_dep`'s `declare_dependency()` call, closing gap (2) -- `declare_dependency()`'s `sources:` kwarg exists specifically for this case (generated files that must be built before ANY consumer of the dependency compiles), and since controller/worker/scheduler all already list `libxce_dep` in their own `dependencies:`, this single change propagates the missing build-order edge to every current and future consumer with no per-component changes needed. (Tried first, and reverted: a `depends: [gen_srcs]` kwarg directly on the `static_library()` call -- rejected immediately by this project's pinned Meson 1.9.1 with 'Got unknown keyword arguments \"depends\"', not supported on `static_library()`/`build_target()` in this version.) Verified: two consecutive from-scratch `src/tools/xce-build.sh lib debug` builds succeeded with Part 1 alone (313/313 lib tests passing) -- but this was INSUFFICIENT, confirmed live by the user's own second `./tools/xce-build.sh all debug` run still failing, now in scheduler/src/{sim_record,core_events}.c.o. After adding Part 2: two consecutive full from-scratch `./tools/xce-build.sh all debug` builds (1173/1173 targets each, the user's own exact repro command) both succeeded with zero failures and no retry; a subsequent incremental `src/tools/xce-build.sh controller asan xce.build --incremental` rebuild (this session's own established validation command) also passed clean, 1331 Tests 0 Failures 0 Ignored, 9/9 meson suites OK, confirming no regression to the controller/xce_test build this session's own xce-test-cli-ux spec work depends on. Two consecutive full-project clean-build successes do not, by the race's own probabilistic nature, provide absolute certainty against some other unlucky ninja ordering recurring -- but Part 2 addresses the actual general-case structural root cause (every consumer of libxce_dep now has a real build-graph edge on the codegen step, not just libxce itself), so it is expected to eliminate this race class project-wide rather than merely reduce its likelihood. Does not address the separate, distinct apiserver<-libxce.a CGO-link race noted in one prior recurrence (2026-07-26) -- that is a different meson target (frontend/apiserver's go_build custom_target referencing @BUILD_ROOT@/lib/libxce.a via an env-var string, not a `dependencies:`/`libxce_dep` relationship) and remains unfixed.", + "root_cause": "THREE separate but related missing-dependency-edge gaps, all in src/lib/meson.build, all following the identical shape -- a multi-output custom_target() (a .c + a paired .h) whose .h output is reached by some OTHER file only via #include, with libxce_src or libxce_dep only ever referencing the .c output's index [0]: (1) the capnp code-generation custom_target (`gen_srcs`, output controller.capnp.c/.h) -- `libxce_src` only listed `gen_srcs[0]`, so libxce's OWN other .c files (utils.c, hook.c, threshold.c, xlist.c, window.c, xceapi.c, wrap.c, zarray.c, htab.c, xstring.c, etc.) that #include controller.capnp.h had no build edge forcing codegen to finish first. (2) The SAME gen_srcs custom_target was also unreachable, via any dependency edge, from EXTERNAL consumers: `libxce_dep = declare_dependency(link_with: libxce)` had no `sources:` kwarg, so `link_with` only ordered a consumer's LINK step after libxce was built, never a consumer's own COMPILE steps -- and controller/worker/scheduler all reach controller.capnp.h directly via a raw `-Ilib`/`-I../lib` include path, formally depending on nothing. Fix (1) alone only closed the race for files inside libxce itself, which is why the user's SECOND from-scratch `all debug` run still failed, now in scheduler instead. (3) Found via a follow-up audit (user asked 'check source code generated by peg'): the SAME shape recurs for all four packcc PEG-parser custom_targets (section_parser, rusage_parser, xce_resreq_parser, xce_select_parser -- each producing a .c + a paired .h: section.h, rusage.h, xce_resreq.h, xce_select_peg.h). `libxce_src` only listed each parser's `[0]` (.c) index; `lib/resreq.c` (confirmed via a project-wide grep to be the ONLY consumer of any of these four headers, entirely inside src/lib/, so no libxce_dep-level fix was needed for this part) #includes all four headers but has no declared build edge to any of the four codegen steps. On an INCREMENTAL build all three gaps are invisible (the generated files from a prior build already exist on disk), which is why this session's own extensive controller-only `--incremental` rebuilds throughout the xce-test-cli-ux spec never hit any of them -- they only manifest on a genuinely from-scratch build (`rm -rf` + fresh `meson setup`, which `xce-build.sh ` does by default without `--incremental`), and even then only when ninja's scheduler happens to pick an unlucky order under parallelism -- explaining the long history of the capnp variant being reported as an intermittent, unfixed 'ninja-race' across at least 5 prior recorded recurrences (2026-07-16 through 2026-07-26) with no meson.build-level fix ever applied, only the workaround 'retry the build once'. A project-wide search confirmed these are the ONLY multi-output C/H code-generation custom_target()s anywhere in the codebase (every other custom_target() -- xceadmin, frontend/apiserver, frontend/xcecli, autoscaler/providers -- produces a single Go-binary/stamp output with no equivalent header-only-reached-via-#include hazard).", + "fix": "FIXED (2026-09-10), in three parts, all in src/lib/meson.build. Part 1: changed `libxce_src`'s `gen_srcs[0]` element to the unindexed `gen_srcs`, closing gap (1) for capnp. Part 2 (the fix that actually closes the general, project-wide capnp race): added `sources: [gen_srcs]` to `libxce_dep`'s `declare_dependency()` call, closing gap (2) -- `declare_dependency()`'s `sources:` kwarg exists specifically for this case (generated files that must be built before ANY consumer of the dependency compiles), and since controller/worker/scheduler all already list `libxce_dep` in their own `dependencies:`, this single change propagates the missing build-order edge to every current and future consumer with no per-component changes needed. Part 3: changed `libxce_src`'s `section_parser[0]`/`rusage_parser[0]`/`xce_resreq_parser[0]`/`xce_select_parser[0]` elements to their unindexed forms, closing gap (3) for all four PEG parsers (no libxce_dep change needed here since nothing outside lib/ consumes these headers). Every part uses the same pattern: passing a multi-output custom_target unindexed into a `sources:`-accepting list still compiles only its recognized-source (.c) output, while ordering the WHOLE enclosing target's (or dependency's every consumer's) build after ALL of that custom_target's outputs are generated. (Tried first, and reverted: a `depends: [gen_srcs]` kwarg directly on the `static_library()` call -- rejected immediately by this project's pinned Meson 1.9.1 with 'Got unknown keyword arguments \"depends\"', not supported on `static_library()`/`build_target()` in this version.) Verified incrementally as each gap was found: two consecutive from-scratch `lib debug` builds passed with Part 1 alone (313/313 lib tests) -- but this was INSUFFICIENT, confirmed live by the user's own second `./tools/xce-build.sh all debug` run still failing in scheduler; after Part 2, two consecutive full from-scratch `./tools/xce-build.sh all debug` builds (1173/1173 targets each, the user's own exact repro command) both succeeded with zero failures and no retry; after adding Part 3 (found via the follow-up PEG audit), a third consecutive full from-scratch `all debug` build also succeeded clean (1173/1173), plus a fresh `lib debug` build + full `xcelib_test` run (313/313) and a fresh incremental `controller asan --incremental` rebuild. That incremental rebuild's FIRST run showed `1331 Tests 1 Failures` in `test_smoke_launch_send_echo_capture_stdout_and_exit_code` ('xce_test did not exit within the harness's safety timeout after quit') -- investigated and confirmed to be an unrelated, system-load-induced timing flake (the controller build ran concurrently with/immediately after the just-completed full `all debug` build): re-running the SAME binary standalone on an otherwise-idle system passed this exact test and the full suite cleanly (1331/1331) twice in a row; this is a pre-existing PTY-harness timing sensitivity, not a defect introduced by any of the three meson.build parts, all of which only affect build-graph ordering, never runtime process-exit behavior. Three consecutive full-project clean-build successes do not, by the race's own probabilistic nature, provide absolute certainty against some other unlucky ninja ordering recurring -- but Parts 1-3 address the actual general-case structural root cause for every multi-output codegen step in the project (every consumer of libxce_dep, plus every internal libxce_src consumer, now has a real build-graph edge on the relevant codegen step), so this is expected to eliminate this entire race class project-wide rather than merely reduce its likelihood. Does not address the separate, distinct apiserver<-libxce.a CGO-link race noted in one prior recurrence (2026-07-26) -- that is a different meson target (frontend/apiserver's go_build custom_target referencing @BUILD_ROOT@/lib/libxce.a via an env-var string, not a `dependencies:`/`libxce_dep` relationship) and remains unfixed; also worth a follow-up ticket for the test_test_cli.c PTY-harness safety-timeout flakiness under system load, observed here but out of this bug's own scope.", "tags": [ "meson", "build-system", diff --git a/src/lib/meson.build b/src/lib/meson.build index 9e0cf787..28aa725f 100644 --- a/src/lib/meson.build +++ b/src/lib/meson.build @@ -76,10 +76,20 @@ libxce_src = [ # bug-1798: pass the whole gen_srcs custom_target (both # before) while ordering the ENTIRE target's build after # BOTH outputs are generated. gen_srcs, - section_parser[0], - rusage_parser[0], - xce_resreq_parser[0], - xce_select_parser[0], + # bug-1798 (same class): each packcc parser custom_target + # below also produces a paired .h output (section.h, + # rusage.h, xce_resreq.h, xce_select_peg.h) that only + # resreq.c #includes -- a separate file from each parser's + # own generated .c. Indexing to [0] dropped every one of + # these headers from libxce_src, leaving resreq.c's compile + # with no declared dependency on any of the four codegen + # steps. Passing each custom_target unindexed compiles only + # its .c output (as before) while ordering this whole + # target's build after both of its outputs. + section_parser, + rusage_parser, + xce_resreq_parser, + xce_select_parser, 'client.c', 'client_failover.c', 'dataq.c',