--- id: "bug-1798" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "lib" severity: "unknown" timestamp: "2026-09-10T06:57:00Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 files: ["src/lib/meson.build"] tags: ["meson", "build-system", "capnp", "controller.capnp.h", "ninja-race", "missing-dependency-edge", "libxce", "src/lib/meson.build", "root-cause-fix"] related: ["bug-1439"] legacy: true gitea_issue: 267 gitea_synced_status: "fixed" --- # bug-1798: Fresh (from-scratch) build fails intermittently with `fatal error: controller.capnp.h: No such… ## Description Fresh (from-scratch) build fails intermittently with `fatal error: controller.capnp.h: No such file or directory`, reported live by the user running `./tools/xce-build.sh all debug` -- first for a libxce-internal consumer (utils.c.o), then AGAIN after the first fix landed, this time for scheduler/src/sim_record.c.o and scheduler/src/core_events.c.o (files entirely outside libxce). This is the root-cause fix for a class of failures already recorded across many prior recurrences in this buglog (see related_bugs) as an unexplained/unfixed 'ninja-race' -- this entry identifies and fixes the actual missing-dependency-edge cause at the meson.build level, in three parts (see fix), and additionally audited the whole project for other instances of this same bug class. ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause THREE separate but related missing-dependency-edge gaps, all in src/lib/meson.build, all following the identical shape -- a multi-output custom_target() (a .c + a paired .h) whose .h output is reached by some OTHER file only via #include, with libxce_src or libxce_dep only ever referencing the .c output's index [0]: (1) the capnp code-generation custom_target (`gen_srcs`, output controller.capnp.c/.h) -- `libxce_src` only listed `gen_srcs[0]`, so libxce's OWN other .c files (utils.c, hook.c, threshold.c, xlist.c, window.c, xceapi.c, wrap.c, zarray.c, htab.c, xstring.c, etc.) that #include controller.capnp.h had no build edge forcing codegen to finish first. (2) The SAME gen_srcs custom_target was also unreachable, via any dependency edge, from EXTERNAL consumers: `libxce_dep = declare_dependency(link_with: libxce)` had no `sources:` kwarg, so `link_with` only ordered a consumer's LINK step after libxce was built, never a consumer's own COMPILE steps -- and controller/worker/scheduler all reach controller.capnp.h directly via a raw `-Ilib`/`-I../lib` include path, formally depending on nothing. Fix (1) alone only closed the race for files inside libxce itself, which is why the user's SECOND from-scratch `all debug` run still failed, now in scheduler instead. (3) Found via a follow-up audit (user asked 'check source code generated by peg'): the SAME shape recurs for all four packcc PEG-parser custom_targets (section_parser, rusage_parser, xce_resreq_parser, xce_select_parser -- each producing a .c + a paired .h: section.h, rusage.h, xce_resreq.h, xce_select_peg.h). `libxce_src` only listed each parser's `[0]` (.c) index; `lib/resreq.c` (confirmed via a project-wide grep to be the ONLY consumer of any of these four headers, entirely inside src/lib/, so no libxce_dep-level fix was needed for this part) #includes all four headers but has no declared build edge to any of the four codegen steps. On an INCREMENTAL build all three gaps are invisible (the generated files from a prior build already exist on disk), which is why this session's own extensive controller-only `--incremental` rebuilds throughout the xce-test-cli-ux spec never hit any of them -- they only manifest on a genuinely from-scratch build (`rm -rf` + fresh `meson setup`, which `xce-build.sh ` does by default without `--incremental`), and even then only when ninja's scheduler happens to pick an unlucky order under parallelism -- explaining the long history of the capnp variant being reported as an intermittent, unfixed 'ninja-race' across at least 5 prior recorded recurrences (2026-07-16 through 2026-07-26) with no meson.build-level fix ever applied, only the workaround 'retry the build once'. A project-wide search confirmed these are the ONLY multi-output C/H code-generation custom_target()s anywhere in the codebase (every other custom_target() -- xceadmin, frontend/apiserver, frontend/xcecli, autoscaler/providers -- produces a single Go-binary/stamp output with no equivalent header-only-reached-via-#include hazard). ## Fix FIXED (2026-09-10), in three parts, all in src/lib/meson.build. Part 1: changed `libxce_src`'s `gen_srcs[0]` element to the unindexed `gen_srcs`, closing gap (1) for capnp. Part 2 (the fix that actually closes the general, project-wide capnp race): added `sources: [gen_srcs]` to `libxce_dep`'s `declare_dependency()` call, closing gap (2) -- `declare_dependency()`'s `sources:` kwarg exists specifically for this case (generated files that must be built before ANY consumer of the dependency compiles), and since controller/worker/scheduler all already list `libxce_dep` in their own `dependencies:`, this single change propagates the missing build-order edge to every current and future consumer with no per-component changes needed. Part 3: changed `libxce_src`'s `section_parser[0]`/`rusage_parser[0]`/`xce_resreq_parser[0]`/`xce_select_parser[0]` elements to their unindexed forms, closing gap (3) for all four PEG parsers (no libxce_dep change needed here since nothing outside lib/ consumes these headers). Every part uses the same pattern: passing a multi-output custom_target unindexed into a `sources:`-accepting list still compiles only its recognized-source (.c) output, while ordering the WHOLE enclosing target's (or dependency's every consumer's) build after ALL of that custom_target's outputs are generated. (Tried first, and reverted: a `depends: [gen_srcs]` kwarg directly on the `static_library()` call -- rejected immediately by this project's pinned Meson 1.9.1 with 'Got unknown keyword arguments "depends"', not supported on `static_library()`/`build_target()` in this version.) Verified incrementally as each gap was found: two consecutive from-scratch `lib debug` builds passed with Part 1 alone (313/313 lib tests) -- but this was INSUFFICIENT, confirmed live by the user's own second `./tools/xce-build.sh all debug` run still failing in scheduler; after Part 2, two consecutive full from-scratch `./tools/xce-build.sh all debug` builds (1173/1173 targets each, the user's own exact repro command) both succeeded with zero failures and no retry; after adding Part 3 (found via the follow-up PEG audit), a third consecutive full from-scratch `all debug` build also succeeded clean (1173/1173), plus a fresh `lib debug` build + full `xcelib_test` run (313/313) and a fresh incremental `controller asan --incremental` rebuild. That incremental rebuild's FIRST run showed `1331 Tests 1 Failures` in `test_smoke_launch_send_echo_capture_stdout_and_exit_code` ('xce_test did not exit within the harness's safety timeout after quit') -- investigated and confirmed to be an unrelated, system-load-induced timing flake (the controller build ran concurrently with/immediately after the just-completed full `all debug` build): re-running the SAME binary standalone on an otherwise-idle system passed this exact test and the full suite cleanly (1331/1331) twice in a row; this is a pre-existing PTY-harness timing sensitivity, not a defect introduced by any of the three meson.build parts, all of which only affect build-graph ordering, never runtime process-exit behavior. Three consecutive full-project clean-build successes do not, by the race's own probabilistic nature, provide absolute certainty against some other unlucky ninja ordering recurring -- but Parts 1-3 address the actual general-case structural root cause for every multi-output codegen step in the project (every consumer of libxce_dep, plus every internal libxce_src consumer, now has a real build-graph edge on the relevant codegen step), so this is expected to eliminate this entire race class project-wide rather than merely reduce its likelihood. Does not address the separate, distinct apiserver<-libxce.a CGO-link race noted in one prior recurrence (2026-07-26) -- that is a different meson target (frontend/apiserver's go_build custom_target referencing @BUILD_ROOT@/lib/libxce.a via an env-var string, not a `dependencies:`/`libxce_dep` relationship) and remains unfixed; also worth a follow-up ticket for the test_test_cli.c PTY-harness safety-timeout flakiness under system load, observed here but out of this bug's own scope. ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-1439 - Gitea: #267