myxce/.wolf/bugs/bug-1798.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	legacy	gitea_issue	gitea_synced_status
bug-1798	fixed	2026-09-24		lib	unknown	2026-09-10T06:57:00Z	2026-09-19T09:00:00Z	1	src/lib/meson.build	meson build-system capnp controller.capnp.h ninja-race missing-dependency-edge libxce src/lib/meson.build root-cause-fix	bug-1439	true	267	fixed

bug-1798: Fresh (from-scratch) build fails intermittently with `fatal error: controller.capnp.h: No such…

Description

Fresh (from-scratch) build fails intermittently with fatal error: controller.capnp.h: No such file or directory, reported live by the user running ./tools/xce-build.sh all debug -- first for a libxce-internal consumer (utils.c.o), then AGAIN after the first fix landed, this time for scheduler/src/sim_record.c.o and scheduler/src/core_events.c.o (files entirely outside libxce). This is the root-cause fix for a class of failures already recorded across many prior recurrences in this buglog (see related_bugs) as an unexplained/unfixed 'ninja-race' -- this entry identifies and fixes the actual missing-dependency-edge cause at the meson.build level, in three parts (see fix), and additionally audited the whole project for other instances of this same bug class.

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

THREE separate but related missing-dependency-edge gaps, all in src/lib/meson.build, all following the identical shape -- a multi-output custom_target() (a .c + a paired .h) whose .h output is reached by some OTHER file only via #include, with libxce_src or libxce_dep only ever referencing the .c output's index [0]: (1) the capnp code-generation custom_target (gen_srcs, output controller.capnp.c/.h) -- libxce_src only listed gen_srcs[0], so libxce's OWN other .c files (utils.c, hook.c, threshold.c, xlist.c, window.c, xceapi.c, wrap.c, zarray.c, htab.c, xstring.c, etc.) that #include controller.capnp.h had no build edge forcing codegen to finish first. (2) The SAME gen_srcs custom_target was also unreachable, via any dependency edge, from EXTERNAL consumers: libxce_dep = declare_dependency(link_with: libxce) had no sources: kwarg, so link_with only ordered a consumer's LINK step after libxce was built, never a consumer's own COMPILE steps -- and controller/worker/scheduler all reach controller.capnp.h directly via a raw -Ilib/-I../lib include path, formally depending on nothing. Fix (1) alone only closed the race for files inside libxce itself, which is why the user's SECOND from-scratch all debug run still failed, now in scheduler instead. (3) Found via a follow-up audit (user asked 'check source code generated by peg'): the SAME shape recurs for all four packcc PEG-parser custom_targets (section_parser, rusage_parser, xce_resreq_parser, xce_select_parser -- each producing a .c + a paired .h: section.h, rusage.h, xce_resreq.h, xce_select_peg.h). libxce_src only listed each parser's [0] (.c) index; lib/resreq.c (confirmed via a project-wide grep to be the ONLY consumer of any of these four headers, entirely inside src/lib/, so no libxce_dep-level fix was needed for this part) #includes all four headers but has no declared build edge to any of the four codegen steps. On an INCREMENTAL build all three gaps are invisible (the generated files from a prior build already exist on disk), which is why this session's own extensive controller-only --incremental rebuilds throughout the xce-test-cli-ux spec never hit any of them -- they only manifest on a genuinely from-scratch build (rm -rf + fresh meson setup, which xce-build.sh <component> <profile> does by default without --incremental), and even then only when ninja's scheduler happens to pick an unlucky order under parallelism -- explaining the long history of the capnp variant being reported as an intermittent, unfixed 'ninja-race' across at least 5 prior recorded recurrences (2026-07-16 through 2026-07-26) with no meson.build-level fix ever applied, only the workaround 'retry the build once'. A project-wide search confirmed these are the ONLY multi-output C/H code-generation custom_target()s anywhere in the codebase (every other custom_target() -- xceadmin, frontend/apiserver, frontend/xcecli, autoscaler/providers -- produces a single Go-binary/stamp output with no equivalent header-only-reached-via-#include hazard).

Fix

FIXED (2026-09-10), in three parts, all in src/lib/meson.build. Part 1: changed libxce_src's gen_srcs[0] element to the unindexed gen_srcs, closing gap (1) for capnp. Part 2 (the fix that actually closes the general, project-wide capnp race): added sources: [gen_srcs] to libxce_dep's declare_dependency() call, closing gap (2) -- declare_dependency()'s sources: kwarg exists specifically for this case (generated files that must be built before ANY consumer of the dependency compiles), and since controller/worker/scheduler all already list libxce_dep in their own dependencies:, this single change propagates the missing build-order edge to every current and future consumer with no per-component changes needed. Part 3: changed libxce_src's section_parser[0]/rusage_parser[0]/xce_resreq_parser[0]/xce_select_parser[0] elements to their unindexed forms, closing gap (3) for all four PEG parsers (no libxce_dep change needed here since nothing outside lib/ consumes these headers). Every part uses the same pattern: passing a multi-output custom_target unindexed into a sources:-accepting list still compiles only its recognized-source (.c) output, while ordering the WHOLE enclosing target's (or dependency's every consumer's) build after ALL of that custom_target's outputs are generated. (Tried first, and reverted: a depends: [gen_srcs] kwarg directly on the static_library() call -- rejected immediately by this project's pinned Meson 1.9.1 with 'Got unknown keyword arguments "depends"', not supported on static_library()/build_target() in this version.) Verified incrementally as each gap was found: two consecutive from-scratch lib debug builds passed with Part 1 alone (313/313 lib tests) -- but this was INSUFFICIENT, confirmed live by the user's own second ./tools/xce-build.sh all debug run still failing in scheduler; after Part 2, two consecutive full from-scratch ./tools/xce-build.sh all debug builds (1173/1173 targets each, the user's own exact repro command) both succeeded with zero failures and no retry; after adding Part 3 (found via the follow-up PEG audit), a third consecutive full from-scratch all debug build also succeeded clean (1173/1173), plus a fresh lib debug build + full xcelib_test run (313/313) and a fresh incremental controller asan --incremental rebuild. That incremental rebuild's FIRST run showed 1331 Tests 1 Failures in test_smoke_launch_send_echo_capture_stdout_and_exit_code ('xce_test did not exit within the harness's safety timeout after quit') -- investigated and confirmed to be an unrelated, system-load-induced timing flake (the controller build ran concurrently with/immediately after the just-completed full all debug build): re-running the SAME binary standalone on an otherwise-idle system passed this exact test and the full suite cleanly (1331/1331) twice in a row; this is a pre-existing PTY-harness timing sensitivity, not a defect introduced by any of the three meson.build parts, all of which only affect build-graph ordering, never runtime process-exit behavior. Three consecutive full-project clean-build successes do not, by the race's own probabilistic nature, provide absolute certainty against some other unlucky ninja ordering recurring -- but Parts 1-3 address the actual general-case structural root cause for every multi-output codegen step in the project (every consumer of libxce_dep, plus every internal libxce_src consumer, now has a real build-graph edge on the relevant codegen step), so this is expected to eliminate this entire race class project-wide rather than merely reduce its likelihood. Does not address the separate, distinct apiserver<-libxce.a CGO-link race noted in one prior recurrence (2026-07-26) -- that is a different meson target (frontend/apiserver's go_build custom_target referencing @BUILD_ROOT@/lib/libxce.a via an env-var string, not a dependencies:/libxce_dep relationship) and remains unfixed; also worth a follow-up ticket for the test_test_cli.c PTY-harness safety-timeout flakiness under system load, observed here but out of this bug's own scope.

Verification

Not recorded (migrated from buglog.json).

References

• Related: bug-1439
• Gitea: #267