Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
41 lines
8.6 KiB
Markdown
41 lines
8.6 KiB
Markdown
---
|
|
id: "bug-1798"
|
|
status: "fixed"
|
|
status_updated: "2026-09-24"
|
|
status_note: ""
|
|
component: "lib"
|
|
severity: "unknown"
|
|
timestamp: "2026-09-10T06:57:00Z"
|
|
last_seen: "2026-09-19T09:00:00Z"
|
|
occurrences: 1
|
|
files: ["src/lib/meson.build"]
|
|
tags: ["meson", "build-system", "capnp", "controller.capnp.h", "ninja-race", "missing-dependency-edge", "libxce", "src/lib/meson.build", "root-cause-fix"]
|
|
related: ["bug-1439"]
|
|
legacy: true
|
|
gitea_issue: 267
|
|
gitea_synced_status: "fixed"
|
|
---
|
|
|
|
# bug-1798: Fresh (from-scratch) build fails intermittently with `fatal error: controller.capnp.h: No such…
|
|
|
|
## Description
|
|
Fresh (from-scratch) build fails intermittently with `fatal error: controller.capnp.h: No such file or directory`, reported live by the user running `./tools/xce-build.sh all debug` -- first for a libxce-internal consumer (utils.c.o), then AGAIN after the first fix landed, this time for scheduler/src/sim_record.c.o and scheduler/src/core_events.c.o (files entirely outside libxce). This is the root-cause fix for a class of failures already recorded across many prior recurrences in this buglog (see related_bugs) as an unexplained/unfixed 'ninja-race' -- this entry identifies and fixes the actual missing-dependency-edge cause at the meson.build level, in three parts (see fix), and additionally audited the whole project for other instances of this same bug class.
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
THREE separate but related missing-dependency-edge gaps, all in src/lib/meson.build, all following the identical shape -- a multi-output custom_target() (a .c + a paired .h) whose .h output is reached by some OTHER file only via #include, with libxce_src or libxce_dep only ever referencing the .c output's index [0]: (1) the capnp code-generation custom_target (`gen_srcs`, output controller.capnp.c/.h) -- `libxce_src` only listed `gen_srcs[0]`, so libxce's OWN other .c files (utils.c, hook.c, threshold.c, xlist.c, window.c, xceapi.c, wrap.c, zarray.c, htab.c, xstring.c, etc.) that #include controller.capnp.h had no build edge forcing codegen to finish first. (2) The SAME gen_srcs custom_target was also unreachable, via any dependency edge, from EXTERNAL consumers: `libxce_dep = declare_dependency(link_with: libxce)` had no `sources:` kwarg, so `link_with` only ordered a consumer's LINK step after libxce was built, never a consumer's own COMPILE steps -- and controller/worker/scheduler all reach controller.capnp.h directly via a raw `-Ilib`/`-I../lib` include path, formally depending on nothing. Fix (1) alone only closed the race for files inside libxce itself, which is why the user's SECOND from-scratch `all debug` run still failed, now in scheduler instead. (3) Found via a follow-up audit (user asked 'check source code generated by peg'): the SAME shape recurs for all four packcc PEG-parser custom_targets (section_parser, rusage_parser, xce_resreq_parser, xce_select_parser -- each producing a .c + a paired .h: section.h, rusage.h, xce_resreq.h, xce_select_peg.h). `libxce_src` only listed each parser's `[0]` (.c) index; `lib/resreq.c` (confirmed via a project-wide grep to be the ONLY consumer of any of these four headers, entirely inside src/lib/, so no libxce_dep-level fix was needed for this part) #includes all four headers but has no declared build edge to any of the four codegen steps. On an INCREMENTAL build all three gaps are invisible (the generated files from a prior build already exist on disk), which is why this session's own extensive controller-only `--incremental` rebuilds throughout the xce-test-cli-ux spec never hit any of them -- they only manifest on a genuinely from-scratch build (`rm -rf` + fresh `meson setup`, which `xce-build.sh <component> <profile>` does by default without `--incremental`), and even then only when ninja's scheduler happens to pick an unlucky order under parallelism -- explaining the long history of the capnp variant being reported as an intermittent, unfixed 'ninja-race' across at least 5 prior recorded recurrences (2026-07-16 through 2026-07-26) with no meson.build-level fix ever applied, only the workaround 'retry the build once'. A project-wide search confirmed these are the ONLY multi-output C/H code-generation custom_target()s anywhere in the codebase (every other custom_target() -- xceadmin, frontend/apiserver, frontend/xcecli, autoscaler/providers -- produces a single Go-binary/stamp output with no equivalent header-only-reached-via-#include hazard).
|
|
|
|
## Fix
|
|
FIXED (2026-09-10), in three parts, all in src/lib/meson.build. Part 1: changed `libxce_src`'s `gen_srcs[0]` element to the unindexed `gen_srcs`, closing gap (1) for capnp. Part 2 (the fix that actually closes the general, project-wide capnp race): added `sources: [gen_srcs]` to `libxce_dep`'s `declare_dependency()` call, closing gap (2) -- `declare_dependency()`'s `sources:` kwarg exists specifically for this case (generated files that must be built before ANY consumer of the dependency compiles), and since controller/worker/scheduler all already list `libxce_dep` in their own `dependencies:`, this single change propagates the missing build-order edge to every current and future consumer with no per-component changes needed. Part 3: changed `libxce_src`'s `section_parser[0]`/`rusage_parser[0]`/`xce_resreq_parser[0]`/`xce_select_parser[0]` elements to their unindexed forms, closing gap (3) for all four PEG parsers (no libxce_dep change needed here since nothing outside lib/ consumes these headers). Every part uses the same pattern: passing a multi-output custom_target unindexed into a `sources:`-accepting list still compiles only its recognized-source (.c) output, while ordering the WHOLE enclosing target's (or dependency's every consumer's) build after ALL of that custom_target's outputs are generated. (Tried first, and reverted: a `depends: [gen_srcs]` kwarg directly on the `static_library()` call -- rejected immediately by this project's pinned Meson 1.9.1 with 'Got unknown keyword arguments "depends"', not supported on `static_library()`/`build_target()` in this version.) Verified incrementally as each gap was found: two consecutive from-scratch `lib debug` builds passed with Part 1 alone (313/313 lib tests) -- but this was INSUFFICIENT, confirmed live by the user's own second `./tools/xce-build.sh all debug` run still failing in scheduler; after Part 2, two consecutive full from-scratch `./tools/xce-build.sh all debug` builds (1173/1173 targets each, the user's own exact repro command) both succeeded with zero failures and no retry; after adding Part 3 (found via the follow-up PEG audit), a third consecutive full from-scratch `all debug` build also succeeded clean (1173/1173), plus a fresh `lib debug` build + full `xcelib_test` run (313/313) and a fresh incremental `controller asan --incremental` rebuild. That incremental rebuild's FIRST run showed `1331 Tests 1 Failures` in `test_smoke_launch_send_echo_capture_stdout_and_exit_code` ('xce_test did not exit within the harness's safety timeout after quit') -- investigated and confirmed to be an unrelated, system-load-induced timing flake (the controller build ran concurrently with/immediately after the just-completed full `all debug` build): re-running the SAME binary standalone on an otherwise-idle system passed this exact test and the full suite cleanly (1331/1331) twice in a row; this is a pre-existing PTY-harness timing sensitivity, not a defect introduced by any of the three meson.build parts, all of which only affect build-graph ordering, never runtime process-exit behavior. Three consecutive full-project clean-build successes do not, by the race's own probabilistic nature, provide absolute certainty against some other unlucky ninja ordering recurring -- but Parts 1-3 address the actual general-case structural root cause for every multi-output codegen step in the project (every consumer of libxce_dep, plus every internal libxce_src consumer, now has a real build-graph edge on the relevant codegen step), so this is expected to eliminate this entire race class project-wide rather than merely reduce its likelihood. Does not address the separate, distinct apiserver<-libxce.a CGO-link race noted in one prior recurrence (2026-07-26) -- that is a different meson target (frontend/apiserver's go_build custom_target referencing @BUILD_ROOT@/lib/libxce.a via an env-var string, not a `dependencies:`/`libxce_dep` relationship) and remains unfixed; also worth a follow-up ticket for the test_test_cli.c PTY-harness safety-timeout flakiness under system load, observed here but out of this bug's own scope.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Related: bug-1439
|
|
- Gitea: #267
|