From f756c4b09fcff21803592c556dcb474dc97d55f3 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sun, 23 Aug 2026 17:21:12 +0000 Subject: [PATCH] chore(wolf): checkpoint buglog.json (1011 bugs) --- .wolf/buglog.json | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 111d617a..ec64fe65 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -16304,7 +16304,7 @@ "error_message": "controller_unit_test: test_conf_trim_keeps_only_n_versions and test_conf_get_saved_tag_no_config (test_config_repl.c) FAIL during an unrelated ASAN controller build/test run (bug-451 fix session)", "file": "src/controller/testsuite/test_config_repl.c", "root_cause": "Not investigated -- discovered incidentally while verifying an unrelated fix (bug-451, persist_job_hook_fun's RUSAGE arm). Neither test_config_repl.c nor config_repl.c were touched this session (confirmed via git diff, zero changes). Coincides with a burst of 'persist: flusher fdatasync FAILED (errno=5 Input/output error)' ERROR lines in the same test run, matching this project's well-documented overlayfs sandbox fdatasync artifact class (same family as bug-924, bug-1470, bug-1553) rather than a genuine regression.", - "fix": "NOT FIXED -- out of scope for the bug-451 session that surfaced it. Needs confirmation this is the same overlayfs artifact (re-run on a real fs) before treating it as a real regression in test_config_repl.c's config-version-trim/tag-save logic. Re-observed 2026-08-17 during resource-value-verification task 1.1 (resval_typeresolve extraction) full controller ASAN build/test run: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3, test_conf_get_saved_tag_no_config Expected 0 Was ), same accompanying persist_flush/flusher fdatasync errno=5 burst. Zero diff in test_config_repl.c/config_repl.c this session either (task 1.1 boundary was testsuite/resval_typeresolve.{c,h}, testsuite/test_resval_typeresolve.c, meson.build only) -- further corroborates the overlayfs sandbox artifact theory over a genuine regression, still unconfirmed on a real fs. Re-observed again 2026-08-17 during resource-value-verification task 2.1 (resupdate fallback + -t flag wiring, GREEN phase): identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst, in the same full `/xce-build controller` run that also confirmed resval_typeresolve_unit_test's own Unity summary at 4 Tests 0 Failures (task 2.1's GREEN evidence). Task 2.1's diff was scoped to testsuite/resval_typeresolve.{c,h} and testsuite/test.c only (confirmed via git diff --stat) -- test_config_repl.c/config_repl.c untouched, per this bug's own do-not-chase guidance in tasks.md's Implementation Notes.\n\nRe-observed a 5th time 2026-08-18 during a full `nix develop ./src --command bash src/tools/xce-build.sh all asan build-asan --incremental` verification run on the scheduler-framework branch (post bug-1584/1585 scheduler fixes). Identical signature: same 2 tests, same Expected/Was values (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. This run recompiled ZERO C source files (ninja reported \"no work to do\" for all C targets -- only the xceadmin custom target/wheel regenerated), so the failure cannot be attributed to any change in this session -- strongest evidence yet that this is purely an environment (overlayfs fdatasync) artifact, not a code regression. Scheduler suite's own single failure in the same run (test_recordstart_symlink_escape_replies_error, bug-1560) is separate and expected. Re-observed a 6th time 2026-08-19 during a controller ASAN build/test run verifying the StartTime/FinishTime persist.c fix (raft-apply-decoupling job-fact timestamp stamping, unrelated files persist.c/test_apply_committed_entry.c) -- identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst. Session diff confirmed scoped to persist.c/test_apply_committed_entry.c only -- further corroborates environment-artifact theory.\n\nRe-observed a 7th time 2026-08-21 verifying a pure output-formatting edit to src/controller/testsuite/test.c (raw %lu jobid printf sites replaced with fmt_jobid()+%s, 10 call sites, zero logic change) via `xce-build.sh controller asan --incremental` on the scheduler-framework branch. Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to test.c only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. controller/xce_test (the binary test.c itself compiles into) built and linked cleanly with zero warnings/errors tied to the fmt_jobid change; only controller_unit_test (a separate binary, testsuite_test_config_repl.c.o) carries these 2 pre-existing failures. Further corroborates environment-artifact theory over regression.", + "fix": "NOT FIXED -- out of scope for the bug-451 session that surfaced it. Needs confirmation this is the same overlayfs artifact (re-run on a real fs) before treating it as a real regression in test_config_repl.c's config-version-trim/tag-save logic. Re-observed 2026-08-17 during resource-value-verification task 1.1 (resval_typeresolve extraction) full controller ASAN build/test run: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3, test_conf_get_saved_tag_no_config Expected 0 Was ), same accompanying persist_flush/flusher fdatasync errno=5 burst. Zero diff in test_config_repl.c/config_repl.c this session either (task 1.1 boundary was testsuite/resval_typeresolve.{c,h}, testsuite/test_resval_typeresolve.c, meson.build only) -- further corroborates the overlayfs sandbox artifact theory over a genuine regression, still unconfirmed on a real fs. Re-observed again 2026-08-17 during resource-value-verification task 2.1 (resupdate fallback + -t flag wiring, GREEN phase): identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst, in the same full `/xce-build controller` run that also confirmed resval_typeresolve_unit_test's own Unity summary at 4 Tests 0 Failures (task 2.1's GREEN evidence). Task 2.1's diff was scoped to testsuite/resval_typeresolve.{c,h} and testsuite/test.c only (confirmed via git diff --stat) -- test_config_repl.c/config_repl.c untouched, per this bug's own do-not-chase guidance in tasks.md's Implementation Notes.\n\nRe-observed a 5th time 2026-08-18 during a full `nix develop ./src --command bash src/tools/xce-build.sh all asan build-asan --incremental` verification run on the scheduler-framework branch (post bug-1584/1585 scheduler fixes). Identical signature: same 2 tests, same Expected/Was values (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. This run recompiled ZERO C source files (ninja reported \"no work to do\" for all C targets -- only the xceadmin custom target/wheel regenerated), so the failure cannot be attributed to any change in this session -- strongest evidence yet that this is purely an environment (overlayfs fdatasync) artifact, not a code regression. Scheduler suite's own single failure in the same run (test_recordstart_symlink_escape_replies_error, bug-1560) is separate and expected. Re-observed a 6th time 2026-08-19 during a controller ASAN build/test run verifying the StartTime/FinishTime persist.c fix (raft-apply-decoupling job-fact timestamp stamping, unrelated files persist.c/test_apply_committed_entry.c) -- identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst. Session diff confirmed scoped to persist.c/test_apply_committed_entry.c only -- further corroborates environment-artifact theory.\n\nRe-observed a 7th time 2026-08-21 verifying a pure output-formatting edit to src/controller/testsuite/test.c (raw %lu jobid printf sites replaced with fmt_jobid()+%s, 10 call sites, zero logic change) via `xce-build.sh controller asan --incremental` on the scheduler-framework branch. Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to test.c only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. controller/xce_test (the binary test.c itself compiles into) built and linked cleanly with zero warnings/errors tied to the fmt_jobid change; only controller_unit_test (a separate binary, testsuite_test_config_repl.c.o) carries these 2 pre-existing failures. Further corroborates environment-artifact theory over regression.\n\nRe-observed an 8th time 2026-08-23 during scheduler-dispatch-commit task 3.3's full controller ASAN build/test verification run (job.c dispatch-failure retry restructuring + new test_dispatch_fail_retry.c). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to job.c and testsuite/ files only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. Further corroborates environment-artifact theory over regression.", "tags": [ "controller", "testsuite", @@ -16320,8 +16320,8 @@ "bug-1470", "bug-1553" ], - "occurrences": 7, - "last_seen": "2026-08-21T16:03:00Z" + "occurrences": 8, + "last_seen": "2026-08-23T17:20:00Z" }, { "id": "bug-912", @@ -18309,7 +18309,7 @@ "error_message": "None yet observed at runtime -- found via code trace during round 3/4 of /kiro-validate-design review of scheduler-dispatch-commit.", "file": "src/controller/job.c", "root_cause": "dispatch_a_job_() (job.c:831-963) calls reference(j) at line 854 after successfully looking up the job. On FOUR of its failure/return paths it never calls a matching dereference(j): workermgr key_find() failing (rc<0, line 888-890), the worker not being found (w==NULL, line 892-894), the worker not being in a ready state (E_XCE_BAD_WORKER, line 897-909), and xce_startjob_req() itself failing (rc<0, line 956-960 -- found in a later review pass than the first three). (Contrast: the nc<=0 path at line 869-873 and the get_queue_exec() failure path at line 917-925 both correctly dereference(j) before returning.) Each occurrence permanently leaks one reference on the job object. Separately and NOT included in this bug (flagged as a distinct open question, not a confirmed defect): the function also never calls dereference(j) on its normal SUCCESS return path -- this may be an intentional reference hand-off to the job entering its dispatched/running lifetime rather than a bug, and needs its own investigation rather than being folded in here.", - "fix": "NOT YET FIXED -- found while designing scheduler-dispatch-commit (.kiro/specs/scheduler-dispatch-commit/), which already restructures exactly these four failure points (to add RTTI_LOCK_WRITE-protected dispatch-failure classification/resolution per that spec Requirement 4, including a new XCE_DISPATCH_FAIL_SITE_RPC_FAILURE classification site for the fourth). design.md now explicitly includes the missing dereference(j) at all four sites as part of that restructuring, closing this leak as a side effect rather than a separate patch.", + "fix": "FIXED -- scheduler-dispatch-commit task 3.3. dispatch_a_job_() (job.c) now closes all four leak sites as part of the retryable-failure classify/resolve restructuring the bug's own notes anticipated: XCE_DISPATCH_FAIL_SITE_WORKER_LOOKUP, _WORKER_NULL, _WORKER_NOT_READY, and _RPC_FAILURE each now lock(j, RTTI_LOCK_WRITE), classify via xce_classify_dispatch_fail(), increment job.dispatch_fail_count under that lock, resolve retryable (within XCE_DISPATCH_FAIL_RETRY_BOUND==3) or terminal (mark_job_fail(), mirroring task 3.2's two sibling terminal sites) as appropriate, unlock(j), then dereference(j) -- closing the leak uniformly across all four. Regression-tested via test_dispatch_fail_retry.c's canary-reference technique (task 3.2's own established pattern) at three of the four sites (WORKER_LOOKUP/WORKER_NULL share one test since a black-box test cannot force which of the two an unregistered worker key actually triggers -- both share the identical resolution code shape). RED confirmed first (all 4 new tests failed against the unmodified four sites, git-stash verified), then GREEN after the fix, full controller_unit_test suite run (1056 tests, filtered only for the pre-existing bug-1429 hang) showed zero new failures beyond the pre-existing bug-1561 pair.", "tags": [ "controller", "reference-counting", @@ -18322,8 +18322,8 @@ "related_bugs": [ "bug-1626" ], - "occurrences": 2, - "last_seen": "2026-08-23T05:50:00Z" + "occurrences": 3, + "last_seen": "2026-08-23T17:20:00Z" }, { "id": "bug-1628",