diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 9e612c93..5b74449a 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -17178,31 +17178,42 @@ { "id": "bug-1578", "timestamp": "2026-08-18T04:45:00Z", - "error_message": "Controller (xce_controller) reproducibly (3/3) exits silently and completely (zero ASAN report, empty stderr, ctrl-1.log ends mid-request with no WARN/ERROR/shutdown line, no core file, cgroup oom_kill=0) within 5-10 seconds after task 6.1's inject_resources() flow (xce_test worker join + resupdate for N resources + quit) completes against a live single-node deployment. Reproduced via the actual run-resource-verify.sh script (2x, default 9-resource set, --keep-running) and via a manual low-level repro feeding cmd_resupdate() the identical xce_test stdin script. Isolated via targeted experiments: a lone controller with zero other interaction stays alive 90s+; controller+scheduler+apiserver deployed with XCE_WORKER_BIN=/bin/true but WITHOUT ever running xce_test survives 100s+; a worker join+quit with NO resupdate at all (no -n/-v pairs) survives 60s+. Only the join+resupdate+quit sequence triggers the crash, isolating it specifically to the resupdate path (very likely cmd_resupdate()'s per-name follow-up xcecli_resval() verification GET-query loop documented in design.md's Test Tooling/C component, or the controller-side opget/resval RPC handling it exercises), not to worker-connect/disconnect lifecycle in general.", - "file": "src/controller/testsuite/test.c (cmd_resupdate's follow-up verification loop) and/or src/controller/opget.c / src/controller/worker.c (the GET/resval RPC path it exercises) -- exact crash site not isolated; no ASAN report was captured in any repro despite an ASAN-instrumented controller build, consistent with an uncatchable termination rather than a caught ASAN abort", - "root_cause": "NOT ROOT-CAUSED -- discovered live during independent adversarial review of resource-value-verification task 6.1 (run-resource-verify.sh's join+inject_resources() addition). The crash is reproducible through task 6.1's own newly-exercised live flow, but the resupdate follow-up verification mechanism it triggers was implemented by an earlier task (2.1/2.2 per design.md's Components table), not task 6.1's Bash-only boundary -- task 6.1 is very likely the first task to exercise this exact code path end-to-end against a full live deployment with a full multi-resource batch, rather than the cause of the underlying defect. Completely silent termination (no ASAN report, no signal-catch message, no log line) despite ASAN instrumentation suggests either an uncatchable external signal (SIGKILL-class) or a crash class ASAN's runtime does not intercept -- needs a debugger/core-dump-based investigation to actually isolate the faulting instruction (ulimit -c was unlimited in the sandbox but no core file was produced by any repro; worth checking core_pattern).", - "fix": "NOT FIXED -- out of boundary for task 6.1's review (run-resource-verify.sh only; the crash is in controller/test.c C code, not the Bash script). Flagging as a CRITICAL, must-address-before-building-further finding: tasks 6.2-6.4 (controller/scheduler/apiserver per-layer verification, per design.md) are wired as direct extension points appended immediately after inject_resources() in run-resource-verify.sh's main() -- if those tasks' controller-side check (Requirement 3's `resval` query) runs more than ~5-10s after injection, it will find the controller already dead, causing spurious FAILs unrelated to whatever those tasks actually implement. Needs a dedicated root-cause task before 6.2 can be built/verified reliably. Workaround for 6.2 in the meantime: query the controller (Requirement 3's `resval` check) IMMEDIATELY after inject_resources() returns, before any scheduler/apiserver bounded-wait polling that could burn the 5-10s window.", + "error_message": "Controller (xce_controller) reproducibly exits via SIGSEGV (si_addr=NULL, si_code=1/SEGV_MAPERR) within ~1s of the last resupdate follow-up verification reply completing, when the synthetic worker then disconnects ('quit'). ROOT-CAUSED via live repro (2/2) with an LD_PRELOAD signal-handler shim (gdb reliably masked the crash for both ASAN and debug/-O0 builds -- ptrace overhead perturbs timing enough to avoid the window; ~20s under gdb = no crash, every time). Resolved backtrace (addr2line against the debug-symbol'd ASAN binary): encode_NameValue <- encode_NameValue_list <- encode_Worker <- capnp_encode_worker (persist.c) <- persist_worker_hook_fun <- xce_run_hooks <- mark_worker_unavail <- handle_worker_leave <- handle_endpoint_error (cluster.c) <- io_worker_dispatch_batch <- io_worker (netio.c).", + "file": "src/controller/sched.c (sched_worker_change(), ~line 548-553) -- crash SITE is persist.c/controller.capnp.c (capnp_encode_worker/encode_Worker/encode_NameValue_list/encode_NameValue) but the DEFECT is in sched.c", + "root_cause": "ROOT-CAUSED, PRE-EXISTING (predates resource-value-verification by ~5 months, NOT introduced by tasks 2.1/2.2/6.1). sched_worker_change() (src/controller/sched.c) aliases w->host.resvals_to_sched/nresvals_to_sched directly to a local xlist_t resvalist's backing array (resvalist.data), pushes it to the scheduler connection, then calls xlist_fini(&resvalist) -- which frees that backing array -- WITHOUT first resetting w->host.resvals_to_sched to NULL / nresvals_to_sched to 0. The correct reset-before-free code exists in the function but is wrapped in a dead `#if 0 /*FIXME*/ ... #endif` block, introduced already-disabled by commit acfbec875 (2026-03-03, 'save works'). This leaves w->host.resvals_to_sched a DANGLING pointer and nresvals_to_sched a stale nonzero count on the worker_t struct for the rest of that worker's lifetime. commit e5fde7b5 (2026-05-20, 'fix(persist): zero nresvals_to_sched after free_name_val_list') attempted to fix exactly this class of bug but edited code INSIDE the already-dead #if 0 block, so the fix silently never took effect. When the worker later disconnects, handle_worker_leave() -> mark_worker_unavail() fires the WORKER_DEAD hook -> persist_worker_hook_fun() -> capnp_encode_worker() -> encode_Worker() -> encode_NameValue_list(), which iterates the stale count over the freed array and dereferences a garbage/NULL entry in encode_NameValue() -> SIGSEGV. Only manifests once a scheduler has connected at least once: on_scheduler_join() (sched.c) registers sched_worker_hook_fun into c->hooksys (gated by a `static int hook_installed` one-shot flag) ONLY on the first scheduler join -- a controller with no scheduler ever connecting never reaches the buggy sched_worker_change(), so w->host.resvals_to_sched stays NULL/0 forever and persist_worker_hook_fun's later read is always safe. This exactly matches run-resource-verify.sh's controller+scheduler+apiserver deployment shape and explains the original isolation matrix precisely. The sibling function apiserver_worker_change() (src/controller/apiserver.c ~line 479-481) does the reset correctly (NOT wrapped in #if 0), which is why apiserver-only interaction never crashed.", + "fix": "FIX IDENTIFIED, NOT YET APPLIED (out of task 6.1's boundary -- lives entirely in sched.c, a file no task in this feature touches). In src/controller/sched.c's sched_worker_change(), replace the dead block:\n #if 0\n /*FIXME*/\n free_name_val_list(&(w->host.resvals_to_sched), &(w->host.nresvals_to_sched));\n #endif\n xlist_fini(&resvalist);\nwith the exact working pattern already used by apiserver_worker_change():\n w->host.nresvals_to_sched = 0;\n w->host.resvals_to_sched = NULL;\n xlist_fini(&resvalist);\nIMPORTANT: do NOT simply uncomment the existing free_name_val_list() call as-is -- that would introduce a DOUBLE FREE instead. free_name_val_list() (src/lib/utils.c:412) already deep-frees every entry AND the outer array (FREEUP(resvals) where resvals == w->host.resvals_to_sched == the identical pointer value as resvalist.data), then sets w->host.resvals_to_sched itself to NULL -- but resvalist.data (the xlist_t's own separate field, still holding that same now-freed address) is untouched by that call, so the very next line's xlist_fini(&resvalist) -> FREEUP(resvalist.data) would free the identical address a second time. This is almost certainly why the original author disabled it with #if 0 rather than deleting it. Secondary/lower-severity, NOT part of this fix: both sched_worker_change() and apiserver_worker_change() leak the individual xce_name_val_t entries collected into resvalist -- xlist_fini() only frees the outer array (FREEUP(l->data)), never each entry's .name/.val/nv allocations -- worth a separate follow-up. Also worth a wider audit: other readers/freers of w->resvals_to_sched (opget.c:128 do_reply_workerlist(), apiserver.c, dataex.c) may have the same latent dangling-pointer exposure, only the WORKER_DEAD->persist path was confirmed live in this repro.", "tags": [ - "controller", - "resupdate", + "CRITICAL", + "capnp-encode", "cmd_resupdate", - "silent-crash", - "sigkill-class", + "controller", + "dangling-pointer", + "dead-code", + "double-free-risk", + "encode_NameValue", + "if-0-fixme", "no-asan-report", + "not-root-caused", + "pre-existing-not-this-feature", "resource-value-verification", + "resupdate", + "reviewer-discovered", + "root-caused", + "sched.c", + "sched_worker_change", + "sigkill-class", + "silent-crash", "task-6.1", "task-6.2-blocker", - "not-root-caused", - "reviewer-discovered", - "CRITICAL" + "use-after-free" ], "related_bugs": [ "bug-1572", "bug-1574", "bug-1577" ], - "occurrences": 1, - "last_seen": "2026-08-18T04:45:00Z" + "occurrences": 2, + "last_seen": "2026-08-18T05:20:00Z" } ] }