chore(wolf): checkpoint buglog.json (1040 bugs)

.wolf/buglog.json

@@ -19105,7 +19105,7 @@
       "timestamp": "2026-08-26T07:20:00Z",
       "error_message": "User-reported live deployment (/tmp/xce-ha-deploy, 3-node HA cluster, build.dbg): after ./scripts/start-all.sh reported 'Controller cluster healthy (primary elected)', ctrl-1's log showed it perpetually retrying ha_connect_finish_peer() against both peer[1] (localhost:9902) and peer[2] (localhost:9903) every ~3s for 8+ minutes straight (120+ attempts, INFO-level 'connecting outbound to peer[N]' logged each time -- the SUCCESS-path log line per ha_connect_finish_peer's own code, not a failure), never once completing a working peer connection. Meanwhile ctrl-2/ctrl-3 were actively, healthily replicating with each other (real AppendEntries traffic, LSN steadily advancing) -- they formed a working 2-of-3 quorum excluding ctrl-1 entirely. `xceadmin scheduler record status` failed with 'no primary available: timed out contacting localhost:9901' (ctrl-1's own client port) even though `xce_test`'s REPL joblist worked fine against the real 2-node quorum.",
       "file": "src/controller/ha.c (ha_connect_finish_peer, reconnect_lane_fn, ha_connect_to_peer)",
-      "root_cause": "PARTIALLY CONFIRMED, NOT FULLY ROOT-CAUSED. Ruled out as a test-harness/environment artifact: independently confirmed via a completely fresh, isolated 3-node deploy (~/tmp/xce-ha-fresh-repro, same build.dbg binaries, same run-ha-soak.sh topology) that the SAME symptom (impossible term jump, standby permanently failing cluster_init_phase2) reproduced ONCE due to a genuine same-session zombie-process port collision (see cerebrum.md's 2026-08-26 Do-Not-Repeat entry) -- but that specific mechanism does NOT apply to /tmp/xce-ha-deploy itself: confirmed via `ss -tlnp` that no live process holds ports 9901-9903 at investigation time, and no OTHER unrelated cluster shares those specific ports (9991-9993 was the colliding zombie's port range, unrelated). Direct on-disk evidence via `waldump` on the STOPPED cluster's data dirs CONFIRMS a real, non-artifactual divergence: ctrl-1's WAL has only 138 entries (lsn 1..138, all term=2, CRC chain intact, no gaps) while ctrl-2 has 2648 entries and ctrl-3 has 2641 entries (both showing term=3 activity in ha-term.bin, one of which -- ctrl-3 -- legitimately recorded voted_for_name='ctrl-2', a NORMAL vote-grant record, not corruption). This proves ctrl-1 was genuinely functioning (as primary, at term=2, given the job data the user's own joblist output showed) up through WAL entry 138, then something caused it to lose contact with both peers, a new election (term=3) happened among ctrl-2/ctrl-3 without it, and ctrl-1's own RECONNECT mechanism (ha_connect_finish_peer, dispatched via the SlowLane per raft-drive-async task 2.1's offload design) never once succeeded in re-establishing a working connection for the remainder of the run, despite logging its own SUCCESS path (endpoint_connect()+iomgr_monitor() both returning >=0) on every single attempt. This strongly suggests the bug is NOT in the initial connect() call itself (which appears to succeed at the socket/iomgr-registration level every time) but in whatever HIGHER-LEVEL protocol step is supposed to follow a successful low-level connect (completing a vote/ctrlhello handshake, registering the peer in standby_endpoints, or some per-peer state that should be cleared/retried differently after a genuine peer-loss) -- that step silently never completes or never gets attempted, so the SAME low-level connect()+iomgr_monitor() cycle repeats every ~3s forever without progress. Could NOT be reproduced live/interactively -- no debugger/ptrace access to the (already-stopped) real deployment's processes was available; this analysis is from static log/WAL evidence only, not a live capture of the actual failure in progress.",
+      "root_cause": "PARTIALLY CONFIRMED, NOT FULLY ROOT-CAUSED. Ruled out as a test-harness/environment artifact: independently confirmed via a completely fresh, isolated 3-node deploy (~/tmp/xce-ha-fresh-repro, same build.dbg binaries, same run-ha-soak.sh topology) that the SAME symptom (impossible term jump, standby permanently failing cluster_init_phase2) reproduced ONCE due to a genuine same-session zombie-process port collision (see cerebrum.md's 2026-08-26 Do-Not-Repeat entry) -- but that specific mechanism does NOT apply to /tmp/xce-ha-deploy itself: confirmed via `ss -tlnp` that no live process holds ports 9901-9903 at investigation time, and no OTHER unrelated cluster shares those specific ports (9991-9993 was the colliding zombie's port range, unrelated). Direct on-disk evidence via `waldump` on the STOPPED cluster's data dirs CONFIRMS a real, non-artifactual divergence: ctrl-1's WAL has only 138 entries (lsn 1..138, all term=2, CRC chain intact, no gaps) while ctrl-2 has 2648 entries and ctrl-3 has 2641 entries (both showing term=3 activity in ha-term.bin, one of which -- ctrl-3 -- legitimately recorded voted_for_name='ctrl-2', a NORMAL vote-grant record, not corruption). This proves ctrl-1 was genuinely functioning (as primary, at term=2, given the job data the user's own joblist output showed) up through WAL entry 138, then something caused it to lose contact with both peers, a new election (term=3) happened among ctrl-2/ctrl-3 without it, and ctrl-1's own RECONNECT mechanism (ha_connect_finish_peer, dispatched via the SlowLane per raft-drive-async task 2.1's offload design) never once succeeded in re-establishing a working connection for the remainder of the run, despite logging its own SUCCESS path (endpoint_connect()+iomgr_monitor() both returning >=0) on every single attempt. This strongly suggests the bug is NOT in the initial connect() call itself (which appears to succeed at the socket/iomgr-registration level every time) but in whatever HIGHER-LEVEL protocol step is supposed to follow a successful low-level connect (completing a vote/ctrlhello handshake, registering the peer in standby_endpoints, or some per-peer state that should be cleared/retried differently after a genuine peer-loss) -- that step silently never completes or never gets attempted, so the SAME low-level connect()+iomgr_monitor() cycle repeats every ~3s forever without progress. Could NOT be reproduced live/interactively -- no debugger/ptrace access to the (already-stopped) real deployment's processes was available; this analysis is from static log/WAL evidence only, not a live capture of the actual failure in progress. FURTHER REPRO ATTEMPTS (2026-08-26, all failed to reproduce): (a) fresh 3-node deploy, kill+restart both standbys simultaneously while primary stays alive -- clean reconnect both times; (b) same, with continuous background jobsub load throughout -- clean reconnect (one transient ECONNREFUSED-then-retry-succeeds cycle observed, not a stuck state); (c) user-supplied exact recipe (submit a mixed array of jobs giving RUN/PEND/DONE states, then xceadmin scheduler record status) repeated 10x against a genuinely healthy full-component deploy (3x controller + worker + scheduler + xceadmin) -- succeeded cleanly every time; (d) 8x fresh-boot-only attempts (no artificial kills at all) immediately running reslist/workerlist/jobsub/joblist in sequence within the first ~5s of boot, matching the user's report that the real failure occurred within MINUTES of start-all.sh with no explicit kill -- all 8 clean, no crash, no elevated reconnect-retry counts. Total: 20+ reproduction attempts across 4 distinct scenario shapes, zero reproductions. This is a genuinely rare, timing-sensitive condition, not reliably triggerable by any tested scenario shape so far.",
       "fix": "NOT YET ATTEMPTED. Needs a live reproduction with process-level tracing (gdb/strace on a debug build) or targeted new logging in ha_connect_finish_peer/reconnect_lane_fn's post-connect handshake path to see exactly which step silently fails to progress after a genuine peer-loss (RDHUP) event, as distinct from initial cold-boot connection establishment (which DOES work correctly, per the fresh-repro soak's clean PASS with normal kill/restart/reconverge cycles). Next investigation step: reproduce a genuine mid-run peer-loss (not a driver-initiated kill, which already reconverges fine per the fresh-repro soak's PASS) -- e.g. a transient network blip or a peer restart landing in an unlucky window -- under gdb/strace, and trace ha_connect_finish_peer's full call chain including whatever runs after endpoint_connect()/iomgr_monitor() succeed (the ctrlhello/vote-channel handshake) to find where it silently stalls.",
       "tags": [
         "controller",
@@ -19134,7 +19134,7 @@
       "timestamp": "2026-08-26T07:35:00Z",
       "error_message": "ctrl-1 segfaulted (SIGSEGV, core dumped -- confirmed via bash's own accurate job-control message, not a misread) within ~1.5s of a genuinely fresh boot, immediately after processing its first client 'ctrlstatus' probe request, during a fresh 10-round --chaotic run-ha-soak.sh --chaotic invocation (build.dbg, no ASAN) launched to verify bug-1696. Driver reported 'FAIL: no primary elected within ~90s' as a consequence (ctrl-1 never came back; ctrl-2 was subsequently Killed by cleanup).",
       "file": "src/controller/ha.c or src/controller/cluster.c (exact function not identified -- crash occurred somewhere in the ctrlstatus request-handling path shortly after raft_cb_become_follower/first_boot=1)",
-      "root_cause": "NOT ROOT-CAUSED -- occurred exactly ONCE and could not be reproduced despite two dedicated repro attempts: (1) re-running the identical build.dbg chaotic soak under run-ha-soak.sh's built-in GDB_WRAP=ctrl-1 debug seam caught only a benign SIGPIPE (gdb's own default stop-on-SIGPIPE behavior firing on a normal peer-disconnect-during-TLS-write event that the real code already ignores via signal(SIGPIPE, SIG_IGN) at netio.c:1250 -- confirmed this was a gdb artifact, not the real crash, since production code never sees this signal); (2) a manual gdb session with `handle SIGPIPE nostop noprint pass` set before `run` (correctly letting SIGPIPE pass through), driving 15+ rapid ctrlstatus probes against a freshly-booted ctrl-1 via xce_test's REPL (mirroring the exact probe pattern run-ha-soak.sh's own wait-primary/discover_primary functions use), did NOT reproduce any crash. (3) A full, from-scratch 10-round --chaotic soak (33 jobs, 10 kill/elect cycles) run under a fresh ASAN rebuild (src/build-asan-1696repro) completed with a clean PASS (committed_divergence=0 committed_job_loss=0) and no SIGSEGV/ASAN report at all. This strongly suggests either a genuinely rare, narrow-window race (possibly specific to cold dynamic-linker/first-request timing that a debug or ASAN build's different memory layout/timing perturbs away) or an environment-specific fluke from that one specific boot -- NOT a deterministic, reliably-triggerable defect. No core file was accessible for postmortem gdb analysis (systemd-coredump is configured as the core_pattern handler but coredumpctl is not installed/reachable in this sandbox).",
+      "root_cause": "NOT ROOT-CAUSED -- occurred exactly ONCE and could not be reproduced despite two dedicated repro attempts: (1) re-running the identical build.dbg chaotic soak under run-ha-soak.sh's built-in GDB_WRAP=ctrl-1 debug seam caught only a benign SIGPIPE (gdb's own default stop-on-SIGPIPE behavior firing on a normal peer-disconnect-during-TLS-write event that the real code already ignores via signal(SIGPIPE, SIG_IGN) at netio.c:1250 -- confirmed this was a gdb artifact, not the real crash, since production code never sees this signal); (2) a manual gdb session with `handle SIGPIPE nostop noprint pass` set before `run` (correctly letting SIGPIPE pass through), driving 15+ rapid ctrlstatus probes against a freshly-booted ctrl-1 via xce_test's REPL (mirroring the exact probe pattern run-ha-soak.sh's own wait-primary/discover_primary functions use), did NOT reproduce any crash. (3) A full, from-scratch 10-round --chaotic soak (33 jobs, 10 kill/elect cycles) run under a fresh ASAN rebuild (src/build-asan-1696repro) completed with a clean PASS (committed_divergence=0 committed_job_loss=0) and no SIGSEGV/ASAN report at all. This strongly suggests either a genuinely rare, narrow-window race (possibly specific to cold dynamic-linker/first-request timing that a debug or ASAN build's different memory layout/timing perturbs away) or an environment-specific fluke from that one specific boot -- NOT a deterministic, reliably-triggerable defect. No core file was accessible for postmortem gdb analysis (systemd-coredump is configured as the core_pattern handler but coredumpctl is not installed/reachable in this sandbox). FURTHER NOTE (2026-08-26): did not recur across the same 20+ reproduction attempts made while investigating bug-1698 (fresh boots, standby kill/restart cycles, load, exact user recipe replay) -- still zero occurrences since the single original sighting.",
       "fix": "NOT ATTEMPTED -- cannot fix an unreproduced crash responsibly. If this recurs, capture a live gdb session (with SIGPIPE explicitly passed through, per this investigation's corrected GDB_WRAP-equivalent technique) or install/access a working coredumpctl-equivalent BEFORE the crashing process is cleaned up, to get a real backtrace with symbols. Also worth checking core_pattern/ulimit -c in the actual execution environment ahead of time so a future occurrence leaves a usable core file automatically.",
       "tags": [
         "controller",