chore(wolf): checkpoint buglog.json (1041 bugs)

.wolf/buglog.json

@@ -19134,7 +19134,7 @@
       "timestamp": "2026-08-26T07:35:00Z",
       "error_message": "ctrl-1 segfaulted (SIGSEGV, core dumped -- confirmed via bash's own accurate job-control message, not a misread) within ~1.5s of a genuinely fresh boot, immediately after processing its first client 'ctrlstatus' probe request, during a fresh 10-round --chaotic run-ha-soak.sh --chaotic invocation (build.dbg, no ASAN) launched to verify bug-1696. Driver reported 'FAIL: no primary elected within ~90s' as a consequence (ctrl-1 never came back; ctrl-2 was subsequently Killed by cleanup).",
       "file": "src/controller/ha.c or src/controller/cluster.c (exact function not identified -- crash occurred somewhere in the ctrlstatus request-handling path shortly after raft_cb_become_follower/first_boot=1)",
-      "root_cause": "NOT ROOT-CAUSED -- occurred exactly ONCE and could not be reproduced despite two dedicated repro attempts: (1) re-running the identical build.dbg chaotic soak under run-ha-soak.sh's built-in GDB_WRAP=ctrl-1 debug seam caught only a benign SIGPIPE (gdb's own default stop-on-SIGPIPE behavior firing on a normal peer-disconnect-during-TLS-write event that the real code already ignores via signal(SIGPIPE, SIG_IGN) at netio.c:1250 -- confirmed this was a gdb artifact, not the real crash, since production code never sees this signal); (2) a manual gdb session with `handle SIGPIPE nostop noprint pass` set before `run` (correctly letting SIGPIPE pass through), driving 15+ rapid ctrlstatus probes against a freshly-booted ctrl-1 via xce_test's REPL (mirroring the exact probe pattern run-ha-soak.sh's own wait-primary/discover_primary functions use), did NOT reproduce any crash. (3) A full, from-scratch 10-round --chaotic soak (33 jobs, 10 kill/elect cycles) run under a fresh ASAN rebuild (src/build-asan-1696repro) completed with a clean PASS (committed_divergence=0 committed_job_loss=0) and no SIGSEGV/ASAN report at all. This strongly suggests either a genuinely rare, narrow-window race (possibly specific to cold dynamic-linker/first-request timing that a debug or ASAN build's different memory layout/timing perturbs away) or an environment-specific fluke from that one specific boot -- NOT a deterministic, reliably-triggerable defect. No core file was accessible for postmortem gdb analysis (systemd-coredump is configured as the core_pattern handler but coredumpctl is not installed/reachable in this sandbox). FURTHER NOTE (2026-08-26): did not recur across the same 20+ reproduction attempts made while investigating bug-1698 (fresh boots, standby kill/restart cycles, load, exact user recipe replay) -- still zero occurrences since the single original sighting.",
+      "root_cause": "NOT ROOT-CAUSED -- occurred exactly ONCE and could not be reproduced despite two dedicated repro attempts: (1) re-running the identical build.dbg chaotic soak under run-ha-soak.sh's built-in GDB_WRAP=ctrl-1 debug seam caught only a benign SIGPIPE (gdb's own default stop-on-SIGPIPE behavior firing on a normal peer-disconnect-during-TLS-write event that the real code already ignores via signal(SIGPIPE, SIG_IGN) at netio.c:1250 -- confirmed this was a gdb artifact, not the real crash, since production code never sees this signal); (2) a manual gdb session with `handle SIGPIPE nostop noprint pass` set before `run` (correctly letting SIGPIPE pass through), driving 15+ rapid ctrlstatus probes against a freshly-booted ctrl-1 via xce_test's REPL (mirroring the exact probe pattern run-ha-soak.sh's own wait-primary/discover_primary functions use), did NOT reproduce any crash. (3) A full, from-scratch 10-round --chaotic soak (33 jobs, 10 kill/elect cycles) run under a fresh ASAN rebuild (src/build-asan-1696repro) completed with a clean PASS (committed_divergence=0 committed_job_loss=0) and no SIGSEGV/ASAN report at all. This strongly suggests either a genuinely rare, narrow-window race (possibly specific to cold dynamic-linker/first-request timing that a debug or ASAN build's different memory layout/timing perturbs away) or an environment-specific fluke from that one specific boot -- NOT a deterministic, reliably-triggerable defect. No core file was accessible for postmortem gdb analysis (systemd-coredump is configured as the core_pattern handler but coredumpctl is not installed/reachable in this sandbox). FURTHER NOTE (2026-08-26): did not recur across the same 20+ reproduction attempts made while investigating bug-1698 (fresh boots, standby kill/restart cycles, load, exact user recipe replay) -- still zero occurrences since the single original sighting. FURTHER NOTE (2026-08-26, cont.): USER-SUPPLIED core dumps from the real deployment (4 core.xce_controller.*.zst files today) confirm this is a real, recurring heap corruption (glibc malloc_printerr: \"unaligned tcache chunk detected\", detected inside endpoint_get()/calloc for a 312-byte endpoint_t -- the abort site is downstream of the actual corruption, not the cause). Confirmed the crash DID occur at least once in this session too (attempt 7 of an earlier 8-attempt loop) but was MISSED by an earlier flawed stderr-grep detection method (setsid nohup ...  & disown fully detaches a process from the shells job table, so bash never prints a Aborted/Segmentation fault line for it -- a real methodology gap, not evidence of rarity). Redone with PROPER job-control-based exit-code detection: 15 attempts under a fresh ASAN build (build-asan-1696repro) = 0 reproductions; 15 more attempts with all 3 nodes wrapped in a properly-SIGPIPE-configured gdb session against build.dbg (where it previously reproduced) = 0 reproductions. Total 30 further attempts, zero catches. Hypothesis: ASANs different allocator/timing may suppress the race entirely (consistent with 0/15 under ASAN); build.dbg without a live gdb/gdb overhead may still need many more attempts or a specific concurrent-connection pattern not yet isolated. bug-1698s stuck-reconnect symptom and this heap corruption are plausibly the SAME root cause (both point at endpoint_t lifecycle -- corrupted/UAFd endpoint state could equally explain a connection that never completes ITS handshake). Next step if revisited: run MANY more attempts under build.dbg with proper exit-code detection (not ASAN, which appears to suppress it), or request the user run the real deployment under a debug build with core dumps enabled and share the NEXT core file for backtrace analysis the same way this session did.",
       "fix": "NOT ATTEMPTED -- cannot fix an unreproduced crash responsibly. If this recurs, capture a live gdb session (with SIGPIPE explicitly passed through, per this investigation's corrected GDB_WRAP-equivalent technique) or install/access a working coredumpctl-equivalent BEFORE the crashing process is cleaned up, to get a real backtrace with symbols. Also worth checking core_pattern/ulimit -c in the actual execution environment ahead of time so a future occurrence leaves a usable core file automatically.",
       "tags": [
         "controller",
@@ -19155,6 +19155,34 @@
       ],
       "occurrences": 1,
       "last_seen": "2026-08-26T07:35:00Z"
+    },
+    {
+      "id": "bug-1700",
+      "timestamp": "2026-08-26T16:35:00Z",
+      "error_message": "User-reported: `xceadmin scheduler record status` (and by the same mechanism, `record start`/`record stop`) fails with `Error: controller returned status -10014` whenever the `--controller`/default address happens to point at a controller that is currently a STANDBY, not the primary. User's expectation: 'xceadmin/libxce should find the correct primary to talk to it' instead of surfacing the raw rejection.",
+      "file": "src/controller/cluster.c (myreq_handler, ~line 406-414)",
+      "root_cause": "FULLY ROOT-CAUSED, gap is server-side only. `-10014` is `E_XCE_STANDBY` (errnox.h, XCE_ERRNO_BASE+13) -- ha_dispatch_gate() (ha.c:4745-4775) correctly rejects OPCode_schedAdmin on a CTRL_STANDBY node (it is absent from that role's opcode allow-list, by design -- schedAdmin is a primary-only forward-to-scheduler operation). The gate's generic caller in myreq_handler (cluster.c:406-414) sends the rejection reply with `reply.status = rc; reply.data_which = RequestReply_data_nodata;` and NEVER calls fill_redirect_reply() (join.c:27-31, the exact same one-line helper OPCode_join's handler already uses successfully) -- so redirectHost/redirectPort are left at their zero-value defaults on EVERY dispatch-gate rejection, for EVERY opcode, not just schedAdmin (OPCode_upgradeConfig has the identical, already-documented gap -- see test_e2e_live_ha_push.py's module docstring, confirmed 2026-07-30-era). Critically, this is NOT a client-side deficiency: xceadmin's redirect-following client logic is ALREADY fully implemented, generic/opcode-agnostic, and well-tested (`xceadmin.discovery.follow_redirects()`, with a full test suite covering 1-hop/3-hop/too-many-hops/timeout cases; `xceadmin.scheduler._send_admin_request()` already calls it for every record_start/stop/status call). `_redirect_target()` (discovery.py:80-89) correctly requires BOTH redirectHost and redirectPort to be non-empty/non-zero to treat a reply as a redirect -- given the gate-rejection reply supplies neither, the client correctly concludes there is nothing to follow and surfaces the bare rejection status as-is. The client is not 'giving up' -- it is working exactly per its documented contract given the reply it receives; the whole path already works end-to-end against a properly-behaving server (test_cli_scheduler_record_start_follows_redirect_to_primary already exercises the client side against a mock that DOES send a redirect).",
+      "fix": "PENDING. Minimal, single-call-site fix: in myreq_handler's generic ha_dispatch_gate rejection path (cluster.c:406-414), when the rejection reason is specifically -XCE_ERR_STANDBY (i.e. this node knows or may know who the primary is) call fill_redirect_reply(&reply, &c->ha) before xce_reply(), exactly mirroring OPCode_join's existing handle_worker_join()-adjacent usage (join.c:277). This fixes OPCode_schedAdmin AND the already-documented OPCode_upgradeConfig gap simultaneously, with ZERO xceadmin/client-side changes needed -- the existing follow_redirects() mechanism will start working immediately once the server populates the fields. Must verify: (a) ha_state_t.primary_host/primary_port are populated/valid on a CTRL_STANDBY node that has learned of a live primary (the common case) vs. a node that genuinely does not know any primary yet (mid-election, or has never heard from one) -- fill_redirect_reply() unconditionally aliases these fields with no NULL/empty guard, so confirm what a standby's primary_host looks like before any primary is ever discovered (likely NULL/empty, in which case the reply's redirectHost would encode as empty/absent and the client's existing `if host and port` guard already handles this correctly as 'no redirect' -- but verify, don't assume); (b) add a regression test proving a schedAdmin request to a standby-with-known-primary now receives a populated redirect reply (mirroring test_join_redirect.c's existing pattern, `test_redirect_reply_carries_primary_address`) and a live xceadmin CLI test (extending test_e2e_live_ha_push.py's pattern) proving `xceadmin scheduler record status` against ANY of the 3 nodes now transparently reaches the primary and succeeds.",
+      "tags": [
+        "controller",
+        "xceadmin",
+        "redirect",
+        "primary-discovery",
+        "OPCode_schedAdmin",
+        "OPCode_upgradeConfig-adjacent",
+        "usability",
+        "found-live",
+        "root-caused",
+        "fix-scoped",
+        "low-risk",
+        "unfixed"
+      ],
+      "related_bugs": [
+        "bug-1698",
+        "bug-854"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-08-26T16:35:00Z"
     }
   ]
 }