chore: update OpenWolf session state (xce-audit-invariants controller pass)

.wolf/buglog.json

@@ -13434,5 +13434,85 @@
     "related_bugs": [],
     "occurrences": 1,
     "last_seen": "2026-07-18T13:00:00.000Z"
+  },
+  {
+    "id": "bug-1450",
+    "timestamp": "2026-07-18",
+    "error_message": "xce-audit-invariants controller pass (dim: async I/O thread-role) found ha_demotion_convergence_fn scheduled directly as a delaymgr task body with no vtask/slow-lane hop: it truncates the WAL (persist_truncate_wal_above, file I/O under p->lock) then calls ha_self_restart (tlog_exit synchronous log flush + execv) -- all inline on the delaymgr thread, the SAME single thread that drives heartbeats and election timeouts cluster-wide for the whole cluster.",
+    "file": "src/controller/ha_convergence.c, src/controller/ha.h",
+    "root_cause": "ha_demotion_convergence_fn (ha_convergence.c) ran its full body -- truncate + self-restart -- synchronously on whatever thread invoked it (the delaymgr worker in production). A prior fix attempt (2026-07-10, see bug-1429) tried moving this to a VTASK_LIVE_BLOCK vtask via c->ha.pool and deadlocked controller_unit_test, because c->ha.pool is never initialized/started by this file's lightweight unit-test fixtures (dc_setup), so a vtask enqueued VTASK_LIVE_BLOCK blocked forever waiting for a worker that never existed; that attempt was reverted.",
+    "fix": "Split ha_demotion_convergence_fn into a dispatcher (fast admin work: single-flight CAS on new ha_state_t field demotion_convergence_inflight, then slowlane_submit) and a new demotion_lane_fn slow-lane worker body running the actual truncate+self-restart (extracted into ha_demotion_convergence_do_run) -- the SAME established slow-lane pattern already used for snapshot send / config-push / peer connect in ha.c (ha_snapshot_send_dispatch, ha_config_push_offloadable, ha_connect_to_peer), not the vtask pool that deadlocked before. New feature flag g_drive_async_demotion_offload (ha_set_demotion_offload_for_test / ha_demotion_offload_enabled, ha.h/ha_convergence.c) gates the offload. IMPORTANT: defaulting this flag to ON (1) was tried first and reproduced a NEW real hang -- raft_cb_become_follower (which schedules this task) is exercised by a very broad set of pre-existing controller_unit_test fixtures across many files that never call slowlane_init, so the offloaded work was silently queued on an uninitialized c->slowlane (queued, never run -- no live workers) and downstream code waiting on the now-never-happening completion hung. Reverted the default to OFF (0), matching the ORIGINAL rollout discipline of the three reference offloads (all shipped OFF-by-default, flipped ON only in a later dedicated compatibility-audit pass -- raft-drive-async task 3.4). The mechanism is correct and covered by dedicated tests (test_demotion_offload_task_returns_promptly / test_demotion_inline_task_blocks_RED in test_demotion_convergence.c) that explicitly flip the flag ON against a fixture with a real slowlane_init'd lane. Flipping the cluster-wide default to ON is a follow-up compatibility audit across the full test suite, explicitly out of scope for this fix. Verified: standalone controller_unit_test binary run directly (bypassing meson) reports 976 Tests 0 Failures 0 Ignored, no hang/timeout.",
+    "tags": [
+      "async-io",
+      "thread-role-violation",
+      "delaymgr",
+      "slowlane",
+      "ha_convergence.c",
+      "demotion",
+      "raft-drive-async-pattern",
+      "xce-audit-invariants",
+      "controller-tests"
+    ],
+    "related_bugs": [
+      "bug-1429",
+      "bug-1451"
+    ],
+    "occurrences": 1,
+    "last_seen": "2026-07-18"
+  },
+  {
+    "id": "bug-1451",
+    "timestamp": "2026-07-18",
+    "error_message": "xce-audit-invariants controller pass (dim: async I/O thread-role) found ha_catchup_continuation_fn scheduled directly as a delaymgr task body with no vtask/slow-lane hop: catchup_stream_range -> ha_stream_wal_to_standby does blocking WAL file reads + socket writes to a standby directly on the delaymgr thread. Flagged as the MORE URGENT of the two violations -- catch-up can stream a variable, potentially large amount of WAL data, so it can block delaymgr (and therefore heartbeats to every OTHER peer) for an unbounded, unpredictable time, a real spurious-election risk.",
+    "file": "src/controller/config_repl.c, src/controller/ha.h",
+    "root_cause": "ha_catchup_continuation_fn (config_repl.c) ran its full body -- re-derive target, then catchup_stream_range's blocking WAL reads/socket writes -- synchronously on whatever thread invoked it (the delaymgr worker in production). Source comment at config_repl.c confirmed a prior VTASK_LIVE_BLOCK vtask-pool attempt here (2026-07-10) also deadlocked controller_unit_test and was reverted (bug-1429), tracked as a known open violation.",
+    "fix": "Split ha_catchup_continuation_fn into a dispatcher (fast admin work: slowlane_submit keyed by peer_idx+1) and a new catchup_lane_fn slow-lane worker body running the original streaming logic unchanged -- the SAME established slow-lane pattern used for snapshot/config-push/reconnect in ha.c, not the vtask pool. c->ha.catchup_inflight[peer_idx] (set by ha_walack_on_ack before scheduling) remains the authoritative per-standby single-flight guard; the lane's own per-key dedup is a redundant backstop, mirroring ha_connect_to_peer's reconnect offload. New feature flag g_drive_async_catchup_offload (ha_set_catchup_offload_for_test / ha_catchup_offload_enabled, ha.h/config_repl.c) gates the offload. Defaulting this flag to ON (1) was tried first and reproduced a real hang: test_convergence_verification.c's pre-existing ACK-driven catch-up fixtures (which directly exercise this exact code path via ha_walack_continuation_for_test) never call slowlane_init, so offloaded work was silently queued and never ran, hanging controller_unit_test at ~90s+ (confirmed reproducible both via meson and a direct standalone binary run with SIGQUIT timeout). Reverted the default to OFF (0), matching the same rollout discipline as bug-1450's demotion-convergence fix and the three original raft-drive-async reference offloads. Mechanism covered by dedicated tests (test_catchup_offload_probe_runs_promptly / test_catchup_inline_probe_blocks_RED in test_config_repl.c) using a REAL, started, single-worker delaymgr (delaymgr_init+delaymgr_start, not just an un-started fixture) plus a probe task queued behind the continuation to directly prove the delaymgr thread is/isn't blocked. Flipping the cluster-wide default to ON is a follow-up compatibility audit, out of scope for this fix. Verified: standalone controller_unit_test binary run directly reports 976 Tests 0 Failures 0 Ignored, no hang/timeout.",
+    "tags": [
+      "async-io",
+      "thread-role-violation",
+      "delaymgr",
+      "slowlane",
+      "config_repl.c",
+      "catchup-continuation",
+      "raft-drive-async-pattern",
+      "xce-audit-invariants",
+      "controller-tests",
+      "spurious-election-risk"
+    ],
+    "related_bugs": [
+      "bug-1429",
+      "bug-1450"
+    ],
+    "occurrences": 1,
+    "last_seen": "2026-07-18"
+  },
+  {
+    "id": "bug-1452",
+    "timestamp": "2026-07-18T00:00:00Z",
+    "error_message": "AddressSanitizer: SEGV on unknown address 0x000000000038 in commit_advance_locked (controller/commit.c:208), called from commit_on_durable_advance (commit.c:341) <- fsync_worker (persist.c:399), a background ASAN thread started by persist_start (persist.c:723) inside test_persist_truncate.c's make_fresh_persist / test_truncate_above_resolves_pending_continuations_as_failed. Reproduced by temporarily flipping BOTH g_drive_async_demotion_offload and g_drive_async_catchup_offload (bug-1450/1451) from their default 0 to 1 and running the standalone controller_unit_test binary directly (976-test suite) -- crashed immediately, before completing the suite. Confirmed absent with both flags at their committed default of 0: the same binary/build (just reverted) completed cleanly, 976 Tests 0 Failures 0 Ignored, twice.",
+    "file": "src/controller/commit.c (crash site, commit_advance_locked); src/controller/persist.c (fsync_worker background thread, apply_committed_entry/restore_job/makeJob chain also appears in a related non-crashing stack from the same investigation); src/controller/ha_convergence.c + config_repl.c (g_drive_async_demotion_offload / g_drive_async_catchup_offload -- the flags whose flip triggers this)",
+    "root_cause": "NOT ROOT-CAUSED -- discovered as a side effect of an xce-audit-invariants Step 4 live-regression attempt (2026-07-18), not investigated further, deliberately out of scope for that session (see bug-1450/1451: flipping either offload flag's default to ON is explicitly scoped to a LATER, dedicated compatibility-audit pass, matching the original rollout discipline of the snapshot/config-push/reconnect offloads). bug-1450/1451 already documented a HANG from flipping these flags ON against pre-existing fixtures that never call slowlane_init (uninitialized c->slowlane silently queues work that's never run). This is a DIFFERENT and more severe symptom -- a hard SEGV, not a hang -- in an apparently-unrelated code path (commit.c's durability-advance callback, invoked from the fsync background thread against a WAL-truncation test fixture), suggesting the two offload flags' interaction with the broader pre-existing test-fixture surface is not limited to the single hang mechanism already documented. Not confirmed whether this reproduces with only ONE of the two flags flipped (both were flipped together when this was found) or whether it is fixture-specific (test_persist_truncate.c) vs a broader live-relevant race -- the crash is inside commit.c/persist.c machinery that IS also exercised by the real live daemon, so this should NOT be assumed to be test-only until someone actually checks whether a live cluster with both flags forced on hits the same SEGV.",
+    "fix": "NOT FIXED -- out of scope for the xce-audit-invariants session that found it (which correctly kept both flags at their default of 0, matching bug-1450/1451's scoping decision; this finding is additional evidence supporting that decision, discovered incidentally while verifying it). Whoever picks up the deferred 'flip demotion/catchup offload to default ON' follow-up audit (referenced in bug-1450/1451) MUST reproduce and root-cause this SEGV as part of that work, in addition to the already-documented hang -- do not treat the hang as the only pre-flip blocker. Suggested next steps: (1) reproduce with each flag flipped individually to isolate which one (or the combination) triggers it; (2) reproduce running only test_persist_truncate.c in isolation (narrow binary via meson test args or a --run filter) to rule out cross-test global-state pollution as a confound; (3) once isolated, read commit_advance_locked (commit.c:208) against what's at offset 0x38 in whatever struct is being dereferenced -- a near-null read at a small fixed offset strongly suggests an uninitialized or already-freed context pointer, consistent with the same 'offload path assumes c->slowlane/commit-engine machinery is live-initialized' class of issue as the documented hang, just manifesting as a crash instead of a stall in this particular fixture.",
+    "tags": [
+      "ha_convergence.c",
+      "config_repl.c",
+      "commit.c",
+      "demotion-offload",
+      "catchup-offload",
+      "raft-drive-async-pattern",
+      "xce-audit-invariants",
+      "controller-tests",
+      "ASAN",
+      "SEGV",
+      "NOT-FIXED",
+      "not-root-caused"
+    ],
+    "related_bugs": [
+      "bug-1429",
+      "bug-1450",
+      "bug-1451"
+    ],
+    "occurrences": 1,
+    "last_seen": "2026-07-18"
   }
 ]