chore: update OpenWolf session state (Group 1 validate-impl batch + bug-1447/1449)

.wolf/buglog.json

@@ -9297,7 +9297,7 @@
     "error_message": "Election Safety violation (two leaders at same term) surfaced by raft-sim-conformance step driver (task 4.2): a duplicated VOTE_RSP grant double-counts a single peer's vote",
     "file": "src/lib/raft/raft.c",
     "root_cause": "raft_on_vote_reply tallies grants with a COUNTER (s->votes_granted++) and explicitly ignores from_idx ((void)from_idx;), so DuplicateMessage of one peer's grant counts that peer twice and a candidate reaches quorum with a single real vote. Diverges from raft.tla where votesGranted is a SET (idempotent). Proven: 3-node, single peer grant duplicated -> candidate becomes LEADER with votes_granted=2.",
-    "fix": "NOT FIXED (out of task 4.2 boundary = raft_sim.c/test_raft.c). Library fix: dedupe per-peer in raft_on_vote_reply (track a granted-by bitset/array keyed on from_idx, or guard against re-counting the same peer in the current term). The conformance harness correctly CATCHES this on adversarial seeds.",
+    "fix": "FIXED (verified 2026-07-18 by validate-impl for raft-sim-conformance): src/lib/raft/raft.c raft_on_vote_reply now has a per-peer vote_replies[from_idx] idempotency guard (dedupes a DuplicateMessage grant) plus a term < s->term DropStaleResponse guard, landed together with bug-1096 in commit b5b7e01 (fix(raft): vote-reply per-peer dedup + stale-term drop). This buglog entry was stale (said NOT FIXED) even though the fix has been live since 2026-06-24 per tasks.md raft-sim-conformance Implementation Notes and bug-1096.fix; corrected here for consistency.",
     "tags": [
       "raft",
       "vote-reply",
@@ -9307,14 +9307,16 @@
       "conformance",
       "needs-library-fix",
       "bug-851-family",
-      "dedup"
+      "dedup",
+      "fixed",
+      "stale-buglog-entry"
     ],
     "related_bugs": [
       "bug-1096",
       "bug-851"
     ],
-    "occurrences": 2,
-    "last_seen": "2026-06-24T13:30:00.000Z"
+    "occurrences": 3,
+    "last_seen": "2026-07-18T00:20:00.000Z"
   },
   {
     "id": "bug-1096",
@@ -13332,5 +13334,105 @@
     ],
     "occurrences": 1,
     "last_seen": "2026-07-17T10:45:00Z"
+  },
+  {
+    "id": "bug-1446",
+    "timestamp": "2026-07-18T00:47:00Z",
+    "error_message": "/kiro-validate-impl ha-commit-durability: test-commit-durability.sh --deterministic reported 3/3 FAIL ('DURABILITY GAP -- primary acked with 0 fdatasync in-window'), i.e. a false RED that looked like a regression of the spec's core invariant on the current build.",
+    "file": "src/tools/test-commit-durability.sh (det_probe_once / date usage) -- test harness only, NOT a product file",
+    "root_cause": "This sandbox's /sbin/date is BusyBox date, which does NOT support GNU date's %N (nanoseconds) format directive -- 'date +%H:%M:%S.%N' literally emits the unexpanded string 'HH:MM:SS.%N' (confirmed by direct invocation). det_probe_once() captures t0/t1 with this call to bound the submit-to-ack window, then _hms_to_sod() converts them to seconds-of-day for the in-window fdatasync count. With the literal '%N' suffix the sub-second component is unparseable/degenerate, so the computed window fails to contain the real fdatasync timestamp even though it lands in the same wall-clock second -- producing syncs=0 (RED) regardless of actual ordering. This is an environment tooling gap (BusyBox vs GNU date), not a code regression: the script assumes GNU date is on PATH.",
+    "fix": "NOT a product bug, no source fix applied (out of this validation's scope; script is owned by xce-test-e2e-primitives per this spec's task 1.3 boundary note). Diagnosed by cross-checking strace's own microsecond timestamps (which DO work correctly, e.g. 'fdatasync(10) = 0' at 00:47:01.128244) against the controller's committed_lsn advance log lines (commit_advance_locked '1 -> 2' at 00:47:01,144) for the same three probes: fdatasync consistently preceded the commit-advance (that gates the client ack) by 16-43ms in all 3 probes -- i.e. the real durable-before-ack ordering the spec requires DOES hold on the current build; the script's verdict was the false signal, not the product. Recommend the script detect/require GNU date (or use 1784335901.164264 / a C helper) rather than assuming %N support, so it fails loud on an incompatible date instead of silently producing a false RED.",
+    "tags": [
+      "test-harness",
+      "false-positive",
+      "busybox",
+      "date",
+      "kiro-validate-impl",
+      "ha-commit-durability",
+      "deterministic-mode"
+    ],
+    "related_bugs": [],
+    "occurrences": 1,
+    "last_seen": "2026-07-18T00:47:00Z"
+  },
+  {
+    "id": "bug-1447",
+    "timestamp": "2026-07-18T01:05:02Z",
+    "error_message": "/kiro-validate-impl raft-drive-async: 10-round SEQUENTIAL HA soak (the fully-supported single-failover lifecycle, hard-fail-on-any-loss) failed at round 9/10 -- 'FAIL: round 9: post-failover submit did not get 3 acks' (0/3 acked across 5 retry attempts). Root traced to ctrl-3 (a standby) crashing: AddressSanitizer DEADLYSIGNAL SEGV on address 0x40 (near-null, READ) in openssl's ssl3_read_bytes <- tls_get_message_header <- state_machine <- handle_ssl_handshake (src/lib/netio.c:548) <- io_handler (netio.c:588) <- io_worker (netio.c:633), thread T2 (an io_worker thread). With ctrl-1 already killed (the round's own kill target) and ctrl-3 now also dead, no 2-of-3 quorum remained for ctrl-2 (freshly elected PRIMARY, term=11) to commit writes -- explaining the observed 0/3-acks symptom as a downstream consequence, not the primary defect.",
+    "file": "src/lib/netio.c (handle_ssl_handshake / io_handler / io_worker -- crash site, shared iomgr epoll dispatch); src/controller/ha.c (ha_connect_prep_peer/ha_connect_finish_peer/reconnect_lane_fn, raft-drive-async Task 2.1/2.2 reconnect-offload -- suspected trigger); src/controller/cluster.c (handle_connection / endpoint_accept -- inbound accept path, alternate suspect)",
+    "root_cause": "CONFIRMED (deterministic regression test + live soak, not speculative). The earlier 'unsynchronized concurrent SSL_do_handshake from two io_worker threads' hypothesis is WRONG and is retracted: iomgr_start() creates exactly ONE io_worker thread per iomgr_t (src/lib/netio.c:iomgr_start), and a single controller process has exactly one such iomgr/thread for all peer connections -- there is no second io_worker thread to race against.\n\nREAL mechanism: io_worker's epoll dispatch loop processes every ready fd from ONE epoll_wait() batch in a plain for-loop calling io_handler() per event (src/lib/netio.c, old io_worker ~line 612-640). A handler for an EARLIER event in that SAME batch can free a DIFFERENT endpoint that also has a ready event LATER in the SAME batch. In production this is exactly src/controller/ha.c:8257-8331 on_standby_connected(), the handshake-complete callback for a freshly-reconnected outbound standby channel (raft-drive-async Task 2.1/2.2's reconnect-offload path): when it detects a stale duplicate outbound channel to the same peer (ha.c:8286-8303, 'Different ep, same peer -> stale duplicate to evict'), it calls iomgr_unmonitor() (ha.c:8320-8321) then endpoint_unref() (ha.c:8327) on the STALE endpoint. iomgr_unmonitor()'s epoll_ctl(EPOLL_CTL_DEL) only stops a FUTURE epoll_wait() from returning events for that fd -- it does NOT invalidate an event for that SAME stale endpoint that is already sitting in the CURRENT batch's stack-local events[] array (very plausible during a reconnect burst, where several peer connections' fds become ready close together right after a new primary election). endpoint_unref() reaching refcnt 0 immediately SSL_free()s ep->ssl and free()s the endpoint_t. The loop's next iteration then calls io_handler() for that now-freed endpoint -- a heap-use-after-free that reads freed memory as 'ep', and (if it still looks like a plausible pointer chain) hands a freed/reused ep->ssl to SSL_do_handshake(), producing exactly the observed ASAN DEADLYSIGNAL SEGV deep inside OpenSSL's ssl3_read_bytes rather than an immediate top-of-function null-ssl crash.\n\nThis is a genuine gap in the bug-1282/1286/1298 hardening family: INVOKE_MSG_HANDLER (netio.h) already refs/unrefs an endpoint around its OWN msg_handler call (bug-1298 note in that macro's comment) so a handler cannot free the SAME ep it is currently dispatched on -- but that bracket never covered a DIFFERENT, already-batched endpoint torn down as a side effect (on_standby_connected's cross-endpoint eviction). io_worker's own epoll dispatch loop held no reference of any kind on the endpoints in its batch. Reproduced deterministically (no live soak needed to hit it) via a new unit test that drives io_worker's real batch-dispatch loop with a synthetic 2-event epoll batch where event 0's handler endpoint_unref()s event 1's endpoint -- confirmed RED pre-fix: ASAN heap-use-after-free at src/lib/netio.c:587 (io_handler reading ep->ssl_handshake_done), freed by src/lib/netio.c:843 (endpoint_unref) called from the injected handler, exactly mirroring the production on_standby_connected -> stale endpoint eviction -> later-batched io_handler() read chain.",
+    "fix": "FIXED (2026-07-18). src/lib/netio.c: extracted io_worker's per-batch dispatch into a new static io_worker_dispatch_batch(struct epoll_event*, int) (~line 639-654), called from io_worker's while(1) loop in place of the old inline for-loop. It now takes an endpoint_ref() on every endpoint in the batch BEFORE dispatching any of them, dispatches all events via io_handler() as before, then endpoint_unref()s every endpoint only AFTER the whole batch has been processed -- mirroring the SAME endpoint_ref/endpoint_unref bracket idiom this codebase already uses for the vtask dispatch path (bug-1298) and INVOKE_MSG_HANDLER (netio.h), applied here to io_worker's own epoll dispatch loop, the one dispatcher that didn't yet have it. With this bracket, on_standby_connected's endpoint_unref() on a stale duplicate can only drop that endpoint's refcount from 2 (owner + io_worker's batch ref) to 1 during the batch -- never to 0 -- so the free is deferred until io_worker's own trailing unref runs after the ENTIRE batch (including that endpoint's own later event) has been safely dispatched. ref/unref are the existing lock-free _Atomic-int operations (netio.c endpoint_ref/endpoint_unref) with no I/O and no lock acquisition, so this adds zero blocking risk to the io_worker hot path (satisfies the async-io steering rule against blocking in vtask/event-loop handlers). Also added a small always-compiled test seam, io_worker_dispatch_batch_for_test() (netio.c + netio.h) -- not XCE_TESTING-gated because netio.c compiles once into libxce.a with no per-target defines (unlike controller/ha.c, which the controller test binary compiles directly with -DXCE_TESTING), so it follows this file's existing convention of unconditionally exporting its internals (endpoint_ref, endpoint_unref, endpoint_mark_tearing_down, etc. are all plain public functions already).\n\nRegression test: src/lib/testsuite/test_netio_batch_dispatch_uaf.c (test_batch_dispatch_survives_cross_endpoint_eviction), registered in src/lib/meson.build and src/lib/testsuite/test.c. Drives io_worker_dispatch_batch_for_test() with a synthetic 2-event batch: event 0 -> an endpoint whose conn_handler calls endpoint_unref() on event 1's endpoint (simulating on_standby_connected's stale-duplicate eviction); event 1 -> an endpoint with refcnt==1 (its only ref is the one event 0's handler drops) whose conn_handler just records that it ran. TDD verified: RED confirmed pre-fix (ASAN heap-use-after-free at netio.c:587 in io_handler, object freed at netio.c:843 endpoint_unref from the injected handler -- captured verbatim before the fix was restored); GREEN confirmed post-fix (both handlers run safely, exit 0, re-confirmed again standalone after both live soaks completed). Full regression: libxce lib suite 294/0/0 (xcelib_test, ASAN, ~/xce.build), controller_unit_test 972/0/0 with zero AddressSanitizer/heap-use-after-free/heap-buffer-overflow/double-free/SEGV hits in the full test log. Live verification beyond the unit test: two independent fresh sequential 10-round /xce-test-ha HA soaks on real ext4 (/home/claude/tmp/xce-bug1447-verify-1: 10/10 rounds, 33 jobs, zero job loss, all 3 nodes led; /home/claude/tmp/xce-bug1447-verify-2: 10/10 rounds, 36 jobs, zero job loss, all 3 nodes led including a forced-election leadership-coverage check on ctrl-3) -- both fully clean, zero crashes, and an explicit grep of every node's logs in both runs for the bug-1447 signature (AddressSanitizer/DEADLYSIGNAL/SEGV/ssl3_read_bytes/handle_ssl_handshake) found zero matches.",
+    "tags": [
+      "ASAN",
+      "FIXED",
+      "SEGV",
+      "endpoint_ref",
+      "handle_ssl_handshake",
+      "io_worker",
+      "kiro-validate-impl",
+      "netio.c",
+      "on_standby_connected",
+      "openssl",
+      "raft-drive-async",
+      "reconnect-offload",
+      "regression-test",
+      "round-9",
+      "same-batch-uaf",
+      "sequential-soak",
+      "ssl3_read_bytes"
+    ],
+    "related_bugs": [
+      "bug-1282",
+      "bug-1286",
+      "bug-1298"
+    ],
+    "occurrences": 1,
+    "last_seen": "2026-07-18T14:30:00.000Z"
+  },
+  {
+    "id": "bug-1448",
+    "timestamp": "2026-07-18T01:54:00Z",
+    "error_message": "/kiro-validate-impl ha-heartbeat-committed-propagation: the xce-build skill's own documented pattern (\"dispatch via a plain Agent-tool call, run_in_background: false, wait for it\") produced an INCOMPLETE build -- ninja log ends with 'ninja: build stopped: interrupted by user.' at 85/192 targets, roughly 14 minutes after launch with zero nix/meson/ninja processes still running and no error in the log. The synchronous Agent call returned to the caller almost immediately (~38s) with text claiming 'The build is running in the background... I'll wait for it to complete' -- i.e. the subagent itself spawned an internal Bash run_in_background job and then ended its own turn, rather than blocking until the command finished.",
+    "file": "src/tools/xce-build.sh (skill invocation pattern, not the script itself); .claude/skills/xce-build/SKILL.md's 'Always dispatch the build via the Agent tool' instruction",
+    "root_cause": "Same defect family as the already-documented xce-test-ha/run-ha-soak.sh 'context: fork' SIGHUP-teardown bug (fixed 2026-07-08), but manifesting through a DIFFERENT mechanism: xce-build's skill doc explicitly avoids 'context: fork' and claims a plain Agent-tool dispatch is safe -- but that safety claim implicitly assumes the subagent's own Bash tool call blocks synchronously for the whole build. In practice the subagent chose to run the long nix/meson/ninja command via its OWN internal run_in_background Bash call and then finished its turn, un-detached (no setsid/nohup/disown) -- so when that subagent's session/turn ended, the still-attached background job's process group got interrupted (ninja saw SIGINT/SIGHUP-equivalent, logged literally as 'interrupted by user'). A second SendMessage asking the same agent to 'wait for it now' resumed a session with nothing left alive to wait for.",
+    "fix": "Relaunched the build via a fresh Agent dispatch whose ONLY instruction was to run `setsid nohup nix develop ./src --command bash src/tools/xce-build.sh controller asan </dev/null >LOG 2>&1 & disown` and return immediately (fire-and-forget), then polled LOG/binary-mtimes independently from the main session via the Monitor tool. That build completed cleanly in ~10 minutes (85/85 ninja targets, clean link, meson test suite started normally) with no interruption. General fix for any long shell command dispatched through the Agent tool that must outlive the dispatching subagent's own turn: always explicit setsid+nohup+disown, never rely on the subagent's internal run_in_background alone or on run_in_background:false at the Agent-tool-call level to guarantee synchronous blocking all the way through.",
+    "tags": [
+      "xce-build",
+      "Agent-tool",
+      "run_in_background",
+      "nohup",
+      "setsid",
+      "disown",
+      "subagent-teardown",
+      "kiro-validate-impl",
+      "ha-heartbeat-committed-propagation",
+      "process-detach"
+    ],
+    "related_bugs": [],
+    "occurrences": 1,
+    "last_seen": "2026-07-18T01:54:00Z"
+  },
+  {
+    "id": "bug-1449",
+    "timestamp": "2026-07-18T09:56:00.000Z",
+    "error_message": "[xce-test-ha] FAIL: round 1: no new primary elected after kill (no serving primary at term>2) -- hit once during ha-elect-ctx-lifecycle kiro-validate-impl soak, did NOT reproduce on immediate retry (10/10 clean)",
+    "file": ".claude/skills/xce-test-ha/run-ha-soak.sh (sequential mode round-1 kill timing) / src/controller/ha.c ha_on_primary_ctrl_fd_rdhup (ha-config-bootstrap Req 2.8 voter-only guard, working as designed)",
+    "root_cause": "Test-harness timing race, not a code defect. Root-caused from the controller logs (not assumed): cold-start election was unusually slow this run -- ctrl-1 only reached CTRL_PRIMARY (term=2) ~10.1s after process start (raft_cb_become_leader). Immediately on becoming primary it began its config-push handshake to both standbys (on_standby_connected: 'outbound standby connection to peer[N] established, ctrlhello sent', same ~10.1s timestamp). The driver's round-1 kill fired only ~1.7s later (its budget is 'submit 3 jobs + verify + walcompare' after primary is detected, no floor on how fast that completes) -- before either standby had received and applied the pushed config. Both standbys were still at config tag=0 when ctrl-1 died, so ha_on_primary_ctrl_fd_rdhup correctly refused to start PreVote/RDHUP re-election per ha-config-bootstrap's own Req 2.8 guard ('no local config (tag=0, n_peers=3) -- voter only, not starting PreVote/RDHUP re-election; awaiting configured primary'), leaving the cluster with no primary and no candidate. This is the guard working exactly as designed against an unconfigured standby self-electing -- the actual gap is that run-ha-soak.sh's sequential mode has no wait for initial 3-of-3 config convergence before round 1's first kill (unlike its wait for FULL reconvergence between later rounds). Sequential mode's round-1 timing has apparently never lined up this unluckily in any of the many other Group-1 kiro-validate-impl soaks run earlier the same session (ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote, raft-sim-conformance, libxce-failover, ha-entity-snapshots, ha-commit-durability, ha-term-safety, controller-status, job-array, job-array-single-wal-record) -- all of which use the same driver script unmodified and ran clean.",
+    "fix": "FIXED (.claude/skills/xce-test-ha/run-ha-soak.sh). Added a round-1-only config-convergence gate that mirrors wait_for_full_convergence's exact polling style (same alive-count guard via pgrep -c -f \"$(run_pattern)\", same STABLE_TICKS=2-consecutive-converged-probes, same sleep 2 backoff, same ~90s/30-iteration bound, same log()/fail() convention) but gates on a NEW signal, cluster_config_converged (all 3 nodes report a non-zero configVersion via a new node_config_tag() single-node ctrlstatus client probe, mirroring node_probe's technique), instead of the existing cluster_committed_converged (committed-prefix catch-up) -- the two convergence signals are independent, and only the config-tag one covers the ha-config-bootstrap Req 2.8 voter-only guard. Added functions node_config_tag() / cluster_config_converged() / wait_for_initial_config_convergence() right after wait_for_full_convergence (~line 556). Call site inserted right after the chaotic-mode branch's closing `fi` and before the 'Steps 7-10 looped' round loop (~line 1897), so it applies only to the sequential/default path (chaotic mode intentionally tolerates this race by design -- 'no new primary' is a COUNTED outcome there, never a hard fail, so it was correctly left untouched). Diff: +77 lines, 0 lines removed -- no C source touched. VERIFIED: 3 back-to-back --sequential 6-round soaks on a genuine ext4 path (/home/claude/tmp, confirmed via `mount`/`df -T`, not overlayfs -- bug-924), all 3 PASS (6 rounds/24 jobs/zero loss/all 3 nodes led, identical verdict each run), and the new 'waiting for 3-of-3 initial config convergence before round 1' / 'confirmed' log lines appeared correctly ahead of every round 1. Cross-checked the actual cold-start timing from controller logs across all 3 runs to size the margin: election (raft_cb_become_leader -> CTRL_PRIMARY) took 10.3-11.7s after process start each run, and the config-push (handle_repl_config_push on both standbys) landed only 40-46ms after election every time -- i.e. the actual condition resolves in well under 100ms, while the new gate's minimum dwell is ~4s (2 ticks x 2s) and its ceiling is ~90s, giving roughly 2 orders of magnitude of margin over the observed timing, not just enough for these 3 runs' luck.",
+    "tags": [
+      "ha-elect-ctx-lifecycle",
+      "xce-test-ha",
+      "run-ha-soak.sh",
+      "ha-config-bootstrap",
+      "test-harness",
+      "flaky",
+      "timing-race",
+      "config-push",
+      "round-1",
+      "sequential-mode"
+    ],
+    "related_bugs": [],
+    "occurrences": 1,
+    "last_seen": "2026-07-18T13:00:00.000Z"
   }
 ]