forked from lei.chen/myxce
chore: update OpenWolf session state (Group 1 validate-impl batch + bug-1447/1449)
This commit is contained in:
parent
426728bb98
commit
8f614b6421
6 changed files with 2822 additions and 350 deletions
|
|
@ -1,7 +1,7 @@
|
|||
# anatomy.md
|
||||
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-07-17T14:29:03.853Z
|
||||
> Files: 682 tracked | Anatomy hits: 0 | Misses: 0
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-07-18T12:26:40.951Z
|
||||
> Files: 683 tracked | Anatomy hits: 0 | Misses: 0
|
||||
|
||||
## ../../../../../../tmp/claude-1000/
|
||||
|
||||
|
|
@ -268,7 +268,7 @@
|
|||
|
||||
## .claude/skills/xce-test-ha/
|
||||
|
||||
- `run-ha-soak.sh` — run-ha-soak.sh — driver for /xce-test-ha: deploy a 3-node XCE HA cluster, then (~29044 tok)
|
||||
- `run-ha-soak.sh` — run-ha-soak.sh — driver for /xce-test-ha: deploy a 3-node XCE HA cluster, then repeatedly kill primary/verify/restart. Sequential mode now gates round 1's first kill on a 3-of-3 initial config-tag convergence wait (bug-1449 fix; mirrors wait_for_full_convergence's poll style) before the round loop, in addition to its existing full-reconvergence wait between later rounds. (~31300 tok)
|
||||
- `SKILL.md` — XCE HA Failover Soak Test (~6845 tok)
|
||||
- `soak-verdict-selftest.sh` — soak-verdict-selftest.sh — deterministic unit test for the verdict parser in (~2637 tok)
|
||||
- `soak-verdict.sh` — soak-verdict.sh — pure (side-effect-free) verdict helpers for run-ha-soak.sh. (~2781 tok)
|
||||
|
|
@ -368,7 +368,7 @@
|
|||
- `INVESTIGATION-failover-job-loss.md` — Investigation: committed jobs lost across controller failover (bug-795 family) (~5299 tok)
|
||||
- `requirements.md` — Requirements Document (~2587 tok)
|
||||
- `research.md` — Gap Analysis — ha-commit-durability (~7773 tok)
|
||||
- `spec.json` (~196 tok)
|
||||
- `spec.json` — phase implementation-complete, validate_impl GO (2026-07-18) (~2100 tok)
|
||||
- `tasks.md` — Implementation Plan (~11353 tok)
|
||||
|
||||
## .kiro/specs/ha-commit-durability/repro-851/
|
||||
|
|
@ -431,7 +431,7 @@
|
|||
- `design.md` — Design Document: ha-entity-snapshots (~8185 tok)
|
||||
- `requirements.md` — Requirements Document (~2216 tok)
|
||||
- `research.md` — Gap Analysis: ha-entity-snapshots (~3038 tok)
|
||||
- `spec.json` (~129 tok)
|
||||
- `spec.json` (~1450 tok)
|
||||
- `tasks.md` — Implementation Plan (~3997 tok)
|
||||
|
||||
## .kiro/specs/ha-heartbeat-committed-propagation/
|
||||
|
|
@ -440,7 +440,7 @@
|
|||
- `design.md` — Design Document — ha-heartbeat-committed-propagation (~4689 tok)
|
||||
- `requirements.md` — Requirements Document (~1763 tok)
|
||||
- `research.md` — Research / Gap Analysis — ha-heartbeat-committed-propagation (~2474 tok)
|
||||
- `spec.json` (~301 tok)
|
||||
- `spec.json` (~2799 tok)
|
||||
- `tasks.md` — Implementation Plan (~2873 tok)
|
||||
|
||||
## .kiro/specs/ha-heartbeat-liveness/
|
||||
|
|
@ -465,7 +465,7 @@
|
|||
- `design.md` — Design Document (~6905 tok)
|
||||
- `requirements.md` — Requirements Document (~3604 tok)
|
||||
- `research.md` — Gap Analysis: ha-prevote (~8259 tok)
|
||||
- `spec.json` (~126 tok)
|
||||
- `spec.json` (~3803 tok)
|
||||
- `tasks.md` — Implementation Plan (~3197 tok)
|
||||
|
||||
## .kiro/specs/ha-repl-hardening/
|
||||
|
|
@ -501,7 +501,7 @@
|
|||
- `design.md` — Design: HA WAL Replication (~1413 tok)
|
||||
- `requirements.md` — Requirements Document (~1087 tok)
|
||||
- `research.md` — Research: HA WAL Replication (~2542 tok)
|
||||
- `spec.json` (~128 tok)
|
||||
- `spec.json` (~2214 tok)
|
||||
- `tasks.md` — Implementation Plan: HA WAL Replication (~1913 tok)
|
||||
|
||||
## .kiro/specs/job-array-single-wal-record/
|
||||
|
|
@ -519,7 +519,7 @@
|
|||
- `design.md` — Design Document — libxce-failover (~8279 tok)
|
||||
- `requirements.md` — Requirements Document (~2933 tok)
|
||||
- `research.md` — Research Log — libxce-failover (~4638 tok)
|
||||
- `spec.json` (~128 tok)
|
||||
- `spec.json` (~787 tok)
|
||||
- `tasks.md` — Implementation Plan (~4086 tok)
|
||||
|
||||
## .kiro/specs/raft-apply-decoupling/
|
||||
|
|
@ -547,7 +547,7 @@
|
|||
- `design.md` — Design Document — raft-drive-async (~9226 tok)
|
||||
- `requirements.md` — Requirements Document (~3369 tok)
|
||||
- `research.md` — Gap Analysis — raft-drive-async (2026-06-25) (~3578 tok)
|
||||
- `spec.json` (~128 tok)
|
||||
- `spec.json` (~1977 tok)
|
||||
- `tasks.md` — Implementation Plan (~10270 tok)
|
||||
|
||||
## .kiro/specs/raft-ha-core-safety/
|
||||
|
|
@ -582,7 +582,7 @@
|
|||
- `design.md` — Design Document (~8280 tok)
|
||||
- `requirements.md` — Requirements Document (~3776 tok)
|
||||
- `research.md` — Gap Analysis — raft-sim-conformance (2026-06-24) (~3557 tok)
|
||||
- `spec.json` — Declares divergence (~428 tok)
|
||||
- `spec.json` — Declares divergence (~1650 tok)
|
||||
- `tasks.md` — Implementation Plan (~4680 tok)
|
||||
|
||||
## .kiro/specs/resval-id-decoupling/
|
||||
|
|
@ -1060,7 +1060,7 @@
|
|||
- `delay.h` — if !defined(_DELAY_H_) (~414 tok)
|
||||
- `kvmgr.h` — if !defined(__KVMGR_H__) (~631 tok)
|
||||
- `myconf.h` — if !defined(_MYCONF_H_) (~560 tok)
|
||||
- `netio.h` — if !defined(_NETIO_H_) (~2469 tok)
|
||||
- `netio.h` — if !defined(_NETIO_H_) (~2608 tok)
|
||||
- `persist.h` — pragma once (~24030 tok)
|
||||
- `typemap.h` — if !defined(_TYPEMAP_H_) (~5207 tok)
|
||||
- `wal_types.h` — pragma once (~2076 tok)
|
||||
|
|
@ -1073,8 +1073,8 @@
|
|||
- `delay.c` — include <errno.h> (~1520 tok)
|
||||
- `jobj.c` — include <errno.h> (~6090 tok)
|
||||
- `kvmgr.c` — include <errno.h> (~3053 tok)
|
||||
- `meson.build` (~1686 tok)
|
||||
- `netio.c` — include <arpa/inet.h> (~8709 tok)
|
||||
- `meson.build` (~1703 tok)
|
||||
- `netio.c` — include <arpa/inet.h> (~9456 tok)
|
||||
- `request.c` — include <errno.h> (~17093 tok)
|
||||
- `vtaskmgr.c` — include <errno.h> (~3822 tok)
|
||||
- `zarray.c` — include <assert.h> (~5924 tok)
|
||||
|
|
@ -1090,12 +1090,13 @@
|
|||
- `test_ctrlstatus.c` (~4821 tok)
|
||||
- `test_jobj_hardening.c` — include <errno.h> (~4939 tok)
|
||||
- `test_kvmgr_reserve.c` — Declares instance (~1260 tok)
|
||||
- `test_netio_batch_dispatch_uaf.c` (~1854 tok)
|
||||
- `test_netio_on_fini.c` (~1328 tok)
|
||||
- `test_raft.c` — include <errno.h> (~98108 tok)
|
||||
- `test_request.c` — include <errno.h> (~3136 tok)
|
||||
- `test_wal_wire.c` — include <errno.h> (~4204 tok)
|
||||
- `test_zarray_reserve.c` — Declares instance (~2341 tok)
|
||||
- `test.c` — include <argp.h> (~1186 tok)
|
||||
- `test.c` — include <argp.h> (~1210 tok)
|
||||
|
||||
## src/scheduler/src/
|
||||
|
||||
|
|
|
|||
|
|
@ -9297,7 +9297,7 @@
|
|||
"error_message": "Election Safety violation (two leaders at same term) surfaced by raft-sim-conformance step driver (task 4.2): a duplicated VOTE_RSP grant double-counts a single peer's vote",
|
||||
"file": "src/lib/raft/raft.c",
|
||||
"root_cause": "raft_on_vote_reply tallies grants with a COUNTER (s->votes_granted++) and explicitly ignores from_idx ((void)from_idx;), so DuplicateMessage of one peer's grant counts that peer twice and a candidate reaches quorum with a single real vote. Diverges from raft.tla where votesGranted is a SET (idempotent). Proven: 3-node, single peer grant duplicated -> candidate becomes LEADER with votes_granted=2.",
|
||||
"fix": "NOT FIXED (out of task 4.2 boundary = raft_sim.c/test_raft.c). Library fix: dedupe per-peer in raft_on_vote_reply (track a granted-by bitset/array keyed on from_idx, or guard against re-counting the same peer in the current term). The conformance harness correctly CATCHES this on adversarial seeds.",
|
||||
"fix": "FIXED (verified 2026-07-18 by validate-impl for raft-sim-conformance): src/lib/raft/raft.c raft_on_vote_reply now has a per-peer vote_replies[from_idx] idempotency guard (dedupes a DuplicateMessage grant) plus a term < s->term DropStaleResponse guard, landed together with bug-1096 in commit b5b7e01 (fix(raft): vote-reply per-peer dedup + stale-term drop). This buglog entry was stale (said NOT FIXED) even though the fix has been live since 2026-06-24 per tasks.md raft-sim-conformance Implementation Notes and bug-1096.fix; corrected here for consistency.",
|
||||
"tags": [
|
||||
"raft",
|
||||
"vote-reply",
|
||||
|
|
@ -9307,14 +9307,16 @@
|
|||
"conformance",
|
||||
"needs-library-fix",
|
||||
"bug-851-family",
|
||||
"dedup"
|
||||
"dedup",
|
||||
"fixed",
|
||||
"stale-buglog-entry"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1096",
|
||||
"bug-851"
|
||||
],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-06-24T13:30:00.000Z"
|
||||
"occurrences": 3,
|
||||
"last_seen": "2026-07-18T00:20:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1096",
|
||||
|
|
@ -13332,5 +13334,105 @@
|
|||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-17T10:45:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1446",
|
||||
"timestamp": "2026-07-18T00:47:00Z",
|
||||
"error_message": "/kiro-validate-impl ha-commit-durability: test-commit-durability.sh --deterministic reported 3/3 FAIL ('DURABILITY GAP -- primary acked with 0 fdatasync in-window'), i.e. a false RED that looked like a regression of the spec's core invariant on the current build.",
|
||||
"file": "src/tools/test-commit-durability.sh (det_probe_once / date usage) -- test harness only, NOT a product file",
|
||||
"root_cause": "This sandbox's /sbin/date is BusyBox date, which does NOT support GNU date's %N (nanoseconds) format directive -- 'date +%H:%M:%S.%N' literally emits the unexpanded string 'HH:MM:SS.%N' (confirmed by direct invocation). det_probe_once() captures t0/t1 with this call to bound the submit-to-ack window, then _hms_to_sod() converts them to seconds-of-day for the in-window fdatasync count. With the literal '%N' suffix the sub-second component is unparseable/degenerate, so the computed window fails to contain the real fdatasync timestamp even though it lands in the same wall-clock second -- producing syncs=0 (RED) regardless of actual ordering. This is an environment tooling gap (BusyBox vs GNU date), not a code regression: the script assumes GNU date is on PATH.",
|
||||
"fix": "NOT a product bug, no source fix applied (out of this validation's scope; script is owned by xce-test-e2e-primitives per this spec's task 1.3 boundary note). Diagnosed by cross-checking strace's own microsecond timestamps (which DO work correctly, e.g. 'fdatasync(10) = 0' at 00:47:01.128244) against the controller's committed_lsn advance log lines (commit_advance_locked '1 -> 2' at 00:47:01,144) for the same three probes: fdatasync consistently preceded the commit-advance (that gates the client ack) by 16-43ms in all 3 probes -- i.e. the real durable-before-ack ordering the spec requires DOES hold on the current build; the script's verdict was the false signal, not the product. Recommend the script detect/require GNU date (or use 1784335901.164264 / a C helper) rather than assuming %N support, so it fails loud on an incompatible date instead of silently producing a false RED.",
|
||||
"tags": [
|
||||
"test-harness",
|
||||
"false-positive",
|
||||
"busybox",
|
||||
"date",
|
||||
"kiro-validate-impl",
|
||||
"ha-commit-durability",
|
||||
"deterministic-mode"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-18T00:47:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1447",
|
||||
"timestamp": "2026-07-18T01:05:02Z",
|
||||
"error_message": "/kiro-validate-impl raft-drive-async: 10-round SEQUENTIAL HA soak (the fully-supported single-failover lifecycle, hard-fail-on-any-loss) failed at round 9/10 -- 'FAIL: round 9: post-failover submit did not get 3 acks' (0/3 acked across 5 retry attempts). Root traced to ctrl-3 (a standby) crashing: AddressSanitizer DEADLYSIGNAL SEGV on address 0x40 (near-null, READ) in openssl's ssl3_read_bytes <- tls_get_message_header <- state_machine <- handle_ssl_handshake (src/lib/netio.c:548) <- io_handler (netio.c:588) <- io_worker (netio.c:633), thread T2 (an io_worker thread). With ctrl-1 already killed (the round's own kill target) and ctrl-3 now also dead, no 2-of-3 quorum remained for ctrl-2 (freshly elected PRIMARY, term=11) to commit writes -- explaining the observed 0/3-acks symptom as a downstream consequence, not the primary defect.",
|
||||
"file": "src/lib/netio.c (handle_ssl_handshake / io_handler / io_worker -- crash site, shared iomgr epoll dispatch); src/controller/ha.c (ha_connect_prep_peer/ha_connect_finish_peer/reconnect_lane_fn, raft-drive-async Task 2.1/2.2 reconnect-offload -- suspected trigger); src/controller/cluster.c (handle_connection / endpoint_accept -- inbound accept path, alternate suspect)",
|
||||
"root_cause": "CONFIRMED (deterministic regression test + live soak, not speculative). The earlier 'unsynchronized concurrent SSL_do_handshake from two io_worker threads' hypothesis is WRONG and is retracted: iomgr_start() creates exactly ONE io_worker thread per iomgr_t (src/lib/netio.c:iomgr_start), and a single controller process has exactly one such iomgr/thread for all peer connections -- there is no second io_worker thread to race against.\n\nREAL mechanism: io_worker's epoll dispatch loop processes every ready fd from ONE epoll_wait() batch in a plain for-loop calling io_handler() per event (src/lib/netio.c, old io_worker ~line 612-640). A handler for an EARLIER event in that SAME batch can free a DIFFERENT endpoint that also has a ready event LATER in the SAME batch. In production this is exactly src/controller/ha.c:8257-8331 on_standby_connected(), the handshake-complete callback for a freshly-reconnected outbound standby channel (raft-drive-async Task 2.1/2.2's reconnect-offload path): when it detects a stale duplicate outbound channel to the same peer (ha.c:8286-8303, 'Different ep, same peer -> stale duplicate to evict'), it calls iomgr_unmonitor() (ha.c:8320-8321) then endpoint_unref() (ha.c:8327) on the STALE endpoint. iomgr_unmonitor()'s epoll_ctl(EPOLL_CTL_DEL) only stops a FUTURE epoll_wait() from returning events for that fd -- it does NOT invalidate an event for that SAME stale endpoint that is already sitting in the CURRENT batch's stack-local events[] array (very plausible during a reconnect burst, where several peer connections' fds become ready close together right after a new primary election). endpoint_unref() reaching refcnt 0 immediately SSL_free()s ep->ssl and free()s the endpoint_t. The loop's next iteration then calls io_handler() for that now-freed endpoint -- a heap-use-after-free that reads freed memory as 'ep', and (if it still looks like a plausible pointer chain) hands a freed/reused ep->ssl to SSL_do_handshake(), producing exactly the observed ASAN DEADLYSIGNAL SEGV deep inside OpenSSL's ssl3_read_bytes rather than an immediate top-of-function null-ssl crash.\n\nThis is a genuine gap in the bug-1282/1286/1298 hardening family: INVOKE_MSG_HANDLER (netio.h) already refs/unrefs an endpoint around its OWN msg_handler call (bug-1298 note in that macro's comment) so a handler cannot free the SAME ep it is currently dispatched on -- but that bracket never covered a DIFFERENT, already-batched endpoint torn down as a side effect (on_standby_connected's cross-endpoint eviction). io_worker's own epoll dispatch loop held no reference of any kind on the endpoints in its batch. Reproduced deterministically (no live soak needed to hit it) via a new unit test that drives io_worker's real batch-dispatch loop with a synthetic 2-event epoll batch where event 0's handler endpoint_unref()s event 1's endpoint -- confirmed RED pre-fix: ASAN heap-use-after-free at src/lib/netio.c:587 (io_handler reading ep->ssl_handshake_done), freed by src/lib/netio.c:843 (endpoint_unref) called from the injected handler, exactly mirroring the production on_standby_connected -> stale endpoint eviction -> later-batched io_handler() read chain.",
|
||||
"fix": "FIXED (2026-07-18). src/lib/netio.c: extracted io_worker's per-batch dispatch into a new static io_worker_dispatch_batch(struct epoll_event*, int) (~line 639-654), called from io_worker's while(1) loop in place of the old inline for-loop. It now takes an endpoint_ref() on every endpoint in the batch BEFORE dispatching any of them, dispatches all events via io_handler() as before, then endpoint_unref()s every endpoint only AFTER the whole batch has been processed -- mirroring the SAME endpoint_ref/endpoint_unref bracket idiom this codebase already uses for the vtask dispatch path (bug-1298) and INVOKE_MSG_HANDLER (netio.h), applied here to io_worker's own epoll dispatch loop, the one dispatcher that didn't yet have it. With this bracket, on_standby_connected's endpoint_unref() on a stale duplicate can only drop that endpoint's refcount from 2 (owner + io_worker's batch ref) to 1 during the batch -- never to 0 -- so the free is deferred until io_worker's own trailing unref runs after the ENTIRE batch (including that endpoint's own later event) has been safely dispatched. ref/unref are the existing lock-free _Atomic-int operations (netio.c endpoint_ref/endpoint_unref) with no I/O and no lock acquisition, so this adds zero blocking risk to the io_worker hot path (satisfies the async-io steering rule against blocking in vtask/event-loop handlers). Also added a small always-compiled test seam, io_worker_dispatch_batch_for_test() (netio.c + netio.h) -- not XCE_TESTING-gated because netio.c compiles once into libxce.a with no per-target defines (unlike controller/ha.c, which the controller test binary compiles directly with -DXCE_TESTING), so it follows this file's existing convention of unconditionally exporting its internals (endpoint_ref, endpoint_unref, endpoint_mark_tearing_down, etc. are all plain public functions already).\n\nRegression test: src/lib/testsuite/test_netio_batch_dispatch_uaf.c (test_batch_dispatch_survives_cross_endpoint_eviction), registered in src/lib/meson.build and src/lib/testsuite/test.c. Drives io_worker_dispatch_batch_for_test() with a synthetic 2-event batch: event 0 -> an endpoint whose conn_handler calls endpoint_unref() on event 1's endpoint (simulating on_standby_connected's stale-duplicate eviction); event 1 -> an endpoint with refcnt==1 (its only ref is the one event 0's handler drops) whose conn_handler just records that it ran. TDD verified: RED confirmed pre-fix (ASAN heap-use-after-free at netio.c:587 in io_handler, object freed at netio.c:843 endpoint_unref from the injected handler -- captured verbatim before the fix was restored); GREEN confirmed post-fix (both handlers run safely, exit 0, re-confirmed again standalone after both live soaks completed). Full regression: libxce lib suite 294/0/0 (xcelib_test, ASAN, ~/xce.build), controller_unit_test 972/0/0 with zero AddressSanitizer/heap-use-after-free/heap-buffer-overflow/double-free/SEGV hits in the full test log. Live verification beyond the unit test: two independent fresh sequential 10-round /xce-test-ha HA soaks on real ext4 (/home/claude/tmp/xce-bug1447-verify-1: 10/10 rounds, 33 jobs, zero job loss, all 3 nodes led; /home/claude/tmp/xce-bug1447-verify-2: 10/10 rounds, 36 jobs, zero job loss, all 3 nodes led including a forced-election leadership-coverage check on ctrl-3) -- both fully clean, zero crashes, and an explicit grep of every node's logs in both runs for the bug-1447 signature (AddressSanitizer/DEADLYSIGNAL/SEGV/ssl3_read_bytes/handle_ssl_handshake) found zero matches.",
|
||||
"tags": [
|
||||
"ASAN",
|
||||
"FIXED",
|
||||
"SEGV",
|
||||
"endpoint_ref",
|
||||
"handle_ssl_handshake",
|
||||
"io_worker",
|
||||
"kiro-validate-impl",
|
||||
"netio.c",
|
||||
"on_standby_connected",
|
||||
"openssl",
|
||||
"raft-drive-async",
|
||||
"reconnect-offload",
|
||||
"regression-test",
|
||||
"round-9",
|
||||
"same-batch-uaf",
|
||||
"sequential-soak",
|
||||
"ssl3_read_bytes"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1282",
|
||||
"bug-1286",
|
||||
"bug-1298"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-18T14:30:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1448",
|
||||
"timestamp": "2026-07-18T01:54:00Z",
|
||||
"error_message": "/kiro-validate-impl ha-heartbeat-committed-propagation: the xce-build skill's own documented pattern (\"dispatch via a plain Agent-tool call, run_in_background: false, wait for it\") produced an INCOMPLETE build -- ninja log ends with 'ninja: build stopped: interrupted by user.' at 85/192 targets, roughly 14 minutes after launch with zero nix/meson/ninja processes still running and no error in the log. The synchronous Agent call returned to the caller almost immediately (~38s) with text claiming 'The build is running in the background... I'll wait for it to complete' -- i.e. the subagent itself spawned an internal Bash run_in_background job and then ended its own turn, rather than blocking until the command finished.",
|
||||
"file": "src/tools/xce-build.sh (skill invocation pattern, not the script itself); .claude/skills/xce-build/SKILL.md's 'Always dispatch the build via the Agent tool' instruction",
|
||||
"root_cause": "Same defect family as the already-documented xce-test-ha/run-ha-soak.sh 'context: fork' SIGHUP-teardown bug (fixed 2026-07-08), but manifesting through a DIFFERENT mechanism: xce-build's skill doc explicitly avoids 'context: fork' and claims a plain Agent-tool dispatch is safe -- but that safety claim implicitly assumes the subagent's own Bash tool call blocks synchronously for the whole build. In practice the subagent chose to run the long nix/meson/ninja command via its OWN internal run_in_background Bash call and then finished its turn, un-detached (no setsid/nohup/disown) -- so when that subagent's session/turn ended, the still-attached background job's process group got interrupted (ninja saw SIGINT/SIGHUP-equivalent, logged literally as 'interrupted by user'). A second SendMessage asking the same agent to 'wait for it now' resumed a session with nothing left alive to wait for.",
|
||||
"fix": "Relaunched the build via a fresh Agent dispatch whose ONLY instruction was to run `setsid nohup nix develop ./src --command bash src/tools/xce-build.sh controller asan </dev/null >LOG 2>&1 & disown` and return immediately (fire-and-forget), then polled LOG/binary-mtimes independently from the main session via the Monitor tool. That build completed cleanly in ~10 minutes (85/85 ninja targets, clean link, meson test suite started normally) with no interruption. General fix for any long shell command dispatched through the Agent tool that must outlive the dispatching subagent's own turn: always explicit setsid+nohup+disown, never rely on the subagent's internal run_in_background alone or on run_in_background:false at the Agent-tool-call level to guarantee synchronous blocking all the way through.",
|
||||
"tags": [
|
||||
"xce-build",
|
||||
"Agent-tool",
|
||||
"run_in_background",
|
||||
"nohup",
|
||||
"setsid",
|
||||
"disown",
|
||||
"subagent-teardown",
|
||||
"kiro-validate-impl",
|
||||
"ha-heartbeat-committed-propagation",
|
||||
"process-detach"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-18T01:54:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1449",
|
||||
"timestamp": "2026-07-18T09:56:00.000Z",
|
||||
"error_message": "[xce-test-ha] FAIL: round 1: no new primary elected after kill (no serving primary at term>2) -- hit once during ha-elect-ctx-lifecycle kiro-validate-impl soak, did NOT reproduce on immediate retry (10/10 clean)",
|
||||
"file": ".claude/skills/xce-test-ha/run-ha-soak.sh (sequential mode round-1 kill timing) / src/controller/ha.c ha_on_primary_ctrl_fd_rdhup (ha-config-bootstrap Req 2.8 voter-only guard, working as designed)",
|
||||
"root_cause": "Test-harness timing race, not a code defect. Root-caused from the controller logs (not assumed): cold-start election was unusually slow this run -- ctrl-1 only reached CTRL_PRIMARY (term=2) ~10.1s after process start (raft_cb_become_leader). Immediately on becoming primary it began its config-push handshake to both standbys (on_standby_connected: 'outbound standby connection to peer[N] established, ctrlhello sent', same ~10.1s timestamp). The driver's round-1 kill fired only ~1.7s later (its budget is 'submit 3 jobs + verify + walcompare' after primary is detected, no floor on how fast that completes) -- before either standby had received and applied the pushed config. Both standbys were still at config tag=0 when ctrl-1 died, so ha_on_primary_ctrl_fd_rdhup correctly refused to start PreVote/RDHUP re-election per ha-config-bootstrap's own Req 2.8 guard ('no local config (tag=0, n_peers=3) -- voter only, not starting PreVote/RDHUP re-election; awaiting configured primary'), leaving the cluster with no primary and no candidate. This is the guard working exactly as designed against an unconfigured standby self-electing -- the actual gap is that run-ha-soak.sh's sequential mode has no wait for initial 3-of-3 config convergence before round 1's first kill (unlike its wait for FULL reconvergence between later rounds). Sequential mode's round-1 timing has apparently never lined up this unluckily in any of the many other Group-1 kiro-validate-impl soaks run earlier the same session (ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote, raft-sim-conformance, libxce-failover, ha-entity-snapshots, ha-commit-durability, ha-term-safety, controller-status, job-array, job-array-single-wal-record) -- all of which use the same driver script unmodified and ran clean.",
|
||||
"fix": "FIXED (.claude/skills/xce-test-ha/run-ha-soak.sh). Added a round-1-only config-convergence gate that mirrors wait_for_full_convergence's exact polling style (same alive-count guard via pgrep -c -f \"$(run_pattern)\", same STABLE_TICKS=2-consecutive-converged-probes, same sleep 2 backoff, same ~90s/30-iteration bound, same log()/fail() convention) but gates on a NEW signal, cluster_config_converged (all 3 nodes report a non-zero configVersion via a new node_config_tag() single-node ctrlstatus client probe, mirroring node_probe's technique), instead of the existing cluster_committed_converged (committed-prefix catch-up) -- the two convergence signals are independent, and only the config-tag one covers the ha-config-bootstrap Req 2.8 voter-only guard. Added functions node_config_tag() / cluster_config_converged() / wait_for_initial_config_convergence() right after wait_for_full_convergence (~line 556). Call site inserted right after the chaotic-mode branch's closing `fi` and before the 'Steps 7-10 looped' round loop (~line 1897), so it applies only to the sequential/default path (chaotic mode intentionally tolerates this race by design -- 'no new primary' is a COUNTED outcome there, never a hard fail, so it was correctly left untouched). Diff: +77 lines, 0 lines removed -- no C source touched. VERIFIED: 3 back-to-back --sequential 6-round soaks on a genuine ext4 path (/home/claude/tmp, confirmed via `mount`/`df -T`, not overlayfs -- bug-924), all 3 PASS (6 rounds/24 jobs/zero loss/all 3 nodes led, identical verdict each run), and the new 'waiting for 3-of-3 initial config convergence before round 1' / 'confirmed' log lines appeared correctly ahead of every round 1. Cross-checked the actual cold-start timing from controller logs across all 3 runs to size the margin: election (raft_cb_become_leader -> CTRL_PRIMARY) took 10.3-11.7s after process start each run, and the config-push (handle_repl_config_push on both standbys) landed only 40-46ms after election every time -- i.e. the actual condition resolves in well under 100ms, while the new gate's minimum dwell is ~4s (2 ticks x 2s) and its ceiling is ~90s, giving roughly 2 orders of magnitude of margin over the observed timing, not just enough for these 3 runs' luck.",
|
||||
"tags": [
|
||||
"ha-elect-ctx-lifecycle",
|
||||
"xce-test-ha",
|
||||
"run-ha-soak.sh",
|
||||
"ha-config-bootstrap",
|
||||
"test-harness",
|
||||
"flaky",
|
||||
"timing-race",
|
||||
"config-push",
|
||||
"round-1",
|
||||
"sequential-mode"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-18T13:00:00.000Z"
|
||||
}
|
||||
]
|
||||
|
|
|
|||
|
|
@ -473,3 +473,33 @@ Found 2089 leftover `/tmp/xce_*` directories (~89GB) during the bug-1445 regress
|
|||
|
||||
## Fixed (2026-07-17) — `controller_unit_test`'s nested `UNITY_BEGIN()`/`UNITY_END()` calls masked most of the suite's exit code; fixed suite-wide via `UnitySetTestFile`
|
||||
Root cause of the 2026-06-11 Unity-quirk note (file misattribution) turned out to be worse than cosmetic: only 21 of 86 `test_*.c` group-function files called their own nested `UNITY_BEGIN()`/`UNITY_END()` (each call resets `Unity.NumberOfTests`/`TestFailures` globally); the other 65 correctly relied on the single outer `UNITY_BEGIN()`/`return UNITY_END();` pair in `controller_test.c:main()`. Net effect: `controller_test.c`'s final `return UNITY_END();` — the value `meson test` actually gates on — only ever reflected whichever small window of tests ran since the LAST nested reset (empirically 16 of 972 total tests in one observed run), silently discarding pass/fail evidence for the other ~956. **Fix**: replaced every nested `UNITY_BEGIN();` with `UnitySetTestFile(__FILE__);` (sets the file-attribution pointer only, no counter reset — `unity_internals.h:546`, already visible via the existing `#include <unity/unity.h>`) and deleted the corresponding `UNITY_END();`; added the same `UnitySetTestFile(__FILE__);` as the first statement to the other 65 group functions that had neither. Left the 2 genuinely-standalone binaries (`test_job_array_sched_push.c`, `test_job_array_apiserver_push.c` — each has its own `int main()`, not registered via `controller_test.c`'s `TEST()` macro) untouched. Verified both effects: (1) file attribution — every `PASS`/`FAIL` banner now names its own true source file; (2) exit-code integrity — the single final summary now reads `972 Tests 0 Failures 0 Ignored OK`, exactly matching the ground-truth `grep -c RUN_TEST(` across all 83 registered group files (976 total RUN_TEST call sites minus 4 belonging to the 2 standalone binaries = 972). Full rebuild + `xce-build.sh all` regression pass clean. **Key learning**: `UnitySetTestFile()` (distinct from `UnityBegin()`, which calls it internally AND resets counters) is the correct primitive when a codebase wants per-file test-source attribution inside a single shared Unity counter/exit-code scope spanning multiple group functions in one binary — never call `UNITY_BEGIN()`/`UNITY_END()` from more than one place per test binary.
|
||||
|
||||
## Key Learning (2026-07-18) — ha-entity-snapshots validated GO; task 7.1's "pending upstream snapshot-boundary recon" blocker was a REAL, since-resolved cross-spec dependency, not a stale checkbox
|
||||
Before trusting tasks.md's `[x] 7.1`, traced the blocker's full lifecycle: commit `3fab291` explicitly BLOCKED task 7.1 citing "committed_divergence is upstream (raft-ha-core-safety task 3)"; that upstream task ("Snapshot-boundary reconciliation", raft-ha-core-safety tasks.md task 3, `[x]`) plus its related task 5.10a were later CLOSED with a real investigation chain (buglog + tasks.md 5.10a note, 2026-07-01/02): a `walcompare` false-positive (no per-member checkpoint-floor awareness, fixed `f3c86a6`), a delaymgr seconds/ms deadline-type-confusion starvation bug (fixed `b9ad018`), a hardcoded 64MB WAL-segment-rotation stall, and a netio.h `INVOKE_MSG_HANDLER` UAF during concurrent snapshot-transfer+election — culminating in a live `--force-snapshot` soak (`committed_divergence=0 committed_job_loss=0 result=PASS`, confirmed real snapshot install) plus a clean 3-round `--chaotic` regression (commit `1439e26`). Only after confirming this closure chain (git log grep for the feature name + the blocking-note's own cited upstream task, not just trusting the checkbox) did full validation proceed: build 192/192, controller suite 962 PASS/0 FAIL (ANSI-stripped, incl. all 9 test_persist_snapshot_entities.c cases), cross-task wiring (`out_entities_restored`, `snapshot_inflight_lsn[]`) verified end-to-end by direct code read against design.md's diagrams, File Structure Plan match exact, requirements traceability complete (12/12 line items). GO written to spec.json. **Key learning**: a task marked `[x]` with a documented external "blocked on upstream X" history needs the upstream spec's own tasks.md/buglog evidence checked before trusting the checkbox — don't assume either (a) blind trust in `[x]`, or (b) that any "phase mislabeling" pattern seen elsewhere in the project (ha-consistency-invariants) automatically applies here; each blocker needs its own resolution evidence traced.
|
||||
|
||||
## Key Learning (2026-07-18) — `ready_for_implementation: false` with no `blocked_on` is often just a never-flipped flag, not a real blocker; check tasks.md completion + git log before assuming supersession
|
||||
libxce-failover spec.json showed all 19 tasks [x] (done since 2026-06-18) but `phase: tasks-generated` and `ready_for_implementation: false` with zero explanation. Investigated whether it was superseded by later HA specs (ha-repl-hardening, raft-drive-async, raft-library) touching similar failover-client territory — it was NOT: git log shows 11 dedicated feature commits (foundation through bug-1436 fix) that later specs build ON TOP of, not duplicate. This is the SAME administrative-staleness pattern already seen in ha-term-safety and controller-status spec.json (both also "first post-implementation validation run, all tasks [x], flag never flipped"). **Do-not-repeat pattern for triage**: `ready_for_implementation: false` + all tasks `[x]` + no `blocked_on` field is a strong signal to check (1) git log for feature-tagged commits proving real implementation happened, (2) whether a live/E2E validation entry already exists in .wolf/memory.md for that spec, (3) whether later specs reference it as a dependency rather than a replacement — before concluding it's blocked or dead. Sibling spec.json files (ha-term-safety, controller-status) are the reference shape for the `validate_impl: {decision, validated_at, note, known_limitations}` block used to close this out.
|
||||
|
||||
## Key Learning (2026-07-18) — ha-commit-durability validated GO after re-verifying the core invariant survived heavy adjacent churn, NOT by trusting the 2026-06-18/07-02 evidence alone
|
||||
Same "all tasks [x], phase stuck at tasks-generated, no validate_impl" administrative-staleness pattern as ha-entity-snapshots/libxce-failover, but this one had a real reason for extra scrutiny: persist.c/ha.c/cluster.c/commit.c (the exact files this spec owns) received substantial edits AFTER the spec's own live validation (2026-06-18 durability fix, 2026-07-02 cold-restart-all real-fs revalidation) — notably bug-1433/1434 (2026-07-11, added `persist_committed_lsn_advance_atomic` CAS-safe committed_lsn advance + removed a force-flush call from the promotion path) and bug-1445 (2026-07-17, a first-boot promotion-check race in the SAME `raft_cb_become_leader` function this spec's promotion-eligibility gate lives in). Resolved by (1) code-level re-verification: `majority_candidate_locked` still caps self at `persist_durable_lsn`, and the newer CAS helper's own code comments explicitly state it preserves "committed_lsn never advances past durable_lsn" — later fixes were written TO respect this spec's invariant, not bypass it; (2) a FRESH live re-run this session (not reused from the stale record) on genuine ext4 (`/home/claude` confirmed backed by `/dev/nvme0n1p2 ext4`, NOT the container's overlay `/`): a 4-round rolling-SIGKILL soak + cold-restart-all, 8/8 PASS, 0 job loss. **Do-not-repeat pattern**: before accepting old live evidence as sufficient for a validate_impl GO, check `git log --oneline -- <owned files>` since the evidence's date — if there's a density of unrelated-feature commits touching the SAME files (not just the same subsystem), don't just infer safety from commit-message claims; re-run at least a short version of the spec's own live check on the CURRENT build.
|
||||
|
||||
## Do-Not-Repeat (2026-07-18, bug-1447) — SSL* objects aren't thread-safe; multi-threaded epoll dispatch touching the same endpoint from two io_worker threads crashes deep inside OpenSSL, not at an obvious null check
|
||||
raft-drive-async sequential 10-round HA soak crashed round 9/10: ctrl-3 (standby) died to an ASAN SEGV (addr 0x40) in ssl3_read_bytes <- tls_get_message_header <- state_machine <- handle_ssl_handshake (src/lib/netio.c:548) <- io_handler <- io_worker, 50ms after ctrl-2's promotion mid reconnect-offload burst (Task 2.1/2.2 ha_connect_prep_peer/ha_connect_finish_peer/reconnect_lane_fn). Ruled out simpler causes by reading code directly, not guessing: ssl_handshake_done resets to 0 on every connect (no stale flag), SSL_new(client_ctx) is fresh on every prep (no cross-fd reuse). Leading hypothesis: unsynchronized concurrent SSL_do_handshake/SSL_read on the same ep->ssl from two io_worker threads (classic multithreaded-epoll double-dispatch) — the crash is a deep, plausible-pointer-chain SEGV inside OpenSSL's state machine, not an immediate null-ssl crash at the top of SSL_do_handshake, which is the fingerprint of a race, not a simple logic bug. Same defect family as fixed bug-1282 (ABBA deadlock netio.c), bug-1286 (NULL ep->owner), bug-1298 (endpoint UAF refcount) — newly exposed by raft-drive-async's own faster reconnect cadence. **Do-not-repeat**: when a crash lands deep inside a non-thread-safe C library's internal state machine (OpenSSL SSL*, similar libs) under a multi-threaded epoll/io_worker dispatch loop, suspect concurrent same-object access before anything else — verify no two threads can be handed the same fd/endpoint's readiness event simultaneously (EPOLLONESHOT/EPOLLEXCLUSIVE, or a per-endpoint mutex around the handshake+read+write path). Logged bug-1447 (NOT-FIXED, blocking); NO-GO written to raft-drive-async spec.json validate_impl with 5-step remediation (reproduce, instrument thread-id+ep-pointer logging, serialize per-endpoint, add regression test, re-run soak to clean 10/10).
|
||||
|
||||
## CORRECTION + FIX (2026-07-18, bug-1447 CLOSED) — the "two io_worker threads" hypothesis above was WRONG; the real bug is a SAME-THREAD, SAME-EPOLL-BATCH use-after-free, and it needed no thread race at all to reproduce
|
||||
Read `iomgr_start` (netio.c) closely before instrumenting anything: it does exactly ONE `pthread_create(&iomgr->worker, ..., io_worker, iomgr)`. A controller process has exactly one iomgr/one io_worker thread for all peer connections — there is no second io_worker thread to race against, so the multithreaded-double-dispatch hypothesis in the entry above is retracted. The real mechanism: `io_worker`'s epoll loop dispatches every fd from ONE `epoll_wait()` batch in a plain for-loop over `io_handler()`, holding no reference on any of them. `ha.c`'s `on_standby_connected` (this spec's own Task 2.1/2.2 reconnect-completion callback) evicts a stale duplicate outbound peer channel via `iomgr_unmonitor()`+`endpoint_unref()` when a fresher reconnect lands — but `epoll_ctl(EPOLL_CTL_DEL)` only stops a FUTURE `epoll_wait()` from returning events for that fd; it does NOT invalidate an event for that SAME stale endpoint already sitting in the CURRENT batch's stack-local `events[]` array (very plausible during a reconnect burst — several peer fds become ready close together right after a new-primary election). `endpoint_unref()` hitting refcnt 0 frees the endpoint (and `SSL_free`s its SSL object) immediately; the loop's NEXT iteration then reads that freed memory in `io_handler`, matching the observed deep OpenSSL SEGV exactly (a freed-but-plausible pointer chain, not an immediate null-ssl crash). Proved this INSTEAD OF the thread-race theory with a deterministic unit test (no live soak, no timing luck needed): `test_netio_batch_dispatch_uaf.c` drives `io_worker`'s real batch-dispatch loop (new seam `io_worker_dispatch_batch_for_test`) with a synthetic 2-event batch where event 0's handler `endpoint_unref()`s event 1's endpoint — confirmed RED (ASAN heap-use-after-free at netio.c:587, freed at netio.c:843) before the fix, GREEN after. **Fix**: extracted `io_worker_dispatch_batch()` (netio.c ~line 639) which now takes `endpoint_ref()` on every endpoint in the batch BEFORE dispatching any of them and `endpoint_unref()`s each one only AFTER the whole batch is processed — same lock-free bracket idiom already used for the vtask dispatch path (bug-1298) and `INVOKE_MSG_HANDLER`, just never applied to io_worker's OWN dispatch loop before. Verified: full lib suite 294/0/0, controller suite 972/0/0 (zero ASAN hits), two independent clean sequential 10-round soaks (33 and 36 jobs, zero loss, all 3 nodes led each run) with an explicit post-hoc grep for the crash signature finding zero matches. **Do-not-repeat**: before chasing a "concurrent access from two threads" hypothesis for a crash inside a single-threaded-by-design event loop, verify the actual thread topology first (`grep -n pthread_create` on the suspected loop's `_start`/`_init` function) — a same-thread, same-batch UAF (an earlier event's handler freeing a struct a LATER event in the identical `epoll_wait()` batch still points to) produces an IDENTICAL-looking deep-library-internals SEGV to a genuine cross-thread race, but is far more likely whenever the crash trigger is "a burst of several connections churning close together" (reconnect bursts, election transitions) rather than sustained steady-state load, and is trivially reproducible with a synthetic 2-event batch instead of a multi-minute live soak. `INVOKE_MSG_HANDLER`-style "ref myself for the duration of my own handler" brackets do NOT protect a DIFFERENT endpoint that handler's side effects tear down — that gap is invisible unless you specifically ask "what if this handler frees something ELSE the batch already has queued."
|
||||
|
||||
## Key Learning (2026-07-18) — ha-wal-replication validated GO right after raft-drive-async's bug-1447 NO-GO; a shared-codebase crash from a sibling spec's soak does NOT automatically block an adjacent spec's own soak
|
||||
Ran the same canonical sequential 10-round `/xce-test-ha` soak (same session, same fresh ASAN binaries, same codebase state) that had just crashed raft-drive-async at round 9/10 with bug-1447 (SEGV in netio.c's handle_ssl_handshake/io_worker, triggered by that spec's own faster reconnect-offload cadence). ha-wal-replication's run went 10/10 clean — 33 jobs, zero loss, all 3 nodes led, and an explicit grep of every node log for bug-1447's exact signature (AddressSanitizer/DEADLYSIGNAL/SEGV/ssl3_read_bytes/handle_ssl_handshake/io_worker) found zero matches. Reasoned explicitly about what this does and doesn't prove: it does NOT prove bug-1447 is fixed or absent (it's a rare timing-dependent race that took the sibling until round 9/10 to trigger once) — but it DOES establish that ha-wal-replication's own boundary (repl.c/ha.c/cluster.c WAL-transmission+ACK+commit path) doesn't touch raft-drive-async's reconnect-offload trigger path, so its own regression contract (soak passes clean) is independently satisfiable. Wrote GO with bug-1447 listed as a known_limitation (shared-codebase risk, owned elsewhere) rather than either ignoring it or letting it block an unrelated spec. **Pattern**: when validating spec B right after spec A hit a live-soak crash on the shared codebase, always re-run B's own soak and explicitly grep for A's crash signature before writing B's verdict — don't assume "codebase is broken" (over-blocking) or "A's finding doesn't apply here" (under-checking) without evidence either way.
|
||||
|
||||
## Key Learning (2026-07-18) — ha-heartbeat-committed-propagation validated GO on a RETRY after a prior validation crashed silently mid-soak with zero diagnosable artifacts
|
||||
A prior `/kiro-validate-impl` attempt for this spec died mid-live-soak ~4 hours before this retry, leaving only an empty stale deploy-dir marker (`~/tmp/xce-ha-heartbeat-committed-deploy`) — no logs, no core, no error trail of any kind. Rather than guess at a root cause or treat the silence itself as evidence of a product bug, ran the FULL validation from scratch (fresh ASAN build, 972/972 unit tests including the 12 heartbeat-owned tests, then a clean 10/10 sequential soak with explicit bug-1447 signature grep) and found nothing — no crash, no job loss, no unusual log lines anywhere in the fresh run. **Pattern**: when a validation pass vanishes with no artifacts, don't spend time forensically reconstructing an unrecoverable crash — clean the stale directory, retry the full pipeline end-to-end, and let a clean reproduction (or lack thereof) speak for itself. A silent crash with zero logs is far more often an agent/harness/environment loss (OOM kill, session death, disconnect) than a reproducible product defect; write it into `known_limitations` as an unexplained-but-not-reproduced event rather than blocking the verdict on it.
|
||||
|
||||
## Do-Not-Repeat (2026-07-18, bug-1446) — a live-soak script's `date +%N` silently degrades to a false verdict on BusyBox `date`; don't trust a single-mode script result without cross-checking an independent timestamp source
|
||||
`test-commit-durability.sh --deterministic` reported 3/3 FAIL ("0 fdatasync in-window") on a build that, by every other signal (code read, and the very same run's own strace log), was correct. Root cause: this sandbox's `/sbin/date` is BusyBox date, which has NO `%N` (nanoseconds) support — `date +%H:%M:%S.%N` emits the literal unexpanded string `...%N`, breaking the script's sub-second submit/ack window arithmetic (`_hms_to_sod`) so it never lines up with the real fdatasync timestamp even when it fired in the same wall-clock second. Diagnosed (not accepted at face value) by cross-checking strace's own microsecond timestamps (unaffected — strace does its own formatting, not shelling out to `date`) against the controller log's `commit_advance_locked` lines for the same 3 probes: fdatasync consistently preceded the commit-advance/ack by 16-43ms in all 3, i.e. the real invariant held; the script was lying. **Do-not-repeat**: when a bash test script computes sub-second timing via `date +%N` (or any GNU-date-specific format), verify `date +%N` actually expands in the current environment BEFORE trusting a RED verdict from it — a BusyBox/musl/alpine-family environment will silently degrade instead of erroring. When in doubt, cross-check against a tool that timestamps independently of the shell's `date` binary (here, `strace -tt`'s own log) rather than assuming the product regressed. Logged as bug-1446 (test-harness only, no product fix).
|
||||
|
||||
## Key Learning (2026-07-18) — ha-prevote validated GO; a spec whose original 2026-06-16 implementation had already been gap-audited in research.md still needed full re-validation because of heavy adjacent churn since
|
||||
tasks.md showed 18/18 (12 leaf + 6 group-header) checked since 2026-06-16, and research.md itself already contained a full 2026-06-18 gap re-audit (27/27 acceptance criteria MET/preserved, file:line evidence) — strong prior evidence, but NOT a substitute for re-validation, because 40+ commits touched raft.c/ha.c/cluster.c since then (raft-commit-safety's G6 vote-freshness closure, ha-term-safety, raft-drive-async, raft-apply-decoupling, ha-elect-ctx-lifecycle, and bug fixes 605/606/660/1421/1424/1430/1435 — several DIRECTLY inside the PreVote functions themselves, e.g. bug-1430's UAF in `ha_prevote_round_timeout_fn`). Re-verified via codegraph that the three pure raft-library PreVote functions (`raft_begin_pre_vote`/`raft_on_pre_vote_request`/`raft_on_pre_vote_reply`) still hold the zero-mutation invariant byte-for-byte, then ran the full pipeline fresh: 293 raft-lib + 972 controller unit tests (52 PreVote-tagged across 5 files) all green, then a clean 10/10 sequential soak with LIVE PreVote activity confirmed straight from the per-node logs (13 WON events, 1 DENIED-with-reason, 2 mid-round-cancel events) rather than just inferred from the absence of job loss — the mid-round-cancel events incidentally live-exercised the frozen-then-resumed re-attach path (Req 4.2) even though the sequential kill-schedule soak has no dedicated SIGSTOP scenario. Also found one legitimate, non-blocking design-doc drift: `raft_on_pre_vote_request`'s shipped signature grew two params (cand_last_log_term/cand_last_log_index) beyond design.md's original snapshot, because a LATER sibling spec (raft-commit-safety, G6) closed a freshness gap by moving the (lastLogTerm,lastLogIndex) check INTO the function itself — strictly stronger than the original design, correctly attributed to the owning spec rather than treated as this spec's regression. **Pattern**: `research.md`'s own historical gap-audit is excellent supporting evidence but has a staleness date — always check `git log --since=<that date>` on the spec's core files before trusting an old audit as current.
|
||||
|
||||
## Key Learning (2026-07-18, bug-1449 fixed) — run-ha-soak.sh had TWO independent convergence signals conflated into one; round 1 was only guarded by the wrong one
|
||||
`wait_for_full_convergence`/`cluster_committed_converged` (gates every round AFTER round 1) checks role-count (1 primary + 2 standby) + committed-LSN catch-up — it says nothing about `configVersion` (the config "tag" a standby needs before `ha_on_primary_ctrl_fd_rdhup`'s ha-config-bootstrap Req 2.8 voter-only guard will let it self-elect). Round 1 had NO wait for that second, independent signal before its first kill, so an unlucky-but-legitimate slow cold-start election (observed 10-12s across 3 verification runs, vs. the ~1.7s job-submit budget the old code implicitly relied on) could let the kill land before config-push finished, correctly triggering the tag=0 guard and producing a false "no primary elected" round-1 failure. Fix mirrors `wait_for_full_convergence`'s exact polling primitive (alive-count via `pgrep -c -f "$(run_pattern)"`, STABLE_TICKS=2, sleep 2, ~90s bound) but gates on a NEW `cluster_config_converged` (all 3 nodes' `configVersion` via ctrlstatus non-zero, probed by a new `node_config_tag()` mirroring `node_probe`'s single-node client-probe technique) — added right after `wait_for_full_convergence` (~line 556) and called once, right before the round loop (~line 1897), so it applies ONLY to sequential mode's round-1 kill (chaotic mode intentionally tolerates this exact race elsewhere — a stalled election there is a COUNTED outcome, never a hard fail, so it was correctly left untouched). **Do-not-repeat**: when a soak/test driver has more than one "is the cluster ready" signal server-side (committed-prefix vs. config-tag here), a wait function gating on ONE does not imply the other is also satisfied — check what the actual guard/precondition being raced depends on before assuming an existing convergence wait already covers it. Verified 3/3 back-to-back clean `--sequential 6` runs on genuine ext4 (`/home/claude/tmp`, confirmed via `mount`/`df -T`, not overlayfs); cross-checked cold-start timing from controller logs across all 3 runs (election 10.3-11.7s after process start, config-push landing only 40-46ms after election every time) to confirm the new gate's ~4s minimum dwell / ~90s ceiling has ~2 orders of magnitude of margin, not just enough for that day's luck.
|
||||
|
|
|
|||
|
|
@ -1,379 +1,154 @@
|
|||
{
|
||||
"session_id": "session-2026-07-17-2354",
|
||||
"started": "2026-07-17T06:54:07.397Z",
|
||||
"session_id": "session-2026-07-18-0309",
|
||||
"started": "2026-07-18T10:09:12.216Z",
|
||||
"files_read": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/scheduler/src/handler.c": {
|
||||
"count": 5,
|
||||
"tokens": 4838,
|
||||
"first_read": "2026-07-17T06:57:21.832Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/request.c": {
|
||||
"count": 1,
|
||||
"tokens": 17093,
|
||||
"first_read": "2026-07-17T06:58:14.535Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/xce.build/lib/controller.capnp.c": {
|
||||
"count": 3,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-17T06:58:42.008Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/frontend/apiserver/backend/wire_dispatch_support.go": {
|
||||
"count": 2,
|
||||
"tokens": 2254,
|
||||
"first_read": "2026-07-17T06:59:17.610Z"
|
||||
},
|
||||
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/1c68567a-5bf5-4a4d-980c-beb4c42e613d/scratchpad/sched_job_array_test.log": {
|
||||
"count": 3,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-17T07:02:48.655Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/scheduler/testsuite/test_job_array_expand.c": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh": {
|
||||
"count": 8,
|
||||
"tokens": 3664,
|
||||
"first_read": "2026-07-17T07:03:38.003Z"
|
||||
"tokens": 30070,
|
||||
"first_read": "2026-07-18T11:55:14.604Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/scheduler/testsuite/meson.build": {
|
||||
"count": 1,
|
||||
"tokens": 1057,
|
||||
"first_read": "2026-07-17T07:09:28.555Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/tasks.md": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c": {
|
||||
"count": 3,
|
||||
"tokens": 15579,
|
||||
"first_read": "2026-07-17T07:12:34.041Z"
|
||||
"tokens": 111605,
|
||||
"first_read": "2026-07-18T11:56:51.745Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/netio.c": {
|
||||
"count": 2,
|
||||
"tokens": 9351,
|
||||
"first_read": "2026-07-18T12:15:26.076Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test_netio_on_connected.c": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-18T12:18:53.976Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_ha_vep_teardown.c": {
|
||||
"count": 1,
|
||||
"tokens": 6218,
|
||||
"first_read": "2026-07-18T12:20:54.636Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/include/netio.h": {
|
||||
"count": 1,
|
||||
"tokens": 2469,
|
||||
"first_read": "2026-07-18T12:23:47.205Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/meson.build": {
|
||||
"count": 1,
|
||||
"tokens": 1686,
|
||||
"first_read": "2026-07-18T12:24:40.560Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test.c": {
|
||||
"count": 1,
|
||||
"tokens": 1186,
|
||||
"first_read": "2026-07-17T09:26:55.161Z"
|
||||
"first_read": "2026-07-18T12:24:41.285Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/requirements.md": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-drive-async/tasks.md": {
|
||||
"count": 1,
|
||||
"tokens": 5216,
|
||||
"first_read": "2026-07-17T09:35:08.346Z"
|
||||
"tokens": 10270,
|
||||
"first_read": "2026-07-18T13:20:33.365Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/design.md": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-drive-async/requirements.md": {
|
||||
"count": 1,
|
||||
"tokens": 14117,
|
||||
"first_read": "2026-07-17T09:35:08.470Z"
|
||||
"tokens": 3369,
|
||||
"first_read": "2026-07-18T13:20:33.747Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/steering/testing.md": {
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/raft-drive-async/design.md": {
|
||||
"count": 1,
|
||||
"tokens": 741,
|
||||
"first_read": "2026-07-17T09:37:25.366Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/frontend/Makefile": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-17T09:37:32.149Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/frontend/apiserver/meson.build": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-17T09:37:55.047Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/spec.json": {
|
||||
"count": 4,
|
||||
"tokens": 444,
|
||||
"first_read": "2026-07-17T09:45:25.298Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c": {
|
||||
"count": 7,
|
||||
"tokens": 111605,
|
||||
"first_read": "2026-07-17T10:13:07.553Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/persist.c": {
|
||||
"count": 2,
|
||||
"tokens": 72815,
|
||||
"first_read": "2026-07-17T10:13:46.398Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_job_array_replay_fence.c": {
|
||||
"count": 6,
|
||||
"tokens": 8045,
|
||||
"first_read": "2026-07-17T10:15:49.401Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_job_array_fence.c": {
|
||||
"count": 1,
|
||||
"tokens": 6110,
|
||||
"first_read": "2026-07-17T10:16:03.085Z"
|
||||
},
|
||||
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/1c68567a-5bf5-4a4d-980c-beb4c42e613d/scratchpad/build_output.log": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-17T10:24:34.640Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-term-safety/requirements.md": {
|
||||
"count": 1,
|
||||
"tokens": 4240,
|
||||
"first_read": "2026-07-17T10:43:37.731Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-term-safety/design.md": {
|
||||
"count": 1,
|
||||
"tokens": 11209,
|
||||
"first_read": "2026-07-17T10:43:38.127Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-term-safety/tasks.md": {
|
||||
"count": 1,
|
||||
"tokens": 14377,
|
||||
"first_read": "2026-07-17T10:43:38.709Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/controller-status/spec.json": {
|
||||
"count": 2,
|
||||
"tokens": 608,
|
||||
"first_read": "2026-07-17T10:43:48.699Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/controller-status/tasks.md": {
|
||||
"count": 1,
|
||||
"tokens": 5073,
|
||||
"first_read": "2026-07-17T10:43:49.288Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/controller-status/requirements.md": {
|
||||
"count": 1,
|
||||
"tokens": 3158,
|
||||
"first_read": "2026-07-17T10:43:56.546Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/controller-status/design.md": {
|
||||
"count": 1,
|
||||
"tokens": 7961,
|
||||
"first_read": "2026-07-17T10:43:56.917Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/clean_failover_preservation.sh": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-17T10:44:37.518Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-build/SKILL.md": {
|
||||
"count": 1,
|
||||
"tokens": 985,
|
||||
"first_read": "2026-07-17T10:44:48.796Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/SKILL.md": {
|
||||
"count": 1,
|
||||
"tokens": 6845,
|
||||
"first_read": "2026-07-17T10:44:48.878Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/tools/xce-build.sh": {
|
||||
"count": 1,
|
||||
"tokens": 1400,
|
||||
"first_read": "2026-07-17T10:46:15.229Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/cluster.c": {
|
||||
"count": 1,
|
||||
"tokens": 18669,
|
||||
"first_read": "2026-07-17T10:46:40.673Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test.c": {
|
||||
"count": 2,
|
||||
"tokens": 30294,
|
||||
"first_read": "2026-07-17T10:46:41.369Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/schema/controller.capnp": {
|
||||
"count": 1,
|
||||
"tokens": 9735,
|
||||
"first_read": "2026-07-17T10:49:36.108Z"
|
||||
},
|
||||
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/1c68567a-5bf5-4a4d-980c-beb4c42e613d/scratchpad/go-frontend-test.log": {
|
||||
"count": 1,
|
||||
"tokens": 0,
|
||||
"first_read": "2026-07-17T10:51:06.457Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-term-safety/spec.json": {
|
||||
"count": 1,
|
||||
"tokens": 140,
|
||||
"first_read": "2026-07-17T10:52:40.905Z"
|
||||
},
|
||||
"/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_repl_unit.c": {
|
||||
"count": 2,
|
||||
"tokens": 6234,
|
||||
"first_read": "2026-07-17T14:28:33.396Z"
|
||||
"tokens": 9226,
|
||||
"first_read": "2026-07-18T13:20:46.201Z"
|
||||
}
|
||||
},
|
||||
"files_written": [
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/scheduler/src/handler.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 214,
|
||||
"at": "2026-07-17T07:00:38.098Z"
|
||||
"tokens": 778,
|
||||
"at": "2026-07-18T11:58:57.869Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/scheduler/testsuite/test_job_array_expand.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.claude/skills/xce-test-ha/run-ha-soak.sh",
|
||||
"action": "edit",
|
||||
"tokens": 338,
|
||||
"at": "2026-07-17T07:06:24.977Z"
|
||||
"tokens": 440,
|
||||
"at": "2026-07-18T11:59:09.650Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/scheduler/testsuite/test_job_array_expand.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/netio.c",
|
||||
"action": "edit",
|
||||
"tokens": 180,
|
||||
"at": "2026-07-17T07:06:43.873Z"
|
||||
"tokens": 858,
|
||||
"at": "2026-07-18T12:23:22.256Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/scheduler/testsuite/test_job_array_expand.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/netio.c",
|
||||
"action": "edit",
|
||||
"tokens": 486,
|
||||
"at": "2026-07-17T07:07:17.840Z"
|
||||
"tokens": 125,
|
||||
"at": "2026-07-18T12:23:42.975Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/scheduler/testsuite/test_job_array_expand.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/netio.h",
|
||||
"action": "edit",
|
||||
"tokens": 127,
|
||||
"at": "2026-07-17T07:07:27.576Z"
|
||||
"tokens": 177,
|
||||
"at": "2026-07-18T12:23:56.474Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/scheduler/testsuite/test_job_array_expand.c",
|
||||
"action": "edit",
|
||||
"tokens": 146,
|
||||
"at": "2026-07-17T07:07:49.982Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/tasks.md",
|
||||
"action": "edit",
|
||||
"tokens": 30,
|
||||
"at": "2026-07-17T07:12:41.344Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/tasks.md",
|
||||
"action": "edit",
|
||||
"tokens": 22,
|
||||
"at": "2026-07-17T07:12:48.003Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/tasks.md",
|
||||
"action": "edit",
|
||||
"tokens": 458,
|
||||
"at": "2026-07-17T07:12:59.595Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/tasks.md",
|
||||
"action": "edit",
|
||||
"tokens": 374,
|
||||
"at": "2026-07-17T09:28:35.315Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/spec.json",
|
||||
"action": "edit",
|
||||
"tokens": 28,
|
||||
"at": "2026-07-17T09:45:32.590Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/spec.json",
|
||||
"action": "edit",
|
||||
"tokens": 353,
|
||||
"at": "2026-07-17T09:45:39.222Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/tasks.md",
|
||||
"action": "edit",
|
||||
"tokens": 370,
|
||||
"at": "2026-07-17T09:46:05.965Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/spec.json",
|
||||
"action": "edit",
|
||||
"tokens": 122,
|
||||
"at": "2026-07-17T10:10:09.616Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_job_array_replay_fence.c",
|
||||
"action": "edit",
|
||||
"tokens": 244,
|
||||
"at": "2026-07-17T10:21:06.304Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_job_array_replay_fence.c",
|
||||
"action": "edit",
|
||||
"tokens": 2134,
|
||||
"at": "2026-07-17T10:21:37.628Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_job_array_replay_fence.c",
|
||||
"action": "edit",
|
||||
"tokens": 50,
|
||||
"at": "2026-07-17T10:21:58.975Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/ha.c",
|
||||
"action": "edit",
|
||||
"tokens": 760,
|
||||
"at": "2026-07-17T10:26:59.162Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/spec.json",
|
||||
"action": "edit",
|
||||
"tokens": 28,
|
||||
"at": "2026-07-17T10:41:45.809Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/job-array-single-wal-record/spec.json",
|
||||
"action": "edit",
|
||||
"tokens": 102,
|
||||
"at": "2026-07-17T10:41:50.412Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/ha-term-safety/spec.json",
|
||||
"action": "edit",
|
||||
"tokens": 644,
|
||||
"at": "2026-07-17T10:52:55.892Z"
|
||||
},
|
||||
{
|
||||
"file": "/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/1c68567a-5bf5-4a4d-980c-beb4c42e613d/scratchpad/ctrlstatus_delta_check.sh",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test_netio_batch_dispatch_uaf.c",
|
||||
"action": "create",
|
||||
"tokens": 403,
|
||||
"at": "2026-07-17T10:58:29.603Z"
|
||||
"tokens": 2013,
|
||||
"at": "2026-07-18T12:24:30.071Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/controller-status/spec.json",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test_netio_batch_dispatch_uaf.c",
|
||||
"action": "edit",
|
||||
"tokens": 28,
|
||||
"at": "2026-07-17T11:01:32.385Z"
|
||||
"tokens": 9,
|
||||
"at": "2026-07-18T12:24:35.840Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/.kiro/specs/controller-status/spec.json",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/meson.build",
|
||||
"action": "edit",
|
||||
"tokens": 515,
|
||||
"at": "2026-07-17T11:01:41.240Z"
|
||||
"tokens": 54,
|
||||
"at": "2026-07-18T12:24:47.635Z"
|
||||
},
|
||||
{
|
||||
"file": "/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/1c68567a-5bf5-4a4d-980c-beb4c42e613d/scratchpad/fix_unity_testfile.py",
|
||||
"action": "create",
|
||||
"tokens": 1438,
|
||||
"at": "2026-07-17T14:12:16.754Z"
|
||||
},
|
||||
{
|
||||
"file": "/tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/1c68567a-5bf5-4a4d-980c-beb4c42e613d/scratchpad/fix_unity_testfile.py",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test.c",
|
||||
"action": "edit",
|
||||
"tokens": 301,
|
||||
"at": "2026-07-17T14:12:29.174Z"
|
||||
"tokens": 41,
|
||||
"at": "2026-07-18T12:24:53.067Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_repl_unit.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/testsuite/test.c",
|
||||
"action": "edit",
|
||||
"tokens": 31,
|
||||
"at": "2026-07-17T14:28:43.500Z"
|
||||
"tokens": 34,
|
||||
"at": "2026-07-18T12:24:57.722Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/controller/testsuite/test_repl_unit.c",
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/netio.c",
|
||||
"action": "edit",
|
||||
"tokens": 14,
|
||||
"at": "2026-07-17T14:29:03.861Z"
|
||||
"tokens": 229,
|
||||
"at": "2026-07-18T12:25:57.407Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/include/netio.h",
|
||||
"action": "edit",
|
||||
"tokens": 161,
|
||||
"at": "2026-07-18T12:26:04.775Z"
|
||||
},
|
||||
{
|
||||
"file": "/home/rshen/projects/xce/worktrees/raft_tla/src/lib/netio.c",
|
||||
"action": "edit",
|
||||
"tokens": 101,
|
||||
"at": "2026-07-18T12:26:40.965Z"
|
||||
}
|
||||
],
|
||||
"edit_counts": {
|
||||
"src/scheduler/src/handler.c": 1,
|
||||
"src/scheduler/testsuite/test_job_array_expand.c": 5,
|
||||
".kiro/specs/job-array-single-wal-record/tasks.md": 5,
|
||||
".kiro/specs/job-array-single-wal-record/spec.json": 5,
|
||||
"src/controller/testsuite/test_job_array_replay_fence.c": 3,
|
||||
"src/controller/ha.c": 1,
|
||||
".kiro/specs/ha-term-safety/spec.json": 1,
|
||||
"../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/1c68567a-5bf5-4a4d-980c-beb4c42e613d/scratchpad/ctrlstatus_delta_check.sh": 1,
|
||||
".kiro/specs/controller-status/spec.json": 2,
|
||||
"../../../../../../tmp/claude-1000/-home-rshen-projects-xce-worktrees-raft-tla/1c68567a-5bf5-4a4d-980c-beb4c42e613d/scratchpad/fix_unity_testfile.py": 2,
|
||||
"src/controller/testsuite/test_repl_unit.c": 2
|
||||
".claude/skills/xce-test-ha/run-ha-soak.sh": 2,
|
||||
"src/lib/netio.c": 4,
|
||||
"src/include/netio.h": 2,
|
||||
"src/lib/testsuite/test_netio_batch_dispatch_uaf.c": 2,
|
||||
"src/lib/meson.build": 1,
|
||||
"src/lib/testsuite/test.c": 2
|
||||
},
|
||||
"anatomy_hits": 29,
|
||||
"anatomy_misses": 8,
|
||||
"repeated_reads_warned": 36,
|
||||
"anatomy_hits": 10,
|
||||
"anatomy_misses": 1,
|
||||
"repeated_reads_warned": 10,
|
||||
"cerebrum_warnings": 0,
|
||||
"stop_count": 24
|
||||
"stop_count": 8
|
||||
}
|
||||
103
.wolf/memory.md
103
.wolf/memory.md
|
|
@ -5034,3 +5034,106 @@
|
|||
| 07:35 | Scripted fix across 83 controller/testsuite/*.c group functions | UnitySetTestFile(__FILE__) added; 19 files' nested UNITY_BEGIN/UNITY_END removed | ~2100 |
|
||||
| 07:36 | Verified: controller test suite 972 Tests 0 Failures 0 Ignored OK (single consolidated summary, matches ground-truth RUN_TEST count) | testlog.txt | ~50 |
|
||||
| 07:37 | Updated .wolf/cerebrum.md | added Unity nested-BEGIN/END masking fix entry | ~450 |
|
||||
|
||||
## Session: 2026-07-18 17:02
|
||||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
| 17:17 | Edited .kiro/specs/raft-sim-conformance/spec.json | expanded (+11 lines) | ~1609 |
|
||||
| 17:19 | Session end: 1 writes across 1 files (spec.json) | 8 reads | ~22642 tok |
|
||||
|
||||
## Session: 2026-07-18 17:27
|
||||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
| 17:28 | Edited .kiro/specs/libxce-failover/spec.json | expanded (+10 lines) | ~787 |
|
||||
| 00:35 | Wrote GO decision to spec.json (validate_impl block) | .kiro/specs/libxce-failover/spec.json | ready_for_implementation false->true, phase tasks-generated->implementation-complete | ~450 |
|
||||
| 17:29 | Session end: 1 writes across 1 files (spec.json) | 0 reads | ~787 tok |
|
||||
| 17:36 | Edited .kiro/specs/ha-entity-snapshots/spec.json | modified commit() | ~1450 |
|
||||
| 00:35 | /kiro-validate-impl ha-entity-snapshots: verified task 7.1's 07-02 blocker genuinely resolved upstream (raft-ha-core-safety 5.10a), built controller (192/192), full suite 962 PASS/0 FAIL, wrote GO decision to spec.json | .kiro/specs/ha-entity-snapshots/spec.json | ready_for_implementation stays true, phase tasks-generated->implementation-complete, cleaned up 331 session /tmp/xce_* dirs | ~14000 |
|
||||
| 17:38 | Session end: 2 writes across 1 files (spec.json) | 9 reads | ~91160 tok |
|
||||
| 00:56 | /kiro-validate-impl ha-commit-durability: build clean (0 :FAIL:/791 PASS), re-verified durable_lsn/CAS invariant intact despite heavy churn since 2026-07-02 (bug-1433/1434/1445), fresh live re-soak on ext4 (4-round rolling+cold-restart-all, 8/8 PASS 0 loss), diagnosed deterministic-mode false RED as BusyBox `date +%N` incompatibility (logged bug-1446), wrote GO to spec.json | .kiro/specs/ha-commit-durability/spec.json, .wolf/buglog.json | phase tasks-generated->implementation-complete, bug-1446 appended | ~55000 |
|
||||
| 17:54 | Session end: 2 writes across 1 files (spec.json) | 15 reads | ~121365 tok |
|
||||
|
||||
## Session: 2026-07-18 18:01
|
||||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
|
||||
## Session: 2026-07-18 18:06
|
||||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
|
||||
## Session: 2026-07-18 18:13
|
||||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
| 18:19 | Edited .kiro/specs/raft-drive-async/spec.json | added error handling | ~1860 |
|
||||
| 01:20 | /kiro-validate-impl raft-drive-async: 10-round sequential HA soak crashed round 9/10 (ASAN SEGV, ssl3_read_bytes via netio.c handle_ssl_handshake, ctrl-3 died mid reconnect-offload burst); logged bug-1447 (NOT-FIXED, blocking, LOCAL to Task 2.1/2.2 reconnect offload); wrote NO-GO to spec.json validate_impl | .kiro/specs/raft-drive-async/spec.json, .wolf/buglog.json | NO-GO | ~1200 |
|
||||
|
||||
## Session: 2026-07-18 18:21
|
||||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
| 18:35 | Edited .kiro/specs/ha-wal-replication/spec.json | added error handling | ~2214 |
|
||||
| 02:10 | /kiro-validate-impl ha-wal-replication: ASAN build clean, controller_unit_test 972/972 PASS, cross-task wiring (repl_add_standby/repl_remove_standby/WalAckReply/repl_on_ack) confirmed by code read, 10-round sequential HA soak PASS (33 jobs, zero loss, all 3 nodes led); explicitly grepped for bug-1447's SEGV signature (raft-drive-async's sibling NO-GO from same session) — did NOT reproduce; wrote GO to spec.json validate_impl, phase advanced tasks-generated → implementation-complete | .kiro/specs/ha-wal-replication/spec.json | GO | ~2000 |
|
||||
| 18:38 | Session end: 1 writes across 1 files (spec.json) | 8 reads | ~13024 tok |
|
||||
| 18:40 | Session end: 1 writes across 1 files (spec.json) | 12 reads | ~24736 tok |
|
||||
| 18:44 | Session end: 1 writes across 1 files (spec.json) | 16 reads | ~155010 tok |
|
||||
| 18:57 | Session end: 1 writes across 1 files (spec.json) | 16 reads | ~155010 tok |
|
||||
| 20:01 | Session end: 1 writes across 1 files (spec.json) | 16 reads | ~155010 tok |
|
||||
| 20:12 | Session end: 1 writes across 1 files (spec.json) | 16 reads | ~155010 tok |
|
||||
| 20:13 | Session end: 1 writes across 1 files (spec.json) | 16 reads | ~155010 tok |
|
||||
| 21:07 | Session end: 1 writes across 1 files (spec.json) | 16 reads | ~155010 tok |
|
||||
| 23:14 | Session end: 1 writes across 1 files (spec.json) | 16 reads | ~155010 tok |
|
||||
| 01:26 | Session end: 1 writes across 1 files (spec.json) | 16 reads | ~155010 tok |
|
||||
| 01:28 | Session end: 1 writes across 1 files (spec.json) | 16 reads | ~155010 tok |
|
||||
| 02:12 | Session end: 1 writes across 1 files (spec.json) | 16 reads | ~155010 tok |
|
||||
|
||||
## Session: 2026-07-18 02:19
|
||||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
| 02:28 | Edited .kiro/specs/ha-heartbeat-committed-propagation/spec.json | modified 0() | ~2754 |
|
||||
| 09:28 | /kiro-validate-impl ha-heartbeat-committed-propagation: GO. 972 unit tests pass, 10/10 sequential HA soak clean (33 jobs, zero loss, all 3 nodes led), zero bug-1447 signature | .kiro/specs/ha-heartbeat-committed-propagation/spec.json | GO | ~180 |
|
||||
| 02:30 | Session end: 1 writes across 1 files (spec.json) | 2 reads | ~7744 tok |
|
||||
| 02:43 | Created .kiro/specs/ha-prevote/spec.json | — | ~3803 |
|
||||
| 09:56 | /kiro-validate-impl ha-prevote: GO. 293 raft-lib + 972 controller unit tests pass (52 PreVote-specific), 10/10 sequential HA soak clean (33 jobs, zero loss, all 3 nodes led, live PreVote WON/DENIED/mid-round-cancel activity observed), zero bug-1447 signature at round 9 | .kiro/specs/ha-prevote/spec.json | GO | ~210 |
|
||||
| 02:45 | Session end: 2 writes across 1 files (spec.json) | 10 reads | ~145243 tok |
|
||||
|
||||
## Session: 2026-07-18 02:57
|
||||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
| 09:56 | kiro-validate-impl ha-elect-ctx-lifecycle: GO (13/13 tasks, 972/0 suite, leak-detect 5/5 detect_leaks=1, HA soak retry 10/10 clean after 1 harness-timing FAIL) | .kiro/specs/ha-elect-ctx-lifecycle/spec.json, .wolf/buglog.json (bug-1449) | GO, phase=implementation-complete | ~45k |
|
||||
|
||||
## Session: 2026-07-18 03:09
|
||||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
| 04:58 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | modified node_config_tag() | ~727 |
|
||||
| 04:59 | Edited .claude/skills/xce-test-ha/run-ha-soak.sh | expanded (+18 lines) | ~411 |
|
||||
| 12:12 | Verified bug-1449 fix: 3x back-to-back `run-ha-soak.sh --sequential 6` on ext4 (/home/claude/tmp) | run-ha-soak.sh, .wolf/buglog.json, .wolf/anatomy.md, .wolf/cerebrum.md | PASS 3/3 (6 rounds/24 jobs/zero loss/all 3 nodes led each run); config-convergence gate confirmed before every round 1; cold-start timing: election 10.3-11.7s, config-push 40-46ms after election (ample margin over ~4s min dwell); cleaned up 3 deploy dirs+logs | ~9000 |
|
||||
| 05:13 | Session end: 2 writes across 1 files (run-ha-soak.sh) | 2 reads | ~142893 tok |
|
||||
| 05:23 | Edited src/lib/netio.c | modified XCE_TESTING() | ~800 |
|
||||
| 05:23 | Edited src/lib/netio.c | modified io_worker_dispatch_batch() | ~117 |
|
||||
| 05:23 | Edited src/include/netio.h | modified XCE_TESTING() | ~165 |
|
||||
| 05:24 | Created src/lib/testsuite/test_netio_batch_dispatch_uaf.c | — | ~1879 |
|
||||
| 05:24 | Edited src/lib/testsuite/test_netio_batch_dispatch_uaf.c | 2→1 lines | ~9 |
|
||||
| 05:24 | Edited src/lib/meson.build | 2→3 lines | ~51 |
|
||||
| 05:24 | Edited src/lib/testsuite/test.c | 2→3 lines | ~39 |
|
||||
| 05:24 | Edited src/lib/testsuite/test.c | 2→3 lines | ~32 |
|
||||
| 05:25 | Edited src/lib/netio.c | modified io_worker_dispatch_batch_for_test() | ~214 |
|
||||
| 05:26 | Edited src/include/netio.h | modified in() | ~150 |
|
||||
| 05:26 | Edited src/lib/netio.c | modified io_worker_dispatch_batch() | ~95 |
|
||||
| 05:30 | Session end: 13 writes across 6 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 8 reads | ~167605 tok |
|
||||
| 06:11 | Session end: 13 writes across 6 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 8 reads | ~167605 tok |
|
||||
| 06:13 | Session end: 13 writes across 6 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 8 reads | ~167605 tok |
|
||||
| 06:19 | Session end: 13 writes across 6 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 8 reads | ~167605 tok |
|
||||
| 12:26 | bug-1447 root-caused: NOT two io_worker threads (only one exists per iomgr) — same-batch UAF: on_standby_connected's endpoint_unref on a stale duplicate peer ep frees it while a LATER event in the SAME epoll_wait() batch still points to it | src/lib/netio.c, src/controller/ha.c (read only) | root cause confirmed via code reading, not yet fixed | ~40k |
|
||||
| 12:26 | RED test established: io_worker_dispatch_batch (pure refactor, no fix yet) + io_worker_dispatch_batch_for_test seam + test_netio_batch_dispatch_uaf.c; ran standalone -> ASAN heap-use-after-free at netio.c:587, freed at netio.c:843 via injected evictor handler | src/lib/netio.c, src/include/netio.h, src/lib/testsuite/test_netio_batch_dispatch_uaf.c, src/lib/meson.build, src/lib/testsuite/test.c | RED confirmed for the right reason | ~15k |
|
||||
| 12:28 | Fix applied: endpoint_ref/endpoint_unref bracket around the whole io_worker batch dispatch (ref all before dispatch, unref all after) | src/lib/netio.c | GREEN: test passes, full lib suite 294/0/0, controller suite 972/0/0, zero ASAN hits | ~20k |
|
||||
| 13:00-13:20 | Live verification: 2x independent fresh sequential 10-round /xce-test-ha soaks on real ext4 (~/tmp), both fully clean, zero crash-signature matches | ~/tmp/xce-bug1447-verify-1 (33 jobs), ~/tmp/xce-bug1447-verify-2 (36 jobs, incl. forced ctrl-3 election) | PASS 2/2, all 3 nodes led each run | ~30k |
|
||||
| 13:24 | Updated .wolf/buglog.json bug-1447 (root_cause + fix rewritten, tags FIXED); updated .kiro/specs/raft-drive-async/spec.json validate_impl NO-GO->GO with full evidence; cleaned up both soak dirs + stray logs | .wolf/buglog.json, .kiro/specs/raft-drive-async/spec.json | bug-1447 FIXED, raft-drive-async GO | ~12k |
|
||||
| 06:24 | Session end: 13 writes across 6 files (run-ha-soak.sh, netio.c, netio.h, test_netio_batch_dispatch_uaf.c, meson.build) | 11 reads | ~190470 tok |
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
Loading…
Add table
Add a link
Reference in a new issue