feat(raft-drive-async): snapshot SlowLane offload + WAL-trim cursor guard (task 3.2)

Part B (LIVE, default-ON): trim_wal_segments now bounds WAL-segment deletion
by repl_trim_lsn(&p->repl, active_seg_lsn) = min(active segment LSN, every
active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn),
after repl_expire_disconnect_slots ages out stale slot guards. floor <=
active_seg_lsn structurally, so the guard can only preserve more segments,
never delete one a live standby still needs — closes the bug-795/828
trim-vs-cursor race for connected/recently-disconnected standbys (bug-1147
stopgap; empty/cursor=0 join still needs entity-bearing snapshots).

Part A (DORMANT, g_drive_async_snapshot_offload default OFF): ha_snapshot_send_
dispatch routes HA_REPL_SEND_SNAPSHOT inline (flag OFF — byte-for-byte legacy,
on io_worker) or to slowlane_submit+snapshot_lane_fn (flag ON). Send body
factored into ha_snapshot_do_send. Per-peer single-flight via
ha_state_t.snapshot_inflight[] CAS + slowlane key; in-flight flag cleared on
all four terminal paths, mirroring the reconnect offload (task 2.1).

Lands dormant like task 3.1: snapshot_lane_fn borrows a standby endpoint it
does not own, so flag-ON would race io_worker RDHUP teardown (cross-thread UAF,
bug-1197 family). Flag OFF keeps send+teardown serialized on the io_worker.
Task 3.4 is the single gate that flips both this and the peerctrl flag ON after
establishing endpoint-lifetime discipline; bug-1198 stays open by design until
then. The offload flag was initially authored default-ON (would ship the UAF)
and corrected to OFF in ha.c and the ha.h accessor doc (bug-1210).

Tests (controller 196/0, 0 new leaks): test_snapshot_offload_runs_on_lane,
test_snapshot_inline_when_flag_off, test_snapshot_offload_dedups_per_peer,
test_trim_guard_preserves_standby_segments,
test_trim_without_standbys_proceeds_normally. Adversarial cpp-review: GO.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
This commit is contained in:
Rongsong Shen 2026-06-26 15:43:24 +00:00
parent e17a6a613c
commit 6c0523b69b
8 changed files with 2046 additions and 850 deletions

View file

@ -63,14 +63,14 @@ Tasks mirror the design's 4-phase migration. The slow lane is foundational (ever
- _Boundary: ReconnectOffload, SlowLane_
- _Depends: 1.2, 1.3_
- [ ] 2.2 Complete the reconnect on the lane worker
- [x] 2.2 Complete the reconnect on the lane worker
- The lane worker performs the bounded connect, fully prepares the endpoint, and registers it via the lock-guarded monitor interface; on failure it clears the in-flight flag and lets the next tick retry without terminating; resolve the inbound-vs-outbound connection race per the connection-ownership rules.
- Observable: a successful reconnect registers the new connection with no lost or reordered queued messages; a failed attempt leaves the controller alive and retrying.
- _Requirements: 2.4, 2.6, 4.4_
- _Boundary: ReconnectOffload_
- _Depends: 2.1_
- [ ] 2.3 Reconnect-offload integration test
- [x] 2.3 Reconnect-offload integration test
- With one peer made unreachable, assert heartbeats to the surviving peers continue within the interval, the delaymgr is not stalled, and reconnect is deduped per peer.
- Observable: test passes; reverting the offload (inline connect) makes heartbeats to survivors stall.
- _Requirements: 2.1, 2.2, 2.5_
@ -87,7 +87,7 @@ Tasks mirror the design's 4-phase migration. The slow lane is foundational (ever
- _Boundary: PeerCtrlDispatch_
- _Depends: 1.2, 1.6_
- [ ] 3.2 Offload snapshot send to the slow lane
- [~] 3.2 Offload snapshot send to the slow lane <!-- Part B (trim guard) LIVE; Part A (offload) DORMANT pending 3.4 flip — see Implementation Notes -->
- Trigger snapshot send + checkpoint to the lane from the vtask entry without holding the per-endpoint block across the transfer; dedup one snapshot per standby.
- Enforce the snapshot/checkpoint concurrency invariant: snapshot reads a frozen term/lsn boundary; segment trim never deletes at/above any in-flight snapshot boundary or a standby catch-up cursor; at most one checkpoint in flight; bound the in-lane snapshot socket write so an unreachable-mid-transfer peer cannot pin a worker.
- **Add the trim cursor-bound guard (bug-1147, stopgap):** `trim_wal_segments` (`persist.c:1374`) currently bounds deletion ONLY by the open segment and consults no standby cursor; before this task moves checkpoint/trim onto a concurrent lane worker, change it to compute `floor = min(all standby acked_lsn, in-flight snapshot boundaries)` under `p->lock` and refuse to delete at/above `floor`. This is strategy-A retention discipline; it does NOT fix empty/far-behind join (cursor=0) — that needs entity-bearing snapshots (separate spec).
@ -139,3 +139,12 @@ Tasks mirror the design's 4-phase migration. The slow lane is foundational (ever
- **1.1**: `drive_async_liveness_reserved(n_peers, n_vcpus) = max(1, min(n_peers−1, n_vcpus−1))` in `ha.c` is the single source of truth for `VTASK_LIVENESS_RESERVED` (floor applied LAST). Consumed by 1.6. `myconf_t` CLI override fields for the new `DEFAULT_*` constants were deferred to the consumer tasks (1.2 connect-timeout, 1.3/1.6) per YAGNI — add each field with its reader.
- **2.1**: reconnect offload is gated by a default-ON kill-switch `g_drive_async_reconnect_offload` (legacy inline path retained for the TDD-RED proof + operational fallback). Per-peer single-flight via `ha_state_t.reconnect_inflight[]` CAS + slowlane key; flag cleared on all 4 terminal paths (prep-fail, lane-fn success/-EAGAIN, -EEXIST, hard-error). Lane fn = `reconnect_lane_fn` → `ha_connect_finish_peer` (happy path: bounded connect + lock-guarded iomgr_monitor; 2.2 adds inbound/outbound race + failure hardening via the documented seam). **FOLLOW-UP (test hygiene, LOW)**: the GREEN reconnect tests in test_ha_liveness.c leak the prepped endpoint on the simulated-success hook path (production-clean; masked by detect_leaks=0) — free the un-adopted ep in teardown when convenient.
- **1.6**: pool reservation lives in vtaskmgr (`vtask_class_t` + `blocking_inflight`/`liveness_reserved` + `vtaskmgr_blocking_acquire/release`); commit folds in via opt-in `commit_set_pool` (kept `commit_engine_init(n_vcpus)` signature — the `n_vcpus` local cap is the genuine pool==NULL fallback for unit fixtures). **Production wiring is in `cluster.c` phase2** (once-guarded, right after `commit_engine_init`): `vtaskmgr_set_liveness_reserved(R)` + `commit_set_pool` — DO NOT remove; without it the reservation is dead. Lock order `p->lock → ce->lock → blk_lock` (cycle-free). New-cap = `n_vcpus−R` (tighter than old `n_vcpus−2` for 5/7-node — intended). LOW deferred: `vtaskmgr_blocking_cap()` reads `liveness_reserved` without `blk_lock` (init-time set-once, benign). When 3.1 dispatches peer_ctrl, REQ_BLOCK slot is released in `run_task` — ensure no double-count vs the commit path.
- **3.1 (landed DORMANT — checkbox stays unchecked until 3.4 gate is green)**: `peer_ctrl_msg_handler` is now a shim gated by `g_drive_async_peerctrl_dispatch_enabled` (**DEFAULT 0 / OFF** — production behavior UNCHANGED; toggle via `ha_set_peerctrl_dispatch_for_test`). Flag OFF → `peer_ctrl_msg_handler_inline` runs on io_worker, byte-for-byte prior path. Flag ON → `peer_ctrl_dispatch_to_pool` builds a `VTASK_IO` task and `vtask_enq_classified`s it (per-endpoint order auto-enforced by `bind_vtask`→`endpoint_block_tasks`). The 3 routers (`vote_reply`/`pre_vote_reply`/`repl_ack`_decode_and_dispatch) take `pkt` as an explicit param now (no cross-thread `ep->fsm_in.pkt` re-read). **Packet ownership**: shim clears `ep->fsm_in.pkt` immediately (sole-owner transfer); `peer_ctrl_vtask_entry` NULLs `iod->pkt` before the body frees it once; new `vtask_t.owns_io_pkt` (default 0; controller sets 1) lets `taskq_item_killer` free an un-executed queued task's packet without double-freeing the libxce endpoint-owned path (bug-1199). `ha_state_t.pool` wired in `cluster.c` to `&c->vtaskmgr`.
- **TWO FLAG-ON GATE ITEMS — MUST be resolved before 3.4 flips the flag ON** (documented with loud code comments; harmless while flag OFF):
1. **bug-1197 (CRITICAL UAF)**: `raft_cb_become_follower` `endpoint_fini()`s peer vote-eps from a pool worker → frees `ep->data` while the single io_worker can concurrently read `ep_j->data` in `peer_ctrl_msg_handler_inline` for a same-epoll-batch queued event. FIX (part of 3.4 enablement): `iomgr_unmonitor(ep)` BEFORE `endpoint_fini(ep)` in every pool-reachable teardown.
2. **bug-1198 (HIGH classification)**: `peer_ctrl_classify` returns `VTASK_LIVENESS` for all, but the CTRL_PRIMARY repl-ack path can run a blocking `HA_REPL_SEND_SNAPSHOT`→`persist_checkpoint` inline under the never-gated liveness reservation → breaks deadlock-freedom. The 3.2 SlowLane offload mechanism now EXISTS but is DORMANT (flag OFF, below), so bug-1198 stays OPEN until 3.4 flips it ON together with the peerctrl flag.
- **3.2 — Part B (trim guard, bug-1147) is LIVE; Part A (snapshot offload) landed DORMANT (checkbox `[~]` until 3.4 gate)**:
- **Part B (production-live, default-ON, no flag)**: `trim_wal_segments` (`persist.c`) now bounds deletion by `floor = repl_trim_lsn(&p->repl, active_seg_lsn, snap_interval_s)` = `min(active_seg_lsn, every active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn)`, after a `repl_expire_disconnect_slots` call to age out stale slot guards. `floor ≤ active_seg_lsn` structurally (repl_trim_lsn caps at its `checkpoint_lsn` arg), so the new `seg_start_lsn < floor` check subsumes the old `< active_seg_lsn` guard and can only PRESERVE more segments — never delete one a live standby still needs (closes the bug-795/828 trim-vs-cursor race for connected/recently-disconnected standbys; does NOT fix empty/cursor=0 join — needs entity-bearing snapshots, separate spec). Lock-safe: `trim_wal_segments` runs with `p->lock` released; `repl_trim_lsn`/`repl_expire_disconnect_slots` take only `r->lock` — no inversion. Tests: `test_trim_guard_preserves_standby_segments`, `test_trim_without_standbys_proceeds_normally` (test_e2e_checkpoint.c).
- **Part A (DORMANT, `g_drive_async_snapshot_offload` DEFAULT 0/OFF)**: `ha_snapshot_send_dispatch` routes `HA_REPL_SEND_SNAPSHOT` inline (flag OFF — byte-for-byte legacy, send on io_worker) or to `slowlane_submit`+`snapshot_lane_fn` (flag ON). Send body factored into `ha_snapshot_do_send` (shared by both paths, holds the XCE_TESTING hook). Per-peer single-flight via `ha_state_t.snapshot_inflight[MAX_HA_CANDIDATES]` CAS + slowlane key `peer_idx+1`; flag cleared on all 4 terminal paths (alloc-fail, -EEXIST, hard-error, lane-fn completion) exactly like reconnect (2.1). Tests: `test_snapshot_offload_runs_on_lane`, `test_snapshot_inline_when_flag_off`, `test_snapshot_offload_dedups_per_peer` (test_ha_liveness.c).
- **WHY DORMANT (flag-ON gate, owned by 3.4)**: `snapshot_lane_fn` captures a BORROWED standby endpoint (unlike reconnect, which owns its ep). With the flag ON the send runs on a lane thread while the io_worker may RDHUP-`endpoint_fini` that ep → cross-thread UAF (bug-1197 family). Flag OFF keeps send+teardown serialized on the io_worker. **3.4 is the single gate** that flips BOTH `g_drive_async_snapshot_offload` AND `g_drive_async_peerctrl_dispatch` ON, after establishing the endpoint-lifetime discipline (refcount or lane-serialized teardown) and proving liveness under pool saturation. Caught + fixed pre-commit: the offload flag was initially authored DEFAULT-ON (would have shipped the UAF) → corrected to OFF in both ha.c and the ha.h accessor doc (bug-1210).

File diff suppressed because it is too large Load diff

View file

@ -3076,3 +3076,399 @@
| 21:35 | Edited src/lib/raft/raft_sim.c | modified switch() | ~231 |
| 21:36 | Edited src/lib/testsuite/test_raft.c | modified for() | ~387 |
| 21:39 | Session end: 191 writes across 10 files (raft.h, test_raft.c, tasks.md, raft_sim.c, raft.c) | 28 reads | ~489546 tok |
## Session: 2026-06-25 23:39
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 23:56 | Created .kiro/specs/raft-apply-decoupling/brief.md | — | ~2139 |
| 23:57 | Created .kiro/specs/job-array-single-wal-record/brief.md | — | ~1330 |
| 23:57 | Edited .kiro/steering/roadmap.md | modified Decision() | ~1162 |
| 23:57 | Edited .kiro/steering/roadmap.md | inline fix | ~117 |
| 06:57 | /kiro-discovery: raft apply-ordering & hook decoupling (Path E) — 2 new spec briefs + roadmap phase | .kiro/specs/raft-apply-decoupling/brief.md, .kiro/specs/job-array-single-wal-record/brief.md, .kiro/steering/roadmap.md | done | ~6k |
| 23:58 | Session end: 4 writes across 2 files (brief.md, roadmap.md) | 8 reads | ~148765 tok |
| 00:06 | Session end: 4 writes across 2 files (brief.md, roadmap.md) | 8 reads | ~148765 tok |
| 00:09 | Created .kiro/specs/raft-apply-decoupling/facts-in-raft.md | — | ~1735 |
| 00:12 | Edited .kiro/specs/raft-log-replication/spec.json | 3→3 lines | ~25 |
| 00:12 | Edited .kiro/specs/raft-log-replication/spec.json | 1→2 lines | ~72 |
| 00:12 | Edited .kiro/specs/raft-commit-safety/spec.json | 3→3 lines | ~25 |
| 00:12 | Edited .kiro/specs/raft-commit-safety/spec.json | 1→2 lines | ~77 |
| 00:13 | Edited .kiro/specs/ha-consistency-invariants/requirements.md | modified deviation() | ~457 |
| 00:13 | Edited .kiro/specs/raft-apply-decoupling/facts-in-raft.md | expanded (+14 lines) | ~493 |
| 00:13 | Edited .kiro/specs/raft-apply-decoupling/brief.md | inline fix | ~224 |
| 07:10 | suggestion #1: reconciled raft-log-replication + raft-commit-safety spec.json phase tasks-generated→implemented (24/24,18/18 done) | .kiro/specs/raft-*/spec.json | done | ~1k |
| 07:11 | suggestion #3: Facts-Apply invariant → criterion 2.9 in ha-consistency-invariants; documented RETAIN apply-on-receipt deviation; reconcile owner=raft-apply-decoupling | requirements.md, facts-in-raft.md, brief.md | done | ~3k |
| 00:14 | Session end: 12 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 11 reads | ~156291 tok |
| 00:18 | Session end: 12 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 11 reads | ~156291 tok |
| 00:30 | Session end: 12 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 11 reads | ~156291 tok |
| 00:35 | Session end: 12 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 11 reads | ~156291 tok |
| 00:41 | Session end: 12 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 11 reads | ~156291 tok |
| 00:48 | Session end: 12 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 11 reads | ~156291 tok |
| 01:01 | Session end: 12 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 11 reads | ~156291 tok |
| 01:11 | Session end: 12 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 11 reads | ~156291 tok |
| 01:17 | Edited .kiro/specs/raft-apply-decoupling/facts-in-raft.md | inline fix | ~70 |
| 01:18 | Edited .kiro/specs/raft-apply-decoupling/facts-in-raft.md | 4→2 lines | ~304 |
| 01:18 | Edited .kiro/specs/raft-apply-decoupling/facts-in-raft.md | modified ownership() | ~1363 |
| 01:19 | Edited .kiro/specs/raft-apply-decoupling/facts-in-raft.md | inline fix | ~54 |
| 07:35 | facts-through-raft design converged: D1 resolved (block-but-durable); risk register A–G (A solved by resync, B open, C reclassified to request-handling correctness) recorded | .kiro/specs/raft-apply-decoupling/facts-in-raft.md | done | ~4k |
| 01:19 | Session end: 16 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 12 reads | ~160289 tok |
| 01:52 | Session end: 16 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 29 reads | ~240766 tok |
| 02:06 | Session end: 16 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 29 reads | ~240766 tok |
| 02:16 | Session end: 16 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 29 reads | ~240766 tok |
| 02:26 | Edited .kiro/specs/raft-apply-decoupling/facts-in-raft.md | expanded (+14 lines) | ~600 |
| 02:27 | Edited .kiro/specs/raft-apply-decoupling/brief.md | modified REQUIREMENT() | ~187 |
| 03:10 | Edited .kiro/specs/raft-apply-decoupling/facts-in-raft.md | modified CRITICAL() | ~1020 |
| 03:10 | Edited .kiro/steering/roadmap.md | modified work() | ~498 |
| 08:05 | blocking-audit: protocol drive NOT fully event-driven. 3 CRITICAL iomgr-blockers (snapshot send ha.c:2702, config-push wait config_repl.c:1332, DNS+connect netio.c:289 on delaymgr) beyond known commit barrier. Split into P1 (commit-barrier async, in raft-apply-decoupling) + P2 (drive hardening, recommend new spec raft-drive-async) | facts-in-raft.md, roadmap.md | done | ~5k |
| 03:10 | Session end: 20 writes across 5 files (brief.md, roadmap.md, facts-in-raft.md, spec.json, requirements.md) | 33 reads | ~256515 tok |
## Session: 2026-06-25 03:17
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 03:33 | Edited .kiro/specs/raft-apply-decoupling/facts-in-raft.md | modified completeness() | ~985 |
| 08:40 | P2 refined: delaymgr/iomgr "trigger-don't-execute" offload; reuse COMMIT_WAIT_RESERVED admission cap (commit.c:195); API gap = need general non-IO vtask submit (vtask_handle_msg is IO-bound); reconnect→capped vtask, snapshot/config-push→slow-lane; heartbeat safe only when BOTH delaymgr+iomgr non-blocking; need partition (unreachable not killed) test | facts-in-raft.md | done | ~3k |
| 03:33 | Session end: 1 writes across 1 files (facts-in-raft.md) | 1 reads | ~5841 tok |
| 03:33 | Session end: 1 writes across 1 files (facts-in-raft.md) | 1 reads | ~5841 tok |
| 03:39 | Session end: 1 writes across 1 files (facts-in-raft.md) | 1 reads | ~5841 tok |
| 03:47 | Session end: 1 writes across 1 files (facts-in-raft.md) | 1 reads | ~5841 tok |
| 03:57 | Session end: 1 writes across 1 files (facts-in-raft.md) | 1 reads | ~5841 tok |
| 04:01 | Edited .kiro/steering/ha-async-io.md | modified SUPERSEDES() | ~792 |
| 09:05 | STEERING CHANGE: ha-async-io.md thread-role contract rewritten to 4-role model (iomgr non-blocking+dispatch / vtask bounded-blocking-under-reservation / delaymgr trigger-only-NEVER-block / slow-lane for long IO). SUPERSEDES old "delaymgr: connect/file IO Permitted" — root cause of P2 blocking findings (delaymgr is BOTH heartbeat-driver AND blocking-sink = conflict). Applies to all components. | .kiro/steering/ha-async-io.md | done | ~2k |
| 04:01 | Session end: 2 writes across 2 files (facts-in-raft.md, ha-async-io.md) | 2 reads | ~9376 tok |
| 04:36 | Created .kiro/specs/raft-drive-async/brief.md | — | ~2576 |
| 04:36 | Edited .kiro/steering/roadmap.md | modified prerequisites() | ~1026 |
| 09:25 | created raft-drive-async brief (P2 promoted, START FIRST); restructured roadmap "Raft Drive & Apply Async family" dependency order: raft-drive-async → raft-apply-decoupling(+P1 keystone) → worker-rejoin-resync(HARD for facts) → job-array-single-wal-record; worker-rejoin-resync elevated to HARD prereq | raft-drive-async/brief.md, roadmap.md | done | ~4k |
| 04:37 | Created .kiro/specs/raft-drive-async/spec.json | — | ~129 |
| 04:37 | Created .kiro/specs/raft-drive-async/requirements.md | — | ~736 |
| 09:40 | /kiro-spec-init raft-drive-async: spec.json + requirements.md scaffolded (phase=initialized); ready for /kiro-spec-requirements | .kiro/specs/raft-drive-async/ | done | ~1k |
| 04:38 | Session end: 6 writes across 6 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 3 reads | ~21521 tok |
| 05:01 | Created .kiro/specs/raft-drive-async/requirements.md | — | ~3304 |
| 05:02 | Edited .kiro/specs/raft-drive-async/spec.json | 8→8 lines | ~55 |
| 09:50 | /kiro-spec-requirements raft-drive-async: 8 reqs generated (liveness during long IO, unreachable-peer/bounded-connect, admission protection, ordering correctness, thread-role compliance, NO-REGRESSION naming existing HA suites, network-partition verification, heartbeat perf). phase=requirements-generated. Confirmed NO partition test exists today (kill-only soak). | requirements.md, spec.json | done | ~3k |
| 05:02 | Session end: 8 writes across 6 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 4 reads | ~25245 tok |
| 06:36 | Created .kiro/specs/raft-drive-async/research.md | — | ~2830 |
| 10:05 | /kiro-validate-gap raft-drive-async: research.md written. KEY GAP: existing deferral idiom delay_task_ms runs on delaymgr (heartbeat thread) — codebase can defer-off-iomgr but only target is a liveness thread; missing a non-liveness slow lane. Building blocks exist (vtask_alloc/init/enq, fsync_worker+push_thread precedent, xce_probe_config_tag bounded-connect). Truly missing: slow-lane executor + iomgr-post(eventfd) marshal-back + partition test. Recommend Option C hybrid, phased (Ph1 bounded-connect first). | research.md | done | ~4k |
| 06:36 | Session end: 9 writes across 7 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 4 reads | ~28277 tok |
| 06:49 | Edited .kiro/specs/raft-drive-async/research.md | inline fix | ~13 |
| 06:49 | Session end: 10 writes across 7 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 4 reads | ~28291 tok |
| 06:56 | Session end: 10 writes across 7 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 4 reads | ~28291 tok |
| 07:02 | Edited .kiro/specs/raft-drive-async/research.md | "VTASK_IO" → "endpoint_block_tasks()" | ~251 |
| 07:03 | Edited .kiro/specs/raft-drive-async/requirements.md | 4→6 lines | ~405 |
| 07:03 | Edited .kiro/specs/raft-drive-async/research.md | "VTASK_IO" → "endpoint_block_tasks" | ~84 |
| 10:20 | R4 de-risked: endpoint_block_tasks/unblock_tasks (netio.c:896) is the explicit per-endpoint sequential primitive (auto on VTASK_IO path) — R4.1 = reuse not new logic. Added R4.2 (sequential) + R4.3 (don't hold block across long I/O) to requirements; clarified inbound-process vs outbound-send order; updated research.md asset map + design trap. | requirements.md, research.md, cerebrum.md | done | ~2k |
| 07:03 | Session end: 13 writes across 7 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 6 reads | ~38869 tok |
| 07:08 | Edited .kiro/specs/raft-drive-async/research.md | "iomgr_post" → "iomgr_monitor" | ~268 |
| 07:08 | Edited .kiro/specs/raft-drive-async/research.md | inline fix | ~88 |
| 07:08 | Edited .kiro/specs/raft-drive-async/research.md | 6→6 lines | ~182 |
| 07:08 | Edited .kiro/specs/raft-drive-async/research.md | inline fix | ~87 |
| 07:09 | Edited .kiro/specs/raft-drive-async/research.md | inline fix | ~45 |
| 07:09 | Edited .kiro/specs/raft-drive-async/research.md | primitive() → iomgr() | ~184 |
| 07:09 | Edited .kiro/specs/raft-drive-async/research.md | "iomgr-post" → "iomgr_monitor" | ~84 |
| 07:09 | Edited .kiro/specs/raft-drive-async/research.md | inline fix | ~87 |
| 07:09 | Edited .kiro/specs/raft-drive-async/requirements.md | inline fix | ~62 |
| 10:35 | CORRECTED earlier overstatement: NO new iomgr-post/eventfd primitive needed. epoll registration already thread-safe via lock-guarded iomgr_monitor/iomgr_unmonitor (netio.c:1046-1082, mylock iomgr->lock); connect→register already runs from delaymgr/main today. Worker prepares endpoint then calls iomgr_monitor. Removed the Medium-High risk item; overall risk L→Medium; the one real gap is now just the slow-lane executor. Updated research.md (all spots) + requirements R4.4. | research.md, requirements.md | done | ~2k |
| 07:10 | Session end: 22 writes across 7 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 7 reads | ~43065 tok |
| 08:12 | Edited .kiro/specs/raft-drive-async/research.md | inline fix | ~187 |
| 08:12 | Edited .kiro/specs/raft-drive-async/research.md | inline fix | ~64 |
| 08:12 | Edited .kiro/specs/raft-drive-async/brief.md | inline fix | ~90 |
| 08:12 | Edited .kiro/specs/raft-drive-async/brief.md | inline fix | ~82 |
| 08:12 | Edited .kiro/specs/raft-drive-async/requirements.md | inline fix | ~78 |
| 08:13 | Edited .kiro/steering/roadmap.md | inline fix | ~120 |
| 10:55 | election-iomgr-refactor scope-check: verdict ADJACENT-SAME-AREA → raft-drive-async stays STANDALONE. e-i-r is fully implemented, fixed STARTUP blocking (pthread election pre-cluster_run), provides reusable primitives (delay_task_ms, on_connected, ha_start_election_async/non-blocking vote fan-out); touched NONE of raft-drive-async's runtime targets; designed under OLD thread-role rule. Resolved fold-in question across research.md/brief.md/requirements.md/roadmap. raft-drive-async ready for /kiro-spec-design. | research.md, brief.md, requirements.md, roadmap.md | done | ~3k |
| 08:13 | Session end: 28 writes across 7 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 12 reads | ~59606 tok |
| 08:18 | Created .kiro/specs/raft-drive-async/design.md | — | ~5952 |
| 11:20 | /kiro-spec-design raft-drive-async: design.md written (brownfield extension). Key synthesis: dedicated slow-lane executor (separate threads) as offload target → R3 structural + "general non-IO vtask submit" SUPERSEDED (not needed) → only genuinely-new component = slowlane.{c,h}. 4-phase migration. phase=design-generated. | design.md, research.md, spec.json | done | ~6k |
| 08:18 | Session end: 29 writes across 8 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 12 reads | ~65983 tok |
| 08:21 | Session end: 29 writes across 8 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 12 reads | ~65983 tok |
| 16:37 | Edited .kiro/specs/raft-drive-async/design.md | expanded (+7 lines) | ~471 |
| 16:37 | Edited .kiro/specs/raft-drive-async/design.md | inline fix | ~199 |
| 16:37 | Edited .kiro/specs/raft-drive-async/design.md | 1→2 lines | ~118 |
| 16:38 | Edited .kiro/specs/raft-drive-async/design.md | modified preservation() | ~227 |
| 16:38 | Edited .kiro/specs/raft-drive-async/design.md | modified load() | ~129 |
| 11:40 | /kiro-validate-design raft-drive-async: GO (must-resolve-before-Phase-3). Applied 3 design fixes: (1) snapshot/checkpoint concurrency invariant (frozen (term,lsn) boundary, trim-safety vs catch-up cursor, single in-flight checkpoint); (2) disambiguated 2 config-push paths (iomgr transport-only IN scope; admin OP_upgradeConfig w/ waiting requester OUT); (3) commit-advance preservation invariant (ack handling stays fast, cross-endpoint independent, P1 owns barrier async) + 2 new tests. | design.md | done | ~3k |
| 16:38 | Session end: 34 writes across 8 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 12 reads | ~67208 tok |
| 16:41 | Session end: 34 writes across 8 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 12 reads | ~67208 tok |
| 16:43 | Edited .kiro/specs/raft-drive-async/design.md | inline fix | ~142 |
| 16:43 | Edited .kiro/specs/raft-drive-async/design.md | 12→14 lines | ~524 |
| 11:55 | re-validate raft-drive-async: fixed phase-sequencing (NEW issue) — slow lane is foundational (reconnect offload also submits to it), reordered P1=SlowLane+BoundedConnect, P2=ReconnectOffload, P3=peer_ctrl→vtask COUPLED with long-op offload (prevents transient R4.3 ep->busy-across-long-IO), P4=partition. Also bounded in-lane snapshot write (lane-saturation guard). | design.md | done | ~2k |
| 16:44 | Session end: 36 writes across 8 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 13 reads | ~74408 tok |
| 17:02 | Created .kiro/specs/raft-drive-async/tasks.md | — | ~2346 |
| 12:15 | /kiro-spec-tasks raft-drive-async: tasks.md written, 4 major / 14 sub-tasks mirroring 4-phase migration (P1 lane+bounded-connect, P2 reconnect-offload, P3 peer_ctrl→vtask+offload coupled, P4 partition+regression). All 8 reqs (1.1-8.2) covered. Independent sanity review = PASS. phase=tasks-generated, awaiting approval. | tasks.md, spec.json | done | ~4k |
| 17:03 | Session end: 37 writes across 9 files (facts-in-raft.md, ha-async-io.md, brief.md, roadmap.md, spec.json) | 14 reads | ~79445 tok |
## Session: 2026-06-26 19:36
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-26 19:57
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-26 20:05
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-26 20:20
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 20:23 | Created ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/project_xce_system_map.md | — | ~954 |
| 20:23 | Edited ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/MEMORY.md | 3→4 lines | ~69 |
| 20:24 | Session end: 2 writes across 2 files (project_xce_system_map.md, MEMORY.md) | 3 reads | ~96344 tok |
| 20:35 | Session end: 2 writes across 2 files (project_xce_system_map.md, MEMORY.md) | 10 reads | ~113110 tok |
| 20:39 | Session end: 2 writes across 2 files (project_xce_system_map.md, MEMORY.md) | 10 reads | ~113110 tok |
| 20:41 | Session end: 2 writes across 2 files (project_xce_system_map.md, MEMORY.md) | 10 reads | ~113110 tok |
| 20:43 | Session end: 2 writes across 2 files (project_xce_system_map.md, MEMORY.md) | 10 reads | ~113110 tok |
| 20:45 | Session end: 2 writes across 2 files (project_xce_system_map.md, MEMORY.md) | 11 reads | ~146282 tok |
| 20:46 | Edited .kiro/specs/raft-drive-async/design.md | inline fix | ~324 |
| 20:47 | Edited .kiro/specs/raft-drive-async/tasks.md | modified guard() | ~352 |
| 20:47 | Edited .kiro/specs/raft-drive-async/design.md | modified preservation() | ~670 |
| 20:47 | Edited .kiro/specs/raft-drive-async/design.md | modified environment() | ~333 |
| 20:48 | Edited .kiro/specs/raft-drive-async/tasks.md | modified gate() | ~219 |
| 20:48 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~134 |
| 20:48 | Edited .kiro/steering/roadmap.md | inline fix | ~390 |
| 20:49 | Edited .kiro/steering/roadmap.md | added 1 condition(s) | ~362 |
| 20:49 | Edited .kiro/steering/roadmap.md | inline fix | ~275 |
| 03:49 | validate-design raft-drive-async: GO + 3 issues folded (deadlock arg, trim-guard bug-1147, in-proc partition); roadmap+buglog updated | design.md tasks.md roadmap.md buglog.json | done | ~4k |
| 20:49 | Session end: 11 writes across 5 files (project_xce_system_map.md, MEMORY.md, design.md, tasks.md, roadmap.md) | 12 reads | ~157275 tok |
| 20:52 | Session end: 11 writes across 5 files (project_xce_system_map.md, MEMORY.md, design.md, tasks.md, roadmap.md) | 12 reads | ~157275 tok |
| 21:10 | Session end: 11 writes across 5 files (project_xce_system_map.md, MEMORY.md, design.md, tasks.md, roadmap.md) | 13 reads | ~157275 tok |
| 21:31 | Edited .kiro/specs/raft-drive-async/design.md | 2→3 lines | ~114 |
| 21:31 | Edited .kiro/specs/raft-drive-async/design.md | modified enum() | ~1106 |
| 21:31 | Edited .kiro/specs/raft-drive-async/design.md | "commit_submit" → "commit.c" | ~450 |
| 21:32 | Edited .kiro/specs/raft-drive-async/design.md | 2→4 lines | ~176 |
| 21:32 | Edited .kiro/specs/raft-drive-async/design.md | inline fix | ~172 |
| 21:32 | Edited .kiro/specs/raft-drive-async/design.md | inline fix | ~36 |
| 21:32 | Edited .kiro/specs/raft-drive-async/design.md | inline fix | ~34 |
| 21:32 | Edited .kiro/specs/raft-drive-async/tasks.md | expanded (+15 lines) | ~484 |
| 21:32 | Edited .kiro/specs/raft-drive-async/tasks.md | 3→3 lines | ~32 |
| 21:33 | Edited .kiro/steering/roadmap.md | inline fix | ~196 |
| 04:33 | folded Option A (LivenessAdmission) into raft-drive-async: pool-level vtask task-class reservation, new task 1.6, 3.1 depends on it; deepens Issue-1 deadlock fix | design.md tasks.md roadmap.md | done | ~5k |
| 21:33 | Session end: 21 writes across 5 files (project_xce_system_map.md, MEMORY.md, design.md, tasks.md, roadmap.md) | 13 reads | ~160275 tok |
| 21:36 | Edited .kiro/specs/raft-drive-async/design.md | "VTASK_LIVENESS_RESERVED" → "R = VTASK_LIVENESS_RESERV" | ~293 |
| 21:37 | Edited .kiro/specs/raft-drive-async/tasks.md | 4→5 lines | ~170 |
| 21:37 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~92 |
| 21:37 | Session end: 24 writes across 5 files (project_xce_system_map.md, MEMORY.md, design.md, tasks.md, roadmap.md) | 13 reads | ~160870 tok |
| 21:40 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~184 |
| 21:41 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~10 |
| 21:42 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~112 |
| 04:43 | kiro-spec-tasks raft-drive-async: reviewed (fixed 3.1 stale bullet, 1.3 (P), 1.1/1.6 dup), sanity PASS, 17 sub-tasks APPROVED | tasks.md spec.json | done | ~3k |
| 21:43 | Session end: 27 writes across 5 files (project_xce_system_map.md, MEMORY.md, design.md, tasks.md, roadmap.md) | 15 reads | ~167987 tok |
| 21:47 | Created src/controller/testsuite/test_drive_async_config.c | — | ~990 |
| 21:47 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~30 |
| 21:47 | Edited src/controller/testsuite/controller_test.c | 4→5 lines | ~30 |
| 21:47 | Edited src/controller/meson.build | 2→3 lines | ~30 |
| 21:48 | Edited src/include/config.h | expanded (+21 lines) | ~334 |
| 21:48 | Edited src/controller/ha.h | modified invariant() | ~451 |
| 21:49 | Edited src/controller/ha.c | added 1 condition(s) | ~279 |
| 21:49 | Edited src/controller/cluster.c | added 1 condition(s) | ~415 |
| 21:50 | raft-drive-async task 1.1: drive-async config constants + pure liveness-reservation clamp helper (RED→GREEN, 8 unit tests) | config.h, ha.h, ha.c, cluster.c, testsuite/test_drive_async_config.c, controller_test.c, meson.build | 8/8 PASS; commit_engine+ha_cluster regression clean | ~12k |
| 21:57 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~14 |
| 21:57 | Edited .kiro/specs/raft-drive-async/tasks.md | expanded (+6 lines) | ~367 |
| 22:01 | Created src/controller/include/slowlane.h | — | ~1021 |
| 22:01 | Edited src/controller/include/slowlane.h | modified Returns() | ~203 |
| 22:01 | Edited src/controller/include/slowlane.h | 3→4 lines | ~31 |
| 22:02 | Created src/controller/testsuite/test_slowlane.c | — | ~2993 |
| 22:02 | Edited src/controller/meson.build | 2→3 lines | ~27 |
| 22:02 | Edited src/controller/meson.build | 5→6 lines | ~18 |
| 22:02 | Edited src/controller/meson.build | 6→7 lines | ~25 |
| 22:03 | Edited src/controller/testsuite/controller_test.c | 1→2 lines | ~20 |
| 22:03 | Edited src/controller/testsuite/controller_test.c | 3→4 lines | ~23 |
| 22:03 | Created src/controller/slowlane.c | — | ~134 |
| 22:05 | Created src/controller/slowlane.c | — | ~2238 |
| 22:06 | Edited src/controller/slowlane.c | 7→5 lines | ~22 |
| 22:06 | Edited src/include/xce/cluster_def.h | 3→4 lines | ~38 |
| 22:06 | Edited src/include/xce/cluster_def.h | 1→2 lines | ~49 |
| 22:06 | Edited src/controller/cluster.c | added 2 condition(s) | ~316 |
| 22:06 | Edited src/controller/cluster.c | expanded (+6 lines) | ~143 |
| 05:09 | task 1.2 SlowLane executor: new slowlane.c/.h + test_slowlane.c (7 tests RED->GREEN), wired init/fini into cluster.c lifecycle + cluster_def.h field | src/controller/slowlane.c, src/controller/include/slowlane.h, src/controller/testsuite/test_slowlane.c, src/controller/cluster.c, src/include/xce/cluster_def.h, src/controller/meson.build, src/controller/testsuite/controller_test.c | GREEN 7/7, full suite 194 pass | ~40k |
| 22:23 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~11 |
| 23:50 | Created src/lib/testsuite/test_netio_bounded_connect.c | — | ~1723 |
| 23:50 | Edited src/lib/testsuite/test.c | 1→2 lines | ~25 |
| 23:51 | Edited src/lib/testsuite/test.c | 2→3 lines | ~32 |
| 23:51 | Edited src/lib/meson.build | 1→2 lines | ~33 |
| 23:51 | Edited src/lib/netio.c | 4→5 lines | ~29 |
| 23:51 | Edited src/lib/netio.c | 4→5 lines | ~26 |
| 23:51 | Edited src/lib/netio.c | added 7 condition(s) | ~847 |
| 23:52 | Edited src/lib/netio.c | modified endpoint_connect() | ~75 |
| 23:52 | Edited src/include/netio.h | expanded (+6 lines) | ~124 |
| 23:52 | Edited src/controller/ha.c | 1→2 lines | ~32 |
| 23:52 | Edited src/lib/client.c | 2→3 lines | ~49 |
| 23:52 | Edited src/lib/xcectx.c | inline fix | ~20 |
| 23:53 | Edited src/lib/client.c | 2→3 lines | ~27 |
| 23:53 | Edited src/lib/xcectx.c | 3→4 lines | ~30 |
| 07:00 | task 1.3 bound peer connect (socket_connect O_NONBLOCK-before-connect + poll bound, +connect_timeout_ms; updated all 5 endpoint_connect callers) | netio.c/.h, ha.c, client.c, xcectx.c, test_netio_bounded_connect.c | GREEN: lib 278/0, controller 194/0, broad build OK | ~30k |
| 00:10 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~10 |
| 00:10 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~9 |
| 00:10 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~11 |
| 00:15 | Created src/lib/testsuite/test_vtask_admission.c | — | ~2167 |
| 00:15 | Edited src/lib/testsuite/test.c | modified setUp() | ~45 |
| 00:15 | Edited src/lib/testsuite/test.c | 5→6 lines | ~35 |
| 00:15 | Edited src/lib/meson.build | 4→5 lines | ~56 |
| 00:16 | Edited src/include/vtaskmgr.h | modified int() | ~279 |
| 00:16 | Edited src/include/vtaskmgr.h | modified struct() | ~62 |
| 00:16 | Edited src/include/vtaskmgr.h | modified struct() | ~193 |
| 00:17 | Edited src/include/vtaskmgr.h | expanded (+28 lines) | ~443 |
| 00:17 | Edited src/lib/vtaskmgr.c | 9→11 lines | ~50 |
| 00:17 | Edited src/lib/vtaskmgr.c | modified vtask_init() | ~69 |
| 00:17 | Edited src/lib/vtaskmgr.c | added 1 condition(s) | ~225 |
| 00:17 | Edited src/lib/vtaskmgr.c | modified if() | ~114 |
| 00:18 | Edited src/lib/vtaskmgr.c | added 14 condition(s) | ~859 |
| 00:19 | Edited src/controller/include/commit.h | 7→12 lines | ~159 |
| 00:19 | Edited src/controller/include/commit.h | expanded (+6 lines) | ~150 |
| 00:19 | Edited src/controller/include/commit.h | expanded (+8 lines) | ~170 |
| 00:19 | Edited src/controller/commit.c | added 4 condition(s) | ~447 |
| 00:19 | Edited src/controller/commit.c | 2→2 lines | ~16 |
| 00:20 | Edited src/controller/commit.c | modified XCE_TESTING() | ~186 |
| 00:20 | Edited src/controller/commit.c | modified XCE_TESTING() | ~37 |
| 00:20 | Edited src/controller/commit.c | 4→5 lines | ~56 |
| 00:20 | Edited src/controller/commit.c | added 1 condition(s) | ~152 |
| 00:20 | Edited src/controller/cluster.c | expanded (+9 lines) | ~252 |
| 00:21 | Created src/controller/testsuite/test_commit_admission_pool.c | — | ~1947 |
| 00:21 | Edited src/controller/testsuite/test_commit_admission_pool.c | 4→4 lines | ~43 |
| 00:22 | Edited src/controller/meson.build | 3→4 lines | ~39 |
| 00:22 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~33 |
| 00:22 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~26 |
| 00:28 | Edited src/controller/testsuite/test_commit_admission_pool.c | modified test_commit_fold_uses_pool_cap() | ~845 |
| 00:42 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~20 |
| 00:42 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~11 |
| 00:43 | Edited .kiro/specs/raft-drive-async/tasks.md | 1→2 lines | ~339 |
| 00:47 | Edited src/controller/ha.h | modified reconnect() | ~347 |
| 00:47 | Edited src/controller/ha.h | modified submits() | ~476 |
| 00:48 | Edited src/controller/ha.c | modified submits() | ~488 |
| 00:48 | Edited src/controller/ha.c | added 10 condition(s) | ~2685 |
| 00:50 | Edited src/controller/testsuite/test_ha_liveness.c | 4→7 lines | ~36 |
| 00:51 | Edited src/controller/testsuite/test_ha_liveness.c | added 5 condition(s) | ~2828 |
| 00:51 | Edited src/controller/testsuite/test_ha_liveness.c | 3→7 lines | ~105 |
| 00:52 | Edited src/controller/testsuite/test_ha_liveness.c | modified test_reconnect_offload_tick_returns_promptly() | ~50 |
| 00:52 | Edited src/controller/testsuite/test_ha_liveness.c | modified test_reconnect_offload_tick_returns_promptly() | ~38 |
| 00:52 | Edited src/controller/testsuite/test_ha_liveness.c | 3→4 lines | ~58 |
| 00:53 | Edited src/controller/testsuite/test_ha_liveness.c | 4→3 lines | ~36 |
| 00:56 | Edited src/controller/ha.h | removed 16 lines | ~40 |
| 00:56 | Edited src/controller/ha.h | modified submits() | ~206 |
| 00:57 | Edited src/controller/ha.c | removed 30 lines | ~40 |
| 00:57 | Edited src/controller/ha.c | modified submits() | ~448 |
| 00:58 | Edited src/controller/ha.c | modified if() | ~242 |
| 07:59 | Task 2.1 reconnect offload: split ha_connect_to_peer→prep/finish/lane, per-peer in-flight dedup, slowlane_submit; flag default ON; 3 tests in test_ha_liveness | ha.c ha.h test_ha_liveness.c | RED captured (inline tick stalls), GREEN 196/196 pass | ~12k |
| 01:08 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~17 |
| 01:08 | Edited .kiro/specs/raft-drive-async/tasks.md | modified success() | ~223 |
## Session: 2026-06-26 01:11
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-26 01:12
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 01:12 | Edited src/controller/testsuite/test_ha_liveness.c | modified criteria() | ~1760 |
| 01:12 | Edited src/controller/testsuite/test_ha_liveness.c | 4→7 lines | ~103 |
| 01:13 | Edited src/controller/testsuite/test_ha_liveness.c | 5→9 lines | ~128 |
| 01:13 | Edited src/controller/ha.c | modified XCE_TESTING() | ~119 |
| 01:13 | Edited src/controller/ha.c | 2→3 lines | ~41 |
| 01:13 | Session end: 5 writes across 2 files (test_ha_liveness.c, ha.c) | 1 reads | ~84302 tok |
| 01:14 | Edited src/controller/ha.c | added 2 condition(s) | ~829 |
| 01:14 | Edited src/controller/ha.c | endpoint_put() → free() | ~101 |
| 01:15 | Edited src/controller/ha.c | modified on_standby_connected_for_test() | ~140 |
| 01:15 | Edited src/controller/ha.c | modified if() | ~54 |
| 01:16 | Edited src/controller/ha.c | modified if() | ~43 |
| 01:18 | Edited src/controller/ha.c | 9→13 lines | ~163 |
| 01:19 | Edited src/controller/ha.c | 6→8 lines | ~68 |
| 01:19 | Edited src/controller/ha.c | modified if() | ~148 |
| 01:20 | Edited src/controller/ha.c | modified if() | ~102 |
| 01:20 | Edited src/controller/ha.c | 6→7 lines | ~84 |
| 01:20 | Edited src/controller/ha.c | modified if() | ~68 |
| 01:21 | Edited src/controller/ha.c | added 1 condition(s) | ~332 |
| 01:21 | Edited src/controller/testsuite/test_ha_liveness.c | 6→9 lines | ~120 |
## Session: 2026-06-26 02:44
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-26 02:44
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-26 02:46
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 02:56 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~14 |
| 03:01 | Edited src/controller/testsuite/test_ha_liveness.c | added 3 condition(s) | ~2567 |
| 03:01 | Edited src/controller/testsuite/test_ha_liveness.c | 4→7 lines | ~118 |
| 10:03 | raft-drive-async task 2.3: sustained reconnect-offload integration tests (4-node, 2 survivors, 3 ticks) | test_ha_liveness.c | GREEN+RED pass, suite 25/0 | ~6k |
| 03:05 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~12 |
| 03:20 | Edited src/controller/ha.h | expanded (+6 lines) | ~186 |
| 03:20 | Edited src/controller/ha.h | expanded (+26 lines) | ~408 |
| 03:21 | Edited src/controller/ha.c | modified ha_reconnect_offload_enabled() | ~387 |
| 03:21 | Edited src/controller/ha.c | 2→3 lines | ~58 |
| 03:21 | Edited src/controller/ha.c | 3→4 lines | ~70 |
| 03:21 | Edited src/controller/ha.c | modified frees() | ~297 |
| 03:22 | Edited src/controller/ha.c | modified ownership() | ~200 |
| 03:22 | Edited src/controller/ha.c | modified frees() | ~311 |
| 03:23 | Edited src/controller/ha.c | added 7 condition(s) | ~2148 |
| 03:24 | Edited src/controller/cluster.c | modified if() | ~134 |
| 03:27 | Edited src/controller/testsuite/test_ha_liveness.c | added 4 condition(s) | ~2788 |
| 03:27 | Edited src/controller/testsuite/test_ha_liveness.c | 3→7 lines | ~108 |
| 03:56 | Edited src/include/vtaskmgr.h | expanded (+10 lines) | ~219 |
| 03:56 | Edited src/lib/vtaskmgr.c | 4→5 lines | ~49 |
| 03:57 | Edited src/lib/vtaskmgr.c | added 1 condition(s) | ~335 |
| 03:57 | Edited src/controller/ha.c | expanded (+7 lines) | ~189 |
| 03:58 | Edited src/controller/ha.c | modified peer_ctrl_classify() | ~397 |
| 03:58 | Edited src/controller/ha.c | modified terminate() | ~431 |
| 03:59 | Edited src/controller/ha.c | modified HAZARD() | ~296 |
| 04:00 | Edited src/controller/testsuite/test_ha_liveness.c | 2→5 lines | ~106 |
| 04:00 | Edited src/controller/testsuite/test_ha_liveness.c | 3→3 lines | ~36 |
| 04:00 | Edited src/controller/testsuite/test_ha_liveness.c | 3→5 lines | ~67 |
| 04:00 | Edited src/controller/testsuite/test_ha_liveness.c | 3→5 lines | ~74 |
| 04:04 | Edited src/controller/ha.c | inline fix | ~6 |
| 04:05 | Edited src/controller/ha.c | inline fix | ~6 |
| 04:13 | Edited .kiro/specs/raft-drive-async/tasks.md | modified FIX() | ~767 |
| 04:31 | Edited src/controller/persist.c | modified runs() | ~806 |
| 04:33 | Edited src/controller/testsuite/test_e2e_checkpoint.c | added error handling | ~218 |
| 04:34 | Edited src/controller/testsuite/test_e2e_checkpoint.c | added error handling | ~1674 |
| 04:35 | Edited src/controller/testsuite/test_e2e_checkpoint.c | file_exists_in() → seg_file_exists() | ~143 |
| 04:35 | Edited src/controller/testsuite/test_e2e_checkpoint.c | inline fix | ~4 |
| 04:37 | Edited src/controller/testsuite/test_e2e_checkpoint.c | modified ckpt_drop_broadcast() | ~226 |
| 04:39 | Edited src/controller/ha.h | modified send() | ~274 |
| 04:41 | Edited src/controller/ha.c | modified ON() | ~311 |
| 04:41 | Edited src/controller/ha.h | modified ON() | ~159 |
## Session: 2026-06-26 05:18
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-26 05:18
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-26 07:08
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-26 07:10
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 08:16 | Edited src/controller/testsuite/test_ha_liveness.c | added 3 condition(s) | ~2312 |
| 08:16 | Edited src/controller/testsuite/test_ha_liveness.c | 4→8 lines | ~112 |
| 08:28 | Edited src/controller/ha.c | added 9 condition(s) | ~1771 |
| 08:28 | Edited src/controller/ha.c | reduced (-15 lines) | ~145 |
| 08:29 | Edited src/controller/ha.c | modified OFF() | ~340 |
| 08:31 | Edited src/controller/ha.c | expanded (+7 lines) | ~432 |
| 08:40 | Edited src/controller/ha.h | thread() → safe() | ~161 |
| 08:40 | Edited src/controller/persist.c | modified runs() | ~110 |
| 08:41 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~42 |
| 08:42 | Edited .kiro/specs/raft-drive-async/tasks.md | inline fix | ~41 |
| 08:42 | Edited .kiro/specs/raft-drive-async/tasks.md | modified completion() | ~830 |
| 08:42 | raft-drive-async 3.2: snapshot SlowLane offload (Part A, DORMANT flag-OFF) + trim cursor-guard (Part B, LIVE) | ha.c, ha.h, persist.c, test_ha_liveness.c, test_e2e_checkpoint.c | 196/0 ctrl tests, 0 new leaks, cpp-review GO; caught+fixed default-ON UAF (bug-1210) | ~62k |

View file

@ -2644,6 +2644,145 @@ void ha_process_standby_state(ha_state_t *ha, int peer_idx,
}
}
/* ───────────────────────────────────────────────────────────────────────
* raft-drive-async Task 3.2 — SnapshotSend slow-lane offload (bug-1198).
*
* The snapshot transfer (a fresh persist_checkpoint → fdatasync, then a
* chunked socket write of the snapshot file) is the one unbounded operation
* reachable from the repl-ack decision loop. Run inline it blocks the
* io_worker/drive thread for the whole transfer (and queues that peer's later
* messages behind it). This offloads it to the bounded slow lane, keyed per
* standby so at most one snapshot per peer is in flight (Req 1.5).
* ─────────────────────────────────────────────────────────────────────── */
typedef struct {
cluster_t *c;
endpoint_t *ep;
int peer_idx;
} snapshot_send_ctx_t;
/* ha_snapshot_do_send: the actual (blocking) snapshot work — shared by the
* inline path and the lane worker so both behave identically. In XCE_TESTING
* builds the send is replaced by ha_repl_send_snapshot_hook(peer_idx). */
static void ha_snapshot_do_send(cluster_t *c, endpoint_t *ep, int peer_idx)
{
#ifdef XCE_TESTING
if (ha_repl_send_snapshot_hook != NULL) {
ha_repl_send_snapshot_hook(peer_idx);
return;
}
#endif
if (c && c->persist.snap_path[0]) {
extern int ha_send_snapshot_to_standby(endpoint_t *, const char *,
uint64_t);
/* Req 4.1: create a FRESH checkpoint so the transferred snapshot
* reflects current state (also trims WAL + rotates old snaps).
* -EBUSY means another checkpoint is in flight — fall back to the
* existing checkpoint_lsn rather than skipping the cycle (at most one
* checkpoint in flight is the existing persist_checkpoint invariant). */
int ckpt_rc = persist_checkpoint(&c->persist);
if (ckpt_rc != 0 && ckpt_rc != -EBUSY)
WARN("HA repl: persist_checkpoint before snapshot send "
"failed rc=%d — sending existing checkpoint", ckpt_rc);
ha_send_snapshot_to_standby(ep, c->persist.snap_path,
c->persist.checkpoint_lsn);
}
}
/*
* snapshot_lane_fn: slow-lane worker entry. Runs the bounded blocking
* snapshot send off the drive thread, then ALWAYS clears the per-peer
* in-flight flag (success or failure) so the dedup never wedges and a later
* heartbeat can retry. Frees the ctx.
*
* ⚠ BORROWED-ENDPOINT LIFETIME HAZARD (bug-1197 family — why this path is
* gated OFF by default). `ep` here is a standby endpoint shared with the
* io_worker epoll loop, NOT one this lane owns (contrast reconnect_lane_fn,
* which owns the ep it creates). If the standby RDHUP-disconnects while this
* transfer runs, the io_worker can endpoint_fini/free `ep` concurrently → UAF.
* Today (flag OFF) the send runs inline ON the io_worker, so teardown and send
* are serialized on one thread and cannot overlap. Flipping
* g_drive_async_snapshot_offload ON is therefore only safe once Task 3.4
* establishes the endpoint-lifetime discipline for the whole pool/lane
* dispatch model (refcount or lane-serialized teardown). Until then this code
* lands DORMANT and is exercised only by unit tests with a stable endpoint. */
static void snapshot_lane_fn(void *arg)
{
snapshot_send_ctx_t *sx = (snapshot_send_ctx_t *)arg;
if (!sx) return;
cluster_t *c = sx->c;
int peer_idx = sx->peer_idx;
ha_snapshot_do_send(c, sx->ep, peer_idx);
/* Clear in-flight LAST so a concurrent heartbeat tick cannot enqueue a
* second snapshot for this peer until this one has fully finished. */
atomic_store_explicit(&c->ha.snapshot_inflight[peer_idx], 0,
memory_order_release);
free(sx);
}
/*
* ha_snapshot_send_dispatch: route a SEND_SNAPSHOT decision either inline
* (flag OFF, production default — runs on the caller/drive thread, unchanged
* legacy behavior) or onto the slow lane (flag ON — non-blocking). The
* per-peer single-flight CAS on ha->snapshot_inflight[] is the authoritative
* dedup; the lane's per-key dedup is a redundant backstop. Mirrors the
* reconnect-offload terminal-path handling (Task 2.1).
*/
static void ha_snapshot_send_dispatch(ha_state_t *ha, cluster_t *c,
endpoint_t *ep, int peer_idx)
{
/* Flag OFF (default) or no cluster handle (no lane) → inline. */
if (!ha_snapshot_offload_enabled() || c == NULL) {
ha_snapshot_do_send(c, ep, peer_idx);
return;
}
/* Per-peer single-flight: CAS 0->1. Already 1 → a snapshot to this peer
* is in flight; enqueue nothing (Req 1.5). */
int expected = 0;
if (!atomic_compare_exchange_strong_explicit(
&ha->snapshot_inflight[peer_idx], &expected, 1,
memory_order_acq_rel, memory_order_acquire)) {
return;
}
snapshot_send_ctx_t *sx = malloc(sizeof(*sx));
if (!sx) {
atomic_store_explicit(&ha->snapshot_inflight[peer_idx], 0,
memory_order_release);
return;
}
sx->c = c; sx->ep = ep; sx->peer_idx = peer_idx;
uint64_t key = (uint64_t)(peer_idx + 1); /* +1: key 0 disables dedup */
int srv = slowlane_submit(&c->slowlane, key, snapshot_lane_fn, sx);
if (srv == -EEXIST) {
/* Lane still holds this peer's key from a prior snapshot task that has
* already cleared snapshot_inflight (its last action) but whose key the
* lane has not yet removed. Our CAS saw 0 and set 1, but the submit is
* refused — release the flag we set so no clear is lost, and let the
* next tick retry. */
free(sx);
atomic_store_explicit(&ha->snapshot_inflight[peer_idx], 0,
memory_order_release);
return;
}
if (srv < 0 && srv != -EAGAIN) {
/* Hard submit failure (alloc / lane stopping). -EAGAIN is NOT a
* failure (still queued, will run). Only on a real failure tear down
* and release the guard so the next tick retries; never self-terminate. */
WARN("HA repl: slowlane_submit snapshot peer[%d] failed: %d — retry "
"next cycle", peer_idx, srv);
free(sx);
atomic_store_explicit(&ha->snapshot_inflight[peer_idx], 0,
memory_order_release);
return;
}
/* 0 or -EAGAIN: enqueued exactly once; snapshot_lane_fn clears the flag. */
}
/*
* ha_dispatch_repl_action_for_standby — execute the decision-loop action for
* one standby on the current heartbeat cycle (Task 3.3).
@ -2714,26 +2853,11 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
INFO("HA repl: standby[%d] lsn=%" PRIu64
" < oldest_wal_lsn=%" PRIu64 " → requesting snapshot transfer",
peer_idx, standby_lsn, oldest_wal_lsn);
#ifdef XCE_TESTING
if (ha_repl_send_snapshot_hook != NULL) {
ha_repl_send_snapshot_hook(peer_idx);
break;
}
#endif
if (c && c->persist.snap_path[0]) {
extern int ha_send_snapshot_to_standby(endpoint_t *, const char *,
uint64_t);
/* Req 4.1: create a FRESH checkpoint so the transferred snapshot
* reflects current state (also trims WAL + rotates old snaps).
* -EBUSY means another checkpoint is in flight — fall back to
* the existing checkpoint_lsn rather than skipping the cycle. */
int ckpt_rc = persist_checkpoint(&c->persist);
if (ckpt_rc != 0 && ckpt_rc != -EBUSY)
WARN("HA repl: persist_checkpoint before snapshot send "
"failed rc=%d — sending existing checkpoint", ckpt_rc);
ha_send_snapshot_to_standby(ep, c->persist.snap_path,
c->persist.checkpoint_lsn);
}
/* Task 3.2: route inline (flag OFF, default) or onto the slow lane
* (flag ON). The send body — fresh checkpoint + chunked transfer, or
* the XCE_TESTING hook — lives in ha_snapshot_do_send so both paths
* are identical. */
ha_snapshot_send_dispatch(ha, c, ep, peer_idx);
break;
case HA_REPL_STREAM_WAL:
@ -3802,6 +3926,40 @@ int ha_reconnect_offload_enabled(void)
memory_order_acquire);
}
/* raft-drive-async task 3.2: SnapshotSend feature flag.
*
* DEFAULT OFF (0): the snapshot transfer runs INLINE on the caller (io_worker/
* drive) thread — byte-for-byte the legacy behavior. This task lands DORMANT,
* exactly like the Task 3.1 peer-control dispatch flag, because the offload
* captures a BORROWED standby endpoint (see snapshot_lane_fn's bug-1197 hazard
* note): moving the transfer to a lane thread while the io_worker may
* RDHUP-free that endpoint is a cross-thread UAF. Today (flag OFF) the send
* and the endpoint teardown are serialized on the single io_worker, so they
* cannot overlap.
*
* Flipping ON is the job of Task 3.4 (receiver-side integration), which
* establishes the endpoint-lifetime discipline for the pool/lane dispatch
* model and proves the transfer no longer stalls other peers' heartbeats/votes
* with the pool near saturation. Until then, bug-1198 (the repl-ack body is
* still unbounded when run inline) remains open by design.
*
* Keyed per standby (Req 1.5) so at most one snapshot per peer is in flight;
* at most one persist_checkpoint in flight (existing atomic guard). Compiled
* in ALL builds; the setter is the test seam to flip RED↔GREEN. */
static _Atomic int g_drive_async_snapshot_offload = 0;
void ha_set_snapshot_offload_for_test(int on)
{
atomic_store_explicit(&g_drive_async_snapshot_offload, on ? 1 : 0,
memory_order_release);
}
int ha_snapshot_offload_enabled(void)
{
return atomic_load_explicit(&g_drive_async_snapshot_offload,
memory_order_acquire);
}
/* raft-drive-async task 3.1: PeerCtrlDispatch feature flag.
*
* DEFAULT OFF (0): the registered peer-control handler runs its routing body
@ -4937,17 +5095,24 @@ static int peer_ctrl_vtask_entry(vtask_t *task)
* everything this handler dispatches is liveness work today.)
*
* ┌─ HIGH FLAG-ON HAZARD (raft-drive-async finding-2; bug-1198) ─────────────┐
* │ LIVENESS is correct ONLY once task 3.2 offloads the snapshot send off the │
* │ pool. TODAY the CTRL_PRIMARY repl-ack path can reach │
* │ HA_REPL_SEND_SNAPSHOT → persist_checkpoint (a BLOCKING fdatasync) INLINE │
* │ in repl_ack_decode_and_dispatch → ha_dispatch_repl_action_for_standby. │
* │ LIVENESS is correct ONLY once the snapshot send runs OFF the pool. The │
* │ CTRL_PRIMARY repl-ack path can reach HA_REPL_SEND_SNAPSHOT → │
* │ persist_checkpoint (a BLOCKING fdatasync) in │
* │ repl_ack_decode_and_dispatch → ha_dispatch_repl_action_for_standby. │
* │ Running that blocking op while tagged VTASK_LIVENESS (never gated) breaks │
* │ the deadlock-freedom reservation — a burst of snapshot sends could occupy │
* │ every liveness worker and starve the acks that advance committed_lsn. │
* │ Task 3.2 offloads HA_REPL_SEND_SNAPSHOT to the SlowLane → the repl-ack │
* │ body becomes bounded → LIVENESS becomes correct. │
* │ DO NOT flip g_drive_async_peerctrl_dispatch_enabled ON until task 3.2 │
* │ has landed (and task 3.4's deadlock gate is green). │
* │ │
* │ Task 3.2 has LANDED the slow-lane offload (ha_snapshot_send_dispatch), │
* │ but it is DORMANT: g_drive_async_snapshot_offload defaults OFF (the send │
* │ still runs INLINE) because the offload borrows a standby endpoint whose │
* │ lifetime is not yet pool-safe (bug-1197). So bug-1198 is NOT yet closed │
* │ in the default build. Task 3.4 is the single gate that flips BOTH │
* │ g_drive_async_snapshot_offload AND g_drive_async_peerctrl_dispatch ON │
* │ together (offload bounds the body → LIVENESS correct) and proves the │
* │ deadlock-freedom + endpoint-lifetime properties under load. │
* │ DO NOT flip g_drive_async_peerctrl_dispatch_enabled ON until task 3.4's │
* │ gate is green WITH the snapshot offload also ON. │
* └──────────────────────────────────────────────────────────────────────────┘
*/
static vtask_class_t peer_ctrl_classify(ha_state_t *ha, io_packet_t *pkt)

View file

@ -273,6 +273,18 @@ typedef struct {
* that peer. Indexed by ABSOLUTE candidate
* index (peers_cfg.candidates[]). 0 at
* init (ha_init memset). */
/* ── New field added by raft-drive-async spec (Task 3.2) ──────── */
_Atomic int snapshot_inflight[MAX_HA_CANDIDATES]; /* Per-peer
* single-flight guard for the offloaded
* snapshot send (Req 1.5): set (CAS
* 0->1) when a snapshot to peer[i] is
* submitted to the slow lane, cleared by
* the lane worker on completion (success
* or failure). While set, the heartbeat
* decision loop enqueues no duplicate
* snapshot for that peer. Indexed by
* ABSOLUTE candidate index. 0 at
* init (ha_init memset). */
} ha_state_t;
/* ── Public API ────────────────────────────────────────────── */
@ -1227,6 +1239,19 @@ void ha_start_primary_heartbeat(cluster_t *c);
void ha_set_reconnect_offload_for_test(int on);
int ha_reconnect_offload_enabled(void);
/*
* ha_set_snapshot_offload_for_test / ha_snapshot_offload_enabled:
* TDD kill-switch for raft-drive-async task 3.2 snapshot slow-lane offload.
* OFF (0, PRODUCTION DEFAULT): snapshot send runs INLINE on the caller
* (io_worker/drive) thread — byte-for-byte the legacy path. The offload
* lands DORMANT because it borrows a standby endpoint that is not yet
* pool-lifetime-safe (bug-1197); so bug-1198 stays open by design until
* Task 3.4 flips this ON together with the peer-control dispatch flag.
* ON (1): snapshot send runs on a slow-lane worker, off the drive thread.
*/
void ha_set_snapshot_offload_for_test(int on);
int ha_snapshot_offload_enabled(void);
/*
* ha_set_peerctrl_dispatch_for_test / ha_peerctrl_dispatch_enabled:
* Feature-flag accessors (raft-drive-async Task 3.1). The dispatch flag gates

View file

@ -1368,9 +1368,22 @@ static void scan_latest_committed_config_ref(persist_t* p)
}
/* Delete WAL segment files whose start LSN is strictly less than the
* currently-open segment's start LSN. We parse the active segment
* filename (seg_path) to get its start LSN and never delete that file,
* even if its start_lsn < checkpoint_lsn. */
* active segment's start LSN AND strictly less than the trim floor.
*
* Bug-1147 (stopgap, raft-drive-async task 3.2): the trim floor is the
* minimum of the active-segment LSN, every active standby's last acked LSN,
* and every recently-disconnected slot's last_disconnected_lsn (strategy-A
* retention discipline). This ensures we NEVER delete a WAL segment that a
* live standby still needs for catch-up (the bug-795/828 committed-job-loss
* family). It does NOT fix the empty/far-behind join case (cursor=0) —
* that requires entity-bearing snapshots (separate spec).
*
* Lock ordering (verified 2026-06-26): persist_checkpoint takes p->lock only
* to snapshot checkpoint_lsn, then RELEASES it before calling
* trim_wal_segments — so this function runs with NO p->lock held.
* repl_trim_lsn and repl_expire_disconnect_slots take only r->lock
* (p->repl.lock); neither ever acquires p->lock or ce->lock, so there is no
* p->lock↔repl.lock (or ce->lock↔repl.lock) inversion. */
static void trim_wal_segments(persist_t* p)
{
/* Extract the start LSN of the currently-open segment from seg_path.
@ -1387,6 +1400,18 @@ static void trim_wal_segments(persist_t* p)
if (sscanf(seg_basename, "seg-%" SCNx64 ".wal", &active_seg_lsn) != 1)
return;
/* Expire stale disconnected slots so their last_disconnected_lsn guards
* don't hold WAL segments indefinitely (slot guard expires at
* last_disconnect_time + snap_interval_s). */
repl_expire_disconnect_slots(&p->repl, time(NULL),
p->config.snap_interval_s);
/* Compute the trim floor: min(active_seg_lsn, all active standby
* acked_lsn, all valid disconnected-slot last_disconnected_lsn).
* Any segment with seg_start_lsn BELOW this floor is disposable. */
uint64_t floor = repl_trim_lsn(&p->repl, active_seg_lsn,
p->config.snap_interval_s);
DIR* d = opendir(p->wal_path);
if (!d) return;
@ -1395,8 +1420,11 @@ static void trim_wal_segments(persist_t* p)
if (strncmp(e->d_name, "seg-", 4) != 0) continue;
uint64_t seg_start_lsn = 0;
if (sscanf(e->d_name, "seg-%" SCNx64 ".wal", &seg_start_lsn) != 1) continue;
/* Never delete the currently-open segment; only trim older ones. */
if (seg_start_lsn < active_seg_lsn) {
/* Only delete segments below BOTH the active segment AND the
* standby-cursor floor (bug-1147 guard). floor <= active_seg_lsn
* by construction (repl_trim_lsn caps at checkpoint_lsn=active_seg_lsn),
* so this check subsumes the old < active_seg_lsn guard. */
if (seg_start_lsn < floor) {
char seg_path[4096];
snprintf(seg_path, sizeof(seg_path), "%s/%s", p->wal_path, e->d_name);
unlink(seg_path);

View file

@ -21,6 +21,24 @@
#include "config.h"
#include "persist.h"
#include "repl.h" /* repl_add_standby / repl_on_ack for trim-guard tests
* (raft-drive-async task 3.2, bug-1147) */
#include "typemap.h" /* wal_catchup_entry_t */
/* Drop hook for repl_broadcast: fixtures with dummy (int) endpoints installed
* via repl_add_standby must install this so persist_wal_async→repl_broadcast does
* not crash on the fake endpoint. Pattern from test_commit_engine.c, test_p2_integration.c. */
static int ckpt_drop_broadcast(void *ep, wal_catchup_entry_t **entries, int count)
{
(void)ep;
(void)entries;
(void)count;
return 0;
}
extern int (*repl_broadcast_send_hook)(void *ep,
wal_catchup_entry_t **entries,
int count);
/* ------------------------------------------------------------------ */
/* Helpers */
@ -178,6 +196,133 @@ static void test_snapshot_rotation_keeps_snap_keep_n(void)
printf("test_snapshot_rotation_keeps_snap_keep_n: PASS\n");
}
/* ------------------------------------------------------------------ */
/* Helper: check if a WAL segment file at a given start LSN exists. */
/* Builds the name "seg-<lsn16>.wal" inside dir. */
/* ------------------------------------------------------------------ */
static int seg_file_exists(const char* wal_dir, uint64_t seg_start_lsn)
{
char path[4096];
snprintf(path, sizeof(path), "%s/seg-%016" PRIx64 ".wal",
wal_dir, seg_start_lsn);
return access(path, F_OK) == 0 ? 1 : 0;
}
/* ------------------------------------------------------------------ */
/* Test 4: standby cursor-bound trim guard — Req 1.1 (task 3.2, */
/* bug-1147). With a standby whose acked_lsn is below the */
/* active segment, segments the standby still needs for */
/* catch-up are NOT deleted. */
/* */
/* RED proof: revert the repl_trim_lsn floor call in trim_wal_segments */
/* → the segment at the standby's acked LSN is deleted because it's */
/* below the active segment → re-creates the bug-795/828 gap where */
/* catch-up hits a missing segment → permanent committed-job loss. */
/* ------------------------------------------------------------------ */
static void test_trim_guard_preserves_standby_segments(void)
{
char tmpdir[256];
persist_t* p = make_ckpt_e2e(tmpdir, sizeof(tmpdir), 2);
TEST_ASSERT_NOT_NULL(p);
/* Install drop hook so repl_broadcast doesn't crash on the dummy ep */
repl_broadcast_send_hook = ckpt_drop_broadcast;
/* Add a standby with acked_lsn = 2 (simulating a standby that has
* caught up through LSN 2 but not beyond) */
int dummy_ep = 0;
int idx = repl_add_standby(&p->repl, (void*)&dummy_ep, "standby1", 7001);
TEST_ASSERT_TRUE(idx >= 0);
repl_on_ack(&p->repl, (void*)&dummy_ep, 2);
/* Write entries to create segments.
* init: current_lsn = 1 (LSN 0 reserved, fresh store)
* write "x" @ LSN 1 → current_lsn = 2
* force → close seg-at-1, open seg-at-2
* write "y" @ LSN 2 → current_lsn = 3
* force → close seg-at-2, open seg-at-3
*
* Now we have 3 segments: seg-...001.wal, seg-...002.wal, seg-...003.wal.
* Active segment = seg-at-3. Standby acked_lsn = 2. */
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "x", 1));
TEST_ASSERT_EQUAL_INT(0, persist_test_force_new_segment(p));
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "y", 1));
TEST_ASSERT_EQUAL_INT(0, persist_test_force_new_segment(p));
/* Trigger checkpoint → trim_wal_segments runs */
TEST_ASSERT_EQUAL_INT(0, persist_checkpoint(p));
/* floor = repl_trim_lsn(active_seg_lsn=3, ...) = min(3, acked=2) = 2.
* So seg_start_lsn < 2 means only seg-at-1 is deleted (1 < 2).
* seg-at-2 (the standby still needs it) is NOT deleted (2 >= 2). */
char wal_path[512];
snprintf(wal_path, sizeof(wal_path), "%s/e2e-ckpt/data/wal", tmpdir);
/* The old segment below the standby's acked LSN IS trimmed */
TEST_ASSERT_FALSE_MESSAGE(
seg_file_exists(wal_path, 1),
"seg-at-1: below standby acked_lsn=2, MUST be trimmed");
/* The segment at the standby's acked LSN is PRESERVED (bug-1147 guard) */
TEST_ASSERT_TRUE_MESSAGE(
seg_file_exists(wal_path, 2),
"seg-at-2: at standby acked_lsn=2, MUST be preserved (bug-1147 guard)");
/* The active segment is never trimmed */
TEST_ASSERT_TRUE_MESSAGE(
seg_file_exists(wal_path, 3),
"seg-at-3: active segment, MUST always be preserved");
repl_broadcast_send_hook = NULL;
teardown_ckpt_e2e(p);
printf("test_trim_guard_preserves_standby_segments: PASS\n");
}
/* ------------------------------------------------------------------ */
/* Test 5: trim with no standbys proceeds as before — Req 1.1 */
/* (task 3.2). With no standbys registered, the floor */
/* defaults to active_seg_lsn, so all old segments are */
/* trimmed. This confirms the guard does not prevent normal */
/* cleanup when no standbys need protection. */
/* ------------------------------------------------------------------ */
static void test_trim_without_standbys_proceeds_normally(void)
{
char tmpdir[256];
persist_t* p = make_ckpt_e2e(tmpdir, sizeof(tmpdir), 2);
TEST_ASSERT_NOT_NULL(p);
/* No standbys — just create segments and checkpoint */
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "a", 1));
TEST_ASSERT_EQUAL_INT(0, persist_test_force_new_segment(p));
TEST_ASSERT_EQUAL_INT(0, persist_wal_async(p, WAL_JOB_UPDATE, "b", 1));
TEST_ASSERT_EQUAL_INT(0, persist_test_force_new_segment(p));
TEST_ASSERT_EQUAL_INT(0, persist_checkpoint(p));
char wal_path[512];
snprintf(wal_path, sizeof(wal_path), "%s/e2e-ckpt/data/wal", tmpdir);
/* Without standbys, floor = active_seg_lsn = 3.
* seg_at_1 < 3 → deleted. seg_at_2 < 3 → deleted too. */
TEST_ASSERT_FALSE_MESSAGE(
seg_file_exists(wal_path, 1),
"seg-at-1: no standbys, MUST be trimmed");
TEST_ASSERT_FALSE_MESSAGE(
seg_file_exists(wal_path, 2),
"seg-at-2: no standbys, MUST be trimmed");
/* Active segment still there */
TEST_ASSERT_TRUE_MESSAGE(
seg_file_exists(wal_path, 3),
"seg-at-3: active segment, MUST always be preserved");
teardown_ckpt_e2e(p);
printf("test_trim_without_standbys_proceeds_normally: PASS\n");
}
/* ------------------------------------------------------------------ */
/* Suite entry point */
/* ------------------------------------------------------------------ */
@ -189,4 +334,6 @@ void test_e2e_checkpoint(void)
RUN_TEST(test_checkpoint_recovery_lsn_continuity);
RUN_TEST(test_checkpoint_wal_trim_after_checkpoint);
RUN_TEST(test_snapshot_rotation_keeps_snap_keep_n);
RUN_TEST(test_trim_guard_preserves_standby_segments);
RUN_TEST(test_trim_without_standbys_proceeds_normally);
}

View file

@ -1745,6 +1745,230 @@ static void test_peerctrl_preserves_per_endpoint_order(void)
printf("test_peerctrl_preserves_per_endpoint_order: PASS\n");
}
/* ───────────────────────────────────────────────────────────────────────
* Task 3.2 — SnapshotSend slow-lane offload (Req 1.1, 1.5; bug-1198).
*
* Probe: a snapshot-send hook (ha_repl_send_snapshot_hook) that records the
* thread it ran on and supports an optional gate so a test can hold the lane
* worker in flight while it submits a second snapshot for the same peer
* (per-peer dedup proof, Req 1.5). Mirrors the peerctrl pc_probe gate.
* ─────────────────────────────────────────────────────────────────────── */
extern int (*ha_repl_send_snapshot_hook)(int peer_idx);
static _Atomic int g_snap_runs = 0;
static _Atomic pthread_t g_snap_last_thread;
static _Atomic int g_snap_gate_on = 0;
static pthread_mutex_t g_snap_mu = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t g_snap_parked_cv = PTHREAD_COND_INITIALIZER;
static pthread_cond_t g_snap_release_cv = PTHREAD_COND_INITIALIZER;
static int g_snap_parked = 0;
static int g_snap_release = 0;
static int snap_probe(int peer_idx)
{
(void)peer_idx;
atomic_store_explicit(&g_snap_last_thread, pthread_self(),
memory_order_relaxed);
if (atomic_load(&g_snap_gate_on)) {
pthread_mutex_lock(&g_snap_mu);
g_snap_parked = 1;
pthread_cond_broadcast(&g_snap_parked_cv);
while (!g_snap_release)
pthread_cond_wait(&g_snap_release_cv, &g_snap_mu);
pthread_mutex_unlock(&g_snap_mu);
}
atomic_fetch_add(&g_snap_runs, 1);
return 0;
}
/* Bounded poll until the gated body parks; 0 on park, -1 on timeout. */
static int snap_wait_parked(int timeout_ms)
{
pthread_mutex_lock(&g_snap_mu);
int parked = 0;
for (int i = 0; i < timeout_ms; i++) {
if (g_snap_parked) { parked = 1; break; }
pthread_mutex_unlock(&g_snap_mu);
struct timespec ts = { .tv_sec = 0, .tv_nsec = 1000000L }; /* 1 ms */
nanosleep(&ts, NULL);
pthread_mutex_lock(&g_snap_mu);
}
pthread_mutex_unlock(&g_snap_mu);
return parked ? 0 : -1;
}
static int snap_wait_runs(int want, int timeout_ms)
{
for (int i = 0; i < timeout_ms; i++) {
if (atomic_load(&g_snap_runs) >= want) break;
struct timespec ts = { .tv_sec = 0, .tv_nsec = 1000000L }; /* 1 ms */
nanosleep(&ts, NULL);
}
return atomic_load(&g_snap_runs);
}
static void snap_release_gate(void)
{
pthread_mutex_lock(&g_snap_mu);
g_snap_release = 1;
pthread_cond_broadcast(&g_snap_release_cv);
pthread_mutex_unlock(&g_snap_mu);
}
static void snap_reset_probe(void)
{
atomic_store(&g_snap_runs, 0);
atomic_store(&g_snap_gate_on, 0);
pthread_mutex_lock(&g_snap_mu);
g_snap_parked = 0;
g_snap_release = 0;
pthread_mutex_unlock(&g_snap_mu);
}
/* Drive one SEND_SNAPSHOT decision: standby tag current but lsn behind the
* oldest WAL window → ha_evaluate_repl_action yields HA_REPL_SEND_SNAPSHOT. */
static int snap_dispatch_once(cluster_t *c, endpoint_t *ep, int peer_idx)
{
c->ha.standby_reported_tag[peer_idx] = 5;
c->ha.standby_reported_lsn[peer_idx] = 10; /* < oldest_wal_lsn below */
return ha_dispatch_repl_action_for_standby(
&c->ha, c, ep, peer_idx,
/*committed_tag=*/5, /*committed_lsn=*/100, /*oldest_wal_lsn=*/50);
}
/*
* Task 3.2 GREEN (Req 1.1, 8.1/8.2): with the offload flag ON, SEND_SNAPSHOT
* is submitted to the slow lane — the send body runs on a LANE worker (a
* DIFFERENT thread from the caller) and the dispatch returns promptly. This
* proves the snapshot transfer no longer blocks the io_worker/drive thread.
*
* RED before Task 3.2 lands: the snapshot ran inline on the caller thread, so
* the different-thread assertion fails.
*/
static void test_snapshot_offload_runs_on_lane(void)
{
cluster_t c;
memset(&c, 0, sizeof(c));
ha_init(&c.ha, &g_cfg, 0);
TEST_ASSERT_EQUAL_INT(0, slowlane_init(&c.slowlane, 2, 4));
endpoint_t ep;
memset(&ep, 0, sizeof(ep));
snap_reset_probe();
ha_repl_send_snapshot_hook = snap_probe;
ha_set_snapshot_offload_for_test(1);
pthread_t caller = pthread_self();
int rc = snap_dispatch_once(&c, &ep, 0);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc,
"Req 8.1/8.2: dispatch must return promptly (0) after lane submit");
int runs = snap_wait_runs(1, 2000);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, runs,
"Req 1.1: the snapshot send body must run on the lane exactly once");
TEST_ASSERT_FALSE_MESSAGE(
pthread_equal(atomic_load_explicit(&g_snap_last_thread,
memory_order_relaxed), caller),
"Req 1.1: the snapshot send must run on a LANE worker, NOT the "
"caller's (io_worker/drive) thread");
ha_set_snapshot_offload_for_test(0); /* restore production default (OFF) */
ha_repl_send_snapshot_hook = NULL;
slowlane_fini(&c.slowlane);
ha_fini(&c.ha);
printf("test_snapshot_offload_runs_on_lane: PASS\n");
}
/*
* Task 3.2 RED side / no-regression (Req 6.x): with the offload flag OFF (the
* production default until Task 3.4 resolves the borrowed-endpoint lifetime),
* SEND_SNAPSHOT runs INLINE on the calling thread — byte-for-byte the
* pre-task behavior.
*/
static void test_snapshot_inline_when_flag_off(void)
{
cluster_t c;
memset(&c, 0, sizeof(c));
ha_init(&c.ha, &g_cfg, 0);
endpoint_t ep;
memset(&ep, 0, sizeof(ep));
snap_reset_probe();
ha_repl_send_snapshot_hook = snap_probe;
ha_set_snapshot_offload_for_test(0); /* explicit OFF */
pthread_t caller = pthread_self();
int rc = snap_dispatch_once(&c, &ep, 0);
TEST_ASSERT_EQUAL_INT(0, rc);
/* Synchronous: the body already ran on THIS thread before returning. */
TEST_ASSERT_EQUAL_INT_MESSAGE(1, atomic_load(&g_snap_runs),
"Req 6.x: with the flag OFF the snapshot send must run inline");
TEST_ASSERT_TRUE_MESSAGE(
pthread_equal(atomic_load_explicit(&g_snap_last_thread,
memory_order_relaxed), caller),
"Req 6.x: with the flag OFF the send must run on the caller's thread");
ha_repl_send_snapshot_hook = NULL;
ha_fini(&c.ha);
printf("test_snapshot_inline_when_flag_off: PASS\n");
}
/*
* Task 3.2 (Req 1.5): per-peer single-flight. While a snapshot to peer 0 is
* in flight on the lane (parked at the gate), a second SEND_SNAPSHOT decision
* for the SAME peer enqueues nothing — the CAS on snapshot_inflight[0] fails.
* Proof: with the first body parked, the run count stays at 0; after release,
* exactly ONE body has run (the duplicate was deduped, not queued).
*/
static void test_snapshot_offload_dedups_per_peer(void)
{
cluster_t c;
memset(&c, 0, sizeof(c));
ha_init(&c.ha, &g_cfg, 0);
TEST_ASSERT_EQUAL_INT(0, slowlane_init(&c.slowlane, 2, 4));
endpoint_t ep;
memset(&ep, 0, sizeof(ep));
snap_reset_probe();
atomic_store(&g_snap_gate_on, 1); /* lane body parks at the gate */
ha_repl_send_snapshot_hook = snap_probe;
ha_set_snapshot_offload_for_test(1);
/* First snapshot — submitted; its body parks in the lane. */
TEST_ASSERT_EQUAL_INT(0, snap_dispatch_once(&c, &ep, 0));
TEST_ASSERT_EQUAL_INT_MESSAGE(0, snap_wait_parked(2000),
"the first snapshot body must reach the gate (running on the lane)");
/* Second snapshot for the SAME peer while the first is in flight. */
TEST_ASSERT_EQUAL_INT(0, snap_dispatch_once(&c, &ep, 0));
/* Give a generous window: the duplicate must NOT run. */
struct timespec ts = { .tv_sec = 0, .tv_nsec = 200 * 1000000L }; /* 200 ms */
nanosleep(&ts, NULL);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, atomic_load(&g_snap_runs),
"Req 1.5: the duplicate snapshot must be deduped while one is in flight");
/* Release; exactly the first body completes. */
snap_release_gate();
int runs = snap_wait_runs(1, 2000);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, runs,
"Req 1.5: exactly one snapshot ran for the peer (duplicate deduped)");
/* And no second body sneaks in after release. */
nanosleep(&ts, NULL);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, atomic_load(&g_snap_runs),
"Req 1.5: the deduped duplicate was dropped, not queued behind the first");
ha_set_snapshot_offload_for_test(0); /* restore production default (OFF) */
ha_repl_send_snapshot_hook = NULL;
slowlane_fini(&c.slowlane);
ha_fini(&c.ha);
printf("test_snapshot_offload_dedups_per_peer: PASS\n");
}
/* ── Suite entry point ──────────────────────────────────────────────────── */
void test_ha_liveness(void)
@ -1782,5 +2006,9 @@ void test_ha_liveness(void)
RUN_TEST(test_peerctrl_dispatch_runs_on_pool);
RUN_TEST(test_peerctrl_inline_when_flag_off);
RUN_TEST(test_peerctrl_preserves_per_endpoint_order);
/* raft-drive-async Task 3.2 — SnapshotSend slow-lane offload (Req 1.1, 1.5, 6.x) */
RUN_TEST(test_snapshot_offload_runs_on_lane);
RUN_TEST(test_snapshot_inline_when_flag_off);
RUN_TEST(test_snapshot_offload_dedups_per_peer);
UNITY_END();
}