Commit graph

147 commits

Author SHA1 Message Date
1cbd7df1a8 fix(job-array-single-wal-record): wire dispatcher was silently dropping compact array pushes
extract_data() (src/lib/request.c), the shared reassembly step every
inbound push passes through inside vtask_entry() before any receiver's
handler runs, had no case for Data_data_jobArray -- it hit
default: rc=-EINVAL. Because vtask_entry()'s callnext check treats any
nonzero return (including a negative errno) as "reassembly complete,
proceed," this never surfaced as an error: the real payload was freed
and the receiver silently handed an empty, zeroed dataset_t, with no
log anywhere in the chain. Tasks 5.1-5.3 were each individually
correct and tested in isolation, but the feature was dead on the real
wire end-to-end.

Adds case Data_data_jobArray: passing the single, unchunked
descriptor straight through (no pagination needed, unlike the
existing LIST-shaped cases) and nulls the source pointer after
transfer -- verified this is not cosmetic, since free_Request()'s
pushmsg case would otherwise double-free it via free_Data()'s live
Data_data_jobArray branch.

Proven with a real xce_encode()/xce_decode() round-trip through the
fixed extract_data() -> sched_handle_dataset() -> sched_on_job_array()
(task 5.3, unmodified) chain, not just a unit-level check on
extract_data() alone. Also confirms task 5.4's future API-server
receive path shares this exact fix, since both go through the same
xcecli_ctx_init() -> vtask_entry() -> extract_data() pipeline.

One self-flagged, non-blocking residual gap (pre-existing since task
5.3, out of this task's request.c-only boundary): dataset->rows for
the jobArray case is never deep-freed downstream, leaking one bounded
xce_job_submit_t per successfully-delivered array push commit.

Task 5.3b of job-array-single-wal-record (Requirements 6.1, 6.2, 6.4, 8.4).
Inserted mid-run after independent review confirmed the gap; user
confirmed sequencing (fix now, before task 5.4).
2026-07-15 10:59:39 +00:00
76862bd9aa feat(job-array-single-wal-record): harden jobj.c job construction primitives
- job_init() now checks makeJobsub()/makeString(ckptinfo) return
  values and propagates failure instead of storing NULL with a
  success return (Req 10.1).
- Root-caused the previously observed makeJobsub() NULL-with-EINVAL
  failure to a test-binary-only hazard: multiple controller test
  files each called job_obj_init() from their own setUp() with no
  cross-file coordination, re-running jobsub_table_init() against an
  already-live lock/table. Fixed via atomic_flag-guarded idempotent
  job_obj_init()/job_obj_fini(), plus a distinct -ENODEV return from
  makeJobsub()/makeJobstartinfo() when the table isn't ready
  (previously fell through to undefined behavior) (Req 10.2).
- Verified (test) MOVE_PTR fields are always NULL on a freshly-built
  submission payload -- no code change needed (Req 10.3).
- Fixed job_destroy()'s accounts leak via the existing
  free_job_account() helper, replacing the stale FIXME (Req 10.4).
- Verified (inspection + test) struct Job's capnp schema has no
  ckptinfo field, so decode_Job/free_Job have nothing to leak there
  -- no code change needed (Req 10.5).

Independently re-verified: reviewer reproduced a pre-existing,
unrelated rtti-library leak-on-init-failure against a stashed
pre-task copy of jobj.c to confirm it predates this change.

Task 1.3 of job-array-single-wal-record.
2026-07-14 08:29:45 +00:00
6529dcf08f feat(ha): reference-counted ha_elect_ctx_t lifecycle (ha-elect-ctx-lifecycle)
Replace the deliberate election-carrier leak with an atomic refcount:
a bias reference held by ha->elect_ctx plus one reference per captured
vote endpoint, released via a new endpoint_t.on_fini teardown-completion
hook (fired once, before ep->data is freed). Both real-election and
PreVote capture sites (raft_cb_send_vote_request,
raft_cb_send_pre_vote_request) increment on capture; both publish sites
(ha_start_election_async, ha_start_prevote_async) now atomic_exchange
the outgoing carrier and release it instead of leaking or freeing
outright, so a still-outstanding vote endpoint keeps the superseded
carrier alive until its own teardown completes.

Verified: 758+ controller tests green, isolated LeakSanitizer run shows
zero ha_elect_ctx_t leaks across many election/PreVote rounds, no
premature-free/UAF under supersession, and a live 10-round HA soak
(36 jobs, zero loss, all 3 nodes served as primary).
2026-07-12 04:25:42 +00:00
0941cf9649 fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss
Found via extended chaotic /xce-test-ha soaking on top of the bug-1419 fix
(same session). Both bugs independently caused the 10-round chaotic soak to
fail; sequential and chaotic now pass cleanly and repeatably.

1. Term-desync liveness deadlock (new): a standby that loses a near-
   simultaneous election race to a peer can legitimately end up at a HIGHER
   term than the peer that actually won and became primary. The primary's
   now-stale-term AppendEntries/heartbeats are correctly rejected by the raft
   library, but every one of those rejected messages was ALSO resetting the
   standby's own liveness deadline (ha_on_heartbeat_received), so its
   liveness timeout never fired and it could never start a fresh,
   higher-term election to let the cluster self-heal — a permanent stall.
   Fix: ha_msg_term_is_stale() gates the liveness reset (and
   ha_discover_primary step-down) on the incoming message's own term stamp,
   skipping it for provably-stale messages, in cluster.c's wal_catchup_entry
   and nop/heartbeat handlers.

2. bug-847 (root-caused 2026-06-30, never fixed): a per-sub-entry `payload`
   Data blob in a multi-entry WAL catch-up batch loses its capnp pointer once
   the message crosses a segment boundary — the receiver decodes
   raw_payload_len=0 for entries at/after the boundary and silently persists
   empty WAL entries (structurally intact: correct LSN/term/CRC chain, empty
   content). Only surfaces when a node rejoins and catches up a real batch,
   which is why it needed a live chaotic soak (not the 2-entry unit test) to
   reproduce. Fix: carry every sub-entry's payload concatenated in ONE
   top-level batchPayloads blob instead of one Data blob per entry — a
   single top-level blob does not exhibit the cross-segment loss. Sliced
   back apart on decode using each sub-entry's own wirePayloadLen, a plain
   scalar that (like lsn/entryType/term) decodes via the generic generated
   decode path and was never subject to the bug. Confirmed via targeted
   diagnostic logging that the corrupted entry arrives already-empty over
   the wire (not a receive-side race), and via a faithful reproduction at
   live JOB_SUBMIT payload size (5×3560B) after the original 2-entry/
   14×872B tests turned out not to cross a segment boundary in this
   environment.

Incidental fixes found and fixed along the way:
- A real self-deadlock (persist_entry_term() re-acquiring p->lock from
  inside a caller that already held it) introduced by an initial, ultimately
  unnecessary WAL-read locking fix attempted before bug-847 was pinned down;
  fixed via a new persist_entry_term_locked() lock-free counterpart. Left in
  place as a genuine (if incidental) hardening of ha_stream_wal_to_standby's
  read consistency.
- .claude/skills/xce-test-ha/SKILL.md's `context: fork` frontmatter caused
  the skill's own forked sub-session teardown to SIGHUP the whole
  xce_controller process group mid-soak, killing all 3 nodes silently with
  no error logged, whenever the soak was launched via the Skill tool itself
  (setsid/nohup inside run-ha-soak.sh only protects against a controlling
  terminal's own SIGHUP, not a parent forked-session's teardown). Removed;
  added explicit guidance to launch long soaks via direct Bash from the main
  session instead.

Verified: full 78-group unit regression sweep clean (0 failures) after each
change; two consecutive clean 10-round chaotic soaks and one clean 10-round
sequential soak on the final binary (zero committed divergence, zero job
loss, all 3 nodes led in both modes).
2026-07-08 15:21:24 +00:00
5f98d1b0ef feat(raft-apply-decoupling): task 7.1 apply-ordering conformance assertions
Extend raft_sim.c's deterministic TLA+-conformance harness with two
fault-injectable checks: no reaction before commit, and idempotent
re-apply of an already-committed entry. Wired as dedicated tests only,
not into raft_sim_check_all's fixed four-invariant contract (owned by
a separate spec). RED/GREEN independently verified by adversarial
review; zero regression to existing scenario functions.
2026-07-05 05:25:32 +00:00
78e8ec817e fix: close bug-847 (un-ignore fixed reproducer), confirm bug-1307 fixed via soak
bug-847 (catch-up WAL batch drops payloads across capnp segment boundaries):
its purpose-built RED reproducer, test_wal_batch_roundtrip_large_payloads
(14 entries x 872-byte payloads, 12.2KB total, genuinely spans multiple
4096-byte capnp segments per capn-malloc.c's create()), was still
TEST_IGNORE_MESSAGE'd. Un-ignoring it and running under ASAN with leak
detection: clean pass, byte-exact payloads, zero leaks, 2/2 runs. Traced the
vendored c-capnproto far-pointer machinery (write_ptr/capn_setp in capn.c)
to confirm the cross-segment struct->Data reference path is correct by
construction — whatever regressed this already got fixed elsewhere (library
bump or an incidental request.c change); the ignore was just never removed.
No production code changed.

bug-1307 (task-5.7 election-jitter data-safety regression: 7 divergences +
4 job losses over 6 chaotic rounds): re-ran the exact reproducing scenario
(6-round chaotic soak, real ext4, ASAN binaries) — 21 jobs, 0 divergence,
0 job loss, clean PASS. Confirms the raft-ha-core-safety Phase 5 work
resolved it.

Both bugs were the last 2 of the 7 flagged in the prior staleness sweep;
all 7 are now closed (6 were already fixed and undetected, 1 needed the
test un-ignored).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
2026-07-03 03:13:29 +00:00
9318f9977e fix: close bug-1401 ctrlstatus leak, bug-1400 jobsub leak; harden bug-1396 harness
bug-1401: xce_free_ctrlstatus(s) was defined as free_CtrlStatus(s), which
only frees the fields inside xce_ctrlstatus_t. The real client reply path
(decode_CtrlStatus_ptr) calloc's the struct itself, so every ctrlstatus call
leaked sizeof(xce_ctrlstatus_t) (120 bytes). Existing unit tests never caught
it because they built/freed the struct via the raw stack-decode + bare
free_CtrlStatus() helpers, never exercising the actual client macro. Fixed
by switching the macro to free_CtrlStatus_ptr(&(s)) and added a regression
test that exercises the real heap-decode path; RED confirmed the exact
120-byte/1-alloc LeakSanitizer signature before the fix.

bug-1400: the originally reported SEGV in cmd_jobsub on malformed positional
input is stale — NULLSTR guards added 2025-11-05 already reject it cleanly.
The real residual defect was submit.start_info leaking on all 4 early-return
validation paths; fixed by routing them through a shared `goto cleanup`
(matches the existing goto-clean idiom in queue_ctrl).

bug-1396 (job-count anomaly after cold-restart-all, not yet root-caused):
instrumented test-commit-durability.sh's assert_job_count() to dump the live
joblist and ACKED_FILE ledger on any count mismatch instead of only the
count, per the bug's own recorded next step.

Verified: controller ASAN build + full controller_unit_test green; lib
xcelib_test 285/285 green, 0 ASAN leaks.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
2026-07-03 02:40:52 +00:00
5cb36794d2 fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389)
ha_send_snapshot_to_standby ignored the network send's return value in its
chunk loop, silently truncating a transfer on a mid-stream error while still
logging "complete" (bug-1386). Fixing the silent drop exposed a second,
independent bug in the same pipeline: ha_handle_snapshot_chunk appended
received chunks in arrival order instead of at their wire-carried
chunk_offset, so out-of-order delivery across the vtask pool corrupted the
reassembled file even though every chunk's bytes were individually correct
(bug-1389). Fixed by checking every fallible call in the send loop and
switching the receiver to pwrite-at-offset with a count-based completion
tracker.

Adds snapverify/snapdiff xce_test commands (offline checkpoint-file
structural validator and byte-level comparator) used to diagnose bug-1389 and
kept as standing diagnostic tools for future snapshot-transfer triage.

g_drive_async_configpush_offload flipped back ON after both fixes, validated
via 6-round chaotic + 3/3 force-snapshot soak (raft-drive-async task 3.3).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
2026-07-03 02:40:12 +00:00
a36234a103 feat(raft-drive-async): LivenessAdmission pool reservation + commit fold (tasks 1.6/1.7)
vtaskmgr gains vtask_class_t (LIVENESS/LIVE_BLOCK/REQ_BLOCK) + a pool-level
blocking reservation: REQ_BLOCK fast-rejects -XCE_ERR_NOT_READY past
n_vcpus-R; LIVENESS never gated (deadlock-freedom). commit_submit folds its
inflight_waits onto the pool via commit_set_pool; production wired in
cluster.c phase2 (vtaskmgr_set_liveness_reserved(R) + commit_set_pool).
Lock order p->lock->ce->lock->blk_lock. Dormant until 3.1. Commit suites
38/30/22/38 green; lib 282/0, controller 196/0, ASAN clean. Phase 1 done.
2026-06-26 07:43:19 +00:00
0781c6c027 feat(raft-drive-async): bound peer connect (O_NONBLOCK+poll), close tasks 1.3/1.4/1.5
socket_connect sets O_NONBLOCK before connect and bounds the attempt with
poll(timeout); new connect_timeout_ms param (0=DEFAULT_CONNECT_TIMEOUT_MS);
all 5 callers updated (ha.c x3, client.c, xcectx.c). Revalidation Trigger:
shared netio helper signature changed. Broad build links all components;
xcelib_test 278/0, controller_unit_test 194/0. 1.4/1.5 unit tests satisfied
by the TDD output of 1.2/1.3 (test_slowlane.c, test_netio_bounded_connect.c).
2026-06-26 07:10:53 +00:00
6563adf8c6 docs(raft-sim-conformance): cosmetic cleanups (LOW review nits)
- sim_tla_action_name: drop brittle per-case 'raft.tla:NNN' line-number comments
  (some had drifted a few lines off; the return string IS the stable raft.tla
  Next-disjunct name, and the function header already cites raft.tla:454).
- test_sim_5_1_partition_heal: the per-follower convergence loop was vacuous —
  it bounded by commit_index, which stays 0 for followers under the leader-only
  commit model, so the loop never ran. Bound by log_count instead (followers DO
  advance the replicated log); Log Matching still guarantees the shared-prefix
  terms agree. Now non-vacuous; still passes.

Test/comment-only; no behavior change. Lib suite Ok:1 Fail:0.
2026-06-25 04:38:43 +00:00
95629797f8 feat(raft-sim-conformance): optional TLC-consumable trace export, default-off (task 8.1) 2026-06-24 16:49:36 +00:00
90197f8595 test(raft-sim-conformance): negative test proving oracle teeth, both directions (task 7.2) 2026-06-24 16:30:20 +00:00
62523892d9 feat(raft-sim-conformance): runtime fault-injection switch (G1 term-stamp, G3 truncate) (task 7.1) 2026-06-24 16:15:13 +00:00
47f68032b7 test(raft-sim-conformance): randomized M-seed sweep + all scenarios, zero violations (task 6.1) 2026-06-24 15:59:22 +00:00
a2933c274d feat(raft-sim-conformance): stale-rejoin scenario, bug-851 G5 term adoption via AE (task 5.5) 2026-06-24 15:50:13 +00:00
b1f289d2cb feat(raft-sim-conformance): Figure-8 commit-safety scenario + restore dropped 5.3 impl (task 5.4)
Adds raft_sim_scenario_figure8: a previous-term entry on a quorum under a
new-term leader is NOT committed until a current-term entry commits (library-
gated by raft_advance_commit; verified by a mutation test). Also restores the
raft_sim_scenario_drop_dup_reorder implementation that commit e445404 (task 5.3)
silently dropped from raft_sim.c (a stray git checkout reverted the uncommitted
impl), which had left the lib build broken with an undefined reference.

Build green: test_sim_5_3_drop_dup_reorder + test_sim_5_4_figure8 both PASS.
2026-06-24 15:38:01 +00:00
e445404439 feat(raft-sim-conformance): drop/duplicate/reorder adversarial scenario (task 5.3) 2026-06-24 15:09:31 +00:00
6976392d77 feat(raft-sim-conformance): restart-churn adversarial scenario (task 5.2) 2026-06-24 14:47:25 +00:00
7d0964dd16 feat(raft-sim-conformance): partition/heal adversarial scenario (task 5.1) 2026-06-24 14:34:22 +00:00
b844e75f3b feat(raft-sim-conformance): raft_sim_run + deterministic replay (task 4.4) 2026-06-24 14:14:15 +00:00
5f67be4fa3 test(raft-sim-conformance): conformance Restart persistence test (task 4.3) 2026-06-24 14:05:19 +00:00
b5b7e01b92 fix(raft): vote-reply per-peer dedup + stale-term drop (bug-1095/1096)
raft_on_vote_reply double-counted duplicated vote-responses (counter ignored
from_idx) and failed to drop stale-term replies, allowing two leaders at the
same term (bug-851 split-brain family). Mirror the existing pre-vote per-peer
dedup pattern (new vote_replies[] set) and add DropStaleResponse (term <
currentTerm). Surfaced by the raft-sim-conformance oracle (task 4.2).

47 raft / 264 overall lib tests pass.
2026-06-24 13:56:24 +00:00
e15bbf8190 feat(raft-sim-conformance): seeded scheduler raft_sim_step + per-step invariant check (task 4.2)
Harness surfaced a real upstream raft.c split-brain bug (bug-1095/1096,
bug-851 family) in raft_on_vote_reply: counter-based vote tally double-counts
a duplicated vote-response and no stale-term drop -> two leaders same term.
Out of this spec's boundary (raft.c protocol); recorded as Implementation Note.
2026-06-24 13:37:03 +00:00
90e7e9c999 feat(raft-sim-conformance): action trace recorder + raft_sim_dump_trace (task 4.1) 2026-06-24 13:10:31 +00:00
f0be46fb32 feat(raft-sim-conformance): raft_sim_check_all first-failure composition (task 3.5) 2026-06-24 13:00:10 +00:00
a38d9a7271 feat(raft-sim-conformance): State Machine Safety invariant checker (task 3.4) 2026-06-24 12:51:58 +00:00
63a9740c85 feat(raft-sim-conformance): Leader Completeness invariant checker (task 3.3) 2026-06-24 12:44:18 +00:00
d9b5fd51e0 test(raft-sim-conformance): focused Log Matching checker test for R3.2 (task 3.2) 2026-06-24 12:35:36 +00:00
25b14a6dca feat(raft-sim-conformance): Election Safety invariant checker (task 3.1) 2026-06-24 12:28:16 +00:00
b450e60d13 feat(raft-sim-conformance): bag-only Drop/Duplicate/Reorder ops (task 2.2) 2026-06-24 12:19:15 +00:00
ae4495104a feat(raft-sim-conformance): bag push/deliver routes vote+AE RPCs through library handlers (task 2.1) 2026-06-24 12:10:11 +00:00
631854303c feat(raft-sim-conformance): seeded PRNG + in-memory adapter callbacks + conformance init (task 1.2) 2026-06-24 11:55:11 +00:00
35c012ddc2 feat(raft-sim-conformance): add conformance-harness data types to raft.h (task 1.1) 2026-06-24 11:40:38 +00:00
8da8fd03dd fix(libxce): bug-1077 ctrlstatus crash from shared reply-union type confusion
ctx->reply.u is a single untagged union shared across opcodes (debugmsg /
get(dataset_t*) / jobid / clusterconf / ctrlstatus alias the same 8 bytes).
Commit 61e1391 added speculative free-guards
`if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(...)` (and the clusterconf
twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted
reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer)
because xce_get_with_condition/xce_resval only cleared the slot on the success
path, not the xce_get_req<0 / cbuf_get<0 early returns. The next ctrlstatus
reply (worker thread) freed that foreign pointer as a CtrlStatus -> ASan
SEGV / stack-use-after-return in free_CtrlStatus (controller.capnp.c:11638).

Fix (root cause, both layers):
- Revert the unsafe free-guards in xce_handle_reply: MOVE_PTR overwrite only,
  never free/deref an untagged union slot by type.
- Clear ctx->reply.u.get = NULL on ALL exit paths of xce_get_with_condition
  and xce_resval via a single `goto done` cleanup (forget-proof; the bug was
  one forgotten failover path). Also drops the now-unused controller.capnp.h.
- Defense-in-depth: handle_get_reply drops a reply when the slot is NULL.

Regression: expose xce_handle_reply and add
test_handle_reply_ctrlstatus_does_not_free_foreign_slot, which poisons the
shared slot with a foreign heap value and proves the ctrlstatus reply handler
does not free it (RED reproduced the exact free_CtrlStatus_ptr double-free;
GREEN after fix). xcelib_test 244/0, controller_unit_test 166/0, all binaries
build clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
2026-06-24 09:44:43 +00:00
61e1391cd0 fix bug-1060 bug-1067 2026-06-24 17:05:43 +08:00
62a927b419 test(raft-commit-safety): simulator invariant tests (tasks 7.1-7.4)
7.1 Figure-8 commit refusal (G4): prior-term entry at a genuine quorum HELD (only the
   current-term gate keeps commit_index at 0), then a current-term entry advances over
   both (indirect commit) with one fire. Closes the Req 6.3 two-phase coverage.
7.2 Leader Completeness: committed prefix on {0,1}; node 0 fails; the stale node 2 is
   DENIED by the freshness gate so only the up-to-date node 1 can win — and it holds the
   full committed prefix (term+bytes).
7.3 freshness lexicographic (vote + prevote): stale-term-higher-index DENIED,
   fresher-term-lower-index GRANTED, tie decided by index; prevote side-effect-free.
7.4 on_commit_advanced trigger: fire-count tracks STRICT advances — held(0)/advance(1)/
   no-op-redrive(1)/advance(2)/non-leader(2).

Test-only (sim_set_log_entry local helper); reviewed in-parent (transient API 529 on the
review subagent). ASAN green (243 tests).
2026-06-22 00:41:11 +00:00
56485e2fa3 feat(raft-commit-safety): (lastLogTerm,lastLogIndex) vote+prevote freshness (tasks 1.3, 4.1-4.2, 5.1-5.2, 6.2)
logOk (raft.tla:285, closes G6): local_last_log_term = log_len>0 ? on_get_entry_term(log_len):0;
logOk = cand_last_log_term > local || (== && cand_last_log_index >= log_len). Lexicographic:
term dominates, tie decided by index. Shared raft_log_ok helper.

Vote (4.2): logOk deny inserted after stale-term/UpdateTerm/role-gate/one-vote-per-term and
BEFORE on_vote_eligible (denies even when the app gate would grant); persist-before-grant intact.
PreVote (5.2): identical logOk, fully side-effect-free (no term/vote/role/commit mutation, no
on_persist). Signatures extended (1.3) on raft_on_vote_request/pre_vote_request; all 39 lib call
sites + sim routing updated; sim captures the candidate's send-time last-log tuple (raft_sim_rpc_t).
Empty-log call sites pass (0,0) → logOk true → existing outcomes preserved. 9 freshness tests.

NOTE: controller cluster.c vote callers still use the old signatures and will NOT compile — an
accepted, design-documented revalidation trigger; the controller adapter plumbing is owned by the
downstream ha-term-safety spec (to be implemented next). Library suite ASAN green (239 tests).
2026-06-22 00:14:26 +00:00
195f660f9a feat(raft-commit-safety): raft_advance_commit Figure-8 current-term rule (tasks 3.1-3.3, 6.1)
AdvanceCommitIndex (raft.tla:259/270, closes G4): leader-only (-EPERM); Max(agreeIndexes)
= highest N with a quorum ((n_peers/2)+1) of match_index>=N, self counted as log_len;
scan log_len downward. CURRENT-TERM RULE: advance commit_index to N only when
on_get_entry_term(N)==s->term, else HOLD (Figure-8 safety — never commit a previous-term
entry on replication count alone; a later current-term commit indirectly commits it).
Monotonic; fires on_commit_advanced once per strict advance; commit_index<=log_len.

3.3: raft_on_append_entries_reply left untouched — verified it never advances commit
(separation, Req 2.6). 6.1: sim commit-driver helper raft_sim_advance_commit added.

Req 6.3 two-phase indirect-commit transition covered end-to-end by task 7.1 (sim).
ASAN green (230 tests).
2026-06-21 23:27:12 +00:00
3ba6f03a9e feat(raft-commit-safety): commit_index state + on_commit_advanced callback + init (tasks 1.1, 1.2, 2.1)
1.1: raft_state_t.commit_index (monotonic, ==WAL LSN, <=log_len, advanced only via
raft_advance_commit, untouched by AE reply) + raft_config_t.initial_commit_index.
1.2: on_commit_advanced(ctx, commit_index) REQUIRED callback — fired only on strict
advance, a TRIGGER for the controller commit barrier (never does durability itself).
2.1: raft_init requires it (-EINVAL if NULL) + restores commit_index from
initial_commit_index. raft_advance_commit prototype declared (impl in 3.2).

Coupled callback wiring to keep the required-callback ripple green: sim wires
sim_on_commit_advanced (+per-node commit_index mirror & fire-count), all test cbs
builders set it, raft_sim_restart_node restores the durable commit position. Vote
signatures unchanged; no commit-advance logic yet. ASAN green (224 tests).
2026-06-21 23:11:17 +00:00
3d6dc9cc56 test(raft-log-replication): close Req 8.3 node-restart with sim restart primitive
raft_sim_restart_node models a process restart: the durable in-memory log[]
(WAL) and persisted (term, voted_for) survive, while volatile consensus state
is cleared by re-running raft_init (returns the node to FOLLOWER). Callbacks,
ctx, idx, and partition flag are preserved.

test_sim_9_5_node_restart_resumes_replication restarts a follower mid-replication
(partial 2-of-3 log), asserts the log survived (load-bearing: log_count==2 would
fail if wiped), role/term restored, then replication resumes and converges with
Log Matching. Closes the 8.3 'node restart while replication proceeds' sub-clause
(8/8 requirement sections now fully covered). ASAN green (220 tests).
2026-06-21 21:49:20 +00:00
9c1a034aa8 test(raft-log-replication): simulator invariant tests (tasks 9.1-9.4)
9.1 replication + partition catch-up: 3-entry quorum replication then Log Matching;
   isolate a follower, heal, assert next_index walks back, catch-up, Log Matching.
9.2 divergent-suffix truncation (G3): follower with a same-index/different-term tail
   converges via conflict-truncate; byte-for-byte prefix equality + Log Matching.
9.3 drop/dup/reorder convergence: dropped AE recovered; duplicate already-have AE is
   a no-op; genuine out-of-order delivery (entry-3 AE before entry-2) rejected via the
   logOk gap with no phantom gap; converges + Log Matching (Req 8.3).
9.4 chaotic-churn G5/bug-851 regression: stale-term node rejoins under a higher-term
   leader, adopts the term via the AE/heartbeat UpdateTerm path (load-bearing: would
   fail as stale-reject if UpdateTerm-on-AE regressed), voted_for cleared, no second
   leader, and a post-converge election campaigns strictly above the old term (cannot
   reuse). Log Matching holds throughout.

Review-applied: genuine reorder in 9.3; active re-election + voted_for assertions in
9.4. ASAN green (219 tests).
2026-06-21 17:06:05 +00:00
6ff07e5843 feat(raft-log-replication): Log Matching assertion helper (task 8.3)
raft_sim_check_log_matching: for every node pair and every shared index i, if
both entries at i have equal term, assert the whole prefix 1..i is term-and-byte
identical (Req 8.4, raft.tla Log Matching). Read-only, framework-agnostic (returns
1/0), bounded. Self-tests: pass on a converged cluster, fail on a hand-divergent
pair sharing (i=2,term=7) but differing at the index-1 prefix. ASAN green.
2026-06-21 16:47:49 +00:00
0e8c6ed05a feat(raft-log-replication): simulator AppendEntries delivery (task 8.2)
raft_sim_deliver_append_entries drains the AE queue, routes each buffered AE to
the (non-partitioned) destination via raft_on_append_entries_request, and routes
the result back to the source via raft_on_append_entries_reply — mirroring the
vote-RPC routing; partitioned endpoints are skipped.

Review-applied (HIGH): raft_sim_ae_t now captures the leader's SEND-time term
(leader_term) at buffer time and delivery uses it instead of the live source
state.term — without this, reorder/churn delivery (9.3/9.4) would inject the
wrong term and defeat UpdateTerm/stale-drop (G5). Smoke test asserts replication
to a connected follower + isolation of a partitioned one. ASAN green.
2026-06-21 16:40:11 +00:00
eeb95591ae feat(raft-log-replication): raft_on_append_entries_reply (tasks 7.1, 7.2)
HandleAppendEntriesResponse (raft.tla:393): stale-term drop first (applied only
when reply_term==s->term, raft.tla:415, Req 5.4 — both lower and higher dropped);
success -> next_index[peer]=match_index+1, match_index[peer]=match_index (Req
5.1-5.2); failure -> next_index[peer]=Max(next-1,1) floored, match_index unchanged
(Req 5.3); no commit advance. uint64 underflow blocked by the >1 floor guard.
3 tests incl. floor-at-1 and lower/higher stale. ASAN green.
2026-06-21 16:26:42 +00:00
3c5185a5c3 feat(raft-log-replication): raft_on_append_entries_request UpdateTerm+logOk+accept (tasks 5.1-5.2, 6.1-6.3)
Complete HandleAppendEntriesRequest (raft.tla:327) + UpdateTerm (raft.tla:406),
implemented as one function to avoid a misleading partial handler. Fixed step
order: UpdateTerm FIRST on every RPC incl. heartbeat (G5, strictly-greater,
clears voted_for, persists, leader step-down fires on_become_follower) ->
stale reject -> candidate step-down -> logOk (G2) -> reject if !logOk ->
accept (already-have / conflict-truncate-then-repersist (G3) / append).
Never advances commit, never does transport I/O.

Review-applied: logOk now includes the explicit prev_log_index <= log_len bound
(raft.tla:330 / Req 4.1) instead of relying solely on the 0-absent sentinel;
added a candidate-equal-term-logOk-fails step-ordering test. 15 tests, ASAN green.
2026-06-21 16:18:02 +00:00
a1aacc0389 feat(raft-log-replication): raft_send_append_entries AppendEntries emit (tasks 4.1, 4.2)
Leader-only emit (raft.tla AppendEntries:204, Req 3.5 — no-op off leader);
prev_log_index=next_index[j]-1, prev_log_term via on_get_entry_term (0/unqueried
at index 0, Req 3.2-3.3); n_entries=1 when next_index[j]<=log_len else 0 heartbeat
(Req 3.4); leader_commit opaque passthrough. Trace fires only on successful send
(consistency with raft_log_append) and non-leader RAFT_OK asserted — both from
review. 4 spy-based tests. ASAN green.
2026-06-21 16:01:17 +00:00
e5b9b83955 feat(raft-log-replication): raft_log_append leader client-entry append (tasks 3.1, 3.2)
Leader-only (-EPERM for follower/candidate, no mutation/callback); index=
log_len+1, term=currentTerm (raft.tla ClientRequest:246, Req 2.1-2.4);
persist-before-act: on_persist_entry called once BEFORE log_len advances,
non-zero return propagated with no phantom entry (Req 2.5); no commit-index
advance, no transport I/O. Spy-based tests for all four branches. ASAN green.
2026-06-21 15:50:33 +00:00
fc87a662cd fix(raft): reject n_peers > RAFT_MAX_PEERS in raft_init
next_index[]/match_index[]/pre_vote_replies[] are fixed-size RAFT_MAX_PEERS
arrays indexed by peer; without an upper-bound check a caller passing
n_peers > 8 would OOB-write in the leader-progress and pre-vote loops.
Add the guard (returns -EINVAL, no mutation) + canary-checked test.

Latent since task 1.1; surfaced by task 2.2 review (first loop writes into
the new arrays). No current caller exceeds the bound (sim caps at 8, HA <= 4).
2026-06-21 15:42:37 +00:00
6ea89569fa feat(raft-log-replication): init leader progress on becoming leader (task 2.2)
next_index[j]=log_len+1, match_index[j]=0 for every peer at both win paths
(single-node RAFT_ELECTED_IMMEDIATELY in raft_start_election; multi-node
RAFT_BECAME_LEADER in raft_on_vote_reply) via shared raft_init_leader_progress
helper (raft.tla BecomeLeader, Req 3.1). Suite green under ASAN.
2026-06-21 15:40:26 +00:00