Lands the PeerCtrlDispatch machinery behind a DEFAULT-OFF flag
(g_drive_async_peerctrl_dispatch_enabled); production behavior is
UNCHANGED. peer_ctrl_msg_handler becomes a thin shim that takes
ownership of ep->fsm_in.pkt and either runs the routing body inline
(flag OFF, byte-for-byte the prior path) or dispatches it to the vtask
pool via vtask_enq_classified (flag ON). The 3 routers now take pkt as
an explicit param (no cross-thread ep->fsm_in.pkt re-read). Per-endpoint
arrival order is enforced by bind_vtask's endpoint_block_tasks.
Packet ownership: shim clears ep->fsm_in.pkt at dispatch (sole-owner
transfer); peer_ctrl_vtask_entry NULLs iod->pkt before the body frees
it once. New vtask_t.owns_io_pkt (default 0) lets taskq_item_killer free
the carried packet for an un-executed queued task at vtaskmgr_fini WITHOUT
double-freeing the libxce endpoint-owned path (bug-1199).
Adversarial review found two FLAG-ON hazards, documented with loud
comments (flag is OFF so production is unaffected), to be resolved before
task 3.4 flips the flag ON:
- bug-1197 (CRITICAL): raft_cb_become_follower endpoint_fini()s peer
vote-eps from a pool worker, racing the io_worker's read of ep->data;
needs iomgr_unmonitor before endpoint_fini in every pool-reachable
teardown.
- bug-1198 (HIGH): peer_ctrl_classify tags all work VTASK_LIVENESS, but
the CTRL_PRIMARY repl-ack path can run a blocking snapshot checkpoint
inline; resolved by task 3.2 (offload snapshot to SlowLane).
Per spec, 3.1 is not 'done' until the task 3.4 deadlock gate is green.
Tests: 3 new peerctrl tests; controller 196/196, lib 282/0; 0 net leaks.
Escalates the single-tick unit tests (15-17) to sustained liveness: with
one peer unreachable (reconnect parked in-flight on the lane for the whole
window), drives 3 consecutive heartbeat ticks against a 4-node cluster
(two survivors) and asserts every survivor keeps receiving heartbeats on
every tick (g_nop_calls == 2*3), each tick returns promptly (delaymgr
never stalled), the reconnect stays in-flight throughout, and is deduped
across all ticks (connect hook submitted exactly once — Req 2.5).
RED arm (offload flag OFF) proves the inline connect blocks the first
tick, so survivors stop getting heartbeats (g_nop_calls < 6). The flag
toggle is the RED<->GREEN switch; production is unchanged (test-only task).
test_ha_liveness suite 23 -> 25 tests; full controller suite 196/196 pass.
When a reconnect registers a new outbound control channel for peer X,
on_standby_connected now evicts any pre-existing standby_endpoints entry
for the same peer_idx (a promoted vote-ep or a stale prior channel),
keeping exactly one live outbound channel per peer (Req 2.6/4.4). The
stale ep is torn down OUTSIDE ha->lock via standby_ep_err_handler_primary
(which re-takes the lock) and reclaimed; resolution is strictly per-peer.
Also fixes endpoint leaks on the endpoint_get() calloc paths: every
endpoint_fini() error path now free(ep) (fini reclaims inner members
only), closing the task-2.1 GREEN-test leak masked by detect_leaks=0.
Tests 18-19 in test_ha_liveness.c assert single-survivor resolution and
per-peer isolation. 196/196 controller unit tests pass.
Heartbeat timer detects a disconnected peer, CAS-acquires a per-peer
single-flight flag, slowlane_submits the bounded connect, and returns — the
delaymgr never blocks on connect. Lane worker (reconnect_lane_fn) does the
bounded endpoint_connect + lock-guarded iomgr_monitor; flag cleared on all
terminal paths (incl. -EEXIST race fix). Default-ON kill-switch retains the
inline path for TDD-RED/fallback. Never self-terminates on failure. 2.2 adds
the inbound/outbound race + failure hardening (documented seam). 196/0 green.
vtaskmgr gains vtask_class_t (LIVENESS/LIVE_BLOCK/REQ_BLOCK) + a pool-level
blocking reservation: REQ_BLOCK fast-rejects -XCE_ERR_NOT_READY past
n_vcpus-R; LIVENESS never gated (deadlock-freedom). commit_submit folds its
inflight_waits onto the pool via commit_set_pool; production wired in
cluster.c phase2 (vtaskmgr_set_liveness_reserved(R) + commit_set_pool).
Lock order p->lock->ce->lock->blk_lock. Dormant until 3.1. Commit suites
38/30/22/38 green; lib 282/0, controller 196/0, ASAN clean. Phase 1 done.
socket_connect sets O_NONBLOCK before connect and bounds the attempt with
poll(timeout); new connect_timeout_ms param (0=DEFAULT_CONNECT_TIMEOUT_MS);
all 5 callers updated (ha.c x3, client.c, xcectx.c). Revalidation Trigger:
shared netio helper signature changed. Broad build links all components;
xcelib_test 278/0, controller_unit_test 194/0. 1.4/1.5 unit tests satisfied
by the TDD output of 1.2/1.3 (test_slowlane.c, test_netio_bounded_connect.c).
Dedicated-pthread pool isolated from the vtask pool; non-blocking slowlane_submit
(0 enqueued / -EEXIST deduped / -EAGAIN cap-full-deferred), admission cap on
running tasks, per-key in-flight dedup, run-exactly-once, drain+join fini. Wired
into controller lifecycle (dormant; consumers land in 2.x/3.x). 7 unit tests,
mutation-verified cap+dedup; full suite 194/0 green, 15x ASAN.
drive_async_liveness_reserved = max(1, min(n_peers-1, n_vcpus-1)) — single
source of truth for VTASK_LIVENESS_RESERVED (deadlock-freedom floor applied
last). New DEFAULT_CONNECT_TIMEOUT_MS / DEFAULT_SLOWLANE_* constants. Resolved
reservation logged at init; under-provisioned (n_vcpus<n_peers) clamp WARNs
loudly, never self-terminates. 8 unit cases, mutation-verified.
- sim_tla_action_name: drop brittle per-case 'raft.tla:NNN' line-number comments
(some had drifted a few lines off; the return string IS the stable raft.tla
Next-disjunct name, and the function header already cites raft.tla:454).
- test_sim_5_1_partition_heal: the per-follower convergence loop was vacuous —
it bounded by commit_index, which stays 0 for followers under the leader-only
commit model, so the loop never ran. Bound by log_count instead (followers DO
advance the replicated log); Log Matching still guarantees the shared-prefix
terms agree. Now non-vacuous; still passes.
Test/comment-only; no behavior change. Lib suite Ok:1 Fail:0.
test_demotion_replay_yields_committed_only flaked (~16% on overlayfs, but
fs-independently too). Two root causes, both test-harness artifacts (NOT product
bugs; production self-restart is execv on a real fs and never overlaps persist
contexts or runs an auto-committer during demotion):
1. single-node commit race: the fixture's live commit engine on_durable_advance
auto-commits on n_peers=1, advancing committed_lsn past the test's boundary so
the 'uncommitted' suffix became committed. Fix: detach on_durable_advance +
re-pin committed_lsn to the test's own committed_boundary, so the suffix stays
genuinely uncommitted (the test's premise).
2. overlayfs fd-incoherence: a fresh persist_t recovering over the same files
reads a stale post-ftruncate size on overlayfs. Fix: hybrid statfs gate —
real fs keeps the full fresh boot-recovery assertion; overlayfs asserts the
same property via the coherent live-context current_lsn.
Test-only; production truncate (ftruncate+fdatasync) untouched. 0/25 runs fail
(was 4/25 with the prior persist_stop-only attempt).
The bug-850/851 forward-gap contiguity guard (persist.c:839, added in 20e59d0b)
correctly rejects a catch-up entry whose lsn is ahead of the WAL tail (-EAGAIN).
7 test_wal_catchup_* tests in test_config_repl.c predate the guard and applied
non-contiguous LSNs (42, 10-12, 1000+) to a fresh persist (current_lsn=1),
failing deterministically. Seed current_lsn to each test's first applied LSN so
the in-order stream is contiguous (mirroring the real catch-up streamer); the
apply->advance path and all asserted semantics are unchanged. Test-only; the
production contiguity guard is untouched. 179 controller tests, 0 failures (x3).
Adds two controller-level regression tests driving the production raft_on_vote_reply
via ha->raft (test_ha_election.c): a duplicated vote-grant from one peer must not
double-count toward quorum (bug-1095) and a stale-term reply must be dropped
(bug-1096) — both asserted on role/promotion (no two leaders per term). Formally
absorbs bug-1095/1096 into the ha-term-safety spec (Implementation Note) mapping to
its Requirement 1 / Objective; the raft.c fix (b5b7e01) is owned there with
protocol-suite regression at both the libxce (test_vote_reply_*) and controller
(test_ha_election) levels. No production source changed.
Adds raft_sim_scenario_figure8: a previous-term entry on a quorum under a
new-term leader is NOT committed until a current-term entry commits (library-
gated by raft_advance_commit; verified by a mutation test). Also restores the
raft_sim_scenario_drop_dup_reorder implementation that commit e445404 (task 5.3)
silently dropped from raft_sim.c (a stray git checkout reverted the uncommitted
impl), which had left the lib build broken with an undefined reference.
Build green: test_sim_5_3_drop_dup_reorder + test_sim_5_4_figure8 both PASS.
raft_on_vote_reply double-counted duplicated vote-responses (counter ignored
from_idx) and failed to drop stale-term replies, allowing two leaders at the
same term (bug-851 split-brain family). Mirror the existing pre-vote per-peer
dedup pattern (new vote_replies[] set) and add DropStaleResponse (term <
currentTerm). Surfaced by the raft-sim-conformance oracle (task 4.2).
47 raft / 264 overall lib tests pass.
Harness surfaced a real upstream raft.c split-brain bug (bug-1095/1096,
bug-851 family) in raft_on_vote_reply: counter-based vote tally double-counts
a duplicated vote-response and no stale-term drop -> two leaders same term.
Out of this spec's boundary (raft.c protocol); recorded as Implementation Note.
Mermaid parse errors in design.md's diagrams:
- Literal `|` in flowchart node labels is read as an edge-label delimiter:
system-flow `term | 0` -> `term | 0`; checker `|{i:...}|` ->
`|{i:...}|` (entity renders as a literal `|`, meaning preserved).
- `;` is a mermaid statement separator and truncated the text it appeared in:
sequence msgs `commit_index += ; apply()` -> `commit_index advanced, apply()`
and `(R5); halt` -> `(R5), then halt`; flowchart label `responses;<br/>` ->
`responses,<br/>`.
Valid edge-label pipes (-->|violation|) and all `;` in prose/tables outside the
mermaid fences are unchanged. All three diagrams now render.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
Ran the live 3-node E2E (the last open task). Result:
PASS: 8 rounds, 30 jobs submitted, zero job loss; all 3 nodes led
(ctrl-1,ctrl-2,ctrl-3) (exit 0)
Live-covered: kill-primary -> promoted standby owns all confirmed jobs
(zero loss across 8 failovers, jobs 3->30, expect-job ok each round);
restart ex-primary -> converged rejoin, 3-of-3 each round, no phantom jobs;
deterministic leadership coverage (ctrl-3 forced via RequestVote-freshness
fence). The soak also served as the live regression for bug-1077 (~40
failover client round-trips, no SEGV) after rebuilding the controller (the
deployed binaries predated that fix).
Not exercised live (remain unit-covered, tasks 2/4/5): withhold-ACK barrier
single-outcome (test_demotion_convergence) and isolate/heal quorum-WARN
(test_convergence_verification). All 8 tasks now [x]; spec closed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
Add an in-situ NOTICE at the config-upgrade restart site
(do_upgrade_config_with_bytes, step 9). Today an upgrade commits the
config-change marker, then each member restarts and re-materializes its
post-upgrade state by replaying its own pre-upgrade WAL under the NEW config;
replay is config-dependent (lost_and_found re-homes orphaned entities), so
members can diverge in materialized state (bug-857). The notice records the
intended fix — materialize a full entity-bearing snapshot at the marker's
(term,lsn) boundary, embed the new config, and install the SAME snapshot on
every member via snapshot_transfer_s — and its hard dependency on
entity-bearing-snapshots. Comment-only; anchors to
.kiro/specs/config-upgrade-snapshot/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
ctx->reply.u is a single untagged union shared across opcodes (debugmsg /
get(dataset_t*) / jobid / clusterconf / ctrlstatus alias the same 8 bytes).
Commit 61e1391 added speculative free-guards
`if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(...)` (and the clusterconf
twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted
reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer)
because xce_get_with_condition/xce_resval only cleared the slot on the success
path, not the xce_get_req<0 / cbuf_get<0 early returns. The next ctrlstatus
reply (worker thread) freed that foreign pointer as a CtrlStatus -> ASan
SEGV / stack-use-after-return in free_CtrlStatus (controller.capnp.c:11638).
Fix (root cause, both layers):
- Revert the unsafe free-guards in xce_handle_reply: MOVE_PTR overwrite only,
never free/deref an untagged union slot by type.
- Clear ctx->reply.u.get = NULL on ALL exit paths of xce_get_with_condition
and xce_resval via a single `goto done` cleanup (forget-proof; the bug was
one forgotten failover path). Also drops the now-unused controller.capnp.h.
- Defense-in-depth: handle_get_reply drops a reply when the slot is NULL.
Regression: expose xce_handle_reply and add
test_handle_reply_ctrlstatus_does_not_free_foreign_slot, which poisons the
shared slot with a foreign heap value and proves the ctrlstatus reply handler
does not free it (RED reproduced the exact free_CtrlStatus_ptr double-free;
GREEN after fix). xcelib_test 244/0, controller_unit_test 166/0, all binaries
build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
Add an optional [dir] argument to the /xce-test-ha skill so the 3-node
soak cluster can be deployed to a caller-chosen path.
- run-ha-soak.sh: deploy dir resolves positional $1 > $XCE_HA_TOP > default;
relative paths absolutized in pure bash (this host's busybox realpath
lacks -m and would mangle nonexistent/relative paths).
- Fail-loud rm -rf guard: writes a .xce-ha-deploy sentinel at deploy time and
refuses to wipe a non-empty dir lacking it (protects a typo'd home/source
path); empty/nonexistent dirs stay safe to (re)deploy into.
- SKILL.md: document /xce-test-ha [rounds] [dir] and the safety semantics.
Verified: guard refuses an unmarked non-empty dir (dir survives, exit 1);
full 5-round soak into a custom dir PASS (18 jobs, zero job loss), cluster
landed in the named dir while the default dir was untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
The post-failover wait-primary/expect-job probes used the full masters list led
by the just-killed node, triggering libxce bug-854 (positional discovery can't
fail over from a dead-first master). Now they use a live-first list excluding the
killed primary.
NOTE: this exposes a deeper, separate post-failover AVAILABILITY issue (not data
loss): the new primary stays stuck in a catch-up loop with a lagging standby
(standby[0] lsn=5 < committed_lsn=6) and does not serve clients, so wait-primary
still times out. That is a catch-up/serving-readiness liveness concern orthogonal
to bug-828's committed-job DATA SAFETY, which is fixed and verified (walcompare
committed-prefix identical, jobs preserved on disk).
