Add io_classify_io_error and propagate endpoint/epoll errors through
endpoint_update_epoll and cbuf_get instead of swallowing them, so client
failover surfaces the real error rather than hanging. Adds netio arm-failure,
close-propagation, and request send-propagation tests.
Pre-existing uncommitted work, committed to establish a clean baseline before
ha-commit-durability implementation.
Desync FAILOVER re-elections by node priority (ha_election_stagger_ms /
ha_failover_stagger_fn) mirroring the startup stagger, with seed jitter and
async prevote start; allow prevote in voter roles through the dispatch gate.
Adds HA role and vote-gate tests (stagger desync by priority, prevote on dead
primary, stickiness-lapse grants).
Pre-existing uncommitted work, committed to establish a clean baseline before
ha-commit-durability implementation.
xce_failover_rotate reconnected with endpoint_fini+endpoint_init+endpoint_connect
but never re-bound ep->iomgr (reset by endpoint_init) nor re-armed the owner/
msg/err handlers + iomgr_monitor. endpoint_connect then dereferenced a NULL
iomgr (->ssl_ctx) → SEGV at netio.c:672 during live master failover. Added
xce_ctx_reconnect() in xcectx.c that mirrors xcecli_start's establishment
(without restarting iomgr/vtaskmgr or touching ctx->state); rotate delegates to
it. Found via live 3-node E2E. bug-610.
Live 3-node E2E revealed the reported 'Invalid argument' bug was still
reachable: after the primary dies, handle_disconnect leaves ctx.state=
XCE_CTX_ERR, so a FRESH xce_* call returns -EINVAL from the xceapi guard
before reaching a primary. The classifier mapped -EINVAL to TERMINAL, so the
loop bailed without rotating/recovering. decide() now detects the known-dead
ctx and treats it as a prompt reconnect cause (RETRY_NOW), surfacing the
distinct cluster-not-ready code on budget exhaustion, never -EINVAL.
Regression test: test_decide_known_dead_ctx_retries_not_terminal.
fsync_worker slept the full wal_sync_ms in one uninterruptible nanosleep()
before re-checking fsync_stop. persist_stop sets the flag and immediately
pthread_join()s, so the join blocked until the in-flight sleep returned —
up to a full wal_sync_ms per teardown. Tests set wal_sync_ms=5000 ("slow
fsync — no races"), so each persist_stop cost ~5s; across the ~7 suites
that use it the cumulative teardown time blew past the 300s controller-test
meson cap, killing the suite with SIGTERM.
Replace the nanosleep with pthread_cond_timedwait on a new fsync_mutex/
fsync_cond (added to persist_t, init in persist_init, destroyed in
persist_fini). persist_stop now sets fsync_stop and signals fsync_cond
under fsync_mutex before joining, so the worker wakes immediately. fdatasync
still runs under p->lock, released before re-acquiring fsync_mutex — no lock
nesting. The "commit: WAIT deadline expired" log lines were unrelated
intentional negative tests (50-300ms), not the cause.
Full controller suite now completes in ~10s (was >300s TIMEOUT); 0 failures.
Logged as bug-604.
- Mark the PreVote denial log-reason nit RESOLVED in ha-prevote tasks notes (47f7338).
- File ha-elect-ctx-lifecycle spec scaffold (spec.json + brief.md) capturing the
pre-existing, bounded, UAF-sensitive election-carrier leak and the refcount/
teardown-ordered-free approach across both election and PreVote paths. Deferred
by decision — not folded into ha-prevote to avoid a risky lifetime refactor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cluster.c denial-reason elimination ordered catch_up before fresh_heartbeat,
disagreeing with raft_cb_vote_eligible (which checks fresh-heartbeat-on-STANDBY
before catch-up). In the rare STANDBY catch-up + fresh-heartbeat overlap the log
could report catch_up while the gate denied on fresh_heartbeat. Expose the pure
read predicate ha_heartbeat_is_fresh and mirror the gate's exact order so the
logged reason can no longer disagree with the actual denial. Decision unchanged;
log-string fidelity only. prevote/vote-gate/failover-prevote suites green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four property-style raft tests proving the PreVote API preserves the safety
invariants: term/voted_for/role never change and on_persist never fires across
mixed begin/request/reply sequences (WON and LOST); a WON round creates no leader
(stays FOLLOWER); and the real-vote higher-term-adopt + step-down + quorum-elect
path is unchanged (bug-339/340 intact). Records the TLA+/checklist consistency
argument (RN-1) in test_raft.c and research.md, citing the tests as evidence.
Raft suite 123/0/0; zero production change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three scenario tests (zero production change): a stale-heartbeat standby GRANTS
(grant<->deny flip vs 5.2) and the won round advances to CTRL_ELECTING (PreVote
does not block legitimate failover); a heartbeat mid-round cancels and re-attaches
as standby with term unchanged; a lost round re-arms liveness and a later genuine
failure still wins (reusable, not one-shot).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two-angle scenario test (zero production change): a healthy PRIMARY and a
fresh-heartbeat STANDBY both DENY a higher-prospective-term preVoteRequest
without adopting the term or demoting; and a resumed standby whose round both
peers deny gets no majority -> stays CTRL_STANDBY/FOLLOWER, term byte-for-byte
unchanged, liveness re-armed. Regresses if the disruptive-server bug returns.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
peer_ctrl_msg_handler gains a CTRL_STANDBY + pre_vote_in_progress branch that
decodes an incoming preVoteReply (no term adoption) and feeds it to
pre_vote_reply_dispatch with the absolute peer index, paralleling the real-vote
reply path. Gives pre_vote_reply_dispatch its production caller (drops the used
attribute). STANDBY-no-round still drops; ELECTING/PRIMARY routing unchanged.
PreVote now drives rounds to a decision end-to-end over the wire.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ha_on_primary_ctrl_fd_rdhup + liveness_poll_fn now begin a PreVote round via
ha_begin_failover -> ha_start_prevote_async instead of electing directly; WAL
truncate + CTRL_ELECTING deferred to PreVote WON (ha_enter_real_election).
Connection teardown + config-less voter guard preserved at the trigger.
ha_on_heartbeat_received cancels an in-progress round and re-attaches as standby
with no term change. Single-node immediate-WON synchronously enters the real
election. PreVote now live; full controller suite 563 pass / 0 fail; bug-339/340
preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add OPCode_preVoteRequest case in cluster.c, mirroring the real-vote case:
freshness floor + reused eligibility/stickiness gate (raft_on_pre_vote_request),
reply over preVoteReply with the voter's UNCHANGED term (no adoption), denial
logging with candidate id + reason. Pure insertion; real-vote case untouched.
6 new vote-gate tests assert deny-while-leader-alive, grant-when-lapsed, and the
no-mutation/term-unchanged contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raft_on_pre_vote_reply: per-peer-deduped tally returning WON at quorum
((n_peers/2)+1), LOST when all non-self peers reply without quorum, else
PENDING; bounds-guards peer_idx. Never fires become-leader/follower, never
mutates term/voted_for/role, never persists. 8 RED-first tests (3- and 5-node,
dedup, out-of-range, purity). Raft PreVote pure core complete.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raft_begin_pre_vote: compute prospective term (term+1) locally without
persisting, arm the pre_vote_* tally with a self-grant, NULL-safely probe each
non-self peer, return RAFT_PRE_VOTE_WON for single-node else RAFT_OK. Stays
FOLLOWER, schedules no timeout, mutates no persistent state. 6 RED-first tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raft_on_pre_vote_request: deny on stale term, else reuse on_vote_eligible
(primary/heartbeat-stickiness/catch-up gates). Never mutates term/voted_for/
role and never persists or adopts a higher term. 6 RED-first raft unit tests
assert the purity invariant; libxce suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append preVoteRequest @38 / preVoteReply @39 to the OPCode enum, reusing the
existing RequestVoteRequest/RequestVoteReply payload structs and union arms
(no new structs, no isPreVote flag). Keeps the real-vote wire path unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cluster_start (which set c->running) is orphaned on the HA myapp_run path, so
the running flag was never set on HA nodes — check_controller() treated the
primary as shutting down and zeroed joblist/workerlist reply chunks. Own the
flag lifecycle in cluster_run, symmetric with the c->running=0 on shutdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two distinct HA livelocks, both verified on a live 3-node ASAN deploy
(kill+restart of primary and standby, plus SIGSTOP wedge injection).
bug-339 — election freshness-floor asymmetry (RDHUP failover path):
After the primary dies the two survivors could never elect a leader — both
only ever became follower while terms climbed in lockstep. The RDHUP candidate
advertised c->persist.committed_lsn, which a standby never advances (only the
primary commit engine writes it), while every voter computed its freshness floor
from max(applied_lsn, last_known_committed_lsn). So each survivor advertised a
stale-low LSN against a high voter floor and the freshness gate denied on both
sides. Fix: compute the candidate LSN via ha_vote_self_freshness so the
advertised value equals the voter floor (ha_on_primary_ctrl_fd_rdhup); also route
the boot-path election (myapp_run) through the same helper for symmetry.
bug-340 — wedged standby endpoint suppressed reconnect (rejoin path):
A killed/restarted peer's outbound endpoint on the primary could wedge (epoll
re-arm ENOENT) yet stay in standby_endpoints, so primary_heartbeat_fn's reconnect
scan treated the peer as still-connected and never re-established a ctrlhello
channel — the restarted node never heard from the primary and re-elected every
heartbeat_timeout (term climbing unbounded). xce_nop_req returns success on
enqueue even when the send can't progress, so the heartbeat layer never noticed.
Fix: primary_heartbeat_fn now evicts a standby that was responsive once
(ts_last_in > 0) but has been silent past heartbeat_timeout — symmetric with the
standby's own liveness poll — and the reconnect scan re-establishes the channel.
Uses existing endpoint_t.ts_last_in (no libnetio change).
Tests (RED->GREEN):
- test_ha_rdhup_integration: candidate advertises freshness floor, not stale committed_lsn
- test_ha_liveness: evicts_silent_standby + keeps_live/keeps_unconfirmed (no false eviction)
Live symptom: jobsub intermittently failed with 'Connection timed out' even
though the cluster had quorum. Root cause: the synchronous commit barrier waits
only DEFAULT_SYNC_TIMEOUT_MS=50ms for durable majority replication, and that was
not configurable. 50ms is too tight for durable sync majority replication —
especially since standbys now fdatasync each replicated entry before ACK — so a
multi-entry jobsub's later WAL entry intermittently misses the window:
commit: WAIT deadline expired without commit (lsn=3 committed_lsn=2 timeout_ms=50)
-> persist hook returns -ETIMEDOUT -> handle_jobsubmit rejects -> client ETIMEDOUT.
Fix:
- Raise DEFAULT_SYNC_TIMEOUT_MS 50 -> 500 (config.h) — realistic for durable sync
majority replication, still bounded for failure detection.
- Add a --sync-timeout-ms controller flag threaded arguments -> myconf ->
persist pcfg (override applied before persist_init); 0 = use default, positive
overrides — so operators can tune per environment.
TDD: literal-pinned persist_config_defaults().sync_timeout_ms == 500. All
timeout-sensitive tests already pass an explicit timeout_ms (none rely on the
default), so none are slowed; full controller suite green under ASAN. No
commit-engine/barrier/replication/wire change beyond the default value + plumbing.
bug-561.
