Closes the two genuine gaps beyond task 10.1's own coverage (standby and
disconnected-scheduler rejection were already tested):
- Forward timeout: adds an XCE_TESTING-only timeout-override seam
(SCHEDADMIN_TIMEOUT_MS) in cluster.c, verified inert in production
builds (-DXCE_TESTING absent from xce_controller's target and every
project-wide arg; the non-test branch is a bare 5000L literal).
test_schedadmin_forward_timeout drives the real 5s (shrunk to 200ms
under test) pthread_cond_timedwait to genuine expiry and asserts the
exact {ok=false, error="request timeout"} shape from AC 10.
- Round-trip coverage: two more outcome-branch tests
(rpc_status_only, business_error) using the same real xce_encode/
xce_decode functions production uses. A literal two-binary
controller<->scheduler integration was judged out of scope (separate
meson targets, incompatible fixtures) since task 9.4's own
test_admin_request.c already white-box-drives the real scheduler
handler rigorously.
Controller suite 3/3 (1093/1093 assertions, up from 1090), scheduler
26/26 unaffected, clang-format clean, zero diff-hygiene noise (git diff
matches git diff -w for cluster.c).
Adds the spec's one and only src/controller/ change: a new
case OPCode_schedAdmin in myreq_handler() (cluster.c) that mirrors the
existing OPCode_upgradeConfig case -- primary-only gate, explicit reply
on every path, stateless pass-through with no WAL entry.
Forwards to the scheduler via a new xce_schedadmin_req() (async send,
mirrors xce_fetchfile_req) and blocks the vtask thread on a condvar with
a 5s deadline while holding c->sched.lock for the round-trip, matching
this spec's own research-phase design decision (parity with
OPCode_upgradeConfig's no-RBAC posture, blocking wait accepted for
low-frequency admin traffic). A new branch in the existing
OPCode_reply case demuxes the scheduler's SchedAdminReply to a
process-global single-slot pending-waiter mechanism, serialized by
c->sched.lock, matching job.c's existing fetchfile-proxy pattern.
Adversarial review independently traced double-free/cleanup, the
timeout/signal race, and reply-opcode demux correctness against source;
found and a follow-up round closed a narrow-window leak via a carefully
lock-ordered unlock->unregister->relock sequence (avoiding an AB-BA
inversion against schedadmin_signal()'s lock acquisition order).
New test_sched_admin_dispatch.c drives the real myreq_handler via the
existing myreq_handler_for_test seam: not-primary rejection (proven via
a poison-pointer sched.ep that would crash if forwarding were ever
attempted), not-connected rejection, and a genuine two-thread round-trip
test exercising the real signal/demux path.
Controller suite 3/3 (1090/1090 assertions), scheduler suite 26/26
unaffected, clang-format clean.
myreq_handler's ReplConfigPushReply branch now routes cack->term
through raft_on_higher_term_observed (same entry point the
heartbeat-reply and WAL-NACK vectors use), regardless of accept/reject,
without altering the existing accept/reject branching.
Adds myreq_handler_for_test (XCE_TESTING-only) to drive the real
dispatcher in tests, following the peer_ctrl_msg_handler_for_test
precedent. Requires cluster.c in controller_unit_test_src plus the
established link-satisfying fake stubs for its unrelated switch-case
dependencies (same pattern already used by test_worker_join.c and
test_commit_barrier_reply.c).
Build the live 3-node chaotic-multi-kill config-upgrade soak scenario
(run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1,
6.4. This needed genuinely new client-side infrastructure: a hand-rolled
capnp send path for OP_upgradeConfig (the auto-generated encoder is a
no-op for its Data fields, worked around the same way
xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig
CLI commands.
Building and running this live surfaced two real production bugs, both
fixed here:
- bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's
primary/standby dispatch guard in cluster.c, making config upgrades
unreachable from any network client.
- bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called
config_activate_committed(), so the node that originates a config
upgrade never refreshed its own in-memory state as part of committing
it. Killing that node before some later, unrelated event happened to
refresh it left its checkpoint permanently stale -- the first live,
empirical reproduction of bug-857's originally hypothesized failure
mode. Fixed by activating before the marker-triggered checkpoint write,
since checkpoint content is built from live runtime tables that only
reflect the reload after activation runs.
A separate, narrower checkpoint divergence (bug-1471, not config-related)
remains open and still blocks a clean soak PASS -- task 8.6 stays
in-progress until it's resolved.
xcesched_push_configs() sent two hardcoded limit1/limits2 test
literals to the scheduler (behind a #if 0/FIXME "we use builtin limit
currently"), ignoring whatever was actually loaded from cluster_config.
Migrate those literals into limits.yaml as real entries and wire
conf4sched->nlimits/limits from c->myconf->cluster_config directly.
cluster_init_phase2 also downgrades its "cluster_config not loaded"
ERROR to INFO — a standby legitimately reaching phase2 before the
primary's config-push has delivered bytes yet is normal cold-boot
behavior, not a failure; it already matches load_primary_config's
INFO-level framing for the same condition.
Wires all 11 task-6.1 CLI-sourced myconf_t fields into
init_controller_tunables() via the codebase's existing 0-sentinel fallback
idiom ((myconf->x > 0) ? myconf->x : <original hardcoded constant>);
omitting all flags reproduces today's exact hardcoded values.
Resolves both gaps flagged in task 6.1's review: resmgr_inst_delta bridges
to the existing tunables.resmgr_inst field by meaning (confirmed via
zarray_init's delta parameter), and xstrings_delta is deliberately left
unwired with an operator-facing WARN rather than adding a dead field --
the underlying xstring_table_init/kv_params_t has no delta/growth-step
concept at all, unlike the kvmgr-backed jobmgr/workermgr/qmgr/resmgr.
This completes clusterconfig-schema-split -- all required tasks (1.1-5.3)
and optional Requirement 6 tasks (6.1-6.2) are done.
Deletes the dead candidates/ncandidates KV-mirror loop from
cluster_build_config; init_controller_tunables() left untouched (it only
ever read hardcoded constants). cluster.c now compiles clean.
Replaces the cluster_config->port fallback (dead after task 1.2) with:
--peers candidate match (unchanged, including single self-referencing
entries for a custom single-node port) -> hardcoded XCE_CONTROLLER_DEFAULT_PORT
(9990) when --peers was never passed at all. The --peers-given-but-local-
name-absent case still returns -EINVAL and refuses to start, unchanged.
Also folds in task 1.3's discovery (HashZArrayConfig/ResmgrConfig capnp
structs left orphaned by tasks 1.1/1.2, breaking codegen) into
requirements.md (new Req 2.5)/design.md/research.md/tasks.md, and fixes
a design.md citation error (validate_peer_count -> xce_parse_peers) found
during task 2.1's review.
raft-drive-async's default-ON pool dispatch lets messages from different
peer endpoints (two vote replies, or a vote reply racing a vote request)
run concurrently on different vtask pool workers. raft_state_t's fields
(votes_granted, replies_received, election_decided, term, role) are plain
non-atomic — the raft library assumes a single-threaded caller. Two
concurrent raft_on_vote_reply() calls could race: one reaches quorum and
promotes to PRIMARY while the other, reading pre-increment vote count,
independently satisfies "all-replied, no quorum" and fires
on_become_follower for the same term, silently reverting the promotion —
every request afterward correctly (but wrongly) rejected as STANDBY.
Add ha->raft_lock (PTHREAD_MUTEX_RECURSIVE — a WON/quorum outcome can
legitimately re-enter the state machine synchronously on the same thread,
e.g. PreVote WON -> real election) around every raft_state_t entry point
on the election/vote path. Verified via 40-round sequential and 40-round
chaotic HA soak, zero committed-job loss, all 3 nodes led.
Found via extended chaotic /xce-test-ha soaking on top of the bug-1419 fix
(same session). Both bugs independently caused the 10-round chaotic soak to
fail; sequential and chaotic now pass cleanly and repeatably.
1. Term-desync liveness deadlock (new): a standby that loses a near-
simultaneous election race to a peer can legitimately end up at a HIGHER
term than the peer that actually won and became primary. The primary's
now-stale-term AppendEntries/heartbeats are correctly rejected by the raft
library, but every one of those rejected messages was ALSO resetting the
standby's own liveness deadline (ha_on_heartbeat_received), so its
liveness timeout never fired and it could never start a fresh,
higher-term election to let the cluster self-heal — a permanent stall.
Fix: ha_msg_term_is_stale() gates the liveness reset (and
ha_discover_primary step-down) on the incoming message's own term stamp,
skipping it for provably-stale messages, in cluster.c's wal_catchup_entry
and nop/heartbeat handlers.
2. bug-847 (root-caused 2026-06-30, never fixed): a per-sub-entry `payload`
Data blob in a multi-entry WAL catch-up batch loses its capnp pointer once
the message crosses a segment boundary — the receiver decodes
raw_payload_len=0 for entries at/after the boundary and silently persists
empty WAL entries (structurally intact: correct LSN/term/CRC chain, empty
content). Only surfaces when a node rejoins and catches up a real batch,
which is why it needed a live chaotic soak (not the 2-entry unit test) to
reproduce. Fix: carry every sub-entry's payload concatenated in ONE
top-level batchPayloads blob instead of one Data blob per entry — a
single top-level blob does not exhibit the cross-segment loss. Sliced
back apart on decode using each sub-entry's own wirePayloadLen, a plain
scalar that (like lsn/entryType/term) decodes via the generic generated
decode path and was never subject to the bug. Confirmed via targeted
diagnostic logging that the corrupted entry arrives already-empty over
the wire (not a receive-side race), and via a faithful reproduction at
live JOB_SUBMIT payload size (5×3560B) after the original 2-entry/
14×872B tests turned out not to cross a segment boundary in this
environment.
Incidental fixes found and fixed along the way:
- A real self-deadlock (persist_entry_term() re-acquiring p->lock from
inside a caller that already held it) introduced by an initial, ultimately
unnecessary WAL-read locking fix attempted before bug-847 was pinned down;
fixed via a new persist_entry_term_locked() lock-free counterpart. Left in
place as a genuine (if incidental) hardening of ha_stream_wal_to_standby's
read consistency.
- .claude/skills/xce-test-ha/SKILL.md's `context: fork` frontmatter caused
the skill's own forked sub-session teardown to SIGHUP the whole
xce_controller process group mid-soak, killing all 3 nodes silently with
no error logged, whenever the soak was launched via the Skill tool itself
(setsid/nohup inside run-ha-soak.sh only protects against a controlling
terminal's own SIGHUP, not a parent forked-session's teardown). Removed;
added explicit guidance to launch long soaks via direct Bash from the main
session instead.
Verified: full 78-group unit regression sweep clean (0 failures) after each
change; two consecutive clean 10-round chaotic soaks and one clean 10-round
sequential soak on the final binary (zero committed divergence, zero job
loss, all 3 nodes led in both modes).
Chain of fixes needed to get --force-snapshot soak to a real live PASS
(raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1):
1. WAL segment rotation was gated on a hardcoded 64MB compile-time
constant; trim_wal_segments() never deletes the active (currently-open)
segment. At soak job counts a segment never rotated, so nothing was
ever physically trimmable and ha_evaluate_repl_action always chose
STREAM_WAL over SEND_SNAPSHOT for an empty-rejoining node. Fixed by
threading a wal_segment_bytes override through persist_config_t
(mirrors the --max-wal-records pattern: config.h default, persist.h
setter, persist.c 3 usage sites + minimum-floor enforcement, 6-point
CLI wiring through main.c/myconf.h/myapp.c/cluster.c) plus a new
--wal-segment-bytes CLI flag. run-ha-soak.sh's --force-snapshot mode
now sets WAL_SEGMENT_BYTES=4096 to force real rotation at soak scale.
2. This unmasked a genuine heap-use-after-free in INVOKE_MSG_HANDLER
(include/netio.h): the macro calls ep->msg_handler(ep) synchronously
(not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing
doesn't cover it) then writes ep->fsm_in.pkt = NULL. A stale vote-reply
handler chain (raft_on_vote_reply -> raft_cb_become_follower ->
endpoint_unref) could drop the endpoint's last ref and free it inside
the handler call, so the trailing write hit freed memory. This is the
first-ever live exercise of a node receiving a snapshot transfer while
mid-election. Fixed by holding a ref across the synchronous call, same
pattern task 5.8 established for the async path.
With both fixes: --force-snapshot soak now PASSES live (committed
divergence=0, job_loss=0, confirmed real "HA: snapshot" install on the
restarted node) — first genuine live proof of this scenario. 3-round
--chaotic regression clean. Full 217-test controller suite green
before/after every fix in the chain.
Builds on the delaymgr starvation fix (b9ad018) and walcompare
checkpoint-floor fix (f3c86a6) from earlier in this investigation.
Adds --max-wal-records (mirrors the existing --wal-sync-ms pattern:
main.c argp option -> myconf_t -> persist_config_set_max_wal_records())
so a deployment can force the WAL-record-count checkpoint trigger to
fire on a job count a soak can actually reach, instead of the
production default of 10000.
Adds a new `--force-snapshot` soak mode (run-ha-soak.sh): submits
enough jobs to cross the configured threshold, kills a non-primary
node, wipes its data dir (a genuine empty rejoin), restarts it, and
asserts the cluster reconverges with zero walcompare divergence and
zero job loss. Fails loud (not a false PASS) if the restarted node's
log shows no actual snapshot install, so the round can't silently
report success without exercising the real scenario.
INVESTIGATION FINDING (not yet resolved, this commit does not close
5.10a or ha-entity-snapshots 7.1): live-tested with temporary
diagnostic logging (since reverted). schedule_checkpoint() correctly
fires once wal_record_count reaches the configured threshold
(confirmed: "wal_record_count=100 max=100" logged), but the delaymgr-
scheduled checkpoint_task_fn never executes afterward -- no checkpoint
ever completes, no snapshot file is ever written, in a live multi-node
deployment. This is a separate, deeper issue in the delaymgr scheduling
path (or its wiring for this specific call shape), not a bug in this
commit's CLI plumbing or soak scaffolding -- both are built, tested,
and confirmed working up to the point where the async checkpoint
should run. Needs its own dedicated investigation (likely src/lib
delaymgr, not persist.c) before --force-snapshot can produce a PASS.
This also has production implications beyond the test scenario: if
confirmed, WAL checkpoints may never fire automatically today, meaning
WAL segments are never trimmed via this path in a live deployment.
controller_unit_test: test_ha_startup 34/0, test_ha_core_safety 28/0,
test_persist_checkpoint 5/0, test_walcompare 17/0 (no regression).
Soak --selftest 17/17.
The io_worker freed an endpoint (endpoint_killer, cluster.c) on connection
teardown while the vtask dispatcher concurrently read ep->busy
(endpoint_is_task_blocked, netio.c) on a queued vtask — a heap-use-after-free
that ABORTed a controller mid-chaotic-soak (round 4), dropping the cluster
below quorum so no primary could be elected.
Replace the accumulation of ad-hoc ep-lifetime patches (bug-1282 lockless busy,
bug-1197 dispatch_inflight drain + deferred teardown vtask) with one principled
invariant: an endpoint lives until its reference count reaches 0.
- endpoint_t.refcnt (_Atomic) + heap_alloc flag; endpoint_get starts a heap ep
at refcnt=1 (the owner ref).
- endpoint_ref/endpoint_unref: the last unref fini's + frees a heap ep.
Embedded eps (client ctx->ep, heap_alloc=0) are never freed here — their
container owns them — so a vtask-balanced ref/unref pair is a safe no-op.
- vtask_enq takes a per-vtask ref on the bound ep BEFORE publishing the task to
the dispatcher; run_task / taskq_item_killer release it. So the dispatcher can
read ep->busy on any queued vtask without the ep being freed underneath it.
- every HEAP-ep teardown site (endpoint_killer, handle_endpoint_error, the
vote/prevote connect-fail paths, the vote-ep loser/teardown paths, the standby
reconnect/eviction paths) now drops its owner ref via endpoint_unref instead of
raw endpoint_fini+free. Several of those also leaked the ep struct before
(endpoint_fini-only); unref fixes the leak too.
Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown
updated — its OFF-path test asserted the old struct leak, now fixed). 6-round
ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 no longer crashes at
round 4, all 6 rounds elect at bounded terms. (The pre-existing iomgr_fini ->
SSL_CTX_free abort under openssl 3.6 in the aggregate test binary is unrelated.)
Note: with the crash gone, the soak now reaches rounds 4-6 for the first time and
surfaces a SEPARATE, pre-existing committed-job-loss/divergence (the bug-795/828
failover catch-up gap; ha-commit-durability domain) — not a refcount regression
(ASAN clean; rounds 1-3 are loss-free).
Root-caused via the non-ASAN debug build + gdb: on a chaotic restart an inbound
endpoint dispatches a request (e.g. walCatchupEntry op 37) while ep->owner
(cluster_t) is NULL — not yet wired at accept, or cleared during teardown. The
handler did `c = ep->owner` then `ha_dispatch_gate(&c->ha, ...)`; with c==NULL,
&c->ha = NULL+offsetof(cluster,ha) = 0x3ff0, a garbage pointer ha_dispatch_gate
dereferenced (atomic_load &ha->role) -> SIGSEGV, dropping the node below quorum
and stalling the election.
Guard c==NULL after decode (so the decoded request is freed, no leak): log a
loud WARN and drop the request — there is no cluster context to dispatch through
and no c->myhost to address a reply. The controller stays alive (never
self-terminate on an operational condition); the peer retries once the endpoint
is owned. Backtrace: ha_dispatch_gate(ha=0x3ff0) <- myreq_handler cluster.c <-
vtask_entry <- run_task.
Second of the two round-4 blockers (bug-1282 deadlock already fixed). Re-running
the 7.2 chaotic soak with both fixes.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Lands the PeerCtrlDispatch machinery behind a DEFAULT-OFF flag
(g_drive_async_peerctrl_dispatch_enabled); production behavior is
UNCHANGED. peer_ctrl_msg_handler becomes a thin shim that takes
ownership of ep->fsm_in.pkt and either runs the routing body inline
(flag OFF, byte-for-byte the prior path) or dispatches it to the vtask
pool via vtask_enq_classified (flag ON). The 3 routers now take pkt as
an explicit param (no cross-thread ep->fsm_in.pkt re-read). Per-endpoint
arrival order is enforced by bind_vtask's endpoint_block_tasks.
Packet ownership: shim clears ep->fsm_in.pkt at dispatch (sole-owner
transfer); peer_ctrl_vtask_entry NULLs iod->pkt before the body frees
it once. New vtask_t.owns_io_pkt (default 0) lets taskq_item_killer free
the carried packet for an un-executed queued task at vtaskmgr_fini WITHOUT
double-freeing the libxce endpoint-owned path (bug-1199).
Adversarial review found two FLAG-ON hazards, documented with loud
comments (flag is OFF so production is unaffected), to be resolved before
task 3.4 flips the flag ON:
- bug-1197 (CRITICAL): raft_cb_become_follower endpoint_fini()s peer
vote-eps from a pool worker, racing the io_worker's read of ep->data;
needs iomgr_unmonitor before endpoint_fini in every pool-reachable
teardown.
- bug-1198 (HIGH): peer_ctrl_classify tags all work VTASK_LIVENESS, but
the CTRL_PRIMARY repl-ack path can run a blocking snapshot checkpoint
inline; resolved by task 3.2 (offload snapshot to SlowLane).
Per spec, 3.1 is not 'done' until the task 3.4 deadlock gate is green.
Tests: 3 new peerctrl tests; controller 196/196, lib 282/0; 0 net leaks.
vtaskmgr gains vtask_class_t (LIVENESS/LIVE_BLOCK/REQ_BLOCK) + a pool-level
blocking reservation: REQ_BLOCK fast-rejects -XCE_ERR_NOT_READY past
n_vcpus-R; LIVENESS never gated (deadlock-freedom). commit_submit folds its
inflight_waits onto the pool via commit_set_pool; production wired in
cluster.c phase2 (vtaskmgr_set_liveness_reserved(R) + commit_set_pool).
Lock order p->lock->ce->lock->blk_lock. Dormant until 3.1. Commit suites
38/30/22/38 green; lib 282/0, controller 196/0, ASAN clean. Phase 1 done.
Dedicated-pthread pool isolated from the vtask pool; non-blocking slowlane_submit
(0 enqueued / -EEXIST deduped / -EAGAIN cap-full-deferred), admission cap on
running tasks, per-key in-flight dedup, run-exactly-once, drain+join fini. Wired
into controller lifecycle (dormant; consumers land in 2.x/3.x). 7 unit tests,
mutation-verified cap+dedup; full suite 194/0 green, 15x ASAN.
drive_async_liveness_reserved = max(1, min(n_peers-1, n_vcpus-1)) — single
source of truth for VTASK_LIVENESS_RESERVED (deadlock-freedom floor applied
last). New DEFAULT_CONNECT_TIMEOUT_MS / DEFAULT_SLOWLANE_* constants. Resolved
reservation logged at init; under-provisioned (n_vcpus<n_peers) clamp WARNs
loudly, never self-terminates. 8 unit cases, mutation-verified.
1.1: WalCatchupEntry gains term@4/prevLogIndex@5/prevLogTerm@6 (append-only).
1.2: RequestVote/PreVote gain lastLogTerm@4/lastLogIndex@5; cluster.c vote callers
fixed to the extended library arity (RESTORES xce_controller + controller_unit_test
compile+link, broken by the raft-commit-safety signature extension); ha.c emits the
candidate's last-log tuple on send.
5 minimal stub log/commit callbacks wired into s_real_cbs so raft_init's new
non-NULL requirement is satisfied — tasks 2.x replace them with real bodies; the
logOk freshness gate is inert until the term seam (1.3) + on_get_entry_term (2.2).
Pre-existing test_repl_unit.c wal_catchup -EAGAIN failures + ASAN leaks are an
overlayfs /tmp fsync env artifact (bug-850), not from this task (untouched paths;
append-only schema cannot affect the @3 entries-list decode). Build/baseline
isolation is blocked by a c-capnproto flake-hash drift — see Implementation Notes.
Append a committed-consume branch to the OP_nop handler (existing liveness/reply
byte-unchanged): ha_consume_heartbeat_committed applies the non-primary gate
(3.4) + stale-term guard (accept hb->term >= ha->term, 3.2), then delegates to
the monotonic ha_observe_heartbeat_committed (1.2). Absent body / zero / failed
gate are no-ops (4.2). No vote-floor edit — safety is the term-guard. 4 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Desync FAILOVER re-elections by node priority (ha_election_stagger_ms /
ha_failover_stagger_fn) mirroring the startup stagger, with seed jitter and
async prevote start; allow prevote in voter roles through the dispatch gate.
Adds HA role and vote-gate tests (stagger desync by priority, prevote on dead
primary, stickiness-lapse grants).
Pre-existing uncommitted work, committed to establish a clean baseline before
ha-commit-durability implementation.
The cluster.c denial-reason elimination ordered catch_up before fresh_heartbeat,
disagreeing with raft_cb_vote_eligible (which checks fresh-heartbeat-on-STANDBY
before catch-up). In the rare STANDBY catch-up + fresh-heartbeat overlap the log
could report catch_up while the gate denied on fresh_heartbeat. Expose the pure
read predicate ha_heartbeat_is_fresh and mirror the gate's exact order so the
logged reason can no longer disagree with the actual denial. Decision unchanged;
log-string fidelity only. prevote/vote-gate/failover-prevote suites green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add OPCode_preVoteRequest case in cluster.c, mirroring the real-vote case:
freshness floor + reused eligibility/stickiness gate (raft_on_pre_vote_request),
reply over preVoteReply with the voter's UNCHANGED term (no adoption), denial
logging with candidate id + reason. Pure insertion; real-vote case untouched.
6 new vote-gate tests assert deny-while-leader-alive, grant-when-lapsed, and the
no-mutation/term-unchanged contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cluster_start (which set c->running) is orphaned on the HA myapp_run path, so
the running flag was never set on HA nodes — check_controller() treated the
primary as shutting down and zeroed joblist/workerlist reply chunks. Own the
flag lifecycle in cluster_run, symmetric with the c->running=0 on shutdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live symptom: jobsub intermittently failed with 'Connection timed out' even
though the cluster had quorum. Root cause: the synchronous commit barrier waits
only DEFAULT_SYNC_TIMEOUT_MS=50ms for durable majority replication, and that was
not configurable. 50ms is too tight for durable sync majority replication —
especially since standbys now fdatasync each replicated entry before ACK — so a
multi-entry jobsub's later WAL entry intermittently misses the window:
commit: WAIT deadline expired without commit (lsn=3 committed_lsn=2 timeout_ms=50)
-> persist hook returns -ETIMEDOUT -> handle_jobsubmit rejects -> client ETIMEDOUT.
Fix:
- Raise DEFAULT_SYNC_TIMEOUT_MS 50 -> 500 (config.h) — realistic for durable sync
majority replication, still bounded for failure detection.
- Add a --sync-timeout-ms controller flag threaded arguments -> myconf ->
persist pcfg (override applied before persist_init); 0 = use default, positive
overrides — so operators can tune per environment.
TDD: literal-pinned persist_config_defaults().sync_timeout_ms == 500. All
timeout-sensitive tests already pass an explicit timeout_ms (none rely on the
default), so none are slowed; full controller suite green under ASAN. No
commit-engine/barrier/replication/wire change beyond the default value + plumbing.
bug-561.
Root cause (found via ha-convergence-recovery task 8 live 3-node E2E): a
controller with no local config (config_version_tag==0) could become PRIMARY
and push an empty config, breaking replication.
Two upstream election-safety gaps, both outside ha-convergence-recovery:
1. RDHUP re-election bypassed the no-config 'voter only' guard. The startup
path (ha_start_election_async) refuses to start an election when
my_config_tag==0 && n_peers>1, but the RDHUP failover path
(ha_on_primary_ctrl_fd_rdhup) called raft_on_leader_lost unconditionally.
Fix: mirror the guard before elect_ctx alloc / hook / raft_on_leader_lost;
a config-less node stays CTRL_STANDBY (voter only) and grants the
configured peer instead of starting (and winning) its own election.
2. Voter freshness gate compared the candidate only against the voter's LOCAL
config, ignoring the cluster's known committed state. Fix: ha_vote_self_freshness
floors the comparison at max(local, learned-committed config_tag / committed_lsn),
so a voter denies any candidate whose config/cluster state is outdated vs
what the cluster has committed. Strictly additive (never denies the freshest
node — Leader Completeness preserves liveness).
TDD: test_rdhup_noconfig_node_does_not_elect + configured-node contrast;
test_ha_vote_gate learned-floor deny + liveness + backward-compat. Full
controller suite green under ASAN. commit.c and ha-convergence-recovery
files untouched.
configAck branch: per-push majority (acked_config_lsn >= push_tag,
1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate,
ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead.
push_config_to_standbys returns the barrier result; upgrade failure ->
send_upgrade_error (rc forwarded), no success ACK, no restart,
committed_config_tag success-only.
Remediation r1 (two CRITICAL review findings):
- F1: wire production config-ACK ingestion — replconfigpushreplymsg
decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack
on the promoted-vote ep; + defensive cluster.c OPCode_reply case.
Closes the unlanded half of ha-config-bootstrap 8.1; without this the
barrier would brick every multi-member config upgrade. No schema change.
- F2: keep the barrier off the iomgr thread — export
push_config_transport_only (non-blocking) for the 5 ha.c + join.c
streaming sites; barrier path now vtask-only (do_upgrade_config).
Wire-path test drives real capnp ACK bytes through the production decode
branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0
FAIL; pre-existing fixture leak only (less than baseline).
cluster_t embeds commit_engine_t; init in cluster_init_phase2 after the
vtaskmgr guard (idempotence-guarded — phase2 can run twice), wiring
repl_set_commit_engine + ha.repl back-pointer before any standby can
register; fini after iomgr/vtask shutdown. persist_wal_sync_append
passes written_lsn; engine-wired path drops -ENODEV and the
committed_lsn store (engine owns advancement); LSN-0 first-entry
routes to legacy per the accepted interim. p2 -ENODEV pin updated to
the Req 6.3 contract. 34/34 + 6/6 green through honest delegation;
census exactly at the 3 validated baselines; production binary links.
Standby sends WalAckReply via xce_reply after each successfully
applied WAL entry. ha_apply_wal_catchup_entry signature extended
with endpoint_t* and cluster_t* parameters. walackreplymsg union
arm added to RequestReply capnp schema and xce_reply_t. Error
entries skip ACK. Dispatch gate test confirms OPCode_walCatchupEntry
allowed in CTRL_STANDBY/ELECTING/INIT.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
OPCode_nop handler now sends stateReport {applied_lsn, config_tag}
instead of nodata. Function in ha.c for unit-test linkability.
Two tests: warm state and zero-state fresh standby (Req 1.4).
Replace stub with ha_apply_wal_catchup_entry() calling apply_wal_entry().
CAS-advance applied_lsn on success; log ERROR and skip on failure.
Standby stays in CTRL_STANDBY throughout. 3 new unit tests.
Enforce heartbeat_timeout_ms >= 2 * heartbeat_interval_ms at startup.
validate_heartbeat_config() in startup_validation.c returns -EINVAL with
error message on violation; wired in cluster_init_phase1 before ha_init().
4 unit tests added to test_ha_startup.c.
- config_repl.c: add handle_repl_config_push — standby handler for
OPCode_replConfigPush. Verifies SHA-256, saves blob to disk via
conf_save_config, reloads via load_primary_config, ACKs primary.
- config_repl.h: declare handle_repl_config_push.
- cluster.c: dispatch OPCode_replConfigPush to handle_repl_config_push;
populate c->myhost_info.name from myconf->name (ctrl-1/ctrl-2/ctrl-3)
instead of the static "controller" placeholder.
- ha.c: fix raft_cb_become_leader push_config_to_standbys call to pass
real config bytes and tag read from disk (was NULL/0 — standbys got
empty push and never saved config).
- typemap.h: add raw_sha256/raw_bytes/raw_bytes_len fields to
repl_config_push_req_t (mirrors upgrade_config_req_t pattern).
- request.c: pre-extract List(UInt8) sha256/bytes from OPCode_replConfigPush
before capn_free() in xce_decode(); free them in xce_free_request().
- helper.c: add all HA/replication opcode names to xce_request_op_name
so "unknown opcode 30" becomes "replConfigPush" in logs.
- cluster_def.h: add embedded myhost_info field to cluster_t struct.
- ha.h: declare ha_start_primary_heartbeat; remove dead
ha_find_and_connect_to_primary declaration.
