Four property-style raft tests proving the PreVote API preserves the safety
invariants: term/voted_for/role never change and on_persist never fires across
mixed begin/request/reply sequences (WON and LOST); a WON round creates no leader
(stays FOLLOWER); and the real-vote higher-term-adopt + step-down + quorum-elect
path is unchanged (bug-339/340 intact). Records the TLA+/checklist consistency
argument (RN-1) in test_raft.c and research.md, citing the tests as evidence.
Raft suite 123/0/0; zero production change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raft_on_pre_vote_reply: per-peer-deduped tally returning WON at quorum
((n_peers/2)+1), LOST when all non-self peers reply without quorum, else
PENDING; bounds-guards peer_idx. Never fires become-leader/follower, never
mutates term/voted_for/role, never persists. 8 RED-first tests (3- and 5-node,
dedup, out-of-range, purity). Raft PreVote pure core complete.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raft_begin_pre_vote: compute prospective term (term+1) locally without
persisting, arm the pre_vote_* tally with a self-grant, NULL-safely probe each
non-self peer, return RAFT_PRE_VOTE_WON for single-node else RAFT_OK. Stays
FOLLOWER, schedules no timeout, mutates no persistent state. 6 RED-first tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
raft_on_pre_vote_request: deny on stale term, else reuse on_vote_eligible
(primary/heartbeat-stickiness/catch-up gates). Never mutates term/voted_for/
role and never persists or adopts a higher term. 6 RED-first raft unit tests
assert the purity invariant; libxce suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cluster_start (which set c->running) is orphaned on the HA myapp_run path, so
the running flag was never set on HA nodes — check_controller() treated the
primary as shutting down and zeroed joblist/workerlist reply chunks. Own the
flag lifecycle in cluster_run, symmetric with the c->running=0 on shutdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Election protocol fixes validated against raft.tla invariants:
- ha_init_raft(): one-time raft state init at startup (re-init would lose
durable election progress); n_peers = full cluster size including self
- handle_request_vote(): higher-term step-down moved BEFORE the role gate
(split-brain fix); standby vote-deny now requires a live primary control
channel (primary_ctrl_fd >= 0) AND a fresh heartbeat
- ha_heartbeat_is_fresh(): derive freshness from ha->last_activity (the
field production actually writes); the orphaned last_heartbeat timespec
had no production writer, leaving the vote-deny gate inert (bug-337);
field removed from ha_state_t
- handle_request_vote(): restore same-candidate idempotent re-grant —
deny only a DIFFERENT candidate in the same term, matching raft.c and
Raft §5.2 votedFor ∈ {Nil, candidateId} (bug-338)
- raft_init(): restore voted_for_term alongside a persisted vote; leaving
it 0 allowed a double vote in the same term across a restart (bug-339)
- myapp_run(): leader_known check treats a vote recorded under this node's
own name as a self-vote — production persists the real candidate name,
the "self" placeholder only appears in test mode
- raft_on_vote_request(): leader steps down on higher-term vote request
instead of denying it
Tests: test_proto_H6_restored_vote_blocks_second_vote added; H1 restore
assertion strengthened; test_ha_vote_gate.c driven through last_activity +
primary_ctrl_fd (the real production fields). Full /xce-build all green
(lib + controller + scheduler + Go suites, 0 failures, ASAN clean).
deploy-ha.sh: start.sh now forwards extra args to the controller binary.
Root cause: task_cmp in delay.c compared nextrun (always 0 for ms-tasks)
instead of nextrun_ts, making all ms-tasks appear equal in the ordered
queue. The heartbeat task was re-enqueued correctly but stuck behind the
liveness poll task in the broken ordering.
Changes:
- delay.c: fix task_cmp to compare nextrun_ts for ms-tasks (ns precision)
- ha.c: use runcount=-1 (infinite repeat) instead of one-shot+re-arm;
simpler and avoids the broken re-arm path entirely
- ha.c: add static atomic idempotency guard to prevent duplicate heartbeat
tasks on re-election
- config_repl.c: add handle_repl_config_push — standby handler for
OPCode_replConfigPush. Verifies SHA-256, saves blob to disk via
conf_save_config, reloads via load_primary_config, ACKs primary.
- config_repl.h: declare handle_repl_config_push.
- cluster.c: dispatch OPCode_replConfigPush to handle_repl_config_push;
populate c->myhost_info.name from myconf->name (ctrl-1/ctrl-2/ctrl-3)
instead of the static "controller" placeholder.
- ha.c: fix raft_cb_become_leader push_config_to_standbys call to pass
real config bytes and tag read from disk (was NULL/0 — standbys got
empty push and never saved config).
- typemap.h: add raw_sha256/raw_bytes/raw_bytes_len fields to
repl_config_push_req_t (mirrors upgrade_config_req_t pattern).
- request.c: pre-extract List(UInt8) sha256/bytes from OPCode_replConfigPush
before capn_free() in xce_decode(); free them in xce_free_request().
- helper.c: add all HA/replication opcode names to xce_request_op_name
so "unknown opcode 30" becomes "replConfigPush" in logs.
- cluster_def.h: add embedded myhost_info field to cluster_t struct.
- ha.h: declare ha_start_primary_heartbeat; remove dead
ha_find_and_connect_to_primary declaration.
Add void (*on_connected)(endpoint_t *ep) field to endpoint_t with
endpoint_set_on_connected() helper. io_handler fires it exactly once
immediately after ssl_handshake_done=1 then nulls the pointer.
Enables ha_start_election_async to send OP_requestVote as soon as
the TLS handshake with a peer completes — no blocking SSL_connect.
6 new unit tests pass; controller suite 1/1 OK.
Add millisecond-granularity delay_task_ms() to delaymgr using
clock_gettime(CLOCK_MONOTONIC) for deadline computation. Extends
delay_task_t with nextrun_ts field; delay_worker distinguishes
ms-tasks from second-tasks via non-zero nextrun_ts. Backward
compatible with existing delay_task callers.
4 delay tests pass (2 new: tolerance and invalid-args).
Provides the timeout primitive for election and committing watchdogs.
xce_replconfig_reply_raw() - primary sends reply with real bytes via endpoint
xce_replconfig_req_ssl() - standby sends request over raw SSL socket
xce_replconfig_reply_ssl_decode() - standby decodes reply from raw SSL
xce_upgrade_config_req_extract() - documents architectural limitation; returns -ENOTSUP
xce_upgrade_config_req_extract_raw() - extract bytes from raw packet for OP_upgradeConfig
Same bypass pattern as existing xce_replconfigpush_req_raw(): build the
capnp message directly via capn_new_list8 + capn_setv8, sidestepping the
encode_Data_ptr path that routes raw bytes through xce_data_t (a
discriminated cluster-data union that cannot carry raw byte arrays).
Unblocks handle_config_request() and do_one_config_fetch() wiring.
Task A research finding: xce_upgrade_config_req_extract(xce_request_t*)
is unimplementable because xce_decode() calls capn_free() before any
handler runs, and decode_Data_ptr converts the wire-format List(UInt8)
into a zero-filled xce_data_t. Task D must capture the raw io_packet_t
bytes before xce_decode tears them down, then call the _raw variant.
Crash: encode_NameValue_list called with count=21, s=NULL on worker
rejoin, causing SIGSEGV in persist_worker_hook_fun.
Root cause: free_name_val_list(xce_name_val_t***, int) accepted count by
value. It zeroed *vals (resvals_to_sched = NULL) but could not zero
nresvals_to_sched. Any caller that freed the scheduler resource snapshot
(opget.c workerlist, reply.c clusterconf, dataex.c, sched.c dispatch)
left nresvals_to_sched stale. When the worker later rejoined and the
WAL HOOK fired, the WAL encoder read count=21 with a NULL pointer.
Fix: change signature to free_name_val_list(xce_name_val_t***, int*)
so both fields are zeroed atomically. Update all 7 call sites.
