myxce/src/controller
Rongsong Shen 327dbb9bb3 fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election
A primary never learns of a follower's higher term via AppendEntries-reject
responses: ha_raft_on_repl_ack only routes AE replies through
raft_on_append_entries_reply for success=1 ACKs, and that function itself
drops any reply whose term != s->term (raft.c stale-term drop) before it
could adopt a higher term anyway. Net effect: once a follower outpaces a
stale primary's term, every AppendEntries the primary sends is rejected
forever and it can never regain write-quorum — a permanent election livelock
(surfaced as "no primary elected at term>8" in HA soak testing).

Adds two fallback term-discovery vectors that fire regardless of AE-reject
outcome:
- StateReport heartbeat-reply now carries the standby's current term
  (controller.capnp, typemap.h) so the primary learns a higher term on every
  convergence cycle.
- New raft_on_higher_term_observed() mirrors the UpdateTerm step (strictly
  greater-than only; touches term/voted_for/role; persists before notifying
  on_become_follower) and is wired into both the heartbeat state-report path
  and the WAL-NACK path in repl_ack_decode_and_dispatch.

Validated: 265 controller + 287 lib unit tests pass; 10-round sequential and
chaotic HA soaks both PASS with zero job loss and full leadership coverage.
2026-07-09 11:56:46 +00:00
..
include fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard 2026-07-08 12:48:44 +00:00
testsuite fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard 2026-07-08 12:48:44 +00:00
apiserver.c add kiro skills 2026-05-21 03:41:57 +00:00
CLAUDE.md docs: close ha-commit-durability Phase 6 (bug-795) + live-validate 4 specs 2026-07-03 02:41:28 +00:00
cluster.c fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss 2026-07-08 15:21:24 +00:00
commit.c feat(raft-apply-decoupling): ordered live apply pump activates apply_committed_entry (task 6.1) 2026-07-04 19:00:43 +00:00
config_repl.c fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss 2026-07-08 15:21:24 +00:00
config_repl.h feat(raft-ha-core-safety): establish snapshot (term,lsn) boundary on install (task 3.2) 2026-07-01 13:08:10 +00:00
ctrl_metrics.c feat: P3-7 ctrl_metrics — 7 atomic counters/gauges + descriptor table + persist wiring 2026-05-08 00:04:06 +00:00
ctrlstatus.c feat(ha-heartbeat-committed-propagation): migrate last_known_committed_config_tag to _Atomic (task 1.1) 2026-06-20 11:27:46 +00:00
dataex.c fix(persist): zero nresvals_to_sched after free_name_val_list 2026-05-20 09:39:07 +00:00
egress.c feat(ha-convergence-recovery): commit-gated scheduler egress — submit_lsn tagging + watermark gate (task 1.2) 2026-06-14 02:33:36 +00:00
ha.c fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election 2026-07-09 11:56:46 +00:00
ha.h fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss 2026-07-08 15:21:24 +00:00
ha_convergence.c feat(ha-term-safety): route AppendEntries intake through library + leaderCommit wire (task 3.2) 2026-06-22 08:42:35 +00:00
job.c fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard 2026-07-08 12:48:44 +00:00
join.c feat(libxce-failover): controller populates primary host/port on E_XCE_REDIRECT replies (task 5.3) 2026-06-17 04:16:26 +00:00
main.c fix(raft-ha-core-safety): close 5.10a — WAL segment rotation + endpoint UAF blocking real snapshot catch-up 2026-07-02 04:11:20 +00:00
meson.build test(raft-apply-decoupling): apply/notify timing integration through the full pump (task 6.3) 2026-07-05 01:13:17 +00:00
misc.c save works 2026-03-05 16:03:37 +08:00
myapp.c fix(raft-apply-decoupling): kill-mid-fact-stream soak + close bug-1416 boot-time TOCTOU (task 7.2) 2026-07-07 03:03:40 +00:00
myconf.c feat(ha-config-bootstrap): implement --vcpus CLI, wire validation, remove vcpus from config (task 2.2) 2026-05-28 04:17:06 +00:00
opget.c fix(persist): zero nresvals_to_sched after free_name_val_list 2026-05-20 09:39:07 +00:00
opupd.c save works 2026-03-03 16:49:26 +08:00
persist.c fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss 2026-07-08 15:21:24 +00:00
queue.c feat(ha-commit-barrier-rollout): commit-first-then-reply in all four handlers (task 3.2) 2026-06-12 10:28:25 +00:00
repl.c feat(ha-term-safety): retire term-0 repl_broadcast live send; carry leader term on replication (task 3.4) 2026-06-22 15:45:14 +00:00
reply.c fix(persist): zero nresvals_to_sched after free_name_val_list 2026-05-20 09:39:07 +00:00
resources.c update controller code 2025-11-05 16:58:57 +08:00
sched.c feat(ha-convergence-recovery): commit-gated scheduler egress — submit_lsn tagging + watermark gate (task 1.2) 2026-06-14 02:33:36 +00:00
slowlane.c feat(raft-drive-async): SlowLane bounded executor for long blocking I/O (task 1.2) 2026-06-26 05:24:00 +00:00
startup_validation.c feat(ha-heartbeat-liveness): add validate_heartbeat_config startup check (task 1.2) 2026-06-01 12:26:29 +00:00
task.c save works 2026-03-04 18:18:53 +08:00
worker.c feat(ha-commit-barrier-rollout): commit-first-then-reply in all four handlers (task 3.2) 2026-06-12 10:28:25 +00:00