myxce/src/controller/testsuite
Rongsong Shen d3da3ca8bd fix(raft-apply-decoupling): prevent apply-divergence fence from being live-cleared (task 5.3)
Add explicit test coverage for the DIVERGENT fence's stay-alive and
restart-only-clear guarantees (task 5.3), which surfaced a critical,
pre-existing safety gap: the sticky recovery_unrecoverable flag is
trigger-agnostic, and two pre-existing live call sites in ha.c
(promotion_fence_timeout_fn's periodic timer, and
raft_cb_become_leader's promotion-eligibility gate) clear it based
purely on structural conditions, never checking the apply verdict.

A node whose local apply dispatch diverges from the committed log,
but whose WAL bytes are otherwise structurally healthy, would have
its fence silently released by the live timer, then be eligible for
promotion to primary on a later election -- serving genuinely
divergent state.

Fix: gate both clear-call-sites on persist_apply_verdict(&c->persist)
!= APPLY_DIVERGENT in addition to their existing structural checks.
An apply-divergence-triggered fence can now only clear via a fresh
process restart's clean persist_recover() replay.

New tests: test_promotion_fence_apply_divergent_survives_structural_catchup
(test_commit_durability.c) covers the timer site; and
test_promotion_refused_when_apply_divergent_even_if_eligible
(test_convergence_verification.c) covers the promotion-gate site,
structurally isolated so it cannot pass for the wrong reason.

Closes out task-group 5 (restart apply-correctness verification).
2026-07-04 13:33:03 +00:00
..
clean_failover_preservation.sh test(ha-term-safety): harness post-failover client probes target a live node (bug-854 workaround) 2026-06-23 04:12:32 +00:00
controller_test.c feat(raft-apply-decoupling): post-replay apply-correctness verification (task 5.1) 2026-07-04 11:32:21 +00:00
ctrlstatus_fmt.h feat(controller-status): render responderTime in cmd_ctrlstatus, log-comparable format (task 6.3) 2026-06-20 09:28:49 +00:00
snapdiff.c fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389) 2026-07-03 02:40:12 +00:00
snapdiff.h fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389) 2026-07-03 02:40:12 +00:00
snapverify.c fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389) 2026-07-03 02:40:12 +00:00
snapverify.h fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389) 2026-07-03 02:40:12 +00:00
test.c fix: close bug-1401 ctrlstatus leak, bug-1400 jobsub leak; harden bug-1396 harness 2026-07-03 02:40:52 +00:00
test_apply_committed_entry.c feat(raft-apply-decoupling): build apply_committed_entry single dispatch seam (task 2.2) 2026-07-03 16:51:47 +00:00
test_apply_correctness_verification.c fix(raft-apply-decoupling): prevent apply-divergence fence from being live-cleared (task 5.3) 2026-07-04 13:33:03 +00:00
test_apply_reapply_idempotency.c test(raft-apply-decoupling): idempotent and ordered re-application of apply seam (task 2.4) 2026-07-04 00:44:55 +00:00
test_apply_seam.c feat(raft-apply-decoupling): extract guarded fact transition logic into reusable cores (task 2.1) 2026-07-03 15:26:22 +00:00
test_cluster_persist.c feat(ha-commit-durability): add --wal-sync-ms flush-cadence override (task 1.1) 2026-06-18 04:46:01 +00:00
test_coldstart_lsn0.c fix bug-1060 bug-1067 2026-06-24 17:05:43 +08:00
test_commit_admission_pool.c feat(raft-drive-async): LivenessAdmission pool reservation + commit fold (tasks 1.6/1.7) 2026-06-26 07:43:19 +00:00
test_commit_barrier_reply.c test(raft-apply-decoupling): durable fact retention under sustained quorum loss (task 4.2) 2026-07-04 07:22:21 +00:00
test_commit_barrier_rollout.c feat(ha-term-safety): append config markers via unified log + derive committed config (task 4.2) 2026-06-22 10:34:10 +00:00
test_commit_durability.c fix(raft-apply-decoupling): prevent apply-divergence fence from being live-cleared (task 5.3) 2026-07-04 13:33:03 +00:00
test_commit_engine.c feat(ha-commit-durability): cap commit self-contribution at durable_lsn + durable-advance commit wake (task 2.2) 2026-06-18 06:29:47 +00:00
test_commit_resolve.c test(raft-apply-decoupling): genuine multi-threaded concurrency tests for commit resolve (task 1.5) 2026-07-03 13:07:44 +00:00
test_config_repl.c fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389) 2026-07-03 02:40:12 +00:00
test_convergence_verification.c fix(raft-apply-decoupling): prevent apply-divergence fence from being live-cleared (task 5.3) 2026-07-04 13:33:03 +00:00
test_ctrl_metrics.c feat: P3-7 ctrl_metrics — 7 atomic counters/gauges + descriptor table + persist wiring 2026-05-08 00:04:06 +00:00
test_ctrlstatus.c feat(ha-heartbeat-committed-propagation): migrate last_known_committed_config_tag to _Atomic (task 1.1) 2026-06-20 11:27:46 +00:00
test_demotion_convergence.c test(raft-apply-decoupling): genuine multi-threaded concurrency tests for commit resolve (task 1.5) 2026-07-03 13:07:44 +00:00
test_drive_async_config.c feat(raft-drive-async): add drive-async config constants + liveness-reservation helper (task 1.1) 2026-06-26 04:58:07 +00:00
test_e2e_capnp.c feat(ha-term-safety): restore controller build + wire vote/AE term wire fields (tasks 1.1, 1.2) 2026-06-22 03:41:49 +00:00
test_e2e_checkpoint.c feat(raft-drive-async): snapshot SlowLane offload + WAL-trim cursor guard (task 3.2) 2026-06-26 15:43:24 +00:00
test_e2e_gc.c test: P0-G end-to-end smoke tests — WAL, checkpoint, hooks, GC, Cap'n Proto (P0-15a-e) 2026-05-07 13:25:17 +00:00
test_e2e_hooks.c fix(controller): extract job-submit commit barrier out of the hook fan-out — stop committed-job-loss (bug-1306) 2026-07-01 00:42:30 +00:00
test_e2e_wal.c test: P0-G end-to-end smoke tests — WAL, checkpoint, hooks, GC, Cap'n Proto (P0-15a-e) 2026-05-07 13:25:17 +00:00
test_egress_filter.c test(raft-ha-core-safety): fix 2 stale tests surfaced by first full-suite ASAN sweep (task 7.1) 2026-07-01 14:31:26 +00:00
test_gc_retention.c feat: JOB_STAT_TRANSIENT, job_should_retain, worker_should_retain, dead_since (P0-5a, P0-8, P0-13) 2026-05-07 12:52:07 +00:00
test_ha_bootstrap.c fix(ha-bootstrap): skip pre-election config probe; unconfigured nodes defer to post-election fetch 2026-05-29 13:15:46 +00:00
test_ha_bootstrap_support.c feat(election-iomgr-refactor): remove old blocking election code (am_i_primary_quorum, vote_request_thread, bootstrap_config_if_needed) — task 5.3 2026-05-30 13:04:19 +00:00
test_ha_cluster.c feat: wire ha_state_t into cluster_t + HA startup/shutdown lifecycle (P1-5) 2026-05-07 14:55:48 +00:00
test_ha_connection_reuse.c feat(ha-single-leader-role): remove CTRL_PRIMARY_COMMITTING enum + remaining decls — compiler-enforced (task 5.2) 2026-06-13 14:41:42 +00:00
test_ha_core_safety.c docs(raft-ha-core-safety): document term-blind bypass safe-by-guard + pin restart identity (task 6.2) 2026-07-01 13:55:06 +00:00
test_ha_ctrlhello.c feat(ha-config-bootstrap): implement OP_ctrlHello state sync handler (task 9.3) 2026-05-29 05:22:17 +00:00
test_ha_dispatch_integration.c test(raft-drive-async): receiver-side dispatch liveness integration tests (task 3.4b) 2026-06-27 00:36:22 +00:00
test_ha_durability.c test(ha-config-bootstrap): RED-phase tests for ha-term.bin durability (task 6.1) 2026-05-28 10:11:56 +00:00
test_ha_election.c test(ha-term-safety): controller Election-Safety regression for bug-1095/1096 + absorb into spec 2026-06-25 02:44:29 +00:00
test_ha_failover.c fix(ha-config-bootstrap,ha-repl-hardening): config-less node must not win election 2026-06-14 08:58:08 +00:00
test_ha_failover_prevote.c feat(ha-prevote): rewire failover triggers through PreVote, defer destructive step (task 4.2) 2026-06-16 06:29:18 +00:00
test_ha_heartbeat_committed.c feat(ha-heartbeat-committed-propagation): standby OP_nop consume branch + term-guard (task 2.2) 2026-06-20 12:02:15 +00:00
test_ha_liveness.c fix(raft-drive-async): close snapshot-lane borrowed-endpoint hazard + catch-up checkpoint-marker bug 2026-07-02 04:54:17 +00:00
test_ha_partition.c test(raft-drive-async): network-partition soak primitive + RED/GREEN scenario (task 4.1) 2026-06-29 04:37:20 +00:00
test_ha_prevote.c feat(ha-term-safety): restore controller build + wire vote/AE term wire fields (tasks 1.1, 1.2) 2026-06-22 03:41:49 +00:00
test_ha_rdhup_integration.c fix(controller): debounce RDHUP-triggered failover — stop the term-escalation storm (bug-1310, task 5.10b) 2026-07-01 01:32:27 +00:00
test_ha_redirect.c feat: ha_should_redirect + join.c HA redirect/nomaster guard (P1-6) 2026-05-07 15:11:53 +00:00
test_ha_repl_push.c test(ha-single-leader-role): sweep all COMMITTING fixtures to direct-PRIMARY contract (task 5.1) 2026-06-13 13:59:42 +00:00
test_ha_role.c feat(controller-status): allow ctrlstatus through ha_dispatch_gate in all roles incl no-quorum (task 2.2) 2026-06-20 05:04:03 +00:00
test_ha_role_lifecycle.c fix(ha-wal-replication): apply_wal_entry must not own the WAL CRC chain (bug-778) 2026-06-19 14:12:53 +00:00
test_ha_start_election_async.c test(ha-single-leader-role): sweep all COMMITTING fixtures to direct-PRIMARY contract (task 5.1) 2026-06-13 13:59:42 +00:00
test_ha_startup.c test(ha-single-leader-role): sweep all COMMITTING fixtures to direct-PRIMARY contract (task 5.1) 2026-06-13 13:59:42 +00:00
test_ha_vep_teardown.c fix(raft-drive-async): fix bug-1376 cold-start self-deadlock, flip peerctrl+snapshot flags ON 2026-07-02 07:18:42 +00:00
test_ha_vote_floor_heartbeat.c test(ha-heartbeat-committed-propagation): verify vote floor consumes heartbeat-raised committed safely (task 3.2) 2026-06-20 12:31:04 +00:00
test_ha_vote_gate.c test(raft-ha-core-safety): fix 2 stale tests surfaced by first full-suite ASAN sweep (task 7.1) 2026-07-01 14:31:26 +00:00
test_ha_vote_send.c feat(election-iomgr-refactor): implement vote_reply_msg_handler 2026-05-30 08:45:43 +00:00
test_hooks.c feat: add HOOK_DETAIL_JOB_STOP/RESUME; fix xce_reply ordering after AFTER_EVENT hooks (P0-4a) 2026-05-07 12:08:52 +00:00
test_join_redirect.c feat(libxce-failover): controller populates primary host/port on E_XCE_REDIRECT replies (task 5.3) 2026-06-17 04:16:26 +00:00
test_p1_integration.c chore: checkpoint all prior work for clean state 2026-06-04 13:04:16 +00:00
test_p2_integration.c fix(ha-repl-hardening): snapshot fallback off-by-one — empty standby must stream, not snapshot 2026-06-14 12:10:40 +00:00
test_p3_hardening.c fix(ha-commit-engine,persist): reserve WAL lsn 0 — first entry starts at lsn 1 2026-06-14 10:46:16 +00:00
test_p3_manual_failover.c feat: P3-6 ha_trigger_manual_failover_for_test() shim + 3 failover cycle tests 2026-05-07 23:55:00 +00:00
test_persist_checkpoint.c feat: implement persist_checkpoint with WAL marker, snapshot, trim, rotation (P0-3) 2026-05-07 11:19:08 +00:00
test_persist_hooks.c fix(controller): extract job-submit commit barrier out of the hook fan-out — stop committed-job-loss (bug-1306) 2026-07-01 00:42:30 +00:00
test_persist_recovery.c fix bug-1060 bug-1067 2026-06-24 17:05:43 +08:00
test_persist_repl.c feat: P2-4 wire repl_broadcast into persist_wal_async/sync_append; last_replicated_crc 2026-05-07 17:54:48 +00:00
test_persist_schema.c feat: add PersistConfig, SnapshotHeader, ReplEnvelope Cap'n Proto schemas (P0-9) 2026-05-07 10:19:33 +00:00
test_persist_snapshot_entities.c test(ha-entity-snapshots): RED tests for load_snapshot restore contract (task 5.1) 2026-07-01 07:56:57 +00:00
test_persist_truncate.c test(raft-apply-decoupling): genuine multi-threaded concurrency tests for commit resolve (task 1.5) 2026-07-03 13:07:44 +00:00
test_persist_types.c feat: add persist.h — WAL types, persist_config_t, slot_key_t (P0-B) 2026-05-07 09:21:36 +00:00
test_persist_wal.c feat: add schedule_checkpoint() with WAL record-count trigger and idempotency CAS guard 2026-05-13 17:56:46 +00:00
test_repl_apply.c fix(ha-term-safety): bug-828 FIXED — remove premature pre-election WAL truncate (committed-job-loss on clean failover) 2026-06-23 04:06:18 +00:00
test_repl_catchup.c feat: P2-5 repl_start/complete_catchup, repl_needs_snapshot — standby catch-up state 2026-05-07 18:14:12 +00:00
test_repl_majority.c feat(ha-term-safety): retire term-0 repl_broadcast live send; carry leader term on replication (task 3.4) 2026-06-22 15:45:14 +00:00
test_repl_unit.c feat(ha-term-safety): retire term-0 repl_broadcast live send; carry leader term on replication (task 3.4) 2026-06-22 15:45:14 +00:00
test_replay_apply_seam.c feat(raft-apply-decoupling): point restart replay at apply_committed_entry seam (task 2.3) 2026-07-04 00:32:46 +00:00
test_slowlane.c feat(raft-drive-async): SlowLane bounded executor for long blocking I/O (task 1.2) 2026-06-26 05:24:00 +00:00
test_snapdiff.c fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389) 2026-07-03 02:40:12 +00:00
test_snapverify.c fix(raft-drive-async): resolve snapshot chunk-drop (bug-1386) and chunk misordering (bug-1389) 2026-07-03 02:40:12 +00:00
test_term_safety_adapter.c fix bug-1060 bug-1067 2026-06-24 17:05:43 +08:00
test_walcompare.c feat(walcompare): add per-member checkpoint-floor awareness (raft-ha-core-safety 5.10a / ha-entity-snapshots 7.1) 2026-07-02 02:59:08 +00:00
test_waldump.c checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
test_worker_join.c add kiro skills 2026-05-21 03:41:57 +00:00
walcompare.c feat(walcompare): add per-member checkpoint-floor awareness (raft-ha-core-safety 5.10a / ha-entity-snapshots 7.1) 2026-07-02 02:59:08 +00:00
walcompare.h feat(walcompare): add per-member checkpoint-floor awareness (raft-ha-core-safety 5.10a / ha-entity-snapshots 7.1) 2026-07-02 02:59:08 +00:00
waldump.c fix bug-1060 bug-1067 2026-06-24 17:05:43 +08:00
waldump.h test(ha-term-safety): cross-member walcompare divergence tool (task 5.3) 2026-06-22 13:05:23 +00:00