myxce/.kiro/specs/ha-convergence-recovery/tasks.md

Implementation Plan: ha-convergence-recovery

TDD ordering per project steering (testing.md): every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first. No (P) markers: groups 2–5 all modify ha.c/ha.h and group 1's tagging touches persist.c (shared with the demotion-path file set via the truncation/recovery seams), so the plan is sequential — the one nominally independent group (egress filter) is placed first instead of parallelized. Verify exact line numbers with grep before editing (this spec's research.md §1 has the 2026-06-11 verified anchors; files drift). Task 8 (E2E) additionally requires ha-single-leader-role to have landed and a live /xce-deploy-ha cluster.

• 1. Egress filter: submit_lsn tagging and commit-watermark gate on scheduler exports

• 1.1 Write RED tests for tagging and the scheduler-export watermark

  • Create testsuite/test_egress_filter.c (XCE_TESTING, AAA structure) and register it in controller_unit_test_src (meson.build); add uint64_t submit_lsn to the job_t wrapper (types.h:483-485 anchor) and declare job_submit_committed / collect_jobs_committed in the dataex header so tests compile
  • Tests: a live-submitted job is tagged with its WAL entry LSN; a job restored via apply_wal_entry/restore_job replay is tagged with the replayed entry's LSN; a job with submit_lsn > committed_lsn is absent from collect_jobs_committed results and from the JOB_NEW incremental push list; the same job appears once committed_lsn advances to cover it; submit_lsn == 0 is always exportable; plain collect_jobs (status/API path) still returns the uncommitted job
  • Observable completion: build succeeds with the new field/header declarations; all new test cases FAIL (red) against unimplemented predicate/tagging
  • Requirements: 3.1, 3.2, 3.3, 3.4

• 1.2 Implement tagging and the watermark gate to green

  • Tagging: persist_job_hook_fun (persist.c:1107-1125 anchor) sets submit_lsn = written_lsn on each submitted job after a successful append; apply_wal_entry (persist.c:729-731 anchor) passes hdr->lsn through to restore_job (job.c:71), which tags the restored job
  • Predicate + wrapper in dataex.c: job_submit_committed = submit_lsn == 0 || submit_lsn <= committed_lsn (one atomic load, lock-free); collect_jobs_committed wraps collect_jobs with the predicate
  • Scheduler-bound consumers only: xcesched_push_jobs (sched.c:183 anchor) collects via the committed-only wrapper; xcesched_push_joblist (JOB_NEW push fed by sched_job_new_fun, sched.c:236-248 anchor) filters its list with the predicate; apiserver.c:83 and opget.c:173 remain on unfiltered collect_jobs
  • Observable completion: all task 1.1 tests PASS; grep confirms no status/API read path calls the committed-only wrapper
  • Requirements: 3.1, 3.2, 3.3, 3.4

• 2. Demotion convergence: self-restart primitive and the truncate-on-demotion path

• 2.1 Write RED tests for the self-restart primitive and demotion sequence

  • Create testsuite/test_demotion_convergence.c and register it in meson; declare controller_saved_argv, ha_self_restart, and the XCE_TESTING seams (ha_restart_hook, ha_fatal_hook) in ha.h so tests compile
  • Tests: become-follower invoked while role is CTRL_PRIMARY → persistence hooks disabled (verify via hooks_registered reset and a hook-counter probe) BEFORE further events are persisted, persist_truncate_wal_above called with the committed_lsn cutoff, WARN with both LSNs emitted, ha_restart_hook invoked exactly once; ha_self_restart with a failing restart hook path → ha_fatal_hook invoked (FATAL contract); become-follower from STANDBY/ELECTING/INIT → no truncation, no restart (regression); integration: append a committed prefix + uncommitted suffix, run the demotion seam, then replay the surviving WAL into a fresh context → committed-only state (provisional entries absent)
  • Observable completion: new cases FAIL (red) — current raft_cb_become_follower (ha.c:627-690 anchor) neither truncates nor disables hooks
  • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.4

• 2.2 Implement the self-restart primitive and demotion convergence to green

  • main.c: save the NULL-terminated argv before argp_parse (main.c:112 anchor); expose controller_saved_argv()
  • ha.c: ha_self_restart(c) — flush logs, execv(saved_argv[0], saved_argv); on exec return → ERROR + abort(); under XCE_TESTING invoke ha_restart_hook (and ha_fatal_hook instead of abort); audit/ensure listener and peer fds are CLOEXEC so the re-exec'd image inherits no live sockets
  • raft_cb_become_follower: read prior role before the role store; was-PRIMARY branch disables the three persistence hooks inline (reuse the ha_demote_to_standby disable sequence, ha.c:1712-1726 anchor) then schedules a one-shot delaymgr convergence task: truncate above committed_lsn → WARN range → ha_self_restart (file I/O and exec never on the callback thread)
  • Observable completion: all task 2.1 tests PASS; non-PRIMARY demotion behavior byte-equivalent to before
  • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7

• 3. Rejoin reconciliation at the CtrlHelloReply intake

• 3.1 Write RED tests for clamped rejoin truncation with once-guard

  • Tests in test_demotion_convergence.c: ha_rejoin_reconcile with local tip > primary boundary → truncation at max(primary_committed, local committed_lsn) (clamp verified with a local committed_lsn ABOVE the reported boundary) and ha_restart_hook invoked once; tip <= boundary → no truncation, no restart, returns 0; second reconcile call in the same process lifetime → no second restart (defensive once-guard); intake integration: delivering a CtrlHelloReply through the cluster.c handler path stores last_known_committed_lsn (existing) AND triggers reconciliation
  • Observable completion: new cases FAIL (red) — no reconciliation exists at the intake (cluster.c:283-298 anchor)
  • Requirements: 2.1, 2.2, 2.3

• 3.2 Implement rejoin reconciliation to green

  • ha.c: ha_rejoin_reconcile(cluster_t *c, uint64_t primary_committed_lsn) — evaluate local_tip = persist.current_lsn - 1 vs boundary; clamped cutoff; WARN range; _Atomic int rejoin_reconciled once-guard; truncation + restart deferred to a delaymgr task (intake may run on iomgr); reuses ha_self_restart from task 2.2
  • cluster.c: CtrlHelloReply intake calls ha_rejoin_reconcile after the existing last_known_committed_lsn store; RDHUP truncation path (ha.c:3416-3420 anchor) left unchanged
  • Observable completion: all task 3.1 tests PASS; a node whose tip matches the boundary joins with zero side effects
  • Depends: 2.2
  • Requirements: 2.1, 2.2, 2.3

• 4. ACK-driven catch-up continuation and re-inclusion trigger

• 4.1 Write RED tests for continuation, single-flight, and parity re-inclusion

  • Create testsuite/test_convergence_verification.c and register it in meson
  • Tests (use the existing ha_repl_stream_wal_hook / xce_wal_stream_request_hook seams to count streaming invocations): a walAck with acked_lsn < max(committed_lsn, last_written) schedules exactly one continuation that streams [acked_lsn+1, target] WITHOUT any intervening heartbeat/stateReport dispatch; a second walAck while the continuation is in flight schedules nothing (single-flight); after the continuation completes, the next lagging walAck schedules again; a walAck with acked_lsn >= target calls repl_complete_catchup and the standby's in_catch_up clears; the heartbeat decision path (ha_dispatch_repl_action_for_standby) still produces HA_REPL_SEND_SNAPSHOT / HA_REPL_STREAM_WAL decisions unchanged (fallback + snapshot ownership pin)
  • Observable completion: new cases FAIL (red) — current walAck branch (ha.c:2737-2754 anchor) only records the ACK; repl_complete_catchup (repl.c:521) has no production caller
  • Requirements: 5.1, 5.2, 5.3, 7.1

• 4.2 Implement the continuation trigger and parity re-inclusion to green

  • ha.c walAck branch: after repl_on_ack, derive target as at ha.c:2727-2732; behind target → schedule the continuation via delaymgr unless the standby's catchup_inflight flag is set; at parity → repl_complete_catchup (ha-commit-engine wiring propagates the membership re-evaluation)
  • config_repl.c: continuation task body — re-validate the endpoint against the registry, re-derive target, call ha_stream_wal_to_standby(ep, wal_path, acked_lsn, target) (internals untouched), clear catchup_inflight on all exit paths
  • ha.h: per-standby _Atomic int catchup_inflight storage with slot identity stable across schedule/run
  • Observable completion: all task 4.1 tests PASS; heartbeat decision loop diff-clean (fallback + snapshot initiator preserved)
  • Requirements: 5.1, 5.2, 5.3, 7.1

• 5. Quorum transition observability

• 5.1 Write RED tests for once-per-transition quorum logging

  • Tests in test_convergence_verification.c (log-capture or counter-hook pattern): dropping standbys below majority logs the loss WARN exactly once with connected/required counts; repeated dispatch-gate rejections while quorum stays lost add no further transition logs; restoring a standby logs the restore WARN once; a second loss/restore cycle logs again (edge detector, not a latch)
  • Observable completion: new cases FAIL (red) — no transition logging exists (ha.c:1507-1525 anchor)
  • Requirements: 8.1, 8.2, 8.3

• 5.2 Implement quorum transition logging to green

  • ha.h: _Atomic int quorum_lost on ha_state_t; ha.c: ha_log_quorum_transition(ha) — CAS edge detection around ha_has_write_quorum, WARN with connected/required counts on each edge only
  • Call sites: promotion standby registration (ha.c:531 anchor — NOTE: this anchor is inside raft_cb_become_leader (ha.c:445-617), promotion code owned and rewritten wholesale by ha-single-leader-role; re-anchor after that spec lands, or coordinate with it if this task is implemented first), RDHUP removal (ha.c:3160-3166 anchor), on_standby_connected (ha.c:3764+ anchor), and the ha_dispatch_gate quorum-rejection branch as backstop
  • Observable completion: all task 5.1 tests PASS; no log line is emitted per rejected operation
  • Requirements: 8.1, 8.2, 8.3

• 6. Verification of existing behaviors (regression pins — no production code)

• 6.1 Write regression tests pinning full fan-out, snapshot fallback, and degraded-mode service

  • Tests in test_convergence_verification.c: repl_broadcast enqueues to every registered active standby and skips only empty slots and in_catch_up (umbrella 6.1 pin); ha_evaluate_repl_action returns HA_REPL_SEND_SNAPSHOT when standby_lsn < oldest_wal_lsn and the snapshot path takes a fresh checkpoint before sending (umbrella 6.3 pin, decision boundary only); ha_dispatch_gate without write quorum admits OPCode_get, heartbeat, the replication opcode allowlist, and the fact-shaped opcodes (OPCode_notify/update/setpreason/clrpreason — admitted without quorum per ha-commit-barrier-rollout's fact allowlist) while rejecting request-shaped opcodes only (OPCode_jobsub/jobcontrol/queuecontrol/hostcontrol/jobdispatch/join) with -XCE_ERR_NOT_READY (umbrella 7.3 pin); repl_complete_catchup clears in_catch_up and the standby re-enters broadcast iteration (umbrella 6.4 mechanism pin, complements task 4's trigger test)
  • Observable completion: all pins PASS against current code without production changes (these encode behavior that already exists; any failure is a real regression or a wrong assumption to escalate)
  • Requirements: 4.1, 6.1, 7.1, 9.1, 9.2

• 7. Full controller regression under ASAN

  • Build and run the complete controller testsuite with ASAN (/xce-build controller); fix any regression from the become-follower changes, the collect-path wrapper, or the walAck branch additions
  • Audit: grep confirms status/API read paths are unfiltered, commit.c is untouched by this spec, and raft_cb_become_follower's non-PRIMARY behavior is unchanged
  • Observable completion: full testsuite green under ASAN; audit results recorded in completion notes
  • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.1, 2.2, 2.3, 3.1, 3.2, 3.3, 3.4, 4.1, 5.1, 5.2, 5.3, 6.1, 7.1, 8.1, 8.2, 8.3, 9.1, 9.2

• 8. 3-node E2E convergence test (extends ha-wal-replication task 5.4)

  • Blocked: manual live-cluster integration test — requires a running /xce-deploy-ha 3-node cluster with kill-primary / isolate / heal scenarios and operator-observed convergence assertions; not automatable in the unit suite (same blocked-manual status as ha-wal-replication task 5.4). Unit-testable in-process halves of these assertions are already covered: single-outcome demotion+replay (test_demotion_convergence), rejoin reconciliation clamp/once-guard, commit-gated egress (test_egress_filter), quorum-loss/restore WARN once (test_convergence_verification). Run on real infra to close 2.4/8.1/8.2/9.1/9.2 end-to-end.
  • DEPENDS on ha-single-leader-role having landed (direct promotion, COMMITTING removed — the demotion/rejoin path asserted here is the post-removal one); requires a live /xce-deploy-ha cluster — manual integration test, not automatable in the unit suite (same constraint as ha-wal-replication 5.4)
  • Deploy 3 nodes; submit confirmed jobs → kill primary → promoted standby owns all confirmed jobs; force a submit to time out at the barrier (withhold standby ACKs) → verify single-outcome convergence: the entry is committed on all members or absent on all members, the requester's failure answer is never retroactively changed, and the scheduler never dispatched the unconfirmed job
  • Restart the killed ex-primary → verify exactly one self-restart/reconciliation cycle and a converged rejoin (WAL tip at the cluster's committed boundary; no phantom jobs in joblist)
  • Isolate the primary from both standbys → submits fail, joblist/status still served, quorum-loss WARN in the log; heal → quorum-restored WARN, submits succeed without intervention
  • Observable completion: all E2E assertions hold on a live /xce-deploy-ha cluster; results recorded in completion notes
  • Depends: 7
  • Requirements: 2.4, 9.1, 9.2, 8.1, 8.2, 3.1
  • DONE 2026-06-24 (live /xce-test-ha, 8 rounds): PASS: 8 rounds, 30 jobs submitted, zero job loss; all 3 nodes led (ctrl-1,ctrl-2,ctrl-3) (exit 0). Live-covered E2E bullets: kill-primary → promoted standby owns all confirmed jobs (zero loss across 8 failovers, jobs 3→30, every round expect-job ok — Req 9.1/9.2/3.1); restart killed ex-primary → converged rejoin, 3-of-3 reconverged each round, no phantom jobs in joblist (exactly N present, never N+extra — Req 2.4 single-outcome on the survive-and-rejoin side); plus deterministic leadership coverage (ctrl-3 forced to lead via the RequestVote-freshness fence maneuver). Build note: required a controller rebuild first — the deployed binaries predated the bug-1077 libxce fix; the soak then also served as the live regression for bug-1077 (~40 failover client round-trips, no SEGV). NOT exercised by this live soak (remain covered by the in-process unit halves, tasks 2/4/5): the withhold-ACK barrier-timeout single-outcome (test_demotion_convergence) and the isolate-from-both-standbys quorum-loss/restore WARN (test_convergence_verification) — both need network partition / ACK injection the soak driver does not drive. Optional future live validation: a partition/heal scenario for Req 8.1/8.2 end-to-end.

Implementation Notes

• Task 1.2 — predicate placed in new egress.c, not dataex.c (design D3 "refinement, not contradiction"): dataex.c defines notify() (dataex.c:384) and collect_confvars(), both already stubbed in the unit-test binary (test_commit_barrier_reply.c, test_worker_join.c). Linking dataex.c into controller_unit_test_src is a duplicate-symbol error, so job_submit_committed/collect_jobs_committed live in a dedicated TU src/controller/egress.c, registered in BOTH controller_src and controller_unit_test_src. Header decls remain in cluster.h (the controller's "dataex header" — there is no dataex.h). Any future scheduler-bound export path adopts the predicate from there.
• Test-link rule: a production symbol that the unit-test suite already stubs cannot have its owning .c added to controller_unit_test_src. Factor newly-testable logic into its own TU instead.
• restore_job gained an entry_lsn parameter (replay tagging). Callers updated: cluster.h decl, job.c def, persist.c extern + apply_wal_entry caller, test_repl_apply.c stub. Any new caller must pass the WAL entry header LSN (hdr->lsn), or 0 if unknown (0 = exportable).
• Build note: the full meson controller suite SIGTERMs at the 300s wall-clock cap on the slow commit-barrier timeout suites (pre-existing). For per-task GREEN confirmation, run controller_unit_test -t <suite> directly with ASAN_OPTIONS=detect_leaks=0 (harness teardown leaks are a pre-existing accepted pattern; the meson gate sets the same option).
• Task 2.1 RED scaffold + anchor drift (post ha-single-leader-role): throwaway stubs live in new src/controller/ha_convergence.c (controller_saved_argv→NULL, ha_self_restart→return 0, seam pointers ha_restart_hook/ha_fatal_hook=NULL) — task 2.2 moves the real controller_saved_argv to main.c and ha_self_restart + the demotion branch to ha.c per design file-layout. A test-only seam raft_cb_become_follower_with_cluster_for_test (preserves raft_c=c) was added to ha.c. Verified anchors: raft_cb_become_follower ha.c:622; ha_demote_to_standby ha.c:1711 (hook-disable sequence to reuse); persist_truncate_wal_above persist.c:1545; role enum CTRL_PRIMARY=0/STANDBY=1/INIT=2/ELECTING=3 (ha.h:59-64); committed_lsn/current_lsn on persist_t in src/include/persist.h; MAX_STANDBYS=3.
• Follow-up for task 2.2: add a CTRL_INIT non-PRIMARY regression case to test_demotion_convergence.c (currently only STANDBY+ELECTING are exercised though the comment says "ELECTING/INIT") and correct/keep the comment accurate. 1.5 (WARN with both LSNs) is verified only by the truncation-cutoff proxy — no log-capture harness exists; covered directly only at E2E (task 8). [DONE in 2.2: INIT case added, comment corrected.]
• Task 2.2 — latent persist bug fixed: persist_truncate_wal_above (persist.c) had a no-op early-return for cutoff_seg_idx < 0 (cutoff strictly below the oldest retained segment — the first real caller is the demotion path, where a demoted ex-primary's committed boundary can sit below its entire retained log). Now deletes all segments, sets current_lsn = cutoff+1, committed_lsn = cutoff — correct truncate-above semantics; reviewed safe for the checkpoint path (P0 load_snapshot restores only LSN counters, no entity state; entity-bearing snapshots out of scope) and the promotion-truncation caller (no data loss; the discriminant cannot conflate "below oldest" with "above newest"). Follow-up (non-blocking, for task 6.1 or 7): add a targeted persist_truncate_wal_above test exercising the cutoff_seg_idx < 0 branch — currently established correct by analysis only, no direct assertion.
• CLOEXEC: controller sockets are already FD_CLOEXEC at creation (socket_create netio.c:219, socket_accept netio.c:308) — the re-exec'd image inherits no live sockets; no netio.c change was needed.
• Disk space: heavy nix builds can fill the root overlay with stale /tmp/nix-shell.* AND /tmp/xce_* test dirs (reviewers reclaimed ~16GB nix-shell + ~570GB stale xce test dirs). If a build/test fails on "No space left on device", clear stale /tmp/nix-shell.* and /tmp/xce_* and retry — not a code failure.
• Task 3.2 — delaymgr one-shot helper + once-guard test strength: ha_rejoin_reconcile lives in ha_convergence.c (alongside ha_self_restart/demotion task, unit-linkable) and defers truncate→WARN→restart to a delaymgr one-shot, reusing the 2.2 pattern. The test helper dc_run_pending_task was fixed to DEQUEUE (ordq_get+free) mirroring the real delaymgr worker (delay.c:65-90) — the prior peek-without-dequeue caused a test-only heap-UAF (re-running an already-freed one-shot); production never re-runs a one-shot, so the body-frees-own-ctx pattern is safe (bug-343). Follow-up (non-blocking, for task 6.1 or 7): add a once-guard test that calls ha_rejoin_reconcile TWICE before draining the delaymgr (tip unchanged) and asserts exactly one task scheduled — the current second-call test runs truncation synchronously between calls, so call 2 exits at the 2.3 early-return and exercises only the structural guard, not the defensive CAS (CAS is present + correct, needed for a duplicate CtrlHelloReply arriving before the deferred truncation runs).
• Task 5.1 done (RED) + 5.2 notes: counter-hook seam ha_quorum_transition_log_hook(int lost,int connected,int required) is the observable (no log-capture harness exists); _Atomic int quorum_lost added to ha_state_t; no-op stub ha_log_quorum_transition in ha_convergence.c. 5.2 must: (a) report connected = repl_active_standby_count(repl) and required = n_peers/2 to match the ha_has_write_quorum BODY (ha.c:~1488-1498) — its header comment says "ceil(n_peers/2)+1" but that's a pre-existing stale comment; do NOT change the quorum math, just mirror the body so the 5.1 tests pass; (b) re-grep the promotion standby-registration call site inside raft_cb_become_leader (rewritten by ha-single-leader-role) — RDHUP removal (~ha.c:3160), on_standby_connected (~ha.c:3764), and the gate-rejection backstop (~ha.c:1511) are verified present; (c) invoke the hook once per WARN emitted (CAS edge-detect on quorum_lost).
• Task 5.2 done (GREEN) — quorum-decision duplication note: ha_log_quorum_transition (ha_convergence.c) re-derives connected = repl_active_standby_count(repl) / required = n_peers/2 / single-node short-circuit as a FAITHFUL same-input mirror of the file-static ha_has_write_quorum (ha.c) — it cannot report a state inconsistent with the gate. Non-blocking maintainability risk: if ha_has_write_quorum's decision ever changes, this mirror must change too. Follow-up (optional cleanup): expose ha_has_write_quorum via a non-static accessor and call it from ha_log_quorum_transition instead of duplicating. Call sites wired (post ha-single-leader-role rewrite): promotion ha.c:567, RDHUP removal ha.c:3180, on_standby_connected ha.c:3837, gate backstop ha.c:1521.
• Task 7 done (GO) — full ASAN regression + boundary audit: full controller suite green under ASAN, established suite-by-suite (50/50 suites, 524 tests, 0 failures, 0 ASAN memory errors) because the aggregate meson controller test SIGTERMs at the pre-existing 300s cap (slow commit-barrier timeout suites serialized). Audits PASS: (1) status/API read paths UNFILTERED — collect_jobs_committed sole caller is sched.c:218 (full push) + the job_submit_committed predicate filter at the JOB_NEW push; apiserver.c:83 + opget.c:173 stay on plain collect_jobs; (2) commit.c/commit.h UNTOUCHED across the whole spec (git diff --stat e558a62..HEAD empty for both); (3) raft_cb_become_follower demotion additions guarded by prior_role == CTRL_PRIMARY && c != NULL (ha.c:687, :711) — STANDBY/ELECTING/INIT byte-equivalent to pre-spec. Production files touched match the design File Structure Plan (with dataex.c→egress.c per D3). No production change needed in task 7.
• Task 8 (live 3-node E2E) — BLOCKED (manual): requires a running /xce-deploy-ha cluster + operator-observed kill/isolate/heal convergence; not automatable in the unit suite (same status as ha-wal-replication 5.4). All unit-testable in-process halves are covered by tasks 1-6. To close: run on real infra and record results.