myxce/.kiro/specs/ha-convergence-recovery/tasks.md

# Implementation Plan: ha-convergence-recovery

> TDD ordering per project steering (`testing.md`): every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first. No `(P)` markers: groups 2–5 all modify `ha.c`/`ha.h` and group 1's tagging touches `persist.c` (shared with the demotion-path file set via the truncation/recovery seams), so the plan is sequential — the one nominally independent group (egress filter) is placed first instead of parallelized.
> Verify exact line numbers with grep before editing (this spec's research.md §1 has the 2026-06-11 verified anchors; files drift).
> Task 8 (E2E) additionally requires **ha-single-leader-role** to have landed and a live `/xce-deploy-ha` cluster.

- [x] 1. Egress filter: submit_lsn tagging and commit-watermark gate on scheduler exports
- [x] 1.1 Write RED tests for tagging and the scheduler-export watermark
  - Create `testsuite/test_egress_filter.c` (XCE_TESTING, AAA structure) and register it in `controller_unit_test_src` (meson.build); add `uint64_t submit_lsn` to the `job_t` wrapper (types.h:483-485 anchor) and declare `job_submit_committed` / `collect_jobs_committed` in the dataex header so tests compile
  - Tests: a live-submitted job is tagged with its WAL entry LSN; a job restored via `apply_wal_entry`/`restore_job` replay is tagged with the replayed entry's LSN; a job with `submit_lsn > committed_lsn` is absent from `collect_jobs_committed` results and from the JOB_NEW incremental push list; the same job appears once `committed_lsn` advances to cover it; `submit_lsn == 0` is always exportable; plain `collect_jobs` (status/API path) still returns the uncommitted job
  - Observable completion: build succeeds with the new field/header declarations; all new test cases FAIL (red) against unimplemented predicate/tagging
  - _Requirements: 3.1, 3.2, 3.3, 3.4_

- [x] 1.2 Implement tagging and the watermark gate to green
  - Tagging: `persist_job_hook_fun` (persist.c:1107-1125 anchor) sets `submit_lsn = written_lsn` on each submitted job after a successful append; `apply_wal_entry` (persist.c:729-731 anchor) passes `hdr->lsn` through to `restore_job` (job.c:71), which tags the restored job
  - Predicate + wrapper in dataex.c: `job_submit_committed` = `submit_lsn == 0 || submit_lsn <= committed_lsn` (one atomic load, lock-free); `collect_jobs_committed` wraps `collect_jobs` with the predicate
  - Scheduler-bound consumers only: `xcesched_push_jobs` (sched.c:183 anchor) collects via the committed-only wrapper; `xcesched_push_joblist` (JOB_NEW push fed by `sched_job_new_fun`, sched.c:236-248 anchor) filters its list with the predicate; apiserver.c:83 and opget.c:173 remain on unfiltered `collect_jobs`
  - Observable completion: all task 1.1 tests PASS; grep confirms no status/API read path calls the committed-only wrapper
  - _Requirements: 3.1, 3.2, 3.3, 3.4_

- [x] 2. Demotion convergence: self-restart primitive and the truncate-on-demotion path
- [x] 2.1 Write RED tests for the self-restart primitive and demotion sequence
  - Create `testsuite/test_demotion_convergence.c` and register it in meson; declare `controller_saved_argv`, `ha_self_restart`, and the `XCE_TESTING` seams (`ha_restart_hook`, `ha_fatal_hook`) in ha.h so tests compile
  - Tests: become-follower invoked while role is CTRL_PRIMARY → persistence hooks disabled (verify via `hooks_registered` reset and a hook-counter probe) BEFORE further events are persisted, `persist_truncate_wal_above` called with the `committed_lsn` cutoff, WARN with both LSNs emitted, `ha_restart_hook` invoked exactly once; `ha_self_restart` with a failing restart hook path → `ha_fatal_hook` invoked (FATAL contract); become-follower from STANDBY/ELECTING/INIT → no truncation, no restart (regression); integration: append a committed prefix + uncommitted suffix, run the demotion seam, then replay the surviving WAL into a fresh context → committed-only state (provisional entries absent)
  - Observable completion: new cases FAIL (red) — current `raft_cb_become_follower` (ha.c:627-690 anchor) neither truncates nor disables hooks
  - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.4_

- [x] 2.2 Implement the self-restart primitive and demotion convergence to green
  - main.c: save the NULL-terminated argv before `argp_parse` (main.c:112 anchor); expose `controller_saved_argv()`
  - ha.c: `ha_self_restart(c)` — flush logs, `execv(saved_argv[0], saved_argv)`; on exec return → ERROR + `abort()`; under XCE_TESTING invoke `ha_restart_hook` (and `ha_fatal_hook` instead of abort); audit/ensure listener and peer fds are CLOEXEC so the re-exec'd image inherits no live sockets
  - `raft_cb_become_follower`: read prior role before the role store; was-PRIMARY branch disables the three persistence hooks inline (reuse the `ha_demote_to_standby` disable sequence, ha.c:1712-1726 anchor) then schedules a one-shot delaymgr convergence task: truncate above `committed_lsn` → WARN range → `ha_self_restart` (file I/O and exec never on the callback thread)
  - Observable completion: all task 2.1 tests PASS; non-PRIMARY demotion behavior byte-equivalent to before
  - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7_

- [x] 3. Rejoin reconciliation at the CtrlHelloReply intake
- [x] 3.1 Write RED tests for clamped rejoin truncation with once-guard
  - Tests in `test_demotion_convergence.c`: `ha_rejoin_reconcile` with local tip > primary boundary → truncation at `max(primary_committed, local committed_lsn)` (clamp verified with a local committed_lsn ABOVE the reported boundary) and `ha_restart_hook` invoked once; tip <= boundary → no truncation, no restart, returns 0; second reconcile call in the same process lifetime → no second restart (defensive once-guard); intake integration: delivering a CtrlHelloReply through the cluster.c handler path stores `last_known_committed_lsn` (existing) AND triggers reconciliation
  - Observable completion: new cases FAIL (red) — no reconciliation exists at the intake (cluster.c:283-298 anchor)
  - _Requirements: 2.1, 2.2, 2.3_

- [x] 3.2 Implement rejoin reconciliation to green
  - ha.c: `ha_rejoin_reconcile(cluster_t *c, uint64_t primary_committed_lsn)` — evaluate `local_tip = persist.current_lsn - 1` vs boundary; clamped cutoff; WARN range; `_Atomic int rejoin_reconciled` once-guard; truncation + restart deferred to a delaymgr task (intake may run on iomgr); reuses `ha_self_restart` from task 2.2
  - cluster.c: CtrlHelloReply intake calls `ha_rejoin_reconcile` after the existing `last_known_committed_lsn` store; RDHUP truncation path (ha.c:3416-3420 anchor) left unchanged
  - Observable completion: all task 3.1 tests PASS; a node whose tip matches the boundary joins with zero side effects
  - _Depends: 2.2_
  - _Requirements: 2.1, 2.2, 2.3_

- [x] 4. ACK-driven catch-up continuation and re-inclusion trigger
- [x] 4.1 Write RED tests for continuation, single-flight, and parity re-inclusion
  - Create `testsuite/test_convergence_verification.c` and register it in meson
  - Tests (use the existing `ha_repl_stream_wal_hook` / `xce_wal_stream_request_hook` seams to count streaming invocations): a walAck with `acked_lsn < max(committed_lsn, last_written)` schedules exactly one continuation that streams `[acked_lsn+1, target]` WITHOUT any intervening heartbeat/stateReport dispatch; a second walAck while the continuation is in flight schedules nothing (single-flight); after the continuation completes, the next lagging walAck schedules again; a walAck with `acked_lsn >= target` calls `repl_complete_catchup` and the standby's `in_catch_up` clears; the heartbeat decision path (`ha_dispatch_repl_action_for_standby`) still produces HA_REPL_SEND_SNAPSHOT / HA_REPL_STREAM_WAL decisions unchanged (fallback + snapshot ownership pin)
  - Observable completion: new cases FAIL (red) — current walAck branch (ha.c:2737-2754 anchor) only records the ACK; `repl_complete_catchup` (repl.c:521) has no production caller
  - _Requirements: 5.1, 5.2, 5.3, 7.1_

- [x] 4.2 Implement the continuation trigger and parity re-inclusion to green
  - ha.c walAck branch: after `repl_on_ack`, derive target as at ha.c:2727-2732; behind target → schedule the continuation via delaymgr unless the standby's `catchup_inflight` flag is set; at parity → `repl_complete_catchup` (ha-commit-engine wiring propagates the membership re-evaluation)
  - config_repl.c: continuation task body — re-validate the endpoint against the registry, re-derive target, call `ha_stream_wal_to_standby(ep, wal_path, acked_lsn, target)` (internals untouched), clear `catchup_inflight` on all exit paths
  - ha.h: per-standby `_Atomic int catchup_inflight` storage with slot identity stable across schedule/run
  - Observable completion: all task 4.1 tests PASS; heartbeat decision loop diff-clean (fallback + snapshot initiator preserved)
  - _Requirements: 5.1, 5.2, 5.3, 7.1_

- [x] 5. Quorum transition observability
- [x] 5.1 Write RED tests for once-per-transition quorum logging
  - Tests in `test_convergence_verification.c` (log-capture or counter-hook pattern): dropping standbys below majority logs the loss WARN exactly once with connected/required counts; repeated dispatch-gate rejections while quorum stays lost add no further transition logs; restoring a standby logs the restore WARN once; a second loss/restore cycle logs again (edge detector, not a latch)
  - Observable completion: new cases FAIL (red) — no transition logging exists (ha.c:1507-1525 anchor)
  - _Requirements: 8.1, 8.2, 8.3_

- [x] 5.2 Implement quorum transition logging to green
  - ha.h: `_Atomic int quorum_lost` on `ha_state_t`; ha.c: `ha_log_quorum_transition(ha)` — CAS edge detection around `ha_has_write_quorum`, WARN with connected/required counts on each edge only
  - Call sites: promotion standby registration (ha.c:531 anchor — NOTE: this anchor is inside `raft_cb_become_leader` (ha.c:445-617), promotion code owned and rewritten wholesale by ha-single-leader-role; re-anchor after that spec lands, or coordinate with it if this task is implemented first), RDHUP removal (ha.c:3160-3166 anchor), `on_standby_connected` (ha.c:3764+ anchor), and the `ha_dispatch_gate` quorum-rejection branch as backstop
  - Observable completion: all task 5.1 tests PASS; no log line is emitted per rejected operation
  - _Requirements: 8.1, 8.2, 8.3_

- [x] 6. Verification of existing behaviors (regression pins — no production code)
- [x] 6.1 Write regression tests pinning full fan-out, snapshot fallback, and degraded-mode service
  - Tests in `test_convergence_verification.c`: `repl_broadcast` enqueues to every registered active standby and skips only empty slots and `in_catch_up` (umbrella 6.1 pin); `ha_evaluate_repl_action` returns HA_REPL_SEND_SNAPSHOT when `standby_lsn < oldest_wal_lsn` and the snapshot path takes a fresh checkpoint before sending (umbrella 6.3 pin, decision boundary only); `ha_dispatch_gate` without write quorum admits `OPCode_get`, heartbeat, the replication opcode allowlist, and the fact-shaped opcodes (`OPCode_notify`/`update`/`setpreason`/`clrpreason` — admitted without quorum per ha-commit-barrier-rollout's fact allowlist) while rejecting request-shaped opcodes only (`OPCode_jobsub`/`jobcontrol`/`queuecontrol`/`hostcontrol`/`jobdispatch`/`join`) with `-XCE_ERR_NOT_READY` (umbrella 7.3 pin); `repl_complete_catchup` clears `in_catch_up` and the standby re-enters broadcast iteration (umbrella 6.4 mechanism pin, complements task 4's trigger test)
  - Observable completion: all pins PASS against current code without production changes (these encode behavior that already exists; any failure is a real regression or a wrong assumption to escalate)
  - _Requirements: 4.1, 6.1, 7.1, 9.1, 9.2_

- [x] 7. Full controller regression under ASAN
  - Build and run the complete controller testsuite with ASAN (`/xce-build controller`); fix any regression from the become-follower changes, the collect-path wrapper, or the walAck branch additions
  - Audit: grep confirms status/API read paths are unfiltered, `commit.c` is untouched by this spec, and `raft_cb_become_follower`'s non-PRIMARY behavior is unchanged
  - Observable completion: full testsuite green under ASAN; audit results recorded in completion notes
  - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.1, 2.2, 2.3, 3.1, 3.2, 3.3, 3.4, 4.1, 5.1, 5.2, 5.3, 6.1, 7.1, 8.1, 8.2, 8.3, 9.1, 9.2_

- [x] 8. 3-node E2E convergence test (extends ha-wal-replication task 5.4)
  - _Blocked: manual live-cluster integration test — requires a running `/xce-deploy-ha` 3-node cluster with kill-primary / isolate / heal scenarios and operator-observed convergence assertions; not automatable in the unit suite (same blocked-manual status as ha-wal-replication task 5.4). Unit-testable in-process halves of these assertions are already covered: single-outcome demotion+replay (test_demotion_convergence), rejoin reconciliation clamp/once-guard, commit-gated egress (test_egress_filter), quorum-loss/restore WARN once (test_convergence_verification). Run on real infra to close 2.4/8.1/8.2/9.1/9.2 end-to-end._
  - **DEPENDS on ha-single-leader-role having landed** (direct promotion, COMMITTING removed — the demotion/rejoin path asserted here is the post-removal one); requires a live `/xce-deploy-ha` cluster — manual integration test, not automatable in the unit suite (same constraint as ha-wal-replication 5.4)
  - Deploy 3 nodes; submit confirmed jobs → kill primary → promoted standby owns all confirmed jobs; force a submit to time out at the barrier (withhold standby ACKs) → verify single-outcome convergence: the entry is committed on all members or absent on all members, the requester's failure answer is never retroactively changed, and the scheduler never dispatched the unconfirmed job
  - Restart the killed ex-primary → verify exactly one self-restart/reconciliation cycle and a converged rejoin (WAL tip at the cluster's committed boundary; no phantom jobs in joblist)
  - Isolate the primary from both standbys → submits fail, `joblist`/status still served, quorum-loss WARN in the log; heal → quorum-restored WARN, submits succeed without intervention
  - Observable completion: all E2E assertions hold on a live `/xce-deploy-ha` cluster; results recorded in completion notes
  - _Depends: 7_
  - _Requirements: 2.4, 9.1, 9.2, 8.1, 8.2, 3.1_
  - **DONE 2026-06-24 (live `/xce-test-ha`, 8 rounds):** `PASS: 8 rounds, 30 jobs submitted, zero job loss; all 3 nodes led (ctrl-1,ctrl-2,ctrl-3)` (exit 0). Live-covered E2E bullets: kill-primary → promoted standby owns **all** confirmed jobs (zero loss across 8 failovers, jobs 3→30, every round `expect-job` ok — Req 9.1/9.2/3.1); restart killed ex-primary → converged rejoin, 3-of-3 reconverged each round, **no phantom jobs** in `joblist` (exactly N present, never N+extra — Req 2.4 single-outcome on the survive-and-rejoin side); plus deterministic leadership coverage (ctrl-3 forced to lead via the RequestVote-freshness fence maneuver). Build note: required a controller rebuild first — the deployed binaries predated the bug-1077 libxce fix; the soak then also served as the live regression for bug-1077 (~40 failover client round-trips, no SEGV). NOT exercised by this live soak (remain covered by the in-process unit halves, tasks 2/4/5): the withhold-ACK barrier-timeout single-outcome (test_demotion_convergence) and the isolate-from-both-standbys quorum-loss/restore WARN (test_convergence_verification) — both need network partition / ACK injection the soak driver does not drive. Optional future live validation: a partition/heal scenario for Req 8.1/8.2 end-to-end.

## Implementation Notes

- **Task 1.2 — predicate placed in new `egress.c`, not `dataex.c`** (design D3 "refinement, not contradiction"): `dataex.c` defines `notify()` (dataex.c:384) and `collect_confvars()`, both already stubbed in the unit-test binary (test_commit_barrier_reply.c, test_worker_join.c). Linking `dataex.c` into `controller_unit_test_src` is a duplicate-symbol error, so `job_submit_committed`/`collect_jobs_committed` live in a dedicated TU `src/controller/egress.c`, registered in BOTH `controller_src` and `controller_unit_test_src`. Header decls remain in `cluster.h` (the controller's "dataex header" — there is no `dataex.h`). Any future scheduler-bound export path adopts the predicate from there.
- **Test-link rule**: a production symbol that the unit-test suite already stubs cannot have its owning `.c` added to `controller_unit_test_src`. Factor newly-testable logic into its own TU instead.
- **`restore_job` gained an `entry_lsn` parameter** (replay tagging). Callers updated: cluster.h decl, job.c def, persist.c extern + apply_wal_entry caller, test_repl_apply.c stub. Any new caller must pass the WAL entry header LSN (`hdr->lsn`), or `0` if unknown (0 = exportable).
- **Build note**: the full meson controller suite SIGTERMs at the 300s wall-clock cap on the slow commit-barrier timeout suites (pre-existing). For per-task GREEN confirmation, run `controller_unit_test -t <suite>` directly with `ASAN_OPTIONS=detect_leaks=0` (harness teardown leaks are a pre-existing accepted pattern; the meson gate sets the same option).
- **Task 2.1 RED scaffold + anchor drift (post ha-single-leader-role)**: throwaway stubs live in new `src/controller/ha_convergence.c` (`controller_saved_argv`→NULL, `ha_self_restart`→return 0, seam pointers `ha_restart_hook`/`ha_fatal_hook`=NULL) — **task 2.2 moves the real `controller_saved_argv` to main.c and `ha_self_restart` + the demotion branch to ha.c** per design file-layout. A test-only seam `raft_cb_become_follower_with_cluster_for_test` (preserves `raft_c=c`) was added to ha.c. Verified anchors: `raft_cb_become_follower` ha.c:622; `ha_demote_to_standby` ha.c:1711 (hook-disable sequence to reuse); `persist_truncate_wal_above` persist.c:1545; role enum CTRL_PRIMARY=0/STANDBY=1/INIT=2/ELECTING=3 (ha.h:59-64); `committed_lsn`/`current_lsn` on `persist_t` in src/include/persist.h; MAX_STANDBYS=3.
- **Follow-up for task 2.2**: add a `CTRL_INIT` non-PRIMARY regression case to `test_demotion_convergence.c` (currently only STANDBY+ELECTING are exercised though the comment says "ELECTING/INIT") and correct/keep the comment accurate. 1.5 (WARN with both LSNs) is verified only by the truncation-cutoff proxy — no log-capture harness exists; covered directly only at E2E (task 8). [DONE in 2.2: INIT case added, comment corrected.]
- **Task 2.2 — latent persist bug fixed**: `persist_truncate_wal_above` (persist.c) had a no-op early-return for `cutoff_seg_idx < 0` (cutoff strictly below the oldest retained segment — the first real caller is the demotion path, where a demoted ex-primary's committed boundary can sit below its entire retained log). Now deletes all segments, sets `current_lsn = cutoff+1`, `committed_lsn = cutoff` — correct truncate-above semantics; reviewed safe for the checkpoint path (P0 `load_snapshot` restores only LSN counters, no entity state; entity-bearing snapshots out of scope) and the promotion-truncation caller (no data loss; the discriminant cannot conflate "below oldest" with "above newest"). **Follow-up (non-blocking, for task 6.1 or 7): add a targeted `persist_truncate_wal_above` test exercising the `cutoff_seg_idx < 0` branch** — currently established correct by analysis only, no direct assertion.
- **CLOEXEC**: controller sockets are already `FD_CLOEXEC` at creation (`socket_create` netio.c:219, `socket_accept` netio.c:308) — the re-exec'd image inherits no live sockets; no netio.c change was needed.
- **Disk space**: heavy nix builds can fill the root overlay with stale `/tmp/nix-shell.*` AND `/tmp/xce_*` test dirs (reviewers reclaimed ~16GB nix-shell + ~570GB stale xce test dirs). If a build/test fails on "No space left on device", clear stale `/tmp/nix-shell.*` and `/tmp/xce_*` and retry — not a code failure.
- **Task 3.2 — delaymgr one-shot helper + once-guard test strength**: `ha_rejoin_reconcile` lives in `ha_convergence.c` (alongside `ha_self_restart`/demotion task, unit-linkable) and defers truncate→WARN→restart to a delaymgr one-shot, reusing the 2.2 pattern. The test helper `dc_run_pending_task` was fixed to DEQUEUE (`ordq_get`+free) mirroring the real delaymgr worker (delay.c:65-90) — the prior peek-without-dequeue caused a test-only heap-UAF (re-running an already-freed one-shot); production never re-runs a one-shot, so the body-frees-own-ctx pattern is safe (bug-343). **Follow-up (non-blocking, for task 6.1 or 7): add a once-guard test that calls `ha_rejoin_reconcile` TWICE before draining the delaymgr (tip unchanged) and asserts exactly one task scheduled** — the current second-call test runs truncation synchronously between calls, so call 2 exits at the 2.3 early-return and exercises only the structural guard, not the defensive CAS (CAS is present + correct, needed for a duplicate CtrlHelloReply arriving before the deferred truncation runs).
- **Task 5.1 done (RED) + 5.2 notes**: counter-hook seam `ha_quorum_transition_log_hook(int lost,int connected,int required)` is the observable (no log-capture harness exists); `_Atomic int quorum_lost` added to ha_state_t; no-op stub `ha_log_quorum_transition` in ha_convergence.c. **5.2 must**: (a) report `connected = repl_active_standby_count(repl)` and `required = n_peers/2` to match the `ha_has_write_quorum` BODY (ha.c:~1488-1498) — its header comment says "ceil(n_peers/2)+1" but that's a pre-existing stale comment; do NOT change the quorum math, just mirror the body so the 5.1 tests pass; (b) re-grep the promotion standby-registration call site inside `raft_cb_become_leader` (rewritten by ha-single-leader-role) — RDHUP removal (~ha.c:3160), `on_standby_connected` (~ha.c:3764), and the gate-rejection backstop (~ha.c:1511) are verified present; (c) invoke the hook once per WARN emitted (CAS edge-detect on `quorum_lost`).
- **Task 5.2 done (GREEN) — quorum-decision duplication note**: `ha_log_quorum_transition` (ha_convergence.c) re-derives `connected = repl_active_standby_count(repl)` / `required = n_peers/2` / single-node short-circuit as a FAITHFUL same-input mirror of the file-static `ha_has_write_quorum` (ha.c) — it cannot report a state inconsistent with the gate. Non-blocking maintainability risk: if `ha_has_write_quorum`'s decision ever changes, this mirror must change too. **Follow-up (optional cleanup): expose `ha_has_write_quorum` via a non-static accessor and call it from `ha_log_quorum_transition`** instead of duplicating. Call sites wired (post ha-single-leader-role rewrite): promotion ha.c:567, RDHUP removal ha.c:3180, on_standby_connected ha.c:3837, gate backstop ha.c:1521.
- **Task 7 done (GO) — full ASAN regression + boundary audit**: full controller suite green under ASAN, established suite-by-suite (50/50 suites, 524 tests, 0 failures, 0 ASAN memory errors) because the aggregate meson `controller test` SIGTERMs at the pre-existing 300s cap (slow commit-barrier timeout suites serialized). Audits PASS: (1) status/API read paths UNFILTERED — `collect_jobs_committed` sole caller is `sched.c:218` (full push) + the `job_submit_committed` predicate filter at the JOB_NEW push; `apiserver.c:83` + `opget.c:173` stay on plain `collect_jobs`; (2) `commit.c`/`commit.h` UNTOUCHED across the whole spec (`git diff --stat e558a62..HEAD` empty for both); (3) `raft_cb_become_follower` demotion additions guarded by `prior_role == CTRL_PRIMARY && c != NULL` (ha.c:687, :711) — STANDBY/ELECTING/INIT byte-equivalent to pre-spec. Production files touched match the design File Structure Plan (with dataex.c→egress.c per D3). No production change needed in task 7.
- **Task 8 (live 3-node E2E) — BLOCKED (manual)**: requires a running `/xce-deploy-ha` cluster + operator-observed kill/isolate/heal convergence; not automatable in the unit suite (same status as ha-wal-replication 5.4). All unit-testable in-process halves are covered by tasks 1-6. To close: run on real infra and record results.