feat(ha-convergence-recovery): self-restart primitive + demotion convergence (task 2.2)

.kiro/specs/ha-convergence-recovery/tasks.md

@@ -18,14 +18,14 @@
   - Observable completion: all task 1.1 tests PASS; grep confirms no status/API read path calls the committed-only wrapper
   - _Requirements: 3.1, 3.2, 3.3, 3.4_
 
-- [ ] 2. Demotion convergence: self-restart primitive and the truncate-on-demotion path
+- [x] 2. Demotion convergence: self-restart primitive and the truncate-on-demotion path
 - [x] 2.1 Write RED tests for the self-restart primitive and demotion sequence
   - Create `testsuite/test_demotion_convergence.c` and register it in meson; declare `controller_saved_argv`, `ha_self_restart`, and the `XCE_TESTING` seams (`ha_restart_hook`, `ha_fatal_hook`) in ha.h so tests compile
   - Tests: become-follower invoked while role is CTRL_PRIMARY → persistence hooks disabled (verify via `hooks_registered` reset and a hook-counter probe) BEFORE further events are persisted, `persist_truncate_wal_above` called with the `committed_lsn` cutoff, WARN with both LSNs emitted, `ha_restart_hook` invoked exactly once; `ha_self_restart` with a failing restart hook path → `ha_fatal_hook` invoked (FATAL contract); become-follower from STANDBY/ELECTING/INIT → no truncation, no restart (regression); integration: append a committed prefix + uncommitted suffix, run the demotion seam, then replay the surviving WAL into a fresh context → committed-only state (provisional entries absent)
   - Observable completion: new cases FAIL (red) — current `raft_cb_become_follower` (ha.c:627-690 anchor) neither truncates nor disables hooks
   - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.4_
 
-- [ ] 2.2 Implement the self-restart primitive and demotion convergence to green
+- [x] 2.2 Implement the self-restart primitive and demotion convergence to green
   - main.c: save the NULL-terminated argv before `argp_parse` (main.c:112 anchor); expose `controller_saved_argv()`
   - ha.c: `ha_self_restart(c)` — flush logs, `execv(saved_argv[0], saved_argv)`; on exec return → ERROR + `abort()`; under XCE_TESTING invoke `ha_restart_hook` (and `ha_fatal_hook` instead of abort); audit/ensure listener and peer fds are CLOEXEC so the re-exec'd image inherits no live sockets
   - `raft_cb_become_follower`: read prior role before the role store; was-PRIMARY branch disables the three persistence hooks inline (reuse the `ha_demote_to_standby` disable sequence, ha.c:1712-1726 anchor) then schedules a one-shot delaymgr convergence task: truncate above `committed_lsn` → WARN range → `ha_self_restart` (file I/O and exec never on the callback thread)
@@ -99,4 +99,7 @@
 - **`restore_job` gained an `entry_lsn` parameter** (replay tagging). Callers updated: cluster.h decl, job.c def, persist.c extern + apply_wal_entry caller, test_repl_apply.c stub. Any new caller must pass the WAL entry header LSN (`hdr->lsn`), or `0` if unknown (0 = exportable).
 - **Build note**: the full meson controller suite SIGTERMs at the 300s wall-clock cap on the slow commit-barrier timeout suites (pre-existing). For per-task GREEN confirmation, run `controller_unit_test -t <suite>` directly with `ASAN_OPTIONS=detect_leaks=0` (harness teardown leaks are a pre-existing accepted pattern; the meson gate sets the same option).
 - **Task 2.1 RED scaffold + anchor drift (post ha-single-leader-role)**: throwaway stubs live in new `src/controller/ha_convergence.c` (`controller_saved_argv`→NULL, `ha_self_restart`→return 0, seam pointers `ha_restart_hook`/`ha_fatal_hook`=NULL) — **task 2.2 moves the real `controller_saved_argv` to main.c and `ha_self_restart` + the demotion branch to ha.c** per design file-layout. A test-only seam `raft_cb_become_follower_with_cluster_for_test` (preserves `raft_c=c`) was added to ha.c. Verified anchors: `raft_cb_become_follower` ha.c:622; `ha_demote_to_standby` ha.c:1711 (hook-disable sequence to reuse); `persist_truncate_wal_above` persist.c:1545; role enum CTRL_PRIMARY=0/STANDBY=1/INIT=2/ELECTING=3 (ha.h:59-64); `committed_lsn`/`current_lsn` on `persist_t` in src/include/persist.h; MAX_STANDBYS=3.
-- **Follow-up for task 2.2**: add a `CTRL_INIT` non-PRIMARY regression case to `test_demotion_convergence.c` (currently only STANDBY+ELECTING are exercised though the comment says "ELECTING/INIT") and correct/keep the comment accurate. 1.5 (WARN with both LSNs) is verified only by the truncation-cutoff proxy — no log-capture harness exists; covered directly only at E2E (task 8).
+- **Follow-up for task 2.2**: add a `CTRL_INIT` non-PRIMARY regression case to `test_demotion_convergence.c` (currently only STANDBY+ELECTING are exercised though the comment says "ELECTING/INIT") and correct/keep the comment accurate. 1.5 (WARN with both LSNs) is verified only by the truncation-cutoff proxy — no log-capture harness exists; covered directly only at E2E (task 8). [DONE in 2.2: INIT case added, comment corrected.]
+- **Task 2.2 — latent persist bug fixed**: `persist_truncate_wal_above` (persist.c) had a no-op early-return for `cutoff_seg_idx < 0` (cutoff strictly below the oldest retained segment — the first real caller is the demotion path, where a demoted ex-primary's committed boundary can sit below its entire retained log). Now deletes all segments, sets `current_lsn = cutoff+1`, `committed_lsn = cutoff` — correct truncate-above semantics; reviewed safe for the checkpoint path (P0 `load_snapshot` restores only LSN counters, no entity state; entity-bearing snapshots out of scope) and the promotion-truncation caller (no data loss; the discriminant cannot conflate "below oldest" with "above newest"). **Follow-up (non-blocking, for task 6.1 or 7): add a targeted `persist_truncate_wal_above` test exercising the `cutoff_seg_idx < 0` branch** — currently established correct by analysis only, no direct assertion.
+- **CLOEXEC**: controller sockets are already `FD_CLOEXEC` at creation (`socket_create` netio.c:219, `socket_accept` netio.c:308) — the re-exec'd image inherits no live sockets; no netio.c change was needed.
+- **Disk space**: heavy nix builds can fill the root overlay with stale `/tmp/nix-shell.*` dirs (a reviewer hit "No space left on device" and reclaimed ~16GB). If a build fails on disk space, clear stale nix-shell temp dirs and retry — not a code failure.