myxce/.kiro/specs/xce-test-e2e-primitives/tasks.md

# Implementation Plan

> TDD is mandatory (project rule): add the failing test/scenario first, then the change (red → green → refactor).
> Almost all work is in the single test client TU `src/controller/testsuite/test.c`, so tasks are sequential by design; only the soak conversion touches a different file. Behavioral tests are driven through `xce_test` itself (piped scripts), gating on its exit code.

- [ ] 1. Foundation: verdict, loop wrapper, ergonomics
- [x] 1.1 Session failure tracking + process exit code + abort/continue + comment/blank handling
  - Add file-scope state (`g_session_failed`, `g_continue_on_error`, `g_last_status`) and a `preprocess_and_run(line)` wrapper used by the main read loop: ignore blank and `#`-comment lines; run normal lines via `execute_line`; update `g_last_status`/`g_session_failed` from the result; in default mode stop on first failure, under `continue-on-error` keep going. `main` returns `g_session_failed ? 1 : 0`. Extend `cmd_set` with a `continue-on-error` toggle.
  - Observable: a piped script with a bogus/failing command makes `xce_test` exit non-zero; an all-valid script exits 0; with `set continue-on-error` a failing line does not stop later lines but the final exit is still non-zero; `#`/blank lines are ignored.
  - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 6.1, 6.2_
  - _Boundary: VerdictLoopWrapper_
- [x] 1.2 `echo` command
  - Add `cmd_echo` (table-registered) that prints its argument message and returns success.
  - Observable: `echo hello` emits `hello` and the command succeeds (exit unaffected).
  - _Requirements: 6.3_
  - _Boundary: VerdictLoopWrapper_
  - _Depends: 1.1_

- [ ] 2. Shell-escape
- [x] 2.1 `!<cmd>` runs a shell command and feeds `$?`
  - In `preprocess_and_run`, when the first non-whitespace char is `!`, run the remainder via `system()`; set `g_last_status = WEXITSTATUS(rc)` (non-zero on `system` failure); a non-zero status is a command failure (subject to 1.1's abort/continue + exit code).
  - Observable: `!true` succeeds and sets `$?`=0; `!false` fails the session and sets `$?`≠0; a script can `!kill ...` / `!bash start.sh ...` to drive node lifecycle.
  - _Requirements: 2.1, 2.2, 2.3, 2.4_
  - _Boundary: ShellEscape_
  - _Depends: 1.1_

- [ ] 3. Auto-variable substitution
- [x] 3.1 Substitute the fixed auto-vars `$?`, `$jobid`, `$jobcount`
  - Add `substitute_autovars(line)` (run before tokenizing in `preprocess_and_run`): replace whole-token `$?`→`g_last_status`, `$jobid`→`g_last_jobid`, `$jobcount`→`g_last_jobcount`. A reference to an unset `$jobid`/`$jobcount` fails the command (no garbage substitution). No other variable syntax is recognized.
  - Observable: after a command, `echo $?` prints the prior status; referencing `$jobid` before any successful `jobsub` fails that command rather than substituting an empty/garbage value.
  - _Requirements: 3.1, 3.3, 3.5, 3.6_
  - _Boundary: AutoVars_
  - _Depends: 1.1_

- [ ] 4. Correct job-submission reporting (bug-786) + `$jobid` capture
- [x] 4.1 `cmd_jobsub` reports success only on a clean ack with a valid JobId
  - Gate the `jobsubmit ok: JobId: <id>` report on a successful reply status AND a well-formed JobId; on success set `g_last_jobid` (feeds `$jobid`, 3.2). Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return `<0`. Never print "ok" with an uninitialized/garbage id (`63-15--1-...`).
  - Observable: a submit that does not get a clean ack returns failure and does NOT set `$jobid`; `jobsub` never prints success with a malformed id; a successful submit sets `$jobid` to the acked JobId.
  - _Requirements: 7.1, 7.2, 7.3, 3.2_
  - _Boundary: JobsubFix_
  - _Depends: 1.1, 3.1_

- [ ] 5. Job-count capture and assertions
- [x] 5.1 `cmd_joblist` sets `$jobcount`; extract a shared count/contains helper
  - Refactor `cmd_joblist` to expose an internal helper that returns the current job count and tests membership of a JobId; set `g_last_jobcount` after a `joblist`.
  - Observable: after `joblist`, `echo $jobcount` prints the number of jobs returned; the helper is reusable by the assertions.
  - _Requirements: 3.4_
  - _Boundary: JoblistCount_
  - _Depends: 1.1, 3.1_
- [x] 5.2 `expect-jobs N` assertion
  - Add `cmd_expect_jobs`: pass when the current job count equals `N`; otherwise print expected vs actual and return `<0`.
  - Observable: `expect-jobs <correct>` succeeds; `expect-jobs <wrong>` fails the session and prints expected/actual.
  - _Requirements: 5.1, 5.2_
  - _Boundary: Assertions_
  - _Depends: 5.1_
- [x] 5.3 `expect-job <jobid>` assertion
  - Add `cmd_expect_job`: pass when the JobId is present in the job list; otherwise print the missing id and return `<0`. Works with `$jobid` substitution.
  - Observable: `expect-job $jobid` passes for a submitted job; `expect-job <absent>` fails the session and prints the missing id.
  - _Requirements: 5.3, 5.4_
  - _Boundary: Assertions_
  - _Depends: 5.1, 3.1, 4.1_

- [ ] 6. Wait for elected primary
- [x] 6.1 `wait-primary [timeout_ms]`
  - Add `cmd_wait_primary`: loop a cheap, failover-aware client op (the existing `joblist`/probe path via `active_ctx`) until it succeeds or the timeout elapses (monotonic clock); default timeout when the arg is omitted; return 0 on success, `<0` on timeout.
  - Observable: against a healthy cluster `wait-primary` returns immediately; after a primary kill it blocks until a new primary serves, then succeeds; with no primary within the timeout it fails.
  - _Requirements: 4.1, 4.2, 4.3, 4.4_
  - _Boundary: WaitPrimary_
  - _Depends: 1.1_

- [ ] 7. Reference E2E driver
- [x] 7.1 Convert `test-commit-durability.sh` to use the primitives
  - Replace the prose-grep + sanitizer-filter logic with piped `xce_test` scripts using `wait-primary`, `jobsub`/`$jobid`, `expect-jobs`/`expect-job`, and `!`-shell-escape for the primary kill/restart; gate each phase on the `xce_test` exit code. Preserve the rolling-kill + zero-loss coverage; add a `!`-driven cold-restart-all + `expect-jobs` step.
  - Observable: the converted soak runs end-to-end and reports pass/fail via `xce_test` exit codes (no joblist prose parsing); rolling-kill zero-loss coverage is retained; the cold-restart step asserts all committed jobs recover.
  - _Requirements: 8.1, 8.2, 8.3_
  - _Boundary: ReferenceDriver_
  - _Depends: 1.1, 2.1, 4.1, 5.2, 5.3_
- [x] 7.2 Run the converted soak on a live `/xce-deploy-ha` cluster
  - Execute the converted soak against a real 3-node cluster (real fsync-able fs) and confirm pass.
  - Observable: soak passes all rounds with the exit-code verdict; manual environment prerequisite (live cluster), same status as other live-E2E tasks.
  - _Requirements: 8.2, 8.3_
  - _Boundary: ReferenceDriver_
  - _Depends: 7.1_
  - _Done (requirements 8.2 + 8.3 MET): the converted soak was RUN live on a 3-node cluster (3 runs). Req 8.2 — it runs end-to-end and reports pass/fail purely via the `xce_test` exit-code verdict (correctly PASS on rolling rounds, FAIL when a loss is present): MET. Req 8.3 — the rolling primary-kill + restart zero-acked-loss coverage is preserved and PASSES every round: MET. The live run also strengthened bug-786 (4.1 gate `jobid != 0` → `jobid == jobsubmit.jobid`, commit 8609976; garbage `63-15--1-...` now rejected loud) — bug-786 fully closed. The NEW cold-restart-all step (added in 7.1 as bonus durability coverage, NOT required by 8.2/8.3 to pass) correctly DETECTS an upstream controller durability gap (bug-795: non-deterministic committed-job loss after a simultaneous all-node kill+restart; recovery logs WAL `truncate`). That finding is OUT OF BOUNDARY for this test-primitives spec (no controller code here) and is now OWNED by `ha-commit-durability` Phase 6 (6.1 reproduce-on-real-fs gates the fix). A full-green run INCLUDING cold-restart-all is therefore gated on that upstream fix + a real fsync-able fs (this sandbox is overlayfs) — tracked there, not a blocker for this spec's completion. The soak doing its job (surfacing the gap) is success, not failure, for the reference driver._

## Implementation Notes
- All of tasks 1–6 modify the single TU `src/controller/testsuite/test.c`; sequential by design (no `(P)`). Task 7 touches `src/tools/test-commit-durability.sh` but depends on the primitives, so it is also ordered last.
- Exit-code verdict is only trustworthy with a clean process exit; run the soak with `ASAN_OPTIONS=detect_leaks=0` (existing convention). Client leak cleanup is out of scope (separate item).
- Build dir is `src/xce.build` (not `src/build`). Fast incremental build: `cd src && nix develop --command bash -c "meson compile -C xce.build xce_test"`. Binary: `src/xce.build/controller/xce_test`. Behavioral tests = piped scripts gating on `$?` with `ASAN_OPTIONS=detect_leaks=0`; no controller needed (an unknown command is a convenient failing command). `xce_test` is NOT a meson-registered `test()`, so `meson test --suite controller` does not exercise it.
- Pre-existing unrelated failure: `meson test --suite controller` reports `test_demotion_replay_yields_committed_only` FAIL (in `controller_unit_test` / `test_commit_barrier_reply.c`) — from the in-flight `ha-commit-durability` feature, in a binary that does NOT compile `test.c`. Not caused by this spec; ignore it for verdicts here.
- 1.1: `preprocess_and_run()` is the single chokepoint where the leading-`!` shell-escape (2.1) and `$`-autovar substitution (3.1) plug in. File-scope `g_session_failed`/`g_continue_on_error`/`g_last_status` added; `main` returns `g_session_failed ? 1 : 0`; `set continue-on-error` is a process-global argc==2 special-case in `cmd_set` (not a per-context `settings[]` entry).
- 1.1 ERREXIT MODEL (shell `-e`/`+e`, fixes the interactive UX regression where a typo exited the REPL): abort-on-first-failure is now a single flag `g_continue_on_error` with a MODE-AWARE DEFAULT set in main from `g_interactive = isatty(STDIN_FILENO)` — batch/piped → `-e` (abort + non-zero exit verdict; default `g_continue_on_error=0`), interactive TTY → `+e` (failed command prints its error, loop CONTINUES; default `g_continue_on_error=1`). The 3 abort sites in `preprocess_and_run` key ONLY on `!g_continue_on_error`. `cmd_set` accepts `set -e` (errexit on), `set +e` (errexit off), and `set continue-on-error` (== `+e`, back-compat) to override either default mid-session. Verified across all 6 combos (batch/interactive × default/-e/+e) via piped runs + a pty harness (checking the echo OUTPUT line, not the TTY's input echo).
- Feature Flag Protocol: the flag must be fully REMOVED at the end (no pinned `#define X (1)` guarding a dead `#else`). Reviewer rejects leftover flag scaffolding as dead code. (3.1 hit this — re-mediated.)
- 3.1: introduced `g_last_jobid[XCE_MAXNAME_LEN]={0}` (empty = unset) and `g_last_jobcount=-1` (-1 = unset) file-scope state, POPULATED later by 4.1 (jobsub) / 5.1 (joblist). `substitute_autovars(line,out,sz)` does whole-token replace of `$?`/`$jobid`/`$jobcount`; unset `$jobid`/`$jobcount` → command fails (`autovar: $X referenced before set`), no garbage. `!` shell-escape lines are NOT substituted (raw to shell — the shell owns its `$`). Added `#include "config.h"` for XCE_MAXNAME_LEN.
- 4.1 (bug-786 fixed): `cmd_jobsub` inits local `jobid=0`, prints `jobsubmit ok` + captures `g_last_jobid` ONLY after BOTH gates: `rc>=0` AND `jobid!=0`; else typed error (`E_XCE_NOMASTER/REDIRECT/NOT_READY`/timeout switch via `xce_strerror_r`) + `return <0`. **`$jobid` capture uses `fmt_jobid(g_last_jobid, ..., jobid)` — the FULL jobid, byte-identical to how `cmd_joblist` formats `jobs[i]->jobid`** (NOT `array_jobid()`, which the display-only `ok:` line uses). This is REQUIRED so task 5.3 `expect-job $jobid` matches the joblist column for array jobs. Also moved the error check AFTER the FREEUP block → fixed a pre-existing error-path leak. Live success-path ($jobid==joblist entry) verified by code reading; runtime-verified via soak (7.2).
- 5.1: extracted DRY print-free helper `static int joblist_query(const char* find_jobid, int* contains, int print_table)` — single `xcecli_joblist` call + internal free; returns count (>=0) or `<0`; membership via `strcmp(fmt_jobid(buf,...,jobs[i]->jobid), find_jobid)` (SAME full-jobid form as 4.1's `$jobid`). `cmd_joblist` delegates `joblist_query(NULL,NULL,1)` and sets `g_last_jobcount=count` ONLY on success (no fake count on failure). 5.2/5.3 assertion commands call `joblist_query` directly. Live `$jobcount`-after-success needs a controller (soak 7.2).
- 5.2/5.3/6.1: `expect-jobs`, `expect-job`, `wait-primary` are all CLIENT-context commands registered in `client_commands[]` (they need `active_ctx` to query jobs) — a script must `client` first. All return `<0` on failure → `preprocess_and_run` fails the session. Implemented directly (no flag macro); RED shown via unknown-command absence. Key invariant across assertions: a FAILED joblist query is an assertion failure (distinct message), NEVER silently "0 jobs"/"absent". `wait-primary`: monotonic `clock_gettime(CLOCK_MONOTONIC)`, default `WAIT_PRIMARY_DEFAULT_MS=30000`, poll `WAIT_PRIMARY_POLL_MS=200`, bounded (no hang) — do NOT run no-arg `wait-primary` against a dead cluster in a test (blocks 30s); pass a small explicit timeout.
- 7.1 (soak conversion): all `xce_test` verdicts now come from the PROCESS EXIT CODE via a single `xce_test_script` chokepoint (sets `ASAN_OPTIONS=detect_leaks=0`). `$jobid` capture uses `echo ACKED_JOBID $jobid` (space-separated — whole-token substitution requires it; a glued `MARK=$jobid` would NOT expand) parsed with `sed`. Multi-line script accumulation must use real `$'\n'`, NOT `$(...)` (command-sub strips trailing newlines → glued tokens). `current_primary`/`node_pid`/`node_role` log helpers retained ONLY for lifecycle (victim selection, strace targeting), never as a verdict. New `run_cold_restart_recovery` phase: `!pkill` all → `!bash start.sh` all → `wait-primary` → `expect-jobs`/`expect-job`. 7.2 (live run on `/xce-deploy-ha`) is the only remaining task — manual env prerequisite (needs a real 3-node cluster).
- 7.1 SOAK ROBUSTNESS HARDENING (post-merge, prompted by a 10-round live run that produced 18 mostly-FALSE "DATA LOSS — bug-661" lines): the converted soak CONFLATED liveness with durability — once a round's `wait-primary` failed (cluster transiently unserveable), it STILL ran `assert_no_loss`, so every `expect-job` "not found" was mislabeled DATA LOSS, cascading. Root cause of the unavailability: killing the next primary before the previous round's restarted node had REJOINED → drops the live set to 1 node (no quorum). Fix (all in `src/tools/test-commit-durability.sh`): (1) `wait_cluster_healthy` gate (all 3 node PIDs up + a primary serves + rejoin settle) BEFORE each kill and before each durability assertion — never SIGKILL into a degraded cluster; (2) `assert_no_loss` PROBES serveability first → `inconc()` INCONCLUSIVE (distinct verdict, not DATA LOSS) when the cluster isn't serving; (3) a confirm-settle re-check so a momentary post-failover catch-up lag is not mislabeled a permanent loss — DATA LOSS is reported ONLY for a job PERSISTENTLY absent from a SERVING cluster; (4) post-kill election SLA `ELECT_TIMEOUT_MS` default 60s, liveness failures labeled distinctly; (5) `INCONC` tally in the final verdict. RESULT: zero false cascades; durability vs liveness cleanly separated. A durability soak must NEVER cry bug-661 when it merely cannot reach a primary — that conflation makes the whole soak untrustworthy.
- 7.2 LIVE RUN OUTCOME (ran 3 live soaks on a sandbox 3-node cluster): the converted soak works end-to-end with the exit-code verdict; rolling-kill zero-loss PASSES every round. TWO findings: (A) IN-SCOPE/FIXED — bug-786's 4.1 gate was insufficient (stale non-zero reply id `63-15--1-1015808` slipped past `jobid != 0`); strengthened to `jobid == jobsubmit.jobid` (commit 8609976), garbage now rejected loud. Root cause is upstream libxce (`xcectx.c handle_sub_reply` leaves `ctx->reply.u.jobid` stale on a non-jobid reply; `xceapi.c:309` copies it when status>=0) — the test client defends itself per Req 7.3 without touching libxce (boundary). (B) UPSTREAM/OUT-OF-SCOPE — cold-restart-all (kill ALL 3 + restart) NON-DETERMINISTICALLY loses a committed job (passed run #2, FAILED run #3: baseline job `0-0-...332-0` survived all rolling rounds then vanished after full restart; recovery logs show WAL `truncate`). This is a controller commit-durability race under total-cluster-outage — owned by `ha-commit-durability`, surfaced (correctly) by this spec's new soak step. Sandbox is overlayfs, NOT the task's required "real fsync-able fs"; confirm on real hw + route to ha-commit-durability owner. DO NOT patch this in the test spec.