myxce/.kiro/specs/xce-test-e2e-primitives/tasks.md

Implementation Plan

TDD is mandatory (project rule): add the failing test/scenario first, then the change (red → green → refactor). Almost all work is in the single test client TU src/controller/testsuite/test.c, so tasks are sequential by design; only the soak conversion touches a different file. Behavioral tests are driven through xce_test itself (piped scripts), gating on its exit code.

• 1. Foundation: verdict, loop wrapper, ergonomics

• 1.1 Session failure tracking + process exit code + abort/continue + comment/blank handling

  • Add file-scope state (g_session_failed, g_continue_on_error, g_last_status) and a preprocess_and_run(line) wrapper used by the main read loop: ignore blank and #-comment lines; run normal lines via execute_line; update g_last_status/g_session_failed from the result; in default mode stop on first failure, under continue-on-error keep going. main returns g_session_failed ? 1 : 0. Extend cmd_set with a continue-on-error toggle.
  • Observable: a piped script with a bogus/failing command makes xce_test exit non-zero; an all-valid script exits 0; with set continue-on-error a failing line does not stop later lines but the final exit is still non-zero; #/blank lines are ignored.
  • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 6.1, 6.2
  • Boundary: VerdictLoopWrapper

• 1.2 echo command

  • Add cmd_echo (table-registered) that prints its argument message and returns success.
  • Observable: echo hello emits hello and the command succeeds (exit unaffected).
  • Requirements: 6.3
  • Boundary: VerdictLoopWrapper
  • Depends: 1.1

• 2. Shell-escape

• 2.1 !<cmd> runs a shell command and feeds $?

  • In preprocess_and_run, when the first non-whitespace char is !, run the remainder via system(); set g_last_status = WEXITSTATUS(rc) (non-zero on system failure); a non-zero status is a command failure (subject to 1.1's abort/continue + exit code).
  • Observable: !true succeeds and sets $?=0; !false fails the session and sets $?≠0; a script can !kill ... / !bash start.sh ... to drive node lifecycle.
  • Requirements: 2.1, 2.2, 2.3, 2.4
  • Boundary: ShellEscape
  • Depends: 1.1

• 3. Auto-variable substitution

• 3.1 Substitute the fixed auto-vars $?, $jobid, $jobcount

  • Add substitute_autovars(line) (run before tokenizing in preprocess_and_run): replace whole-token $?→g_last_status, $jobid→g_last_jobid, $jobcount→g_last_jobcount. A reference to an unset $jobid/$jobcount fails the command (no garbage substitution). No other variable syntax is recognized.
  • Observable: after a command, echo $? prints the prior status; referencing $jobid before any successful jobsub fails that command rather than substituting an empty/garbage value.
  • Requirements: 3.1, 3.3, 3.5, 3.6
  • Boundary: AutoVars
  • Depends: 1.1

• 4. Correct job-submission reporting (bug-786) + $jobid capture

• 4.1 cmd_jobsub reports success only on a clean ack with a valid JobId

  • Gate the jobsubmit ok: JobId: <id> report on a successful reply status AND a well-formed JobId; on success set g_last_jobid (feeds $jobid, 3.2). Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return <0. Never print "ok" with an uninitialized/garbage id (63-15--1-...).
  • Observable: a submit that does not get a clean ack returns failure and does NOT set $jobid; jobsub never prints success with a malformed id; a successful submit sets $jobid to the acked JobId.
  • Requirements: 7.1, 7.2, 7.3, 3.2
  • Boundary: JobsubFix
  • Depends: 1.1, 3.1

• 5. Job-count capture and assertions

• 5.1 cmd_joblist sets $jobcount; extract a shared count/contains helper

  • Refactor cmd_joblist to expose an internal helper that returns the current job count and tests membership of a JobId; set g_last_jobcount after a joblist.
  • Observable: after joblist, echo $jobcount prints the number of jobs returned; the helper is reusable by the assertions.
  • Requirements: 3.4
  • Boundary: JoblistCount
  • Depends: 1.1, 3.1

• 5.2 expect-jobs N assertion

  • Add cmd_expect_jobs: pass when the current job count equals N; otherwise print expected vs actual and return <0.
  • Observable: expect-jobs <correct> succeeds; expect-jobs <wrong> fails the session and prints expected/actual.
  • Requirements: 5.1, 5.2
  • Boundary: Assertions
  • Depends: 5.1

• 5.3 expect-job <jobid> assertion

  • Add cmd_expect_job: pass when the JobId is present in the job list; otherwise print the missing id and return <0. Works with $jobid substitution.
  • Observable: expect-job $jobid passes for a submitted job; expect-job <absent> fails the session and prints the missing id.
  • Requirements: 5.3, 5.4
  • Boundary: Assertions
  • Depends: 5.1, 3.1, 4.1

• 6. Wait for elected primary

• 6.1 wait-primary [timeout_ms]

  • Add cmd_wait_primary: loop a cheap, failover-aware client op (the existing joblist/probe path via active_ctx) until it succeeds or the timeout elapses (monotonic clock); default timeout when the arg is omitted; return 0 on success, <0 on timeout.
  • Observable: against a healthy cluster wait-primary returns immediately; after a primary kill it blocks until a new primary serves, then succeeds; with no primary within the timeout it fails.
  • Requirements: 4.1, 4.2, 4.3, 4.4
  • Boundary: WaitPrimary
  • Depends: 1.1

• 7. Reference E2E driver

• 7.1 Convert test-commit-durability.sh to use the primitives

  • Replace the prose-grep + sanitizer-filter logic with piped xce_test scripts using wait-primary, jobsub/$jobid, expect-jobs/expect-job, and !-shell-escape for the primary kill/restart; gate each phase on the xce_test exit code. Preserve the rolling-kill + zero-loss coverage; add a !-driven cold-restart-all + expect-jobs step.
  • Observable: the converted soak runs end-to-end and reports pass/fail via xce_test exit codes (no joblist prose parsing); rolling-kill zero-loss coverage is retained; the cold-restart step asserts all committed jobs recover.
  • Requirements: 8.1, 8.2, 8.3
  • Boundary: ReferenceDriver
  • Depends: 1.1, 2.1, 4.1, 5.2, 5.3

• 7.2 Run the converted soak on a live /xce-deploy-ha cluster

  • Execute the converted soak against a real 3-node cluster (real fsync-able fs) and confirm pass.
  • Observable: soak passes all rounds with the exit-code verdict; manual environment prerequisite (live cluster), same status as other live-E2E tasks.
  • Requirements: 8.2, 8.3
  • Boundary: ReferenceDriver
  • Depends: 7.1
  • Done (requirements 8.2 + 8.3 MET): the converted soak was RUN live on a 3-node cluster (3 runs). Req 8.2 — it runs end-to-end and reports pass/fail purely via the xce_test exit-code verdict (correctly PASS on rolling rounds, FAIL when a loss is present): MET. Req 8.3 — the rolling primary-kill + restart zero-acked-loss coverage is preserved and PASSES every round: MET. The live run also strengthened bug-786 (4.1 gate jobid != 0 → jobid == jobsubmit.jobid, commit 8609976; garbage 63-15--1-... now rejected loud) — bug-786 fully closed. The NEW cold-restart-all step (added in 7.1 as bonus durability coverage, NOT required by 8.2/8.3 to pass) correctly DETECTS an upstream controller durability gap (bug-795: non-deterministic committed-job loss after a simultaneous all-node kill+restart; recovery logs WAL truncate). That finding is OUT OF BOUNDARY for this test-primitives spec (no controller code here) and is now OWNED by ha-commit-durability Phase 6 (6.1 reproduce-on-real-fs gates the fix). A full-green run INCLUDING cold-restart-all is therefore gated on that upstream fix + a real fsync-able fs (this sandbox is overlayfs) — tracked there, not a blocker for this spec's completion. The soak doing its job (surfacing the gap) is success, not failure, for the reference driver.

Implementation Notes

• All of tasks 1–6 modify the single TU src/controller/testsuite/test.c; sequential by design (no (P)). Task 7 touches src/tools/test-commit-durability.sh but depends on the primitives, so it is also ordered last.
• Exit-code verdict is only trustworthy with a clean process exit; run the soak with ASAN_OPTIONS=detect_leaks=0 (existing convention). Client leak cleanup is out of scope (separate item).
• Build dir is src/xce.build (not src/build). Fast incremental build: cd src && nix develop --command bash -c "meson compile -C xce.build xce_test". Binary: src/xce.build/controller/xce_test. Behavioral tests = piped scripts gating on $? with ASAN_OPTIONS=detect_leaks=0; no controller needed (an unknown command is a convenient failing command). xce_test is NOT a meson-registered test(), so meson test --suite controller does not exercise it.
• Pre-existing unrelated failure: meson test --suite controller reports test_demotion_replay_yields_committed_only FAIL (in controller_unit_test / test_commit_barrier_reply.c) — from the in-flight ha-commit-durability feature, in a binary that does NOT compile test.c. Not caused by this spec; ignore it for verdicts here.
• 1.1: preprocess_and_run() is the single chokepoint where the leading-! shell-escape (2.1) and $-autovar substitution (3.1) plug in. File-scope g_session_failed/g_continue_on_error/g_last_status added; main returns g_session_failed ? 1 : 0; set continue-on-error is a process-global argc==2 special-case in cmd_set (not a per-context settings[] entry).
• 1.1 ERREXIT MODEL (shell -e/+e, fixes the interactive UX regression where a typo exited the REPL): abort-on-first-failure is now a single flag g_continue_on_error with a MODE-AWARE DEFAULT set in main from g_interactive = isatty(STDIN_FILENO) — batch/piped → -e (abort + non-zero exit verdict; default g_continue_on_error=0), interactive TTY → +e (failed command prints its error, loop CONTINUES; default g_continue_on_error=1). The 3 abort sites in preprocess_and_run key ONLY on !g_continue_on_error. cmd_set accepts set -e (errexit on), set +e (errexit off), and set continue-on-error (== +e, back-compat) to override either default mid-session. Verified across all 6 combos (batch/interactive × default/-e/+e) via piped runs + a pty harness (checking the echo OUTPUT line, not the TTY's input echo).
• Feature Flag Protocol: the flag must be fully REMOVED at the end (no pinned #define X (1) guarding a dead #else). Reviewer rejects leftover flag scaffolding as dead code. (3.1 hit this — re-mediated.)
• 3.1: introduced g_last_jobid[XCE_MAXNAME_LEN]={0} (empty = unset) and g_last_jobcount=-1 (-1 = unset) file-scope state, POPULATED later by 4.1 (jobsub) / 5.1 (joblist). substitute_autovars(line,out,sz) does whole-token replace of $?/$jobid/$jobcount; unset $jobid/$jobcount → command fails (autovar: $X referenced before set), no garbage. ! shell-escape lines are NOT substituted (raw to shell — the shell owns its $). Added #include "config.h" for XCE_MAXNAME_LEN.
• 4.1 (bug-786 fixed): cmd_jobsub inits local jobid=0, prints jobsubmit ok + captures g_last_jobid ONLY after BOTH gates: rc>=0 AND jobid!=0; else typed error (E_XCE_NOMASTER/REDIRECT/NOT_READY/timeout switch via xce_strerror_r) + return <0. $jobid capture uses fmt_jobid(g_last_jobid, ..., jobid) — the FULL jobid, byte-identical to how cmd_joblist formats jobs[i]->jobid (NOT array_jobid(), which the display-only ok: line uses). This is REQUIRED so task 5.3 expect-job $jobid matches the joblist column for array jobs. Also moved the error check AFTER the FREEUP block → fixed a pre-existing error-path leak. Live success-path ($jobid==joblist entry) verified by code reading; runtime-verified via soak (7.2).
• 5.1: extracted DRY print-free helper static int joblist_query(const char* find_jobid, int* contains, int print_table) — single xcecli_joblist call + internal free; returns count (>=0) or <0; membership via strcmp(fmt_jobid(buf,...,jobs[i]->jobid), find_jobid) (SAME full-jobid form as 4.1's $jobid). cmd_joblist delegates joblist_query(NULL,NULL,1) and sets g_last_jobcount=count ONLY on success (no fake count on failure). 5.2/5.3 assertion commands call joblist_query directly. Live $jobcount-after-success needs a controller (soak 7.2).
• 5.2/5.3/6.1: expect-jobs, expect-job, wait-primary are all CLIENT-context commands registered in client_commands[] (they need active_ctx to query jobs) — a script must client first. All return <0 on failure → preprocess_and_run fails the session. Implemented directly (no flag macro); RED shown via unknown-command absence. Key invariant across assertions: a FAILED joblist query is an assertion failure (distinct message), NEVER silently "0 jobs"/"absent". wait-primary: monotonic clock_gettime(CLOCK_MONOTONIC), default WAIT_PRIMARY_DEFAULT_MS=30000, poll WAIT_PRIMARY_POLL_MS=200, bounded (no hang) — do NOT run no-arg wait-primary against a dead cluster in a test (blocks 30s); pass a small explicit timeout.
• 7.1 (soak conversion): all xce_test verdicts now come from the PROCESS EXIT CODE via a single xce_test_script chokepoint (sets ASAN_OPTIONS=detect_leaks=0). $jobid capture uses echo ACKED_JOBID $jobid (space-separated — whole-token substitution requires it; a glued MARK=$jobid would NOT expand) parsed with sed. Multi-line script accumulation must use real $'\n', NOT $(...) (command-sub strips trailing newlines → glued tokens). current_primary/node_pid/node_role log helpers retained ONLY for lifecycle (victim selection, strace targeting), never as a verdict. New run_cold_restart_recovery phase: !pkill all → !bash start.sh all → wait-primary → expect-jobs/expect-job. 7.2 (live run on /xce-deploy-ha) is the only remaining task — manual env prerequisite (needs a real 3-node cluster).
• 7.1 SOAK ROBUSTNESS HARDENING (post-merge, prompted by a 10-round live run that produced 18 mostly-FALSE "DATA LOSS — bug-661" lines): the converted soak CONFLATED liveness with durability — once a round's wait-primary failed (cluster transiently unserveable), it STILL ran assert_no_loss, so every expect-job "not found" was mislabeled DATA LOSS, cascading. Root cause of the unavailability: killing the next primary before the previous round's restarted node had REJOINED → drops the live set to 1 node (no quorum). Fix (all in src/tools/test-commit-durability.sh): (1) wait_cluster_healthy gate (all 3 node PIDs up + a primary serves + rejoin settle) BEFORE each kill and before each durability assertion — never SIGKILL into a degraded cluster; (2) assert_no_loss PROBES serveability first → inconc() INCONCLUSIVE (distinct verdict, not DATA LOSS) when the cluster isn't serving; (3) a confirm-settle re-check so a momentary post-failover catch-up lag is not mislabeled a permanent loss — DATA LOSS is reported ONLY for a job PERSISTENTLY absent from a SERVING cluster; (4) post-kill election SLA ELECT_TIMEOUT_MS default 60s, liveness failures labeled distinctly; (5) INCONC tally in the final verdict. RESULT: zero false cascades; durability vs liveness cleanly separated. A durability soak must NEVER cry bug-661 when it merely cannot reach a primary — that conflation makes the whole soak untrustworthy.
• 7.2 LIVE RUN OUTCOME (ran 3 live soaks on a sandbox 3-node cluster): the converted soak works end-to-end with the exit-code verdict; rolling-kill zero-loss PASSES every round. TWO findings: (A) IN-SCOPE/FIXED — bug-786's 4.1 gate was insufficient (stale non-zero reply id 63-15--1-1015808 slipped past jobid != 0); strengthened to jobid == jobsubmit.jobid (commit 8609976), garbage now rejected loud. Root cause is upstream libxce (xcectx.c handle_sub_reply leaves ctx->reply.u.jobid stale on a non-jobid reply; xceapi.c:309 copies it when status>=0) — the test client defends itself per Req 7.3 without touching libxce (boundary). (B) UPSTREAM/OUT-OF-SCOPE — cold-restart-all (kill ALL 3 + restart) NON-DETERMINISTICALLY loses a committed job (passed run #2, FAILED run #3: baseline job 0-0-...332-0 survived all rolling rounds then vanished after full restart; recovery logs show WAL truncate). This is a controller commit-durability race under total-cluster-outage — owned by ha-commit-durability, surfaced (correctly) by this spec's new soak step. Sandbox is overlayfs, NOT the task's required "real fsync-able fs"; confirm on real hw + route to ha-commit-durability owner. DO NOT patch this in the test spec.