lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(xce-test-e2e-primitives): record 7.2 live-soak findings (bug-786 strengthened; cold-restart-all durability gap routed upstream)

Browse source
This commit is contained in:
Rongsong Shen 2026-06-19 17:02:31 +00:00
parent 860997634a
commit c70fb52f03
1 changed files with 2 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.kiro/specs/xce-test-e2e-primitives/tasks.md
View file

@ -81,6 +81,7 @@
- _Requirements: 8.2, 8.3_ - _Requirements: 8.2, 8.3_
- _Boundary: ReferenceDriver_ - _Boundary: ReferenceDriver_
- _Depends: 7.1_ - _Depends: 7.1_
- _Blocked: PARTIAL — the converted soak was RUN live (3-node cluster) and works: it runs end-to-end and reports pass/fail via the exit-code verdict; rolling-kill zero-loss coverage PASSES every round. Two findings: (1) IN-SCOPE & FIXED — the live run surfaced that task 4.1's bug-786 gate (`jobid != 0`) was insufficient against a stale non-zero reply id (`63-15--1-1015808`); strengthened to require `jobid == jobsubmit.jobid` (commit 8609976) and re-verified (garbage now rejected loud). (2) UPSTREAM, OUT OF BOUNDARY — the new cold-restart-all step NON-DETERMINISTICALLY detects a committed-job loss after a full kill-all+restart (a job that returned `jobsubmit ok` and survived all rolling rounds vanishes; recovery logs show WAL `truncate`). This is a controller commit-durability behavior under total-cluster-outage recovery — owned by spec `ha-commit-durability`, NOT this test-primitives spec (which changes no controller code). The primitives are working correctly by surfacing it. Confirmation needs a REAL fsync-able fs (this sandbox is overlayfs, the task's stated manual prerequisite is unmet) + the ha-commit-durability owner. Full-green 7.2 is blocked on that upstream investigation + a proper live environment._
## Implementation Notes ## Implementation Notes
- All of tasks 1–6 modify the single TU `src/controller/testsuite/test.c`; sequential by design (no `(P)`). Task 7 touches `src/tools/test-commit-durability.sh` but depends on the primitives, so it is also ordered last. - All of tasks 1–6 modify the single TU `src/controller/testsuite/test.c`; sequential by design (no `(P)`). Task 7 touches `src/tools/test-commit-durability.sh` but depends on the primitives, so it is also ordered last.
@ -94,3 +95,4 @@
- 5.1: extracted DRY print-free helper `static int joblist_query(const char* find_jobid, int* contains, int print_table)` — single `xcecli_joblist` call + internal free; returns count (>=0) or `<0`; membership via `strcmp(fmt_jobid(buf,...,jobs[i]->jobid), find_jobid)` (SAME full-jobid form as 4.1's `$jobid`). `cmd_joblist` delegates `joblist_query(NULL,NULL,1)` and sets `g_last_jobcount=count` ONLY on success (no fake count on failure). 5.2/5.3 assertion commands call `joblist_query` directly. Live `$jobcount`-after-success needs a controller (soak 7.2). - 5.1: extracted DRY print-free helper `static int joblist_query(const char* find_jobid, int* contains, int print_table)` — single `xcecli_joblist` call + internal free; returns count (>=0) or `<0`; membership via `strcmp(fmt_jobid(buf,...,jobs[i]->jobid), find_jobid)` (SAME full-jobid form as 4.1's `$jobid`). `cmd_joblist` delegates `joblist_query(NULL,NULL,1)` and sets `g_last_jobcount=count` ONLY on success (no fake count on failure). 5.2/5.3 assertion commands call `joblist_query` directly. Live `$jobcount`-after-success needs a controller (soak 7.2).
- 5.2/5.3/6.1: `expect-jobs`, `expect-job`, `wait-primary` are all CLIENT-context commands registered in `client_commands[]` (they need `active_ctx` to query jobs) — a script must `client` first. All return `<0` on failure → `preprocess_and_run` fails the session. Implemented directly (no flag macro); RED shown via unknown-command absence. Key invariant across assertions: a FAILED joblist query is an assertion failure (distinct message), NEVER silently "0 jobs"/"absent". `wait-primary`: monotonic `clock_gettime(CLOCK_MONOTONIC)`, default `WAIT_PRIMARY_DEFAULT_MS=30000`, poll `WAIT_PRIMARY_POLL_MS=200`, bounded (no hang) — do NOT run no-arg `wait-primary` against a dead cluster in a test (blocks 30s); pass a small explicit timeout. - 5.2/5.3/6.1: `expect-jobs`, `expect-job`, `wait-primary` are all CLIENT-context commands registered in `client_commands[]` (they need `active_ctx` to query jobs) — a script must `client` first. All return `<0` on failure → `preprocess_and_run` fails the session. Implemented directly (no flag macro); RED shown via unknown-command absence. Key invariant across assertions: a FAILED joblist query is an assertion failure (distinct message), NEVER silently "0 jobs"/"absent". `wait-primary`: monotonic `clock_gettime(CLOCK_MONOTONIC)`, default `WAIT_PRIMARY_DEFAULT_MS=30000`, poll `WAIT_PRIMARY_POLL_MS=200`, bounded (no hang) — do NOT run no-arg `wait-primary` against a dead cluster in a test (blocks 30s); pass a small explicit timeout.
- 7.1 (soak conversion): all `xce_test` verdicts now come from the PROCESS EXIT CODE via a single `xce_test_script` chokepoint (sets `ASAN_OPTIONS=detect_leaks=0`). `$jobid` capture uses `echo ACKED_JOBID $jobid` (space-separated — whole-token substitution requires it; a glued `MARK=$jobid` would NOT expand) parsed with `sed`. Multi-line script accumulation must use real `$'\n'`, NOT `$(...)` (command-sub strips trailing newlines → glued tokens). `current_primary`/`node_pid`/`node_role` log helpers retained ONLY for lifecycle (victim selection, strace targeting), never as a verdict. New `run_cold_restart_recovery` phase: `!pkill` all → `!bash start.sh` all → `wait-primary` → `expect-jobs`/`expect-job`. 7.2 (live run on `/xce-deploy-ha`) is the only remaining task — manual env prerequisite (needs a real 3-node cluster). - 7.1 (soak conversion): all `xce_test` verdicts now come from the PROCESS EXIT CODE via a single `xce_test_script` chokepoint (sets `ASAN_OPTIONS=detect_leaks=0`). `$jobid` capture uses `echo ACKED_JOBID $jobid` (space-separated — whole-token substitution requires it; a glued `MARK=$jobid` would NOT expand) parsed with `sed`. Multi-line script accumulation must use real `$'\n'`, NOT `$(...)` (command-sub strips trailing newlines → glued tokens). `current_primary`/`node_pid`/`node_role` log helpers retained ONLY for lifecycle (victim selection, strace targeting), never as a verdict. New `run_cold_restart_recovery` phase: `!pkill` all → `!bash start.sh` all → `wait-primary` → `expect-jobs`/`expect-job`. 7.2 (live run on `/xce-deploy-ha`) is the only remaining task — manual env prerequisite (needs a real 3-node cluster).
- 7.2 LIVE RUN OUTCOME (ran 3 live soaks on a sandbox 3-node cluster): the converted soak works end-to-end with the exit-code verdict; rolling-kill zero-loss PASSES every round. TWO findings: (A) IN-SCOPE/FIXED — bug-786's 4.1 gate was insufficient (stale non-zero reply id `63-15--1-1015808` slipped past `jobid != 0`); strengthened to `jobid == jobsubmit.jobid` (commit 8609976), garbage now rejected loud. Root cause is upstream libxce (`xcectx.c handle_sub_reply` leaves `ctx->reply.u.jobid` stale on a non-jobid reply; `xceapi.c:309` copies it when status>=0) — the test client defends itself per Req 7.3 without touching libxce (boundary). (B) UPSTREAM/OUT-OF-SCOPE — cold-restart-all (kill ALL 3 + restart) NON-DETERMINISTICALLY loses a committed job (passed run #2, FAILED run #3: baseline job `0-0-...332-0` survived all rolling rounds then vanished after full restart; recovery logs show WAL `truncate`). This is a controller commit-durability race under total-cluster-outage — owned by `ha-commit-durability`, surfaced (correctly) by this spec's new soak step. Sandbox is overlayfs, NOT the task's required "real fsync-able fs"; confirm on real hw + route to ha-commit-durability owner. DO NOT patch this in the test spec.