Rongsong Shen
20e59d0b42
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
21 lines
2 KiB
Markdown
21 lines
2 KiB
Markdown
# Research / Discovery — xce-test-e2e-primitives
|
|
|
|
## Discovery type
|
|
LIGHT (extension to an existing single-file test client). No external research needed.
|
|
|
|
## Key findings (from src/controller/testsuite/test.c)
|
|
- Dispatch: `execute_line(line)` → `parse_argv` → `find_command(argv[0])` → `command->func(argc, argv)`; handlers return `int` (`<0` = errno-style error). Command table is `command_t {name, func, doc}` arrays per context.
|
|
- Main read loop reads a line, `rc = execute_line(s)`, and on `rc<0` only `SET_ERROR(-rc)` + continues; `main` returns 0 → **failures never affect the exit code** (the core gap for scriptable E2E).
|
|
- `cmd_jobsub`/`cmd_joblist` already drive the client via `active_ctx` and libxce client-side master-rotation/failover — `wait-primary` and the assertions can reuse these paths.
|
|
- bug-786: `cmd_jobsub` can print `jobsubmit ok: JobId: 63-15--1-...` (uninitialized id) on a non-clean ack; controller then logs `handle_jobsubmit fail to lock job <garbage>`. Root: success is reported without verifying the reply status/jobid.
|
|
|
|
## Design decisions
|
|
- **Extend-in-place, single TU.** Add a `preprocess_and_run` wrapper (blank/`#`/`!`/auto-var-substitution) + file-scope verdict state; new commands as ordinary table entries; `main` returns `g_session_failed ? 1 : 0`. No new files, no new opcode, no JSON.
|
|
- **Fixed auto-vars only** (`$?`, `$jobid`, `$jobcount`) — substitution, not a variable system; unset reference fails (no garbage).
|
|
- **Shell-escape via `system()`**, status → `$?`; node lifecycle stays in the shell.
|
|
- **`wait-primary`** loops an existing failover-aware client op until success/timeout (monotonic clock).
|
|
- Exit-code trustworthiness depends on a clean exit; reuse `ASAN_OPTIONS=detect_leaks=0` for the soak (client leak cleanup is separate/out of scope).
|
|
|
|
## Risks
|
|
- `system()` shell-escape = arbitrary execution — acceptable for a dev/test client only.
|
|
- Auto-var substitution must be whole-token and conservative to avoid surprising expansions.
|