fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool, which hardcodes VTASK_LIVENESS for every inbound peer-control packet including vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in persist_promotion_advance_committed) was reachable from that never-blocks task class -- same bug family as bug-1198. Removed the force-flush; round-3 (durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL: persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path and does bounded pread() -- lower risk (page-cache-hot) but strictly still blocking I/O; not safe to remove outright (reopens bug-1043-family job loss), needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask). bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop, lock-free) and routed both call sites through it. Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
parent
ea0ac81c92
commit
40f510fb14
25 changed files with 1617 additions and 226 deletions
|
|
@ -3,7 +3,7 @@
|
|||
> TDD is mandatory (project rule): add the failing test/scenario first, then the change (red → green → refactor).
|
||||
> Almost all work is in the single test client TU `src/controller/testsuite/test.c`, so tasks are sequential by design; only the soak conversion touches a different file. Behavioral tests are driven through `xce_test` itself (piped scripts), gating on its exit code.
|
||||
|
||||
- [ ] 1. Foundation: verdict, loop wrapper, ergonomics
|
||||
- [x] 1. Foundation: verdict, loop wrapper, ergonomics
|
||||
- [x] 1.1 Session failure tracking + process exit code + abort/continue + comment/blank handling
|
||||
- Add file-scope state (`g_session_failed`, `g_continue_on_error`, `g_last_status`) and a `preprocess_and_run(line)` wrapper used by the main read loop: ignore blank and `#`-comment lines; run normal lines via `execute_line`; update `g_last_status`/`g_session_failed` from the result; in default mode stop on first failure, under `continue-on-error` keep going. `main` returns `g_session_failed ? 1 : 0`. Extend `cmd_set` with a `continue-on-error` toggle.
|
||||
- Observable: a piped script with a bogus/failing command makes `xce_test` exit non-zero; an all-valid script exits 0; with `set continue-on-error` a failing line does not stop later lines but the final exit is still non-zero; `#`/blank lines are ignored.
|
||||
|
|
@ -16,7 +16,7 @@
|
|||
- _Boundary: VerdictLoopWrapper_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 2. Shell-escape
|
||||
- [x] 2. Shell-escape
|
||||
- [x] 2.1 `!<cmd>` runs a shell command and feeds `$?`
|
||||
- In `preprocess_and_run`, when the first non-whitespace char is `!`, run the remainder via `system()`; set `g_last_status = WEXITSTATUS(rc)` (non-zero on `system` failure); a non-zero status is a command failure (subject to 1.1's abort/continue + exit code).
|
||||
- Observable: `!true` succeeds and sets `$?`=0; `!false` fails the session and sets `$?`≠0; a script can `!kill ...` / `!bash start.sh ...` to drive node lifecycle.
|
||||
|
|
@ -24,7 +24,7 @@
|
|||
- _Boundary: ShellEscape_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 3. Auto-variable substitution
|
||||
- [x] 3. Auto-variable substitution
|
||||
- [x] 3.1 Substitute the fixed auto-vars `$?`, `$jobid`, `$jobcount`
|
||||
- Add `substitute_autovars(line)` (run before tokenizing in `preprocess_and_run`): replace whole-token `$?`→`g_last_status`, `$jobid`→`g_last_jobid`, `$jobcount`→`g_last_jobcount`. A reference to an unset `$jobid`/`$jobcount` fails the command (no garbage substitution). No other variable syntax is recognized.
|
||||
- Observable: after a command, `echo $?` prints the prior status; referencing `$jobid` before any successful `jobsub` fails that command rather than substituting an empty/garbage value.
|
||||
|
|
@ -32,7 +32,7 @@
|
|||
- _Boundary: AutoVars_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 4. Correct job-submission reporting (bug-786) + `$jobid` capture
|
||||
- [x] 4. Correct job-submission reporting (bug-786) + `$jobid` capture
|
||||
- [x] 4.1 `cmd_jobsub` reports success only on a clean ack with a valid JobId
|
||||
- Gate the `jobsubmit ok: JobId: <id>` report on a successful reply status AND a well-formed JobId; on success set `g_last_jobid` (feeds `$jobid`, 3.2). Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return `<0`. Never print "ok" with an uninitialized/garbage id (`63-15--1-...`).
|
||||
- Observable: a submit that does not get a clean ack returns failure and does NOT set `$jobid`; `jobsub` never prints success with a malformed id; a successful submit sets `$jobid` to the acked JobId.
|
||||
|
|
@ -40,7 +40,7 @@
|
|||
- _Boundary: JobsubFix_
|
||||
- _Depends: 1.1, 3.1_
|
||||
|
||||
- [ ] 5. Job-count capture and assertions
|
||||
- [x] 5. Job-count capture and assertions
|
||||
- [x] 5.1 `cmd_joblist` sets `$jobcount`; extract a shared count/contains helper
|
||||
- Refactor `cmd_joblist` to expose an internal helper that returns the current job count and tests membership of a JobId; set `g_last_jobcount` after a `joblist`.
|
||||
- Observable: after `joblist`, `echo $jobcount` prints the number of jobs returned; the helper is reusable by the assertions.
|
||||
|
|
@ -60,7 +60,7 @@
|
|||
- _Boundary: Assertions_
|
||||
- _Depends: 5.1, 3.1, 4.1_
|
||||
|
||||
- [ ] 6. Wait for elected primary
|
||||
- [x] 6. Wait for elected primary
|
||||
- [x] 6.1 `wait-primary [timeout_ms]`
|
||||
- Add `cmd_wait_primary`: loop a cheap, failover-aware client op (the existing `joblist`/probe path via `active_ctx`) until it succeeds or the timeout elapses (monotonic clock); default timeout when the arg is omitted; return 0 on success, `<0` on timeout.
|
||||
- Observable: against a healthy cluster `wait-primary` returns immediately; after a primary kill it blocks until a new primary serves, then succeeds; with no primary within the timeout it fails.
|
||||
|
|
@ -68,7 +68,7 @@
|
|||
- _Boundary: WaitPrimary_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 7. Reference E2E driver
|
||||
- [x] 7. Reference E2E driver
|
||||
- [x] 7.1 Convert `test-commit-durability.sh` to use the primitives
|
||||
- Replace the prose-grep + sanitizer-filter logic with piped `xce_test` scripts using `wait-primary`, `jobsub`/`$jobid`, `expect-jobs`/`expect-job`, and `!`-shell-escape for the primary kill/restart; gate each phase on the `xce_test` exit code. Preserve the rolling-kill + zero-loss coverage; add a `!`-driven cold-restart-all + `expect-jobs` step.
|
||||
- Observable: the converted soak runs end-to-end and reports pass/fail via `xce_test` exit codes (no joblist prose parsing); rolling-kill zero-loss coverage is retained; the cold-restart step asserts all committed jobs recover.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue