fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool, which hardcodes VTASK_LIVENESS for every inbound peer-control packet including vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in persist_promotion_advance_committed) was reachable from that never-blocks task class -- same bug family as bug-1198. Removed the force-flush; round-3 (durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL: persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path and does bounded pread() -- lower risk (page-cache-hot) but strictly still blocking I/O; not safe to remove outright (reopens bug-1043-family job loss), needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask). bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop, lock-free) and routed both call sites through it. Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
parent
ea0ac81c92
commit
40f510fb14
25 changed files with 1617 additions and 226 deletions
147
.claude/skills/xce-audit-invariants/SKILL.md
Normal file
147
.claude/skills/xce-audit-invariants/SKILL.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
---
|
||||
name: xce-audit-invariants
|
||||
description: "Audit a component's documented design constraints/invariants, fix confirmed violations, and prove the fix with build+unit+live HA regression. Usage: /xce-audit-invariants [component] [soak-rounds]. Component: controller (default, richest invariant surface), or any src/<component> with a CLAUDE.md/steering invariant section."
|
||||
allowed-tools: Read, Write, Edit, Glob, Grep, Bash, Agent, TaskCreate, TaskUpdate
|
||||
argument-hint: "[component] [soak-rounds]"
|
||||
---
|
||||
|
||||
# XCE Design-Invariant Audit & Fix
|
||||
|
||||
Runs the same loop used to close bug-1417/1418/1419/1430/1431/1432: read the
|
||||
documented invariants → audit code against them in parallel → fix confirmed
|
||||
violations → prove it with a full rebuild, the unit suite, **and** a live HA
|
||||
soak (both chaotic and sequential modes). Code review alone is not sufficient
|
||||
proof here — several of the bugs above were only found by a live soak after
|
||||
the code-level fixes already looked correct.
|
||||
|
||||
## Args
|
||||
|
||||
- `[component]` (default `controller`) — which `src/<component>` to audit.
|
||||
`controller` is the primary target: it is the only component with a rich,
|
||||
explicit invariant surface today (`.kiro/steering/ha-async-io.md`,
|
||||
`persistence.md`, `error-handling.md`, plus `src/controller/CLAUDE.md`'s
|
||||
Process-Lifecycle invariant and TLA+ Protocol Verification Checklist). If
|
||||
invoked against another component, first check whether it has an equivalent
|
||||
invariant section in its `CLAUDE.md` — if not, say so and stop rather than
|
||||
inventing invariants to audit against.
|
||||
- `[soak-rounds]` (default `10`) — rounds for the final HA regression gate
|
||||
(only meaningful for `controller`; clamped to 5-50 by `/xce-test-ha`).
|
||||
|
||||
## Step 1 — Gather the invariant sources (do not skip)
|
||||
|
||||
Read, in full, before auditing anything:
|
||||
- `src/<component>/CLAUDE.md` — component-specific invariants (for
|
||||
`controller`: the "Process Lifecycle" DESIGN INVARIANT block and the full
|
||||
"Protocol Verification Checklist" under the TLA+ section).
|
||||
- The steering files that apply — for `controller`: `.kiro/steering/ha-async-io.md`
|
||||
(thread-role contract: iomgr/delaymgr must never block inline), `persistence.md`
|
||||
(WAL/commit/durability invariants), `error-handling.md` (ReturnError/errnox
|
||||
pattern, no silent error swallowing).
|
||||
- `.wolf/cerebrum.md`'s Key Learnings and Do-Not-Repeat sections — prior
|
||||
audits already found real gaps between what these docs claim and what the
|
||||
code does (e.g. a comment claiming a delaymgr task "starts immediately"
|
||||
when `delay_task_ms` actually delays its first fire — bug-1432). Treat
|
||||
every invariant-adjacent comment as a claim to verify, not a fact.
|
||||
- `.wolf/buglog.json` — grep for the component's tag to avoid re-auditing
|
||||
something already closed, and to recognize a recurrence of a known bug
|
||||
family (e.g. `bug-828`/`bug-851`/`bug-1060` class committed-job-loss bugs).
|
||||
|
||||
## Step 2 — Parallel audit dispatch
|
||||
|
||||
Fan out one Agent-tool subagent per invariant *dimension* (not per file) so
|
||||
findings stay independently verifiable. For `controller`, the dimensions that
|
||||
have historically found real violations are:
|
||||
|
||||
1. **Async I/O thread-role** — does any iomgr/delaymgr callback do blocking
|
||||
work inline instead of dispatching to a `VTASK_LIVE_BLOCK` vtask?
|
||||
2. **Process lifecycle** — does any code path call `exit`/`abort`/`_exit`/
|
||||
`tlog_exit` as a *designed response* to an operational condition (corrupt
|
||||
WAL, lost quorum, failed catch-up) instead of fence-and-stay-alive?
|
||||
3. **Raft/TLA+ safety** — walk the Protocol Verification Checklist in
|
||||
`controller/CLAUDE.md` line by line against the current `ha.c`/`persist.c`
|
||||
(election safety, leader completeness / no-self-truncation, commit rule,
|
||||
commit ordering, durability gating, freshness gate).
|
||||
4. **Commit/apply decoupling** — is entity materialization (`apply_committed_entry`)
|
||||
ever reachable before an entry's LSN is `<= persist->committed_lsn`, on
|
||||
any ingest path (standby WAL receipt, catch-up, heartbeat)?
|
||||
5. **Error handling** — silent swallowed errors, missing `ReturnError`, or a
|
||||
fail-open where the invariant docs require fail-loud-and-fence.
|
||||
|
||||
Each subagent must report file:line evidence for every claimed violation —
|
||||
no speculative findings. Use the Agent tool directly (general-purpose or
|
||||
Explore subagent type); do not invoke the Workflow tool for this unless the
|
||||
user has explicitly asked for multi-agent orchestration in their own words.
|
||||
|
||||
## Step 3 — Triage and fix
|
||||
|
||||
- Dedupe overlapping findings across dimensions.
|
||||
- For each CONFIRMED violation, fix it following the established patterns in
|
||||
this codebase rather than inventing new ones:
|
||||
- Blocking-work-on-delaymgr → the vtask dispatch pattern (`vtask_alloc` +
|
||||
`vtask_init` + `vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK)`, with
|
||||
an inline-fallback + WARN on alloc/enqueue failure — never silently drop
|
||||
scheduled work).
|
||||
- Self-termination on an operational condition → fence-and-retry (set a
|
||||
sticky non-promotable flag, log loud, schedule a retry via `delay_task_ms`,
|
||||
stay alive) — see `ha_self_restart_enter_fence_and_retry` for the
|
||||
reference shape.
|
||||
- A `delay_task_ms`-scheduled task whose doc/comment claims to run
|
||||
"immediately" → verify against `delay_task_ms`'s real semantics (first
|
||||
fire is AFTER `delay_ms` elapses, not before) before trusting the
|
||||
comment; if genuine immediacy is required and the task body only does
|
||||
non-blocking work, call it once synchronously before arming the periodic
|
||||
timer.
|
||||
- After each fix (or small batch of related fixes), rebuild with
|
||||
`/xce-build <component>` and confirm 0 test failures before moving on —
|
||||
don't accumulate multiple unverified fixes.
|
||||
|
||||
## Step 4 — Live regression (mandatory, not optional)
|
||||
|
||||
Code-level correctness is not sufficient proof for `controller`/HA work —
|
||||
bug-1431/1432 in this project's history were invisible to unit tests and the
|
||||
initial code audit, and only surfaced under a live chaotic soak. Run, in
|
||||
this order, using the `/xce-test-ha` skill:
|
||||
|
||||
1. `/xce-test-ha [soak-rounds] <scratch-dir>` in **chaotic** mode
|
||||
(`--chaotic`) — the adversarial overlapping-kill stress mode. Must reach
|
||||
`result=PASS` (`committed_divergence=0` **and** `committed_job_loss=0`).
|
||||
2. `/xce-test-ha [soak-rounds] <scratch-dir>` in **sequential** mode
|
||||
(default) — the supported single-failover lifecycle, including the
|
||||
leadership-coverage maneuver (all 3 nodes must lead at least once). Must
|
||||
print the terminal `PASS:` line.
|
||||
|
||||
Launch each soak directly from the main session via Bash with
|
||||
`setsid nohup ... & disown` (never through a forked/backgrounded sub-agent —
|
||||
see the `/xce-test-ha` skill's own warning about SIGHUP killing the cluster
|
||||
mid-round), then poll the log periodically (`ScheduleWakeup`, 3-5 min
|
||||
intervals) rather than blocking synchronously — a 10-round soak takes
|
||||
several minutes per mode.
|
||||
|
||||
**On FAIL**, root-cause from evidence, not guesswork:
|
||||
- Pull the exact failing round's timestamps from the driver log, then grep
|
||||
the per-node controller logs (`<dir>/logs/ctrl-*.log`) around that window
|
||||
for the relevant subsystem (`committed_lsn`, `safe_committed_lsn`,
|
||||
`converged`, `commit_advance_locked`, etc.).
|
||||
- Distinguish a **true durability violation** (`committed_divergence>0`, or a
|
||||
job still missing at final settlement) from a **visibility-latency gap**
|
||||
(`committed_divergence=0`, job reappears by final settlement — still a
|
||||
hard FAIL per this project's philosophy, but the fix must close the
|
||||
*window*, never unsafely expose an unconfirmed old-term entry early just
|
||||
to make the check pass — see bug-1431/1432 for the reference shape of a
|
||||
correct fix).
|
||||
- Fix, rebuild, re-run the SAME mode that failed before re-running the other.
|
||||
Repeat until both modes PASS.
|
||||
|
||||
## Step 5 — Record and report
|
||||
|
||||
- Append every genuinely new root-caused bug to `.wolf/buglog.json` (id,
|
||||
error_message, file, root_cause, fix, tags, related_bugs) — check the
|
||||
existing entry count first so the new `bug-NNNN` id doesn't collide.
|
||||
- Append durable, non-obvious learnings to `.wolf/cerebrum.md`'s Key
|
||||
Learnings section (anchor the edit near an existing dated entry to avoid
|
||||
the "## Key Learnings" heading collision — the file has one per dated
|
||||
session block).
|
||||
- Append a one-line summary to `.wolf/memory.md`.
|
||||
- Final report to the user: violations found (with file:line), how each was
|
||||
fixed, unit test pass count, and both soak VERDICT lines verbatim. Do not
|
||||
claim "fixed" for anything the soak didn't actually exercise.
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
name: xce-test-ha
|
||||
description: "End-to-end HA failover soak test for an XCE controller cluster: deploy 3 nodes, submit jobs, then repeatedly kill the primary and verify zero committed-job loss across 5-10 rounds. Usage: /xce-test-ha [rounds] [dir]"
|
||||
description: "End-to-end HA failover soak test for an XCE controller cluster: deploy 3 nodes, submit jobs, then repeatedly kill the primary and verify zero committed-job loss across 5-50 rounds. Usage: /xce-test-ha [rounds] [dir]"
|
||||
model: haiku
|
||||
---
|
||||
|
||||
|
|
@ -23,7 +23,7 @@ A green run is `PASS: N rounds, M jobs submitted, zero job loss` (exit 0).
|
|||
|
||||
## Arguments
|
||||
|
||||
- Optional `[rounds]` — number of kill/restart rounds (clamped to 5–10, default 6).
|
||||
- Optional `[rounds]` — number of kill/restart rounds (clamped to 5–50, default 6).
|
||||
- Optional `[dir]` — deploy directory for the 3-node cluster
|
||||
(default `/home/claude/tmp/xce-ha-deploy`). Maps to `XCE_HA_TOP` /
|
||||
`run-ha-soak.sh`'s positional dir arg. Relative paths are resolved to
|
||||
|
|
@ -250,7 +250,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, or `factstream` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–10) | `6` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` |
|
||||
| `XCE_HA_JOBS` | jobs per submit batch | `3` |
|
||||
| `XCE_HA_WAITMS` | `wait-primary` budget per probe | `30000` |
|
||||
| `XCE_HA_FACTSTREAM_ROUNDS` | `--factstream` kill-mid-fact-stream rounds | `2` |
|
||||
|
|
@ -277,7 +277,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
5. **Initial workload** — drives `xce_test`:
|
||||
`set masters … / client / wait-primary / jobsub ×N / joblist (expect-job) / walcompare`.
|
||||
Every acked `JobId` is tracked.
|
||||
6–11. **Kill loop** (5–10 rounds): discover the live primary → `kill -9` it →
|
||||
6–11. **Kill loop** (5–50 rounds): discover the live primary → `kill -9` it →
|
||||
restart the node downed in the *previous* round (so the cluster returns to 3
|
||||
members) → wait for a new primary → `verify_no_loss` (expect-job every tracked
|
||||
JobId) → submit another batch → `verify_no_loss` again → `walcompare`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue