diff --git a/.claude/skills/xce-audit-invariants/SKILL.md b/.claude/skills/xce-audit-invariants/SKILL.md index 1d20de65..0fc04f2b 100644 --- a/.claude/skills/xce-audit-invariants/SKILL.md +++ b/.claude/skills/xce-audit-invariants/SKILL.md @@ -1,6 +1,6 @@ --- name: xce-audit-invariants -description: "Audit a component's documented design constraints/invariants, fix confirmed violations, and prove the fix with build+unit+live HA regression. Usage: /xce-audit-invariants [component] [soak-rounds]. Component: controller (default, richest invariant surface), or any src/ with a CLAUDE.md/steering invariant section." +description: "Audit a component's documented design constraints/invariants, fix confirmed violations, and prove the fix with build+unit+live HA regression. Usage: /xce-audit-invariants [component] [soak-rounds]. Component: controller (default, richest invariant surface) or scheduler (driver state-equivalence) both have a persisted dimension list in this skill; any other src/ with a CLAUDE.md/steering invariant section works too, dimensions improvised from what Step 1 finds." allowed-tools: Read, Write, Edit, Glob, Grep, Bash, Agent, TaskCreate, TaskUpdate argument-hint: "[component] [soak-rounds]" --- @@ -17,13 +17,16 @@ the code-level fixes already looked correct. ## Args - `[component]` (default `controller`) — which `src/` to audit. - `controller` is the primary target: it is the only component with a rich, - explicit invariant surface today (`.kiro/steering/ha-async-io.md`, + `controller` is the richest target (`.kiro/steering/ha-async-io.md`, `persistence.md`, `error-handling.md`, plus `src/controller/CLAUDE.md`'s - Process-Lifecycle invariant and TLA+ Protocol Verification Checklist). If - invoked against another component, first check whether it has an equivalent - invariant section in its `CLAUDE.md` — if not, say so and stop rather than - inventing invariants to audit against. + Process-Lifecycle invariant and TLA+ Protocol Verification Checklist). + `scheduler` also has a persisted dimension list below (driver + state-equivalence, core I/O isolation, picker/allocator rendezvous). For + any OTHER component, first check whether it has an equivalent invariant + section in its `CLAUDE.md` — if not, say so and stop rather than + inventing invariants to audit against (as of 2026-08-24, `apiserver` and + `worker` have none — this is itself worth flagging to the user rather than + silently no-op'ing). - `[soak-rounds]` (default `10`) — rounds for the final HA regression gate (only meaningful for `controller`; clamped to 5-50 by `/xce-test-ha`). @@ -32,11 +35,15 @@ the code-level fixes already looked correct. Read, in full, before auditing anything: - `src//CLAUDE.md` — component-specific invariants (for `controller`: the "Process Lifecycle" DESIGN INVARIANT block and the full - "Protocol Verification Checklist" under the TLA+ section). + "Protocol Verification Checklist" under the TLA+ section; for `scheduler`: + the "Drivers" bullet's DESIGN INVARIANT callout). - The steering files that apply — for `controller`: `.kiro/steering/ha-async-io.md` (thread-role contract: iomgr/delaymgr must never block inline), `persistence.md` (WAL/commit/durability invariants), `error-handling.md` (ReturnError/errnox - pattern, no silent error swallowing). + pattern, no silent error swallowing). For `scheduler`: + `.kiro/steering/scheduler-driver-equivalence.md` (normal-mode driver.c vs + sim-mode sim_loop.c state-level parity contract, known violations, testing + discipline). - `.wolf/cerebrum.md`'s Key Learnings and Do-Not-Repeat sections — prior audits already found real gaps between what these docs claim and what the code does (e.g. a comment claiming a delaymgr task "starts immediately" @@ -88,6 +95,62 @@ have historically found real violations are: each `WAL_*` fact type's payload struct against the full entity fields a client can observe post-restart (`xcecli`/apiserver-visible fields especially — that's how bug-1678's dropped `allocs`/EXEC_HOST surfaced). +8. **Decision Log fidelity** (new 2026-08-25) — walk `.wolf/cerebrum.md`'s + Decision Log section (`## Decision Log`) end-to-end and confirm current + controller code hasn't silently drifted from any stated architectural + decision — this dimension is deliberately open-ended (re-read the WHOLE + section fresh each audit, don't work off a fixed list here, since the log + grows over sessions). Two seed candidates worth an initial look: (a) **no + client reply-wait timeout as a fix for a hung request** — does any code + path add a deadline/timeout wrapper around a slow request as the actual + fix, instead of failing loud at the real transport/source (distinct from + dimension 5's silent-swallow check — this is "papering over with a + timeout," an anti-pattern in its own right)? (b) **commit-first-then-reply + ordering** — for every job/queue/host control-op handler, is a success + reply ever sent before `commit_submit(WAIT)` actually resolves (distinct + from dimension 4, which checks *materialization* timing against + `committed_lsn`, not *reply* timing — a handler can get apply-ordering + right and still leak an early reply on a different code path)? Worker- + reported fact entries are explicitly EXEMPT from the commit-first rule + (pure Raft semantics, retained/retried indefinitely, no commit timeout) + — don't flag those as violations. + +For `scheduler`, the dimensions are (added 2026-08-24, first used to close +bug-1613/bug-1681 and confirm bug-1611's partial-fix status): + +1. **Driver state-equivalence** — normal-mode `driver.c` and simulation-mode + `sim_loop.c` drive the identical core session/decision-callback protocol + ("one core, two consumers") and MUST produce identical decisions AND + identical resource-accounting state (`w->resInUse`, `core->sharedInUse`, + `job->accounts`), not just a matching decision sequence — see + `scheduler-driver-equivalence.md`'s DESIGN INVARIANT in full. Audit method + given directly by that doc's own "Testing discipline" section: enumerate + every call `driver.c`'s normal-mode round makes beyond the core state + machine itself (anything into `scheduler.c`/`resource.c`/`plugin.c`), then + confirm `sim_loop.c`'s round has an equivalent call in the same window — + and the reverse direction too. Known violations as of 2026-08-24: bug-1613 + (FIXED), bug-1681 (FIXED, plugin_begin_session/plugin_end_session parity), + bug-1611 (PARTIALLY fixed — reconciliation only works within a single + replay round, follow-up spec `scheduler-sim-eager-reconciliation` scoped + but not designed), bug-1682 (a distinct, NOT-this-dimension guard-field + bug found adjacent to the bug-1681 fix — don't re-report it under this + dimension). Re-verify each known violation's CURRENT status before + re-reporting it — a spec can land and fix one without the buglog entry + ever being updated (this happened to bug-1613). +2. **Core module I/O isolation** — `core.c`/`core_events.c`/`core_session.c` + (plus `core_clock.c`'s one sanctioned `clock_gettime()` exception) must do + zero controller I/O (no `netio`/`xceapi`/`client`/`delay`/`idgen`). Already + MECHANICALLY enforced continuously by `test_core_isolation.c` (source-text + scan + `nm(1)` undefined-symbol check) — stronger than a periodic audit. + Only worth a subagent pass if you suspect the mechanical test itself has a + gap (e.g. a new core-family file added without being wired into that + test's file list) — check `meson.build`'s `sched_core_isolation_lib` + membership first before spending a full audit dimension on this. +3. **Picker→allocator `jbuf` capacity-1 rendezvous** — the project's + determinism invariant, not a tuning knob (`scheduler/CLAUDE.md`). Audit: + does anything raise the `jbuf` cbuf capacity above 1, or bypass the + rendezvous with a second concurrent producer/consumer? Re-derive Req 5 AC + 7 in `.kiro/specs/scheduler-core/requirements.md` before touching this. Each subagent must report file:line evidence for every claimed violation — no speculative findings. Use the Agent tool directly (general-purpose or @@ -117,7 +180,16 @@ user has explicitly asked for multi-agent orchestration in their own words. `/xce-build ` and confirm 0 test failures before moving on — don't accumulate multiple unverified fixes. -## Step 4 — Live regression (mandatory, not optional) +## Step 4 — Live regression (mandatory for `controller`; not applicable to `scheduler`) + +`scheduler` has no live-HA-soak equivalent — Step 3's full rebuild + full +meson test suite (with RED/GREEN independently verified per fix, not just +narrated) is the complete proof bar for a `scheduler` audit; stop there and +proceed to Step 5. Only add a live check for a `scheduler` finding if the +specific violation genuinely can't be exercised by the meson suite (e.g. a +finding that only manifests via `xce_scheduler -i -s `'s REPL) — in +that case run the real binary against a targeted `.replay` fixture, not the +`/xce-test-ha` cluster soak below (that soak is controller-specific). Code-level correctness is not sufficient proof for `controller`/HA work — bug-1431/1432 in this project's history were invisible to unit tests and the