myxce/.claude/skills/xce-audit-invariants/SKILL.md

name	description	allowed-tools	argument-hint
xce-audit-invariants	Audit a component's documented design constraints/invariants, fix confirmed violations, and prove the fix with build+unit+live HA regression. Usage: /xce-audit-invariants [component] [soak-rounds]. Component: controller (default, richest invariant surface) or scheduler (driver state-equivalence) both have a persisted dimension list in this skill; any other src/<component> with a CLAUDE.md/steering invariant section works too, dimensions improvised from what Step 1 finds.	Read, Write, Edit, Glob, Grep, Bash, Agent, TaskCreate, TaskUpdate	[component] [soak-rounds]

XCE Design-Invariant Audit & Fix

Runs the same loop used to close bug-1417/1418/1419/1430/1431/1432: read the documented invariants → audit code against them in parallel → fix confirmed violations → prove it with a full rebuild, the unit suite, and a live HA soak (both chaotic and sequential modes). Code review alone is not sufficient proof here — several of the bugs above were only found by a live soak after the code-level fixes already looked correct.

Args

• [component] (default controller) — which src/<component> to audit. controller is the richest target (.kiro/steering/ha-async-io.md, persistence.md, error-handling.md, plus src/controller/CLAUDE.md's Process-Lifecycle invariant and TLA+ Protocol Verification Checklist). scheduler also has a persisted dimension list below (driver state-equivalence, core I/O isolation, picker/allocator rendezvous). For any OTHER component, first check whether it has an equivalent invariant section in its CLAUDE.md — if not, say so and stop rather than inventing invariants to audit against (as of 2026-08-24, apiserver and worker have none — this is itself worth flagging to the user rather than silently no-op'ing).
• [soak-rounds] (default 10) — rounds for the final HA regression gate (only meaningful for controller; clamped to 5-50 by /xce-test-ha).

Step 1 — Gather the invariant sources (do not skip)

Read, in full, before auditing anything:

• src/<component>/CLAUDE.md — component-specific invariants (for controller: the "Process Lifecycle" DESIGN INVARIANT block and the full "Protocol Verification Checklist" under the TLA+ section; for scheduler: the "Drivers" bullet's DESIGN INVARIANT callout).
• The steering files that apply — for controller: .kiro/steering/ha-async-io.md (thread-role contract: iomgr/delaymgr must never block inline), persistence.md (WAL/commit/durability invariants), error-handling.md (ReturnError/errnox pattern, no silent error swallowing). For scheduler: .kiro/steering/scheduler-driver-equivalence.md (normal-mode driver.c vs sim-mode sim_loop.c state-level parity contract, known violations, testing discipline).
• .wolf/cerebrum.md's Key Learnings and Do-Not-Repeat sections — prior audits already found real gaps between what these docs claim and what the code does (e.g. a comment claiming a delaymgr task "starts immediately" when delay_task_ms actually delays its first fire — bug-1432). Treat every invariant-adjacent comment as a claim to verify, not a fact.
• .wolf/buglog.json — grep for the component's tag to avoid re-auditing something already closed, and to recognize a recurrence of a known bug family (e.g. bug-828/bug-851/bug-1060 class committed-job-loss bugs).

Step 2 — Parallel audit dispatch

Fan out one Agent-tool subagent per invariant dimension (not per file) so findings stay independently verifiable. For controller, the dimensions that have historically found real violations are:

1. Async I/O thread-role — does any iomgr/delaymgr callback do blocking work inline instead of dispatching to a VTASK_LIVE_BLOCK vtask?
2. Process lifecycle — does any code path call exit/abort/_exit/ tlog_exit as a designed response to an operational condition (corrupt WAL, lost quorum, failed catch-up) instead of fence-and-stay-alive?
3. Raft/TLA+ safety — walk the Protocol Verification Checklist in controller/CLAUDE.md line by line against the current ha.c/persist.c (election safety, leader completeness / no-self-truncation, commit rule, commit ordering, durability gating, freshness gate).
4. Commit/apply decoupling — is entity materialization (apply_committed_entry) ever reachable before an entry's LSN is <= persist->committed_lsn, on any ingest path (standby WAL receipt, catch-up, heartbeat)?
5. Error handling — silent swallowed errors, missing ReturnError, or a fail-open where the invariant docs require fail-loud-and-fence.
6. Object lifetime within a dispatch batch (new 2026-07-18, bug-1447) — any loop that processes a batch of items fetched in one call (epoll events[], a vtask drain, a peer-list scan) where one item's handler can free/unref an object that a LATER item in the SAME batch still references. iomgr_unmonitor()/endpoint_unref() only affects future dispatch, not an event already sitting in the current batch array — see ha-async-io.md "Same-batch endpoint lifetime" for the reference shape (io_worker_dispatch_batch's ref-whole-batch-before-dispatch fix). Check other batch-processing loops (vtaskmgr's drain, any peer-list iteration that can trigger eviction) for the same shape, not just io_worker.
7. Field-complete apply parity (new 2026-08-24, bug-1678) — for every WAL fact type, is every field of the entity it materializes actually (a) encoded into the WAL payload at write time, not just held in the in-memory struct, and (b) decoded and adopted on BOTH apply seams (apply_committed_entry() boot replay and apply_wal_entry() standby catch-up)? persist_apply_verdict()'s APPLY_CLEAN only checks entity count + cursor position — see persistence.md's "APPLY_CLEAN is entity-count + cursor-position only" note — so a field silently dropped between dispatch-time mutation and WAL encode is invisible to it. Check each WAL_* fact type's payload struct against the full entity fields a client can observe post-restart (xcecli/apiserver-visible fields especially — that's how bug-1678's dropped allocs/EXEC_HOST surfaced).
8. Decision Log fidelity (new 2026-08-25) — walk .wolf/cerebrum.md's Decision Log section (## Decision Log) end-to-end and confirm current controller code hasn't silently drifted from any stated architectural decision — this dimension is deliberately open-ended (re-read the WHOLE section fresh each audit, don't work off a fixed list here, since the log grows over sessions). Two seed candidates worth an initial look: (a) no client reply-wait timeout as a fix for a hung request — does any code path add a deadline/timeout wrapper around a slow request as the actual fix, instead of failing loud at the real transport/source (distinct from dimension 5's silent-swallow check — this is "papering over with a timeout," an anti-pattern in its own right)? (b) commit-first-then-reply ordering — for every job/queue/host control-op handler, is a success reply ever sent before commit_submit(WAIT) actually resolves (distinct from dimension 4, which checks materialization timing against committed_lsn, not reply timing — a handler can get apply-ordering right and still leak an early reply on a different code path)? Worker- reported fact entries are explicitly EXEMPT from the commit-first rule (pure Raft semantics, retained/retried indefinitely, no commit timeout) — don't flag those as violations.
9. Redirect-completeness on role-gated rejections (new 2026-08-26, bug-1700) — for every opcode absent from a role's ha_dispatch_gate() allow-list (ha.c, e.g. the CTRL_STANDBY case's switch), does the generic gate-rejection reply path in myreq_handler() (cluster.c, right after the ha_dispatch_gate() call) populate redirect_host/ redirect_port via fill_redirect_reply() (join.c) when the rejection reason is -XCE_ERR_STANDBY? Every client-side redirect-follower in this codebase (xceadmin.discovery.follow_redirects(), libxce's own failover path) is already generic/opcode-agnostic and starts working automatically the instant the server supplies a redirect — a bare rejection status with no redirect fields silently strands every one of them, and this is a defect regardless of how well the client is written. This exact gap shipped independently, under two different opcodes, before either was caught (OPCode_upgradeConfig — found and documented but left unfixed in test_e2e_live_ha_push.py's module docstring; OPCode_schedAdmin — bug-1700, the one that actually got fixed) — audit method: check the ONE generic gate-rejection call site first (not opcode-by-opcode), then confirm every currently-shipped primary-only opcode benefits from whatever that site does. Also check api-standards.md's "Adding a New Opcode" step 6 hasn't drifted from what the code actually does.

For scheduler, the dimensions are (added 2026-08-24, first used to close bug-1613/bug-1681 and confirm bug-1611's partial-fix status):

1. Driver state-equivalence — normal-mode driver.c and simulation-mode sim_loop.c drive the identical core session/decision-callback protocol ("one core, two consumers") and MUST produce identical decisions AND identical resource-accounting state (w->resInUse, core->sharedInUse, job->accounts), not just a matching decision sequence — see scheduler-driver-equivalence.md's DESIGN INVARIANT in full. Audit method given directly by that doc's own "Testing discipline" section: enumerate every call driver.c's normal-mode round makes beyond the core state machine itself (anything into scheduler.c/resource.c/plugin.c), then confirm sim_loop.c's round has an equivalent call in the same window — and the reverse direction too. Known violations as of 2026-08-24: bug-1613 (FIXED), bug-1681 (FIXED, plugin_begin_session/plugin_end_session parity), bug-1611 (PARTIALLY fixed — reconciliation only works within a single replay round, follow-up spec scheduler-sim-eager-reconciliation scoped but not designed), bug-1682 (a distinct, NOT-this-dimension guard-field bug found adjacent to the bug-1681 fix — don't re-report it under this dimension). Re-verify each known violation's CURRENT status before re-reporting it — a spec can land and fix one without the buglog entry ever being updated (this happened to bug-1613).
2. Core module I/O isolation — core.c/core_events.c/core_session.c (plus core_clock.c's one sanctioned clock_gettime() exception) must do zero controller I/O (no netio/xceapi/client/delay/idgen). Already MECHANICALLY enforced continuously by test_core_isolation.c (source-text scan + nm(1) undefined-symbol check) — stronger than a periodic audit. Only worth a subagent pass if you suspect the mechanical test itself has a gap (e.g. a new core-family file added without being wired into that test's file list) — check meson.build's sched_core_isolation_lib membership first before spending a full audit dimension on this.
3. Picker→allocator jbuf capacity-1 rendezvous — the project's determinism invariant, not a tuning knob (scheduler/CLAUDE.md). Audit: does anything raise the jbuf cbuf capacity above 1, or bypass the rendezvous with a second concurrent producer/consumer? Re-derive Req 5 AC 7 in .kiro/specs/scheduler-core/requirements.md before touching this.

Each subagent must report file:line evidence for every claimed violation — no speculative findings. Use the Agent tool directly (general-purpose or Explore subagent type); do not invoke the Workflow tool for this unless the user has explicitly asked for multi-agent orchestration in their own words.

Step 3 — Triage and fix

• Dedupe overlapping findings across dimensions.
• For each CONFIRMED violation, fix it following the established patterns in this codebase rather than inventing new ones:
  • Blocking-work-on-delaymgr → the vtask dispatch pattern (vtask_alloc + vtask_init + vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK), with an inline-fallback + WARN on alloc/enqueue failure — never silently drop scheduled work).
  • Self-termination on an operational condition → fence-and-retry (set a sticky non-promotable flag, log loud, schedule a retry via delay_task_ms, stay alive) — see ha_self_restart_enter_fence_and_retry for the reference shape.
  • A delay_task_ms-scheduled task whose doc/comment claims to run "immediately" → verify against delay_task_ms's real semantics (first fire is AFTER delay_ms elapses, not before) before trusting the comment; if genuine immediacy is required and the task body only does non-blocking work, call it once synchronously before arming the periodic timer.
• After each fix (or small batch of related fixes), rebuild with /xce-build <component> and confirm 0 test failures before moving on — don't accumulate multiple unverified fixes.

Step 4 — Live regression (mandatory for controller; not applicable to scheduler)

scheduler has no live-HA-soak equivalent — Step 3's full rebuild + full meson test suite (with RED/GREEN independently verified per fix, not just narrated) is the complete proof bar for a scheduler audit; stop there and proceed to Step 5. Only add a live check for a scheduler finding if the specific violation genuinely can't be exercised by the meson suite (e.g. a finding that only manifests via xce_scheduler -i -s <replay>'s REPL) — in that case run the real binary against a targeted .replay fixture, not the /xce-test-ha cluster soak below (that soak is controller-specific).

Code-level correctness is not sufficient proof for controller/HA work — bug-1431/1432 in this project's history were invisible to unit tests and the initial code audit, and only surfaced under a live chaotic soak. Run, in this order, using the /xce-test-ha skill:

1. /xce-test-ha [soak-rounds] <scratch-dir> in chaotic mode (--chaotic) — the adversarial overlapping-kill stress mode. Must reach result=PASS (committed_divergence=0 and committed_job_loss=0).
2. /xce-test-ha [soak-rounds] <scratch-dir> in sequential mode (default) — the supported single-failover lifecycle, including the leadership-coverage maneuver (all 3 nodes must lead at least once). Must print the terminal PASS: line.

Launch each soak directly from the main session via Bash with setsid nohup ... & disown (never through a forked/backgrounded sub-agent — see the /xce-test-ha skill's own warning about SIGHUP killing the cluster mid-round), then poll the log periodically (ScheduleWakeup, 3-5 min intervals) rather than blocking synchronously — a 10-round soak takes several minutes per mode.

On FAIL, root-cause from evidence, not guesswork:

• Pull the exact failing round's timestamps from the driver log, then grep the per-node controller logs (<dir>/logs/ctrl-*.log) around that window for the relevant subsystem (committed_lsn, safe_committed_lsn, converged, commit_advance_locked, etc.).
• Distinguish a true durability violation (committed_divergence>0, or a job still missing at final settlement) from a visibility-latency gap (committed_divergence=0, job reappears by final settlement — still a hard FAIL per this project's philosophy, but the fix must close the window, never unsafely expose an unconfirmed old-term entry early just to make the check pass — see bug-1431/1432 for the reference shape of a correct fix).
• Fix, rebuild, re-run the SAME mode that failed before re-running the other. Repeat until both modes PASS.

Step 5 — Record and report

• Append every genuinely new root-caused bug to .wolf/buglog.json (id, error_message, file, root_cause, fix, tags, related_bugs) — check the existing entry count first so the new bug-NNNN id doesn't collide.
• Append durable, non-obvious learnings to .wolf/cerebrum.md's Key Learnings section (anchor the edit near an existing dated entry to avoid the "## Key Learnings" heading collision — the file has one per dated session block).
• Append a one-line summary to .wolf/memory.md.
• Final report to the user: violations found (with file:line), how each was fixed, unit test pass count, and both soak VERDICT lines verbatim. Do not claim "fixed" for anything the soak didn't actually exercise.