Commit graph

5 commits

Author SHA1 Message Date
71a814f302
docs(steering): add field-complete apply-parity invariant; extend xce-audit-invariants
persistence.md's APPLY_CLEAN verdict only checks entity count and apply-
cursor position, not per-field completeness -- bug-1678 (job allocation
silently dropped on restart replay despite APPLY_CLEAN reporting healthy)
fell through exactly this gap. Documents the sharper invariant online-
apply and replay-apply must produce field-complete equivalent state, not
just matching counts/cursor, and adds it as a standing audit dimension
(#7) to xce-audit-invariants' controller dispatch list.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
2026-08-25 06:40:20 +00:00
9f539583d3
docs: sync skills and steering docs with bug-1562..1566 findings
- .claude/skills/xce-build/SKILL.md: --incremental flag,
  xceadmin wheel building, corrected the standalone-install
  guidance (the earlier --system-site-packages venv approach was
  wrong -- nix devShell packages are $PYTHONPATH entries, not
  merged into site-packages).
- .claude/skills/xce-deploy-ha/SKILL.md: $BASH_BIN resolution,
  worker/scheduler startup, xceadmin usage against a deployment,
  troubleshooting rows for the newly-fixed bugs.
- .kiro/steering/ha-async-io.md: bug-1562's standby liveness
  watchdog re-arm invariant.
- .kiro/steering/persistence.md: bug-1563 (fsync backoff) and
  bug-1564 (persist_flush durable-advance callback).
- .kiro/steering/build-deployment.md: xceadmin wheel docs,
  bug-1565's --incremental flag, deploy dir tree updates.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-17 10:21:25 +00:00
40f510fb14 fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
2026-07-11 12:16:28 +00:00
4338a97d0e docs(steering): sync HA protocol with ha-term-safety — per-entry term, (term,lsn) reconcile, commit-source ownership, bug-828 truncate rule 2026-06-23 04:48:06 +00:00
20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00