Commit graph

18 commits

Author SHA1 Message Date
7512dc83f0 docs(specs): kiro-validate-impl GO closure for 8 HA/raft specs
libxce-failover, raft-sim-conformance, ha-commit-durability,
ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote,
ha-elect-ctx-lifecycle, and ha-entity-snapshots all had 100% of tasks
checked but had never run a formal /kiro-validate-impl closure pass
(spec.json stuck at phase "tasks-generated", some with stale
ready_for_implementation flags). Each now has a fresh validate_impl
GO decision backed by an ASAN build, full standalone test-suite run,
and a live 10-round sequential HA soak with zero committed-job loss.
2026-07-18 13:27:36 +00:00
9f530f5742 docs: fix unchecked group-header checkboxes across 8 fully-implemented specs
controller-status, ha-commit-engine, ha-heartbeat-committed-propagation,
ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and
raft-ha-core-safety each had every numbered sub-task marked [x] but their
group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic
gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real
unchecked sub-tasks in any of the eight files. No task content changed.
2026-07-07 08:45:41 +00:00
13c6e89309 docs: close ha-commit-durability Phase 6 (bug-795) + live-validate 4 specs
ha-commit-durability: closes the "pending confirmation under simultaneous
total outage" caveat. Reproduced cold-restart-all (kill all 3 members +
restart) 5 times against a live 3-node cluster on genuine ext4 (not the
sandbox's overlayfs) — zero committed-job loss across 40 rolling-failover
rounds + 5 cold-restart-all cycles. Confirms bug-795 was an overlayfs
fdatasync artifact, not a real durability gap; no production fix needed.
Commit Durability is now proven for both rolling single-node failover and
simultaneous total-cluster outage (src/controller/CLAUDE.md updated).

Live-validated and marked complete against a real 3-node cluster:
- controller-status 6.4: off-primary committed fidelity + responder
  timestamp, confirmed via a real failover
- ha-heartbeat-committed-propagation 4.1: standby committedLsn tracks the
  primary via heartbeat within the 3s interval
- ha-wal-replication 5.4: standby WAL replication verified via committedLsn
  match (job-facing ops are correctly primary-only by design; standbys
  return E_XCE_STANDBY, not a bug)
- libxce-failover 6.2: live 3-node failover E2E, jobsub raced against a
  primary kill resolves to exactly one job, no duplication

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
2026-07-03 02:41:28 +00:00
20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00
223e1b8578 docs(libxce-failover): record live-E2E round-2 fixes (bug-609 EINVAL, bug-610 reconnect SEGV) 2026-06-17 06:54:28 +00:00
16f4ba67c7 docs(libxce-failover): log bug-609 (known-dead EINVAL) + record 6.2 live-E2E partial status 2026-06-17 04:46:45 +00:00
c616f41364 test(libxce-failover): consolidate unit suite green + jitter staggering coverage (task 6.1) 2026-06-17 04:26:00 +00:00
5d1d1e3f12 feat(libxce-failover): controller populates primary host/port on E_XCE_REDIRECT replies (task 5.3) 2026-06-17 04:16:26 +00:00
fdb8938059 feat(libxce-failover): best-effort single-shot reconnect for fire-and-forget ops (notify/heartbeat/nop) (task 5.2) 2026-06-17 04:08:58 +00:00
4345974b54 feat(libxce-failover): wrap 18 reply-bearing xcecli_* ops in failover loop under per-ctx lock; correct nop OpCategory (task 5.1) 2026-06-17 03:57:51 +00:00
8783ce5a9d feat(libxce-failover): reconnect/rejoin hook — public setter + post-reconnect invocation before resume (task 4.1) 2026-06-17 03:46:05 +00:00
12284bc3f5 feat(libxce-failover): reconnect recovery — ctx ERR->RUN reset + stale wbuf drain (fixes post-failover EINVAL) (task 3.2) 2026-06-17 03:36:24 +00:00
84c5e32c2c feat(libxce-failover): retry decision loop — pure decide (budget/backoff/jitter, D6 exhaustion) + thin next wrapper (task 3.1) 2026-06-17 03:27:38 +00:00
cec2e647dc feat(libxce-failover): capture redirect hint in handle_reply (before free, before status return); verify endpoint-error wake (task 2.3) 2026-06-17 03:11:18 +00:00
c7e11a4e9e feat(libxce-failover): endpoint rotator — hint-first target selection + master rotation; keep api_ctx_t decls private (task 2.2) 2026-06-17 02:56:48 +00:00
fa2ac05036 feat(libxce-failover): 4-action status classifier with attempt-aware DUP-replay rule (task 2.1) 2026-06-17 02:42:54 +00:00
d11c6e1d89 feat(libxce-failover): reusable failover test harness — fake masters, status taxonomy, concurrency runner (task 1.2) 2026-06-17 02:32:19 +00:00
387ccfa8dc feat(libxce-failover): foundation — per-ctx failover state, redirect-hint fields, remove orphaned retry primitive (task 1.1) 2026-06-17 00:05:41 +00:00