From dbc631664d5b545224a5325ac884fbe326fc13cb Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 13 Jun 2026 14:58:55 +0000 Subject: [PATCH] docs(ha-single-leader-role): sync steering + contributor docs to three-state model + own-term gate (task 6.1) --- .claude/skills/xce-deploy-ha/SKILL.md | 2 +- .kiro/specs/ha-single-leader-role/tasks.md | 2 +- .kiro/steering/ha-async-io.md | 9 ++++----- .kiro/steering/roadmap.md | 4 ++-- src/controller/CLAUDE.md | 2 +- src/docs/cluster-config-split-analysis.md | 1 - 6 files changed, 9 insertions(+), 11 deletions(-) diff --git a/.claude/skills/xce-deploy-ha/SKILL.md b/.claude/skills/xce-deploy-ha/SKILL.md index 33d3020..2a3c6f4 100644 --- a/.claude/skills/xce-deploy-ha/SKILL.md +++ b/.claude/skills/xce-deploy-ha/SKILL.md @@ -95,7 +95,7 @@ grep "primary control channel established" "$TOP/logs/ctrl-2.log" "$TOP/logs/ctr ``` **Healthy cluster signs:** -- `ctrl-1.log`: `→ CTRL_PRIMARY_COMMITTING` then `→ CTRL_PRIMARY` +- `ctrl-1.log`: `HA FSM (raft): → CTRL_PRIMARY (term=...)` (or legacy `HA FSM: → CTRL_PRIMARY (term=...)`) - `ctrl-1.log`: `Config committed on majority (tag=..., pushed=2/2 standbys)` - `controller/ctrl-2/conf/cluster.bin.*` and `controller/ctrl-3/conf/cluster.bin.*` exist - `ctrl-2.log` / `ctrl-3.log`: `handle_repl_config_push: config saved and loaded` diff --git a/.kiro/specs/ha-single-leader-role/tasks.md b/.kiro/specs/ha-single-leader-role/tasks.md index 4d32a9f..a419d38 100644 --- a/.kiro/specs/ha-single-leader-role/tasks.md +++ b/.kiro/specs/ha-single-leader-role/tasks.md @@ -66,7 +66,7 @@ - _Requirements: 1.2_ - [ ] 6. Documentation sync and grep verification -- [ ] 6.1 Synchronize steering and contributor docs to the three-state model +- [x] 6.1 Synchronize steering and contributor docs to the three-state model - `.kiro/steering/ha-async-io.md`: heartbeat re-arm sentence (:51 — "PRIMARY only"); Replication Consistency section (:57 — replace the COMMITTING→PRIMARY convergence-promotion sentence with the own-term no-op gate: promotion appends WAL_NOOP, confirmation waits behind `committed_lsn >= writable_lsn`); dispatch-gate table (:68 — remove the CTRL_PRIMARY_COMMITTING row; note the PRIMARY no-quorum fact allowlist); Raft-to-code mapping (:79 — role set without PRIMARY_COMMITTING) - `src/controller/CLAUDE.md:40` role-mapping table; `.claude/skills/xce-deploy-ha/SKILL.md:98` expected log line (now a single `→ CTRL_PRIMARY`); `src/docs/cluster-config-split-analysis.md:34` config listing (drop the deleted knob) - Observable completion: `grep -rn "COMMITTING" .kiro/steering/ src/controller/CLAUDE.md .claude/skills/xce-deploy-ha/ src/docs/` returns zero matches diff --git a/.kiro/steering/ha-async-io.md b/.kiro/steering/ha-async-io.md index 5710237..0ecfa89 100644 --- a/.kiro/steering/ha-async-io.md +++ b/.kiro/steering/ha-async-io.md @@ -48,13 +48,13 @@ Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send Bidirectional heartbeat with Raft-style AppendEntries semantics: -- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY or PRIMARY_COMMITTING. +- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY. - **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence. - **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`. ## Replication Consistency (Two-Tier) -**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. `CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY` promotion gates on majority convergence (n_peers/2 converged standbys). Single-node mode returns -ENODEV (degraded). +**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes `CTRL_PRIMARY` directly and, at promotion, appends a `WAL_NOOP` entry of its current term and arms the commit engine's writable-LSN gate (`commit_set_writable_lsn`). Confirmation of any operation then waits behind `committed_lsn >= writable_lsn` — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded). **Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push. @@ -64,8 +64,7 @@ Bidirectional heartbeat with Raft-style AppendEntries semantics: | Role | Allowed | |------|---------| -| **CTRL_PRIMARY** | All opcodes | -| **CTRL_PRIMARY_COMMITTING** | Replication only (heartbeat, replConfig, walAck, hello); job/queue ops rejected | +| **CTRL_PRIMARY** | All opcodes (client ops subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` is still admitted) | | **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes | | **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer | @@ -76,7 +75,7 @@ This ensures a campaigning node never rejects state push from the current primar | TLA+ Variable | XCE Code | |--------------|----------| | `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | -| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY_COMMITTING, PRIMARY, STANDBY) | +| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY, STANDBY) | | `votedFor[i]` | `ha->voted_for_name[]` | | `log[i]` | WAL segments in `persist->wal_path` | | `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index e607ae0..c7c4b0a 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -15,7 +15,7 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin - [x] ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; **IMPLEMENTED 2026-06-12**: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes) - [x] ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; **IMPLEMENTED 2026-06-13**: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied) -- [x] ha-single-leader-role — promotion no-op + own-term gate, CTRL_PRIMARY_COMMITTING removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks) +- [x] ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks) - [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks) ## Future Specs / Backlog (dependency order) @@ -35,7 +35,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o - [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ - [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_ - [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_ -- [ ] steering sync — `ha-async-io.md` still documents CTRL_PRIMARY_COMMITTING and the two-tier model; update after ha-consistency-invariants lands. +- [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1. ## Superseded / Absorbed Items diff --git a/src/controller/CLAUDE.md b/src/controller/CLAUDE.md index 228ccba..0d8dd74 100644 --- a/src/controller/CLAUDE.md +++ b/src/controller/CLAUDE.md @@ -37,7 +37,7 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft | TLA+ Variable | XCE Code | Location | |--------------|----------|----------| | `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:139` | -| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY_COMMITTING,PRIMARY,STANDBY} | `ha.h:139` | +| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY,STANDBY} | `ha.h:139` | | `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:161` | | `log[i]` | WAL segments in `persist->wal_path` | `persist.c` | | `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | `persist.h` | diff --git a/src/docs/cluster-config-split-analysis.md b/src/docs/cluster-config-split-analysis.md index fc7bdb3..03118c0 100644 --- a/src/docs/cluster-config-split-analysis.md +++ b/src/docs/cluster-config-split-analysis.md @@ -31,7 +31,6 @@ ha_config_t ha_cfg = { .heartbeat_interval_ms = HA_DEFAULT_HEARTBEAT_INTERVAL_MS, .heartbeat_timeout_ms = HA_DEFAULT_HEARTBEAT_TIMEOUT_MS, .election_timeout_ms = HA_DEFAULT_ELECTION_TIMEOUT_MS, - .committing_timeout_ms = HA_DEFAULT_COMMITTING_TIMEOUT_MS, ... }; ```