docs(ha-single-leader-role): sync steering + contributor docs to three-state model + own-term gate (task 6.1)

This commit is contained in:
Rongsong Shen 2026-06-13 14:58:55 +00:00
parent 93dfc02717
commit dbc631664d
6 changed files with 9 additions and 11 deletions

View file

@ -95,7 +95,7 @@ grep "primary control channel established" "$TOP/logs/ctrl-2.log" "$TOP/logs/ctr
``` ```
**Healthy cluster signs:** **Healthy cluster signs:**
- `ctrl-1.log`: `→ CTRL_PRIMARY_COMMITTING` then `→ CTRL_PRIMARY` - `ctrl-1.log`: `HA FSM (raft): → CTRL_PRIMARY (term=...)` (or legacy `HA FSM: → CTRL_PRIMARY (term=...)`)
- `ctrl-1.log`: `Config committed on majority (tag=..., pushed=2/2 standbys)` - `ctrl-1.log`: `Config committed on majority (tag=..., pushed=2/2 standbys)`
- `controller/ctrl-2/conf/cluster.bin.*` and `controller/ctrl-3/conf/cluster.bin.*` exist - `controller/ctrl-2/conf/cluster.bin.*` and `controller/ctrl-3/conf/cluster.bin.*` exist
- `ctrl-2.log` / `ctrl-3.log`: `handle_repl_config_push: config saved and loaded` - `ctrl-2.log` / `ctrl-3.log`: `handle_repl_config_push: config saved and loaded`

View file

@ -66,7 +66,7 @@
- _Requirements: 1.2_ - _Requirements: 1.2_
- [ ] 6. Documentation sync and grep verification - [ ] 6. Documentation sync and grep verification
- [ ] 6.1 Synchronize steering and contributor docs to the three-state model - [x] 6.1 Synchronize steering and contributor docs to the three-state model
- `.kiro/steering/ha-async-io.md`: heartbeat re-arm sentence (:51 — "PRIMARY only"); Replication Consistency section (:57 — replace the COMMITTING→PRIMARY convergence-promotion sentence with the own-term no-op gate: promotion appends WAL_NOOP, confirmation waits behind `committed_lsn >= writable_lsn`); dispatch-gate table (:68 — remove the CTRL_PRIMARY_COMMITTING row; note the PRIMARY no-quorum fact allowlist); Raft-to-code mapping (:79 — role set without PRIMARY_COMMITTING) - `.kiro/steering/ha-async-io.md`: heartbeat re-arm sentence (:51 — "PRIMARY only"); Replication Consistency section (:57 — replace the COMMITTING→PRIMARY convergence-promotion sentence with the own-term no-op gate: promotion appends WAL_NOOP, confirmation waits behind `committed_lsn >= writable_lsn`); dispatch-gate table (:68 — remove the CTRL_PRIMARY_COMMITTING row; note the PRIMARY no-quorum fact allowlist); Raft-to-code mapping (:79 — role set without PRIMARY_COMMITTING)
- `src/controller/CLAUDE.md:40` role-mapping table; `.claude/skills/xce-deploy-ha/SKILL.md:98` expected log line (now a single `→ CTRL_PRIMARY`); `src/docs/cluster-config-split-analysis.md:34` config listing (drop the deleted knob) - `src/controller/CLAUDE.md:40` role-mapping table; `.claude/skills/xce-deploy-ha/SKILL.md:98` expected log line (now a single `→ CTRL_PRIMARY`); `src/docs/cluster-config-split-analysis.md:34` config listing (drop the deleted knob)
- Observable completion: `grep -rn "COMMITTING" .kiro/steering/ src/controller/CLAUDE.md .claude/skills/xce-deploy-ha/ src/docs/` returns zero matches - Observable completion: `grep -rn "COMMITTING" .kiro/steering/ src/controller/CLAUDE.md .claude/skills/xce-deploy-ha/ src/docs/` returns zero matches

View file

@ -48,13 +48,13 @@ Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send
Bidirectional heartbeat with Raft-style AppendEntries semantics: Bidirectional heartbeat with Raft-style AppendEntries semantics:
- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY or PRIMARY_COMMITTING. - **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence. - **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`. - **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
## Replication Consistency (Two-Tier) ## Replication Consistency (Two-Tier)
**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. `CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY` promotion gates on majority convergence (n_peers/2 converged standbys). Single-node mode returns -ENODEV (degraded). **Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes `CTRL_PRIMARY` directly and, at promotion, appends a `WAL_NOOP` entry of its current term and arms the commit engine's writable-LSN gate (`commit_set_writable_lsn`). Confirmation of any operation then waits behind `committed_lsn >= writable_lsn` — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded).
**Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push. **Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.
@ -64,8 +64,7 @@ Bidirectional heartbeat with Raft-style AppendEntries semantics:
| Role | Allowed | | Role | Allowed |
|------|---------| |------|---------|
| **CTRL_PRIMARY** | All opcodes | | **CTRL_PRIMARY** | All opcodes (client ops subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` is still admitted) |
| **CTRL_PRIMARY_COMMITTING** | Replication only (heartbeat, replConfig, walAck, hello); job/queue ops rejected |
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes | | **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes |
| **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer | | **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer |
@ -76,7 +75,7 @@ This ensures a campaigning node never rejects state push from the current primar
| TLA+ Variable | XCE Code | | TLA+ Variable | XCE Code |
|--------------|----------| |--------------|----------|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | | `currentTerm[i]` | `ha->term` (_Atomic uint64_t) |
| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY_COMMITTING, PRIMARY, STANDBY) | | `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY, STANDBY) |
| `votedFor[i]` | `ha->voted_for_name[]` | | `votedFor[i]` | `ha->voted_for_name[]` |
| `log[i]` | WAL segments in `persist->wal_path` | | `log[i]` | WAL segments in `persist->wal_path` |
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | | `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) |

View file

@ -15,7 +15,7 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin
- [x] ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; **IMPLEMENTED 2026-06-12**: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes) - [x] ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; **IMPLEMENTED 2026-06-12**: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes)
- [x] ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; **IMPLEMENTED 2026-06-13**: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied) - [x] ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; **IMPLEMENTED 2026-06-13**: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied)
- [x] ha-single-leader-role — promotion no-op + own-term gate, CTRL_PRIMARY_COMMITTING removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks) - [x] ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
- [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks) - [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
## Future Specs / Backlog (dependency order) ## Future Specs / Backlog (dependency order)
@ -35,7 +35,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
- [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ - [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_
- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_ - [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
- [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_ - [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_
- [ ] steering sync — `ha-async-io.md` still documents CTRL_PRIMARY_COMMITTING and the two-tier model; update after ha-consistency-invariants lands. - [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
## Superseded / Absorbed Items ## Superseded / Absorbed Items

View file

@ -37,7 +37,7 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
| TLA+ Variable | XCE Code | Location | | TLA+ Variable | XCE Code | Location |
|--------------|----------|----------| |--------------|----------|----------|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:139` | | `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:139` |
| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY_COMMITTING,PRIMARY,STANDBY} | `ha.h:139` | | `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY,STANDBY} | `ha.h:139` |
| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:161` | | `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:161` |
| `log[i]` | WAL segments in `persist->wal_path` | `persist.c` | | `log[i]` | WAL segments in `persist->wal_path` | `persist.c` |
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | `persist.h` | | `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | `persist.h` |

View file

@ -31,7 +31,6 @@ ha_config_t ha_cfg = {
.heartbeat_interval_ms = HA_DEFAULT_HEARTBEAT_INTERVAL_MS, .heartbeat_interval_ms = HA_DEFAULT_HEARTBEAT_INTERVAL_MS,
.heartbeat_timeout_ms = HA_DEFAULT_HEARTBEAT_TIMEOUT_MS, .heartbeat_timeout_ms = HA_DEFAULT_HEARTBEAT_TIMEOUT_MS,
.election_timeout_ms = HA_DEFAULT_ELECTION_TIMEOUT_MS, .election_timeout_ms = HA_DEFAULT_ELECTION_TIMEOUT_MS,
.committing_timeout_ms = HA_DEFAULT_COMMITTING_TIMEOUT_MS,
... ...
}; };
``` ```