forked from lei.chen/myxce
docs(ha-single-leader-role): sync steering + contributor docs to three-state model + own-term gate (task 6.1)
This commit is contained in:
parent
93dfc02717
commit
dbc631664d
6 changed files with 9 additions and 11 deletions
|
|
@ -95,7 +95,7 @@ grep "primary control channel established" "$TOP/logs/ctrl-2.log" "$TOP/logs/ctr
|
||||||
```
|
```
|
||||||
|
|
||||||
**Healthy cluster signs:**
|
**Healthy cluster signs:**
|
||||||
- `ctrl-1.log`: `→ CTRL_PRIMARY_COMMITTING` then `→ CTRL_PRIMARY`
|
- `ctrl-1.log`: `HA FSM (raft): → CTRL_PRIMARY (term=...)` (or legacy `HA FSM: → CTRL_PRIMARY (term=...)`)
|
||||||
- `ctrl-1.log`: `Config committed on majority (tag=..., pushed=2/2 standbys)`
|
- `ctrl-1.log`: `Config committed on majority (tag=..., pushed=2/2 standbys)`
|
||||||
- `controller/ctrl-2/conf/cluster.bin.*` and `controller/ctrl-3/conf/cluster.bin.*` exist
|
- `controller/ctrl-2/conf/cluster.bin.*` and `controller/ctrl-3/conf/cluster.bin.*` exist
|
||||||
- `ctrl-2.log` / `ctrl-3.log`: `handle_repl_config_push: config saved and loaded`
|
- `ctrl-2.log` / `ctrl-3.log`: `handle_repl_config_push: config saved and loaded`
|
||||||
|
|
|
||||||
|
|
@ -66,7 +66,7 @@
|
||||||
- _Requirements: 1.2_
|
- _Requirements: 1.2_
|
||||||
|
|
||||||
- [ ] 6. Documentation sync and grep verification
|
- [ ] 6. Documentation sync and grep verification
|
||||||
- [ ] 6.1 Synchronize steering and contributor docs to the three-state model
|
- [x] 6.1 Synchronize steering and contributor docs to the three-state model
|
||||||
- `.kiro/steering/ha-async-io.md`: heartbeat re-arm sentence (:51 — "PRIMARY only"); Replication Consistency section (:57 — replace the COMMITTING→PRIMARY convergence-promotion sentence with the own-term no-op gate: promotion appends WAL_NOOP, confirmation waits behind `committed_lsn >= writable_lsn`); dispatch-gate table (:68 — remove the CTRL_PRIMARY_COMMITTING row; note the PRIMARY no-quorum fact allowlist); Raft-to-code mapping (:79 — role set without PRIMARY_COMMITTING)
|
- `.kiro/steering/ha-async-io.md`: heartbeat re-arm sentence (:51 — "PRIMARY only"); Replication Consistency section (:57 — replace the COMMITTING→PRIMARY convergence-promotion sentence with the own-term no-op gate: promotion appends WAL_NOOP, confirmation waits behind `committed_lsn >= writable_lsn`); dispatch-gate table (:68 — remove the CTRL_PRIMARY_COMMITTING row; note the PRIMARY no-quorum fact allowlist); Raft-to-code mapping (:79 — role set without PRIMARY_COMMITTING)
|
||||||
- `src/controller/CLAUDE.md:40` role-mapping table; `.claude/skills/xce-deploy-ha/SKILL.md:98` expected log line (now a single `→ CTRL_PRIMARY`); `src/docs/cluster-config-split-analysis.md:34` config listing (drop the deleted knob)
|
- `src/controller/CLAUDE.md:40` role-mapping table; `.claude/skills/xce-deploy-ha/SKILL.md:98` expected log line (now a single `→ CTRL_PRIMARY`); `src/docs/cluster-config-split-analysis.md:34` config listing (drop the deleted knob)
|
||||||
- Observable completion: `grep -rn "COMMITTING" .kiro/steering/ src/controller/CLAUDE.md .claude/skills/xce-deploy-ha/ src/docs/` returns zero matches
|
- Observable completion: `grep -rn "COMMITTING" .kiro/steering/ src/controller/CLAUDE.md .claude/skills/xce-deploy-ha/ src/docs/` returns zero matches
|
||||||
|
|
|
||||||
|
|
@ -48,13 +48,13 @@ Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send
|
||||||
|
|
||||||
Bidirectional heartbeat with Raft-style AppendEntries semantics:
|
Bidirectional heartbeat with Raft-style AppendEntries semantics:
|
||||||
|
|
||||||
- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY or PRIMARY_COMMITTING.
|
- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
|
||||||
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
|
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
|
||||||
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
|
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
|
||||||
|
|
||||||
## Replication Consistency (Two-Tier)
|
## Replication Consistency (Two-Tier)
|
||||||
|
|
||||||
**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. `CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY` promotion gates on majority convergence (n_peers/2 converged standbys). Single-node mode returns -ENODEV (degraded).
|
**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes `CTRL_PRIMARY` directly and, at promotion, appends a `WAL_NOOP` entry of its current term and arms the commit engine's writable-LSN gate (`commit_set_writable_lsn`). Confirmation of any operation then waits behind `committed_lsn >= writable_lsn` — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded).
|
||||||
|
|
||||||
**Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.
|
**Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.
|
||||||
|
|
||||||
|
|
@ -64,8 +64,7 @@ Bidirectional heartbeat with Raft-style AppendEntries semantics:
|
||||||
|
|
||||||
| Role | Allowed |
|
| Role | Allowed |
|
||||||
|------|---------|
|
|------|---------|
|
||||||
| **CTRL_PRIMARY** | All opcodes |
|
| **CTRL_PRIMARY** | All opcodes (client ops subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` is still admitted) |
|
||||||
| **CTRL_PRIMARY_COMMITTING** | Replication only (heartbeat, replConfig, walAck, hello); job/queue ops rejected |
|
|
||||||
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes |
|
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes |
|
||||||
| **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer |
|
| **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer |
|
||||||
|
|
||||||
|
|
@ -76,7 +75,7 @@ This ensures a campaigning node never rejects state push from the current primar
|
||||||
| TLA+ Variable | XCE Code |
|
| TLA+ Variable | XCE Code |
|
||||||
|--------------|----------|
|
|--------------|----------|
|
||||||
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) |
|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) |
|
||||||
| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY_COMMITTING, PRIMARY, STANDBY) |
|
| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY, STANDBY) |
|
||||||
| `votedFor[i]` | `ha->voted_for_name[]` |
|
| `votedFor[i]` | `ha->voted_for_name[]` |
|
||||||
| `log[i]` | WAL segments in `persist->wal_path` |
|
| `log[i]` | WAL segments in `persist->wal_path` |
|
||||||
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) |
|
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) |
|
||||||
|
|
|
||||||
|
|
@ -15,7 +15,7 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin
|
||||||
|
|
||||||
- [x] ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; **IMPLEMENTED 2026-06-12**: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes)
|
- [x] ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; **IMPLEMENTED 2026-06-12**: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes)
|
||||||
- [x] ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; **IMPLEMENTED 2026-06-13**: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied)
|
- [x] ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; **IMPLEMENTED 2026-06-13**: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied)
|
||||||
- [x] ha-single-leader-role — promotion no-op + own-term gate, CTRL_PRIMARY_COMMITTING removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
|
- [x] ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
|
||||||
- [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
|
- [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
|
||||||
|
|
||||||
## Future Specs / Backlog (dependency order)
|
## Future Specs / Backlog (dependency order)
|
||||||
|
|
@ -35,7 +35,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
|
||||||
- [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_
|
- [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_
|
||||||
- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
|
- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
|
||||||
- [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_
|
- [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_
|
||||||
- [ ] steering sync — `ha-async-io.md` still documents CTRL_PRIMARY_COMMITTING and the two-tier model; update after ha-consistency-invariants lands.
|
- [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
|
||||||
|
|
||||||
## Superseded / Absorbed Items
|
## Superseded / Absorbed Items
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -37,7 +37,7 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
|
||||||
| TLA+ Variable | XCE Code | Location |
|
| TLA+ Variable | XCE Code | Location |
|
||||||
|--------------|----------|----------|
|
|--------------|----------|----------|
|
||||||
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:139` |
|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:139` |
|
||||||
| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY_COMMITTING,PRIMARY,STANDBY} | `ha.h:139` |
|
| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY,STANDBY} | `ha.h:139` |
|
||||||
| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:161` |
|
| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:161` |
|
||||||
| `log[i]` | WAL segments in `persist->wal_path` | `persist.c` |
|
| `log[i]` | WAL segments in `persist->wal_path` | `persist.c` |
|
||||||
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | `persist.h` |
|
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | `persist.h` |
|
||||||
|
|
|
||||||
|
|
@ -31,7 +31,6 @@ ha_config_t ha_cfg = {
|
||||||
.heartbeat_interval_ms = HA_DEFAULT_HEARTBEAT_INTERVAL_MS,
|
.heartbeat_interval_ms = HA_DEFAULT_HEARTBEAT_INTERVAL_MS,
|
||||||
.heartbeat_timeout_ms = HA_DEFAULT_HEARTBEAT_TIMEOUT_MS,
|
.heartbeat_timeout_ms = HA_DEFAULT_HEARTBEAT_TIMEOUT_MS,
|
||||||
.election_timeout_ms = HA_DEFAULT_ELECTION_TIMEOUT_MS,
|
.election_timeout_ms = HA_DEFAULT_ELECTION_TIMEOUT_MS,
|
||||||
.committing_timeout_ms = HA_DEFAULT_COMMITTING_TIMEOUT_MS,
|
|
||||||
...
|
...
|
||||||
};
|
};
|
||||||
```
|
```
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue