lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

update documents

Browse source
This commit is contained in:
Rongsong Shen 2026-08-10 17:51:51 +08:00
parent c705cadadc
commit d70bf6a402
51 changed files with 995 additions and 1997 deletions
Download patch file Download diff file Expand all files Collapse all files

18
.kiro/specs/controller-status/design.md
View file

@ -47,8 +47,8 @@
### Existing Architecture Analysis
- **Opcode-only read precedent:** `clusterconf` is sent with no request body (`Request_body_nomsg`), dispatched in `cluster.c` `switch(request.opcode)`, handled by `handle_clusterconf` (`reply.c:10`), and returned via `xce_reply`. The client side is `xce_clusterconf` → `xcecli_clusterconf` (`XCECLI_FAILOVER_CALL`, `is_write=false`). **`ctrlstatus` clones this pattern.**
- **No-alloc state-report precedent:** `ha_nop_reply_state_report` (`ha.c:1784`) reads HA atomics into a stack struct and points the reply union at it without heap allocation. `handle_ctrlstatus` follows this model (a small bounded peer list is the only allocation).
- **Dispatch gate (the critical seam):** `ha_dispatch_gate` (`ha.c:2059`) is an allow-list keyed by role. The `CTRL_STANDBY` branch defaults to `-XCE_ERR_STANDBY`; `CTRL_INIT`/`CTRL_ELECTING` default to `-XCE_ERR_NOT_READY`; the `CTRL_PRIMARY` no-quorum branch allow-lists only specific reads. `clusterconf` is **not** in any standby allow-list — it is implicitly primary-only-when-serving. A status query must **not** inherit that; it must be explicitly allow-listed in every role branch.
- **No-alloc state-report precedent:** `ha_nop_reply_state_report` (`ha.c:3802`) reads HA atomics into a stack struct and points the reply union at it without heap allocation. `handle_ctrlstatus` follows this model (a small bounded peer list is the only allocation).
- **Dispatch gate (the critical seam):** `ha_dispatch_gate` (`ha.c:4525`) is an allow-list keyed by role. The `CTRL_STANDBY` branch defaults to `-XCE_ERR_STANDBY`; `CTRL_INIT`/`CTRL_ELECTING` default to `-XCE_ERR_NOT_READY`; the `CTRL_PRIMARY` no-quorum branch allow-lists only specific reads. `clusterconf` is **not** in any standby allow-list — it is implicitly primary-only-when-serving. A status query must **not** inherit that; it must be explicitly allow-listed in every role branch.
- **Config content checksum already exists:** `config_version_tag` = first 8 bytes of SHA-256 over the committed config blob; full digest in `config_sha256[32]`, round-trip-verified on load (`config_repl.c`).
### Pattern & dependency direction
@ -249,18 +249,18 @@ Live E2E (2026-06-20): a standby's `ctrlstatus` reported `committedLsn: 0` / `co
### Boundary Commitments (delta)
- **This spec owns (added):** the off-primary field-sourcing fix inside `ctrlstatus_fill` (read the *learned* committed values when not primary); a new read-only `CtrlStatus.responderTimeMs` field stamped at snapshot time; its render in `cmd_ctrlstatus`.
- **Out of Boundary (firm):** making the off-primary committed value **live-fresh** — i.e. consuming `HeartbeatMsg.committedLsn` / extending `OP_nop` to refresh `last_known_committed_lsn` on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to **ha-heartbeat-liveness** / **ha-repl-hardening**. controller-status reports the honest best-known value only.
- **Allowed Dependencies (added):** read access to `ha->last_known_committed_lsn` (`_Atomic uint64_t`, `ha.h:170`) and `ha->last_known_committed_config_tag` (`uint64_t`, `ha.h:172`); `clock_gettime(CLOCK_REALTIME)`; the existing `xcelog.h` wall-clock format for client rendering.
- **Revalidation Triggers (added):** any change to where/when `last_known_committed_lsn`/`last_known_committed_config_tag` are written (today: `cluster_apply_ctrlhello_reply` at join/RDHUP, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat.
- **Out of Boundary (firm):** making the off-primary committed value **live-fresh** — i.e. consuming `HeartbeatMsg.committedLsn` / extending `OP_nop` to refresh `last_known_committed_lsn` on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to **ha-heartbeat-liveness** / **ha-repl-hardening**. controller-status reports the honest best-known value only. (Note: superseded by **ha-heartbeat-committed-propagation** — that spec implemented exactly this, adding `ha_consume_heartbeat_committed`/`ha_observe_heartbeat_committed`, invoked from the primary's heartbeat in `cluster.c`, which now refreshes `last_known_committed_lsn`/`last_known_committed_config_tag` on every heartbeat, term-guarded and monotonic. See that spec's design.md.)
- **Allowed Dependencies (added):** read access to `ha->last_known_committed_lsn` (`_Atomic uint64_t`, `ha.h:183`) and `ha->last_known_committed_config_tag` (`ha.h:185`); `clock_gettime(CLOCK_REALTIME)`; the existing `xcelog.h` wall-clock format for client rendering. (Note: `last_known_committed_config_tag` is now also `_Atomic uint64_t` — migrated by ha-heartbeat-committed-propagation from the plain `uint64_t` this design was written against.)
- **Revalidation Triggers (added):** any change to where/when `last_known_committed_lsn`/`last_known_committed_config_tag` are written (today: `cluster_apply_ctrlhello_reply` at join/RDHUP, `ha_observe_heartbeat_committed` on every heartbeat, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat. (Note: the heartbeat writer was added by ha-heartbeat-committed-propagation, superseding the "join/RDHUP only" premise below.)
### Locked design decisions (from research items)
1. **Req 10 sourcing → role-branch (not uniform `max()`).** When `responderRole == CTRL_PRIMARY`, keep the node's own authoritative live markers (`persist.committed_lsn`, `ha->committed_config_tag`). Otherwise report the learned pair (`ha->last_known_committed_lsn`, `ha->last_known_committed_config_tag`). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the `last_known_*` read precedent at `ha.c:2570`, `ha.c:4126`.)
2. **Req 10 staleness caveat (documented behavior, not a bug).** `last_known_committed_lsn` is refreshed at **CtrlHello join / RDHUP-failover**, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The `0`-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports `0` = unknown, per Req 1.3 / 10.2).
1. **Req 10 sourcing → role-branch (not uniform `max()`).** When `responderRole == CTRL_PRIMARY`, keep the node's own authoritative live markers (`persist.committed_lsn`, `ha->committed_config_tag`). Otherwise report the learned pair (`ha->last_known_committed_lsn`, `ha->last_known_committed_config_tag`). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the `last_known_*` read precedent at `ha.c:5319` (`ha_vote_self_freshness`), `ha.c:6732`.)
2. **Req 10 staleness caveat (documented behavior, not a bug).** `last_known_committed_lsn` is refreshed at **CtrlHello join / RDHUP-failover**, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The `0`-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports `0` = unknown, per Req 1.3 / 10.2). (Note: superseded by **ha-heartbeat-committed-propagation** — `last_known_committed_lsn`/`_config_tag` are now also refreshed on every heartbeat via `ha_observe_heartbeat_committed`, so the join/RDHUP-only staleness window described here no longer applies; see that spec's design.md.)
3. **Req 11 wire type → `UInt64` epoch-milliseconds.** Numeric, locale-safe; the client formats it. (Pre-formatted `Text` was the alternative; rejected to keep the wire value machine-comparable and avoid encoding a locale into the reply.)
4. **Req 11 clock → `CLOCK_REALTIME`, stamped in `ctrlstatus_fill` at snapshot time.** NOT `CLOCK_MONOTONIC` (which `ha_on_heartbeat_received` uses for liveness) — monotonic is not wall-clock and cannot be correlated to logs. The client renders the millis to local `YYYY-MM-DD HH:MM:SS,mmm` using the `xcelog.h` convention so it is directly comparable to a log line.
### Research item carried to implementation
- Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h` and reuse it (or match its `YYYY-MM-DD HH:MM:SS,mmm` form) in `cmd_ctrlstatus` so the printed timestamp is byte-comparable to `ctrl-N.log`. `last_known_committed_config_tag` is non-atomic (`ha.h:172`, written under lock at `ha_convergence.c:377`); the status read is a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path.
- Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h` and reuse it (or match its `YYYY-MM-DD HH:MM:SS,mmm` form) in `cmd_ctrlstatus` so the printed timestamp is byte-comparable to `ctrl-N.log`. `last_known_committed_config_tag` was non-atomic at design time (written under lock in `ha_convergence.c`); the status read was planned as a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path. (Note: superseded by **ha-heartbeat-committed-propagation**, which migrated the field to `_Atomic uint64_t` at `ha.h:185`, written via `atomic_store_explicit` at `ha_convergence.c:368` and `ha.c:9011`; the as-built `ctrlstatus_fill` correctly reads it with `atomic_load_explicit`, not a bare aligned load.)
### Schema delta (Cap'n Proto)
Append one field to `CtrlStatus` (next free ordinal `@13`, after `peerCount @12`; append-only, never reorder):
@ -310,4 +310,4 @@ No new files (the capnp field is generated; all logic slots into existing TUs).
**Integration / round-trip (libxce):** `xcecli_ctrlstatus` decodes the new `responderTimeMs` field end-to-end (no manual decode — codegen), proving wire compatibility.
**E2E (live `/xce-deploy-ha`):** reproduce the original finding — submit jobs so the primary commits, query a **standby**; assert the standby's `committedLsn` is non-zero and tracks (within the join-refresh caveat) rather than `0`, and that the printed `responder timestamp` matches the standby's own `ctrl-N.log` line for that query.
**E2E (live `/xce-deploy-ha`):** reproduce the original finding — submit jobs so the primary commits, query a **standby**; assert the standby's `committedLsn` is non-zero and tracks (within the join-refresh caveat) rather than `0`, and that the printed `responder timestamp` matches the standby's own `ctrl-N.log` line for that query. (Note: the "join-refresh caveat" is narrower than described above — ha-heartbeat-committed-propagation added per-heartbeat refresh, so tracking is now near-live rather than join/RDHUP-only; see that spec's design.md.)