From 423b560353b6f7c7255abdaf2e5b03246aa5f775 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 20 Jun 2026 03:51:27 +0000 Subject: [PATCH] =?UTF-8?q?spec(controller-status):=20design=20=E2=80=94?= =?UTF-8?q?=20new=20ctrlstatus=20opcode,=20dispatch-gate=20all-roles,=20ve?= =?UTF-8?q?rsion+SHA256=20checksum,=20wait-primary=20probe=20swap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .kiro/specs/controller-status/design.md | 231 ++++++++++++++++++++++ .kiro/specs/controller-status/research.md | 14 ++ .kiro/specs/controller-status/spec.json | 10 +- 3 files changed, 250 insertions(+), 5 deletions(-) create mode 100644 .kiro/specs/controller-status/design.md diff --git a/.kiro/specs/controller-status/design.md b/.kiro/specs/controller-status/design.md new file mode 100644 index 0000000..50f1f1b --- /dev/null +++ b/.kiro/specs/controller-status/design.md @@ -0,0 +1,231 @@ +# Design Document — controller-status (`ctrlstatus`) + +## Overview + +**Purpose:** add a single, cheap, read-only client→controller query — `ctrlstatus`, an XCE analog of LSF `lsid` — that returns basic cluster/controller identity and health: who is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and content checksum, and a few HA health/progress fields (term, commit/apply position, fenced state, cluster name). + +**Users:** operators inspecting a running cluster, and test/E2E authors orchestrating failover (including `wait-primary`, which switches its probe to this query). + +**Impact:** today there is no status query — callers infer "is there a serving primary" indirectly (e.g. by listing jobs). This adds a new opcode by cloning the existing **opcode-only read** pattern of `clusterconf`, reads only already-maintained HA/config state (no behavior change), and crucially makes the query answerable on **any** node by adding it to the HA dispatch-gate allow-lists. + +### Goals +- A new read-only opcode returning the status fields of Requirements 2–6 in one request/response (1.1, 1.4). +- Answerable on **any** node and across failover — a standby reports the primary it knows (2.3, 7.1–7.3). +- Report a content checksum of the committed configuration so same-version divergence is detectable (5.2, 5.3). +- A `ctrlstatus` test-client command (8.x) and a `wait-primary` probe switched to this query (9.x). + +### Non-Goals +- No operator-facing CLI/REST `lsid` surface (Go tier) — later pass. +- No change to HA election, replication, or commit behavior — the query only **reads** existing state. +- No new configuration checksum computation — the committed-config SHA-256 already exists; this only exposes it. + +## Boundary Commitments + +### This Spec Owns +- A new Cap'n Proto opcode `ctrlstatus` and its reply payload (`CtrlStatus`, `CtrlPeer`). +- The controller handler `handle_ctrlstatus` that snapshots local HA/config state into the reply. +- The **dispatch-gate allow-list entries** that let `ctrlstatus` answer in every controller role (incl. no-quorum). +- The libxce client API (`xce_ctrlstatus_req` / `xce_ctrlstatus` / `xcecli_ctrlstatus` + free helper). +- The `ctrlstatus` test-client command and the `wait-primary` probe switch. + +### Out of Boundary +- The Go CLI/REST `lsid` surface (later spec). +- Any change to how HA state (role, term, primary identity, committed config tag, commit/apply LSN, fence) is **produced or maintained** — this spec only reads it. +- The existing `clusterconf` opcode and its resource/queue/worker payload (left unchanged). + +### Allowed Dependencies +- The Cap'n Proto schema + build-time codegen (`src/schema/controller.capnp` → generated `controller.capnp.h/.c`). +- The existing request/reply plumbing (`init_reply_by_req`, `xce_reply`, opcode-only `*_req` senders) and client failover wrapper (`XCECLI_FAILOVER_CALL`, `is_write=false`). +- Read access to existing HA state (`ha_state_t`), config (`myconf_t`: `committed_config_tag`, `config_sha256`, `peers_cfg`), and persist accessors (`persist_recovery_unrecoverable`, `persist_durable_lsn`). + +### Revalidation Triggers +- Any change to the `CtrlStatus`/`CtrlPeer` reply shape or the `ctrlstatus` opcode ordinal → re-check libxce decode + the test command + any future consumer. +- Any change to the dispatch-gate role allow-lists → re-check that `ctrlstatus` still answers in every role and that no other opcode's gating widened. +- A change to the meaning of `committed_config_tag` / `config_sha256` → re-check the version/checksum fields. + +## Architecture + +### Existing Architecture Analysis +- **Opcode-only read precedent:** `clusterconf` is sent with no request body (`Request_body_nomsg`), dispatched in `cluster.c` `switch(request.opcode)`, handled by `handle_clusterconf` (`reply.c:10`), and returned via `xce_reply`. The client side is `xce_clusterconf` → `xcecli_clusterconf` (`XCECLI_FAILOVER_CALL`, `is_write=false`). **`ctrlstatus` clones this pattern.** +- **No-alloc state-report precedent:** `ha_nop_reply_state_report` (`ha.c:1784`) reads HA atomics into a stack struct and points the reply union at it without heap allocation. `handle_ctrlstatus` follows this model (a small bounded peer list is the only allocation). +- **Dispatch gate (the critical seam):** `ha_dispatch_gate` (`ha.c:2059`) is an allow-list keyed by role. The `CTRL_STANDBY` branch defaults to `-XCE_ERR_STANDBY`; `CTRL_INIT`/`CTRL_ELECTING` default to `-XCE_ERR_NOT_READY`; the `CTRL_PRIMARY` no-quorum branch allow-lists only specific reads. `clusterconf` is **not** in any standby allow-list — it is implicitly primary-only-when-serving. A status query must **not** inherit that; it must be explicitly allow-listed in every role branch. +- **Config content checksum already exists:** `config_version_tag` = first 8 bytes of SHA-256 over the committed config blob; full digest in `config_sha256[32]`, round-trip-verified on load (`config_repl.c`). + +### Pattern & dependency direction +Extend-in-place along the existing request path. Dependency direction (unchanged): `schema → libxce (request/reply/client) → controller handler`; the test client depends on libxce. No upward dependencies introduced. + +```mermaid +graph LR + TestClient[xce_test ctrlstatus / wait-primary] --> Libxce[libxce xcecli_ctrlstatus] + Libxce --> Failover[XCECLI_FAILOVER_CALL is_write false] + Failover --> Gate[ha_dispatch_gate allow-list all roles] + Gate --> Handler[handle_ctrlstatus snapshot local state] + Handler --> HAState[ha_state_t and myconf_t read-only] + Handler --> Reply[xce_reply CtrlStatus] +``` + +## File Structure Plan + +### Modified Files +- `src/schema/controller.capnp` — add `OPCode ctrlstatus @40`; new `struct CtrlStatus $C.mapname("xce_ctrlstatus_t")` and `struct CtrlPeer $C.mapname("xce_ctrlpeer_t")`; reply-union member `ctrlstatus @17: CtrlStatus`. (Build regenerates `controller.capnp.h/.c` → `xce_ctrlstatus_t`, `RequestReply_data_ctrlstatus`, `free_CtrlStatus`.) +- `src/controller/reply.c` — add `handle_ctrlstatus` (snapshot local HA/config state into a `CtrlStatus` reply; small peer list; `xce_reply`). Mirrors `handle_clusterconf`/`ha_nop_reply_state_report`. +- `src/controller/cluster.c` — add `case OPCode_ctrlstatus: rc = handle_ctrlstatus(...)` to the opcode switch. +- `src/controller/ha.c` — add `OPCode_ctrlstatus` to the `ha_dispatch_gate` allow-lists for `CTRL_PRIMARY` (incl. the no-quorum branch), `CTRL_STANDBY`, `CTRL_ELECTING`, `CTRL_INIT`. **(the requirements-critical change)** +- `src/lib/request.c` + `src/include/request.h` — add `xce_ctrlstatus_req` (opcode-only sender, clone of `xce_clusterconf_req`). +- `src/lib/xceapi.c` — add `xce_ctrlstatus` (send req, `cbuf_get` status, `MOVE_PTR` reply union). +- `src/lib/client.c` + `src/include/client.h` — add `xcecli_ctrlstatus` (`XCECLI_FAILOVER_CALL`, `is_write=false`) and the `xcecli_free_ctrlstatus` macro. +- `src/controller/testsuite/test.c` — add `cmd_ctrlstatus` + table registration; switch `cmd_wait_primary`'s probe (currently `joblist_query(...)` at the deadline-loop call site) to `xcecli_ctrlstatus`. + +### New Files +- None (capnp C types are generated; all handlers/APIs slot into existing TUs). + +## System Flows + +### Status query (any role) + dispatch gate +```mermaid +sequenceDiagram + participant C as Client + participant F as Failover wrapper + participant N as Controller node + participant G as ha_dispatch_gate + participant H as handle_ctrlstatus + C->>F: xcecli_ctrlstatus + F->>N: ctrlstatus request (opcode-only) + N->>G: gate(opcode, role) + G-->>N: ALLOW (all roles, incl no-quorum) + N->>H: snapshot local state + H-->>C: CtrlStatus reply + Note over C,N: On a standby, primaryName = its known primary (or "unknown"); responderRole = STANDBY +``` +Gate decision: `ctrlstatus` is allow-listed in every role branch and does not require quorum or a committed read — it returns the node's local view even when fenced or without quorum (precisely when an operator most wants it). + +### wait-primary probe (Req 9) +`wait-primary` keeps its existing monotonic-deadline poll loop and timeout/exit semantics; only the probe changes from `joblist_query` to `xcecli_ctrlstatus`. Success predicate: the status reply reports a **non-`"unknown"`** current primary. On success it prints the served primary's identity. (Rationale + caveat below.) + +## Requirements Traceability + +| Requirement | Summary | Components | Interfaces | +|-------------|---------|------------|------------| +| 1.1, 1.2, 1.4 | Single-call read-only focused status | `handle_ctrlstatus`, `CtrlStatus` schema | opcode `ctrlstatus`, reply `CtrlStatus` | +| 1.3 | "unknown" for undeterminable fields | `handle_ctrlstatus` | empty Text/sentinel mapping | +| 2.1, 2.2, 2.3 | Current primary identity (unknown when unelected; standby's known primary) | `handle_ctrlstatus` (reads `ha->primary_host` under `ha->lock`) | `CtrlStatus.primaryName` | +| 3.1, 3.2, 3.3 | Answered-by node + role, distinct from primary | `handle_ctrlstatus` | `CtrlStatus.responderName`, `responderRole` | +| 4.1, 4.2 | Configured controllers + count | `handle_ctrlstatus` (reads `peers_cfg`) | `CtrlStatus.peers`, `peerCount` | +| 5.1, 5.2, 5.3 | Config version + content checksum | `handle_ctrlstatus` (reads `committed_config_tag`, `config_sha256`) | `CtrlStatus.configVersion`, `configChecksum` | +| 6.1–6.4 | Term, commit/apply, fenced, cluster name | `handle_ctrlstatus` | `CtrlStatus.term`, `committedLsn`, `appliedLsn`, `fenced`, `clusterName` | +| 7.1, 7.2, 7.3 | Failover-aware any-node delivery | `xcecli_ctrlstatus`, dispatch-gate allow-lists | `XCECLI_FAILOVER_CALL` (is_write=false), gate entries | +| 8.1, 8.2, 8.3 | Test-client `ctrlstatus` command | `cmd_ctrlstatus` | client command + exit code | +| 9.1–9.4 | wait-primary uses the status query | `cmd_wait_primary` probe swap | `xcecli_ctrlstatus` | + +## Components and Interfaces + +| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts | +|-----------|-------|--------|--------------|------------------|-----------| +| `CtrlStatus`/`CtrlPeer` schema | Schema | Reply payload for the status query | 2–6 | capnp codegen (P0) | API (reply) | +| `handle_ctrlstatus` | Controller | Snapshot local HA/config state into the reply | 1–6 | `ha_state_t`, `myconf_t`, persist accessors (P0) | Service | +| dispatch-gate allow-lists | Controller HA | Permit `ctrlstatus` in every role | 7 | `ha_dispatch_gate` (P0) | Service | +| `xce_ctrlstatus` / `xcecli_ctrlstatus` | libxce | Failover-aware client call + free | 1, 7 | failover wrapper (P0) | Service | +| `cmd_ctrlstatus` | Test client | Print the status | 8 | `xcecli_ctrlstatus` (P0) | State | +| `cmd_wait_primary` probe | Test client | Use status as the serving-primary probe | 9 | `xcecli_ctrlstatus` (P0) | State | + +### Schema — reply payload (Cap'n Proto) +```capnp +struct CtrlPeer $C.mapname("xce_ctrlpeer_t") { + name @0: Text; + host @1: Text; + port @2: UInt32; +} + +struct CtrlStatus $C.mapname("xce_ctrlstatus_t") { + clusterName @0: Text; + primaryName @1: Text; # empty ⇒ "unknown" (no elected/known primary) + responderName @2: Text; # the node that answered + responderRole @3: UInt8; # ctrl_role_t: INIT/ELECTING/PRIMARY/STANDBY + term @4: UInt64; + configVersion @5: UInt64; # committed_config_tag (8-byte SHA-256 prefix) + configChecksum @6: Data; # full committed-config SHA-256 (32 bytes) + committedLsn @7: UInt64; + appliedLsn @8: UInt64; + fenced @9: Bool; # in_catch_up OR recovery_unrecoverable + peers @10: List(CtrlPeer) $C.mapname("peers.list") $C.maplistcount("peers.count"); + peerCount @11: UInt32; +} +``` +- New OPCode ordinal: `ctrlstatus @40` (next free after `preVoteReply @39`). New reply-union member: `ctrlstatus @17` (next free after `walackreplymsg @16`). Field ordinals are append-only — never reuse/reorder. + +#### `handle_ctrlstatus` — Service contract +| Field | Detail | +|-------|--------| +| Intent | Build a `CtrlStatus` from the responding node's local HA/config state and reply | +| Requirements | 1.1–1.4, 2.1–2.3, 3.1–3.3, 4.1–4.2, 5.1–5.3, 6.1–6.4 | + +**Responsibilities & Constraints** +- Read-only: must not modify any HA, persist, or config state (1.2). +- Field sourcing: `clusterName`/`responderName` from `c->myconf`/own peerinfo; `responderRole`, `term`, `appliedLsn`, `committedLsn`, `committed_config_tag` via `atomic_load_explicit(..., acquire)`; `primaryName`/peers as below; `fenced = in_catch_up || persist_recovery_unrecoverable(...)`; `configChecksum` from `config_sha256[32]`. +- **Locking:** `primary_host`/`primary_port` are guarded by `ha->lock` — take it for a consistent snapshot, copy out, release. All other fields are atomics or immutable config. +- **"unknown" mapping (1.3, 2.2, 2.3):** if `primary_host[0] == '\0'` (no known primary), emit an empty `primaryName`, which the client renders as `"unknown"`. +- **Peers (4.x):** iterate `c->myconf->peers_cfg.candidates[0..count-1]` → one `CtrlPeer{name,host,port}` each; set `peerCount`. +- Answerable in every role; returns the local view even when fenced/no-quorum. + +**Dependencies** +- Inbound: `cluster.c` opcode switch — routes the request (P0). +- Outbound: `ha_state_t`, `myconf_t`, persist accessors — read-only snapshot (P0). +- Outbound: `init_reply_by_req`, `xce_reply` — reply encode/send (P0). + +#### Dispatch-gate allow-lists — Service contract +- Add `OPCode_ctrlstatus` to the allow-list in `ha_dispatch_gate` for `CTRL_PRIMARY` (including the no-quorum branch), `CTRL_STANDBY`, `CTRL_ELECTING`, and `CTRL_INIT` (7.1, 7.2). +- Constraint: the change is scoped strictly to `ctrlstatus`; no other opcode's gating may widen. `ctrlstatus` never requires quorum or a committed read. + +#### libxce client API — Service contract +```c +/* opcode-only request sender (clone of xce_clusterconf_req) */ +int xce_ctrlstatus_req(endpoint_t* ep); +/* send + await reply; on success MOVE_PTR the reply union out */ +int xce_ctrlstatus(xce_ctx_t* ctx, xce_ctrlstatus_t** status); +/* failover-aware read (is_write=false) */ +int xcecli_ctrlstatus(xcecli_ctx_t c, xce_ctrlstatus_t** status); +#define xcecli_free_ctrlstatus(s) xce_free_ctrlstatus(s) +``` +- Preconditions: valid client context. +- Postconditions: on success `*status` owns a decoded `CtrlStatus` (caller frees); on failure returns a typed negative (7.3) and `*status` untouched. +- Invariants: `is_write=false` — retryable/rotatable across the configured controllers. + +#### `cmd_ctrlstatus` (test client) — State contract +- Calls `xcecli_ctrlstatus`; on success prints the fields in human-readable form (8.1), framing the responder as `answered by ()` distinct from `Primary:` (3.3); renders empty `primaryName` as `unknown`; renders `configChecksum` as hex. Returns 0 on success, `<0` on failure with the error printed (8.2, 8.3). Frees the reply. + +#### `cmd_wait_primary` probe swap — State contract +- Replace the single `joblist_query(...)` probe call in the deadline loop with `xcecli_ctrlstatus`. +- **Success predicate (9.1, 9.2):** the status reply reports a non-`"unknown"` `primaryName`. On success, complete and print the served primary's identity. +- **Preserved (9.4):** the timeout-argument parsing, monotonic deadline, poll interval, log-level quieting, and success/failure exit codes are unchanged; only the probe call and the success message change. +- **Decision & caveat:** the predicate is "a known primary is reported" rather than "the responder is itself the primary," because the failover read may legitimately be answered by a standby; requiring `responderRole == PRIMARY` could starve the predicate if a standby always answers first. The accepted caveat is a brief window where a standby may report a just-failed primary before it clears; this is acceptable for a "wait until a primary exists" gate and is strictly more informative than the prior job-list probe (it names the primary). + +## Data Models + +The only new data is the reply payload above. `CtrlStatus` is a flat value object (no persistence, no aggregate); `CtrlPeer` is a bounded value list (≤ configured controller count, typically 3). `configChecksum` is the 32-byte committed-config SHA-256; `configVersion` is its 8-byte prefix as a `UInt64`. Two nodes with identical committed config content produce identical `configChecksum` and `configVersion`; differing content differs in both (5.3). + +## Error Handling + +- **No reachable controller (7.3):** `xcecli_ctrlstatus` returns the failover wrapper's typed negative (e.g. no-master/timeout); `cmd_ctrlstatus` prints it and returns non-zero (8.3); `wait-primary` keeps polling until its deadline (9.3). +- **Undeterminable field (1.3):** emit the defined "unknown"/empty value, never uninitialized data. +- **Standby with no known primary (2.3):** `primaryName` empty → "unknown"; the call still succeeds (the status itself is valid). +- **Read-only guarantee (1.2):** the handler takes only read locks/atomic loads; no state mutation on any path. + +## Testing Strategy + +### Unit / handler +- `handle_ctrlstatus` on a PRIMARY returns `primaryName==self`, `responderRole==PRIMARY`, populated peers/version/checksum (2.1, 3.x, 4.x, 5.x). +- `handle_ctrlstatus` on a STANDBY with a known primary returns that primary and `responderRole==STANDBY`; with no known primary returns empty `primaryName` (2.2, 2.3). +- `fenced` reflects `in_catch_up`/`recovery_unrecoverable` (6.3); the handler mutates no state (1.2). +- `configChecksum`/`configVersion` equal across two nodes with identical config, differ when content differs (5.3). + +### Integration +- Dispatch-gate: a `ctrlstatus` request is answered (not `-XCE_ERR_STANDBY`/`-XCE_ERR_NOT_READY`) on STANDBY, ELECTING, INIT, and PRIMARY-no-quorum nodes; no other opcode's gating changed (7.1, 7.2). +- libxce round-trip: `xcecli_ctrlstatus` decodes all fields; `is_write=false` rotates across configured controllers; no-reachable-controller returns a typed failure (7.3). + +### E2E (live `/xce-deploy-ha`) +- `ctrlstatus` against a healthy 3-node cluster prints the primary, peers, version+checksum; querying a standby still reports the primary (2.3, 8.1). +- `wait-primary` (status-probe) returns immediately on a healthy cluster and names the primary; after a primary kill, blocks until a new primary is reported, then names it; times out when none (9.1–9.4). Directly useful to the `ha-commit-durability` Phase 6 liveness/>60s-election investigation. + +## Security Considerations + +`ctrlstatus` exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing `clusterconf`/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode. diff --git a/.kiro/specs/controller-status/research.md b/.kiro/specs/controller-status/research.md index b2f24c2..0b381ed 100644 --- a/.kiro/specs/controller-status/research.md +++ b/.kiro/specs/controller-status/research.md @@ -66,3 +66,17 @@ This is Option B in practice: reuse the `clusterconf` *plumbing pattern* (reques - `src/lib/xceapi.c:400` — `xce_ctrlstatus` (`MOVE_PTR` reply union). - `src/lib/client.c:315` — `xcecli_ctrlstatus` (`XCECLI_FAILOVER_CALL`, is_write=false); `src/include/client.h:55,90` decl + free macro. - `src/controller/testsuite/test.c` — `cmd_ctrlstatus` command + registration; swap `cmd_wait_primary` probe at `test.c:1161`. + +--- + +## Design Synthesis (decisions resolved for design.md) + +The 6 carried research items are resolved as: +1. **Reply field set**: expose BOTH `configVersion` (the 8-byte `committed_config_tag`) AND `configChecksum` (the full 32-byte `config_sha256`) — version for quick compare, full digest for strong same-version divergence detection (Req 5.3). +2. **Dispatch-gate semantics**: `ctrlstatus` is allow-listed in ALL role branches (PRIMARY incl. no-quorum, STANDBY, ELECTING, INIT) and bypasses the quorum/serving/fence gate — a status read must answer precisely when the cluster is degraded. Each role returns its local view (PRIMARY→self; STANDBY→its known primary or unknown; ELECTING/INIT→unknown). +3. **Handler placement & shape**: `handle_ctrlstatus` in `reply.c` next to `handle_clusterconf`, following the no-alloc `ha_nop_reply_state_report` model; only the small bounded peer list allocates. `primary_host`/`port` read under `ha->lock`; everything else atomic/immutable. +4. **"unknown" representation**: empty `Text primaryName` on the wire; the client renders it as `"unknown"` (Req 1.3/2.2/2.3). +5. **wait-primary predicate**: success = reply reports a non-"unknown" primary (NOT "responder==PRIMARY", which could starve if a standby answers first under failover rotation). Accepted caveat: a brief standby-stale-primary window; still strictly more informative than the old joblist probe. Preserves all existing wait-primary timeout/exit semantics (Req 9.4). +6. **`committed_lsn` access**: read the persist atomic directly (no accessor exists); optionally add a thin `persist_committed_lsn()` — minor, implementer's choice. + +**Build-vs-adopt**: ADOPT the `clusterconf` opcode-only plumbing pattern + the `state_report` no-alloc handler model; BUILD only the new schema structs, handler body, dispatch-gate entries, and the thin client/test wiring. No new infrastructure; config checksum reused (not built). diff --git a/.kiro/specs/controller-status/spec.json b/.kiro/specs/controller-status/spec.json index ac8c14c..7bd507f 100644 --- a/.kiro/specs/controller-status/spec.json +++ b/.kiro/specs/controller-status/spec.json @@ -1,16 +1,16 @@ { - "feature_name": "controller-lsid", + "feature_name": "controller-status", "created_at": "2026-06-19T00:00:00Z", "updated_at": "2026-06-19T00:00:00Z", "language": "en", - "phase": "initialized", + "phase": "design-generated", "approvals": { "requirements": { - "generated": false, - "approved": false + "generated": true, + "approved": true }, "design": { - "generated": false, + "generated": true, "approved": false }, "tasks": {