From 4b75bfa4dbe139e0a9b8be4dc4105c0e05fd12a2 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 20 Jun 2026 04:22:28 +0000 Subject: [PATCH] =?UTF-8?q?spec(controller-status):=20tasks=20=E2=80=94=20?= =?UTF-8?q?schema,=20handler,=20dispatch-gate,=20libxce=20any-node+authori?= =?UTF-8?q?tative,=20ctrlstatus=20cmd,=20wait-primary=20swap,=20E2E=20(san?= =?UTF-8?q?ity-review=20PASS)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .kiro/specs/controller-status/spec.json | 6 +-- .kiro/specs/controller-status/tasks.md | 69 +++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 3 deletions(-) create mode 100644 .kiro/specs/controller-status/tasks.md diff --git a/.kiro/specs/controller-status/spec.json b/.kiro/specs/controller-status/spec.json index 7bd507f..50e3589 100644 --- a/.kiro/specs/controller-status/spec.json +++ b/.kiro/specs/controller-status/spec.json @@ -3,7 +3,7 @@ "created_at": "2026-06-19T00:00:00Z", "updated_at": "2026-06-19T00:00:00Z", "language": "en", - "phase": "design-generated", + "phase": "tasks-generated", "approvals": { "requirements": { "generated": true, @@ -11,10 +11,10 @@ }, "design": { "generated": true, - "approved": false + "approved": true }, "tasks": { - "generated": false, + "generated": true, "approved": false } }, diff --git a/.kiro/specs/controller-status/tasks.md b/.kiro/specs/controller-status/tasks.md new file mode 100644 index 0000000..8d23dae --- /dev/null +++ b/.kiro/specs/controller-status/tasks.md @@ -0,0 +1,69 @@ +# Implementation Plan + +> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor. +> The work runs along a single request path (schema → controller handler → dispatch gate → libxce client → test command → wait-primary), so tasks are sequential by design; no `(P)` markers. The one HA-critical change is the dispatch-gate allow-list (task 2.2). + +- [ ] 1. Foundation: status reply schema + codegen +- [ ] 1.1 Add the `ctrlstatus` opcode and the status reply payload to the schema + - Add a new request opcode `ctrlstatus` (next free ordinal) and a new reply-union member (next free ordinal), plus a lean status payload struct carrying: cluster name, primary name, responder name and role, term, loaded config version and content checksum, committed config version, commit and apply positions, fenced flag, and a bounded list of configured controllers (name, host, port) with a count. + - Keep the payload focused on identity/health only — it must NOT carry the resource/queue/worker inventory. + - Observable: the build regenerates the C types for the new payload and opcode (the generated status type, its reply-union tag, and a free helper exist) and the schema compiles clean. + - _Requirements: 1.1, 1.4, 2.1, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 6.1, 6.2, 6.3, 6.4_ + +- [ ] 2. Controller: status handler + any-role dispatch +- [ ] 2.1 Implement the controller status handler + - Add a handler that snapshots the responding node's local HA and config state into the status reply: responder name and role; current primary (taken under the HA lock; empty when none is known/elected); term; loaded config version + content checksum; committed config version; commit and apply positions; fenced state (catch-up or unrecoverable-fence); and the configured controllers list with count. It must be read-only (no state mutation) and answer in every role. + - Route the new opcode to this handler in the controller request dispatch switch. + - TDD: unit tests assert the field values for a PRIMARY (primary == self, role == primary, populated peers/version/checksum), a STANDBY with a known primary (reports that primary, role == standby), and a node with no known primary (empty primary → "unknown"); and assert the handler mutates no state. + - Observable: the unit tests pass for the primary, standby, and unknown-primary cases, and a no-mutation assertion holds. + - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1, 6.2, 6.3, 6.4_ + - _Boundary: handle_ctrlstatus_ +- [ ] 2.2 Allow the status query through the HA dispatch gate in every role + - Add the status opcode to the dispatch-gate allow-lists for primary (including the no-quorum branch), standby, electing, and initializing — so a status read always answers (including when fenced or without quorum), rather than being rejected as standby/not-ready. Scope the change strictly to this read-only opcode. + - TDD: an integration test confirms the status request is answered (not rejected) on a standby, an electing node, an initializing node, and a primary without quorum; and a guard test confirms no other opcode's gating changed. + - Observable: the gate test shows the status query answered in all four role conditions and unchanged behavior for at least one previously primary-only opcode. + - _Requirements: 7.1, 7.2_ + - _Boundary: ha_dispatch_gate_ + - _Depends: 2.1_ + +- [ ] 3. libxce client API +- [ ] 3.1 Add the failover-aware any-node status call + - Add the opcode-only request sender, the send-and-decode call, and the failover-aware client read (retryable, non-write) that returns whatever reachable node answers, plus the free helper. On no reachable controller, return a typed failure without a partial/empty success. + - TDD: a round-trip test decodes every status field from a handler reply; a no-reachable-controller test returns a typed failure. + - Observable: the client decodes all fields from a live/loopback handler reply and surfaces a typed error when no controller is reachable. + - _Requirements: 1.1, 7.1, 7.2, 7.3_ + - _Boundary: libxce client_ + - _Depends: 1.1, 2.1, 2.2_ +- [ ] 3.2 Add the authoritative reach-the-primary status call + - Add a client call that returns only an authoritative reply (responder role == primary). When the first answer is from a non-primary, follow the redirect — target the reported primary via the reply's peer list, otherwise rotate the configured controller list — until the primary answers or the failover budget is exhausted. + - TDD: a test where the first reachable node is a standby confirms the call follows the hint to the primary and returns a primary-role reply; a no-serving-primary test returns failure (never a standby hint as the authoritative result). + - Observable: the authoritative call returns a reply whose responder role is primary even when a standby answered first, and fails (does not return a hint) when no primary serves. + - _Requirements: 2.3, 9.1, 9.2_ + - _Boundary: libxce client_ + - _Depends: 3.1_ + +- [ ] 4. Test client command + wait-primary integration +- [ ] 4.1 Add the `ctrlstatus` client command + - Add a client command that issues the any-node status call and prints the fields in human-readable form: the current primary, the answering node framed as "answered by ()" distinct from the primary, the configured controllers and count, the loaded config version and committed config version, the content checksum as hex, term, commit/apply positions, fenced state, and cluster name; render an empty primary as "unknown". The command exit status reflects success or failure of the query. + - Observable: piping `ctrlstatus` through the client prints the status fields (with "answered by" and "unknown" rendering) and exits zero on success / non-zero with the error printed on failure. + - _Requirements: 1.3, 3.3, 5.1, 5.2, 8.1, 8.2, 8.3_ + - _Boundary: cmd_ctrlstatus_ + - _Depends: 3.1_ +- [ ] 4.2 Switch wait-primary to the authoritative status probe + - Replace the wait-primary probe (currently a job-list call) with the authoritative reach-the-primary status call. Success is an authoritative reply from the primary; on success, print the primary's identity. A standby/hint answer is not success — the probe follows the redirect or keeps polling until the primary answers or the deadline elapses. Preserve the existing timeout-argument handling, monotonic deadline, poll interval, log-quieting, and success/failure exit semantics. + - Observable: against a healthy cluster wait-primary returns and names the primary; a malformed timeout still errors; the existing exit-code behavior is unchanged; and it does not succeed from a standby-only answer. + - _Requirements: 9.1, 9.2, 9.3, 9.4_ + - _Boundary: cmd_wait_primary_ + - _Depends: 3.2_ + +- [ ] 5. Validation (live cluster) +- [ ] 5.1 End-to-end status + authoritative wait-primary on a live cluster + - Run against a live 3-node cluster: `ctrlstatus` on a healthy cluster shows the correct primary, peers, and version+checksum; querying via a standby still reports the primary with the "answered by " framing; the version+checksum match across nodes with identical config; and the authoritative wait-primary returns immediately on a healthy cluster, blocks until the NEW primary itself answers after a primary kill (without false-succeeding on a standby's stale primary view), and times out when no primary serves. + - Observable: the live run shows a correct status (primary/peers/checksum) including via a standby, and the authoritative wait-primary names the new primary after a kill without a false success; manual environment prerequisite (live cluster), same status as other live-E2E tasks. + - _Requirements: 2.3, 5.3, 7.1, 7.2, 8.1, 9.1, 9.2, 9.3, 9.4_ + - _Depends: 4.1, 4.2_ + +## Implementation Notes +- All new C types are generated from the schema (task 1.1) — no hand-written struct headers. Build regenerates `controller.capnp.h/.c`. +- Task 2.2 is the one HA-critical change (request-handling path): keep it scoped to the single read-only opcode and prove no other opcode's gating widened. +- The authoritative redirect (3.2) is what makes wait-primary trustworthy (no stale-standby false success) — it is the resolution of the design-review decision; do not collapse it back into a plain any-node read for wait-primary.