lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

spec(controller-status): tasks — schema, handler, dispatch-gate, libxce any-node+authoritative, ctrlstatus cmd, wait-primary swap, E2E (sanity-review PASS)

Browse source
This commit is contained in:
Rongsong Shen 2026-06-20 04:22:28 +00:00
parent ce8c333be0
commit 4b75bfa4db
2 changed files with 72 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

69
.kiro/specs/controller-status/tasks.md Normal file
View file

@ -0,0 +1,69 @@
# Implementation Plan
> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor.
> The work runs along a single request path (schema → controller handler → dispatch gate → libxce client → test command → wait-primary), so tasks are sequential by design; no `(P)` markers. The one HA-critical change is the dispatch-gate allow-list (task 2.2).
- [ ] 1. Foundation: status reply schema + codegen
- [ ] 1.1 Add the `ctrlstatus` opcode and the status reply payload to the schema
- Add a new request opcode `ctrlstatus` (next free ordinal) and a new reply-union member (next free ordinal), plus a lean status payload struct carrying: cluster name, primary name, responder name and role, term, loaded config version and content checksum, committed config version, commit and apply positions, fenced flag, and a bounded list of configured controllers (name, host, port) with a count.
- Keep the payload focused on identity/health only — it must NOT carry the resource/queue/worker inventory.
- Observable: the build regenerates the C types for the new payload and opcode (the generated status type, its reply-union tag, and a free helper exist) and the schema compiles clean.
- _Requirements: 1.1, 1.4, 2.1, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 6.1, 6.2, 6.3, 6.4_
- [ ] 2. Controller: status handler + any-role dispatch
- [ ] 2.1 Implement the controller status handler
- Add a handler that snapshots the responding node's local HA and config state into the status reply: responder name and role; current primary (taken under the HA lock; empty when none is known/elected); term; loaded config version + content checksum; committed config version; commit and apply positions; fenced state (catch-up or unrecoverable-fence); and the configured controllers list with count. It must be read-only (no state mutation) and answer in every role.
- Route the new opcode to this handler in the controller request dispatch switch.
- TDD: unit tests assert the field values for a PRIMARY (primary == self, role == primary, populated peers/version/checksum), a STANDBY with a known primary (reports that primary, role == standby), and a node with no known primary (empty primary → "unknown"); and assert the handler mutates no state.
- Observable: the unit tests pass for the primary, standby, and unknown-primary cases, and a no-mutation assertion holds.
- _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1, 6.2, 6.3, 6.4_
- _Boundary: handle_ctrlstatus_
- [ ] 2.2 Allow the status query through the HA dispatch gate in every role
- Add the status opcode to the dispatch-gate allow-lists for primary (including the no-quorum branch), standby, electing, and initializing — so a status read always answers (including when fenced or without quorum), rather than being rejected as standby/not-ready. Scope the change strictly to this read-only opcode.
- TDD: an integration test confirms the status request is answered (not rejected) on a standby, an electing node, an initializing node, and a primary without quorum; and a guard test confirms no other opcode's gating changed.
- Observable: the gate test shows the status query answered in all four role conditions and unchanged behavior for at least one previously primary-only opcode.
- _Requirements: 7.1, 7.2_
- _Boundary: ha_dispatch_gate_
- _Depends: 2.1_
- [ ] 3. libxce client API
- [ ] 3.1 Add the failover-aware any-node status call
- Add the opcode-only request sender, the send-and-decode call, and the failover-aware client read (retryable, non-write) that returns whatever reachable node answers, plus the free helper. On no reachable controller, return a typed failure without a partial/empty success.
- TDD: a round-trip test decodes every status field from a handler reply; a no-reachable-controller test returns a typed failure.
- Observable: the client decodes all fields from a live/loopback handler reply and surfaces a typed error when no controller is reachable.
- _Requirements: 1.1, 7.1, 7.2, 7.3_
- _Boundary: libxce client_
- _Depends: 1.1, 2.1, 2.2_
- [ ] 3.2 Add the authoritative reach-the-primary status call
- Add a client call that returns only an authoritative reply (responder role == primary). When the first answer is from a non-primary, follow the redirect — target the reported primary via the reply's peer list, otherwise rotate the configured controller list — until the primary answers or the failover budget is exhausted.
- TDD: a test where the first reachable node is a standby confirms the call follows the hint to the primary and returns a primary-role reply; a no-serving-primary test returns failure (never a standby hint as the authoritative result).
- Observable: the authoritative call returns a reply whose responder role is primary even when a standby answered first, and fails (does not return a hint) when no primary serves.
- _Requirements: 2.3, 9.1, 9.2_
- _Boundary: libxce client_
- _Depends: 3.1_
- [ ] 4. Test client command + wait-primary integration
- [ ] 4.1 Add the `ctrlstatus` client command
- Add a client command that issues the any-node status call and prints the fields in human-readable form: the current primary, the answering node framed as "answered by <node> (<role>)" distinct from the primary, the configured controllers and count, the loaded config version and committed config version, the content checksum as hex, term, commit/apply positions, fenced state, and cluster name; render an empty primary as "unknown". The command exit status reflects success or failure of the query.
- Observable: piping `ctrlstatus` through the client prints the status fields (with "answered by" and "unknown" rendering) and exits zero on success / non-zero with the error printed on failure.
- _Requirements: 1.3, 3.3, 5.1, 5.2, 8.1, 8.2, 8.3_
- _Boundary: cmd_ctrlstatus_
- _Depends: 3.1_
- [ ] 4.2 Switch wait-primary to the authoritative status probe
- Replace the wait-primary probe (currently a job-list call) with the authoritative reach-the-primary status call. Success is an authoritative reply from the primary; on success, print the primary's identity. A standby/hint answer is not success — the probe follows the redirect or keeps polling until the primary answers or the deadline elapses. Preserve the existing timeout-argument handling, monotonic deadline, poll interval, log-quieting, and success/failure exit semantics.
- Observable: against a healthy cluster wait-primary returns and names the primary; a malformed timeout still errors; the existing exit-code behavior is unchanged; and it does not succeed from a standby-only answer.
- _Requirements: 9.1, 9.2, 9.3, 9.4_
- _Boundary: cmd_wait_primary_
- _Depends: 3.2_
- [ ] 5. Validation (live cluster)
- [ ] 5.1 End-to-end status + authoritative wait-primary on a live cluster
- Run against a live 3-node cluster: `ctrlstatus` on a healthy cluster shows the correct primary, peers, and version+checksum; querying via a standby still reports the primary with the "answered by <standby>" framing; the version+checksum match across nodes with identical config; and the authoritative wait-primary returns immediately on a healthy cluster, blocks until the NEW primary itself answers after a primary kill (without false-succeeding on a standby's stale primary view), and times out when no primary serves.
- Observable: the live run shows a correct status (primary/peers/checksum) including via a standby, and the authoritative wait-primary names the new primary after a kill without a false success; manual environment prerequisite (live cluster), same status as other live-E2E tasks.
- _Requirements: 2.3, 5.3, 7.1, 7.2, 8.1, 9.1, 9.2, 9.3, 9.4_
- _Depends: 4.1, 4.2_
## Implementation Notes
- All new C types are generated from the schema (task 1.1) — no hand-written struct headers. Build regenerates `controller.capnp.h/.c`.
- Task 2.2 is the one HA-critical change (request-handling path): keep it scoped to the single read-only opcode and prove no other opcode's gating widened.
- The authoritative redirect (3.2) is what makes wait-primary trustworthy (no stale-standby false success) — it is the resolution of the design-review decision; do not collapse it back into a plain any-node read for wait-primary.