spec(controller-status): tasks — schema, handler, dispatch-gate, libxce any-node+authoritative, ctrlstatus cmd, wait-primary swap, E2E (sanity-review PASS)

.kiro/specs/controller-status/spec.json

@@ -3,7 +3,7 @@
   "created_at": "2026-06-19T00:00:00Z",
   "updated_at": "2026-06-19T00:00:00Z",
   "language": "en",
-  "phase": "design-generated",
+  "phase": "tasks-generated",
   "approvals": {
     "requirements": {
       "generated": true,
@@ -11,10 +11,10 @@
     },
     "design": {
       "generated": true,
-      "approved": false
+      "approved": true
     },
     "tasks": {
-      "generated": false,
+      "generated": true,
       "approved": false
     }
   },

.kiro/specs/controller-status/tasks.md

@@ -0,0 +1,69 @@
+# Implementation Plan
+
+> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor.
+> The work runs along a single request path (schema → controller handler → dispatch gate → libxce client → test command → wait-primary), so tasks are sequential by design; no `(P)` markers. The one HA-critical change is the dispatch-gate allow-list (task 2.2).
+
+- [ ] 1. Foundation: status reply schema + codegen
+- [ ] 1.1 Add the `ctrlstatus` opcode and the status reply payload to the schema
+  - Add a new request opcode `ctrlstatus` (next free ordinal) and a new reply-union member (next free ordinal), plus a lean status payload struct carrying: cluster name, primary name, responder name and role, term, loaded config version and content checksum, committed config version, commit and apply positions, fenced flag, and a bounded list of configured controllers (name, host, port) with a count.
+  - Keep the payload focused on identity/health only — it must NOT carry the resource/queue/worker inventory.
+  - Observable: the build regenerates the C types for the new payload and opcode (the generated status type, its reply-union tag, and a free helper exist) and the schema compiles clean.
+  - _Requirements: 1.1, 1.4, 2.1, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 6.1, 6.2, 6.3, 6.4_
+
+- [ ] 2. Controller: status handler + any-role dispatch
+- [ ] 2.1 Implement the controller status handler
+  - Add a handler that snapshots the responding node's local HA and config state into the status reply: responder name and role; current primary (taken under the HA lock; empty when none is known/elected); term; loaded config version + content checksum; committed config version; commit and apply positions; fenced state (catch-up or unrecoverable-fence); and the configured controllers list with count. It must be read-only (no state mutation) and answer in every role.
+  - Route the new opcode to this handler in the controller request dispatch switch.
+  - TDD: unit tests assert the field values for a PRIMARY (primary == self, role == primary, populated peers/version/checksum), a STANDBY with a known primary (reports that primary, role == standby), and a node with no known primary (empty primary → "unknown"); and assert the handler mutates no state.
+  - Observable: the unit tests pass for the primary, standby, and unknown-primary cases, and a no-mutation assertion holds.
+  - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1, 6.2, 6.3, 6.4_
+  - _Boundary: handle_ctrlstatus_
+- [ ] 2.2 Allow the status query through the HA dispatch gate in every role
+  - Add the status opcode to the dispatch-gate allow-lists for primary (including the no-quorum branch), standby, electing, and initializing — so a status read always answers (including when fenced or without quorum), rather than being rejected as standby/not-ready. Scope the change strictly to this read-only opcode.
+  - TDD: an integration test confirms the status request is answered (not rejected) on a standby, an electing node, an initializing node, and a primary without quorum; and a guard test confirms no other opcode's gating changed.
+  - Observable: the gate test shows the status query answered in all four role conditions and unchanged behavior for at least one previously primary-only opcode.
+  - _Requirements: 7.1, 7.2_
+  - _Boundary: ha_dispatch_gate_
+  - _Depends: 2.1_
+
+- [ ] 3. libxce client API
+- [ ] 3.1 Add the failover-aware any-node status call
+  - Add the opcode-only request sender, the send-and-decode call, and the failover-aware client read (retryable, non-write) that returns whatever reachable node answers, plus the free helper. On no reachable controller, return a typed failure without a partial/empty success.
+  - TDD: a round-trip test decodes every status field from a handler reply; a no-reachable-controller test returns a typed failure.
+  - Observable: the client decodes all fields from a live/loopback handler reply and surfaces a typed error when no controller is reachable.
+  - _Requirements: 1.1, 7.1, 7.2, 7.3_
+  - _Boundary: libxce client_
+  - _Depends: 1.1, 2.1, 2.2_
+- [ ] 3.2 Add the authoritative reach-the-primary status call
+  - Add a client call that returns only an authoritative reply (responder role == primary). When the first answer is from a non-primary, follow the redirect — target the reported primary via the reply's peer list, otherwise rotate the configured controller list — until the primary answers or the failover budget is exhausted.
+  - TDD: a test where the first reachable node is a standby confirms the call follows the hint to the primary and returns a primary-role reply; a no-serving-primary test returns failure (never a standby hint as the authoritative result).
+  - Observable: the authoritative call returns a reply whose responder role is primary even when a standby answered first, and fails (does not return a hint) when no primary serves.
+  - _Requirements: 2.3, 9.1, 9.2_
+  - _Boundary: libxce client_
+  - _Depends: 3.1_
+
+- [ ] 4. Test client command + wait-primary integration
+- [ ] 4.1 Add the `ctrlstatus` client command
+  - Add a client command that issues the any-node status call and prints the fields in human-readable form: the current primary, the answering node framed as "answered by <node> (<role>)" distinct from the primary, the configured controllers and count, the loaded config version and committed config version, the content checksum as hex, term, commit/apply positions, fenced state, and cluster name; render an empty primary as "unknown". The command exit status reflects success or failure of the query.
+  - Observable: piping `ctrlstatus` through the client prints the status fields (with "answered by" and "unknown" rendering) and exits zero on success / non-zero with the error printed on failure.
+  - _Requirements: 1.3, 3.3, 5.1, 5.2, 8.1, 8.2, 8.3_
+  - _Boundary: cmd_ctrlstatus_
+  - _Depends: 3.1_
+- [ ] 4.2 Switch wait-primary to the authoritative status probe
+  - Replace the wait-primary probe (currently a job-list call) with the authoritative reach-the-primary status call. Success is an authoritative reply from the primary; on success, print the primary's identity. A standby/hint answer is not success — the probe follows the redirect or keeps polling until the primary answers or the deadline elapses. Preserve the existing timeout-argument handling, monotonic deadline, poll interval, log-quieting, and success/failure exit semantics.
+  - Observable: against a healthy cluster wait-primary returns and names the primary; a malformed timeout still errors; the existing exit-code behavior is unchanged; and it does not succeed from a standby-only answer.
+  - _Requirements: 9.1, 9.2, 9.3, 9.4_
+  - _Boundary: cmd_wait_primary_
+  - _Depends: 3.2_
+
+- [ ] 5. Validation (live cluster)
+- [ ] 5.1 End-to-end status + authoritative wait-primary on a live cluster
+  - Run against a live 3-node cluster: `ctrlstatus` on a healthy cluster shows the correct primary, peers, and version+checksum; querying via a standby still reports the primary with the "answered by <standby>" framing; the version+checksum match across nodes with identical config; and the authoritative wait-primary returns immediately on a healthy cluster, blocks until the NEW primary itself answers after a primary kill (without false-succeeding on a standby's stale primary view), and times out when no primary serves.
+  - Observable: the live run shows a correct status (primary/peers/checksum) including via a standby, and the authoritative wait-primary names the new primary after a kill without a false success; manual environment prerequisite (live cluster), same status as other live-E2E tasks.
+  - _Requirements: 2.3, 5.3, 7.1, 7.2, 8.1, 9.1, 9.2, 9.3, 9.4_
+  - _Depends: 4.1, 4.2_
+
+## Implementation Notes
+- All new C types are generated from the schema (task 1.1) — no hand-written struct headers. Build regenerates `controller.capnp.h/.c`.
+- Task 2.2 is the one HA-critical change (request-handling path): keep it scoped to the single read-only opcode and prove no other opcode's gating widened.
+- The authoritative redirect (3.2) is what makes wait-primary trustworthy (no stale-standby false success) — it is the resolution of the design-review decision; do not collapse it back into a plain any-node read for wait-primary.