myxce/.kiro/specs/controller-status/tasks.md

Implementation Plan

TDD is mandatory (project rule): add the failing C test in testsuite/ (or a piped xce_test behavioral script) before the change — red → green → refactor. The work runs along a single request path (schema → controller handler → dispatch gate → libxce client → test command → wait-primary), so tasks are sequential by design; no (P) markers. The one HA-critical change is the dispatch-gate allow-list (task 2.2).

• 1. Foundation: status reply schema + codegen

• 1.1 Add the ctrlstatus opcode and the status reply payload to the schema

  • Add a new request opcode ctrlstatus (next free ordinal) and a new reply-union member (next free ordinal), plus a lean status payload struct carrying: cluster name, primary name, responder name and role, term, loaded config version and content checksum, committed config version, commit and apply positions, fenced flag, and a bounded list of configured controllers (name, host, port) with a count.
  • Keep the payload focused on identity/health only — it must NOT carry the resource/queue/worker inventory.
  • Observable: the build regenerates the C types for the new payload and opcode (the generated status type, its reply-union tag, and a free helper exist) and the schema compiles clean.
  • Requirements: 1.1, 1.4, 2.1, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 6.1, 6.2, 6.3, 6.4

• 2. Controller: status handler + any-role dispatch

• 2.1 Implement the controller status handler

  • Add a handler that snapshots the responding node's local HA and config state into the status reply: responder name and role; current primary (taken under the HA lock; empty when none is known/elected); term; loaded config version + content checksum; committed config version; commit and apply positions; fenced state (catch-up or unrecoverable-fence); and the configured controllers list with count. It must be read-only (no state mutation) and answer in every role.
  • Route the new opcode to this handler in the controller request dispatch switch.
  • TDD: unit tests assert the field values for a PRIMARY (primary == self, role == primary, populated peers/version/checksum), a STANDBY with a known primary (reports that primary, role == standby), and a node with no known primary (empty primary → "unknown"); and assert the handler mutates no state.
  • Observable: the unit tests pass for the primary, standby, and unknown-primary cases, and a no-mutation assertion holds.
  • Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1, 6.2, 6.3, 6.4
  • Boundary: handle_ctrlstatus

• 2.2 Allow the status query through the HA dispatch gate in every role

  • Add the status opcode to the dispatch-gate allow-lists for primary (including the no-quorum branch), standby, electing, and initializing — so a status read always answers (including when fenced or without quorum), rather than being rejected as standby/not-ready. Scope the change strictly to this read-only opcode.
  • TDD: an integration test confirms the status request is answered (not rejected) on a standby, an electing node, an initializing node, and a primary without quorum; and a guard test confirms no other opcode's gating changed.
  • Observable: the gate test shows the status query answered in all four role conditions and unchanged behavior for at least one previously primary-only opcode.
  • Requirements: 7.1, 7.2
  • Boundary: ha_dispatch_gate
  • Depends: 2.1

• 3. libxce client API

• 3.1 Add the failover-aware any-node status call

  • Add the opcode-only request sender, the send-and-decode call, and the failover-aware client read (retryable, non-write) that returns whatever reachable node answers, plus the free helper. On no reachable controller, return a typed failure without a partial/empty success.
  • TDD: a round-trip test decodes every status field from a handler reply; a no-reachable-controller test returns a typed failure.
  • Observable: the client decodes all fields from a live/loopback handler reply and surfaces a typed error when no controller is reachable.
  • Requirements: 1.1, 7.1, 7.2, 7.3
  • Boundary: libxce client
  • Depends: 1.1, 2.1, 2.2

• 3.2 Add the authoritative reach-the-primary status call

  • Add a client call that returns only an authoritative reply (responder role == primary). When the first answer is from a non-primary, follow the redirect — target the reported primary via the reply's peer list, otherwise rotate the configured controller list — until the primary answers or the failover budget is exhausted.
  • TDD: a test where the first reachable node is a standby confirms the call follows the hint to the primary and returns a primary-role reply; a no-serving-primary test returns failure (never a standby hint as the authoritative result).
  • Observable: the authoritative call returns a reply whose responder role is primary even when a standby answered first, and fails (does not return a hint) when no primary serves.
  • Requirements: 2.3, 9.1, 9.2
  • Boundary: libxce client
  • Depends: 3.1

• 4. Test client command + wait-primary integration

• 4.1 Add the ctrlstatus client command

  • Add a client command that issues the any-node status call and prints the fields in human-readable form: the current primary, the answering node framed as "answered by ()" distinct from the primary, the configured controllers and count, the loaded config version and committed config version, the content checksum as hex, term, commit/apply positions, fenced state, and cluster name; render an empty primary as "unknown". The command exit status reflects success or failure of the query.
  • Observable: piping ctrlstatus through the client prints the status fields (with "answered by" and "unknown" rendering) and exits zero on success / non-zero with the error printed on failure.
  • Requirements: 1.3, 3.3, 5.1, 5.2, 8.1, 8.2, 8.3
  • Boundary: cmd_ctrlstatus
  • Depends: 3.1

• 4.2 Switch wait-primary to the authoritative status probe

  • Replace the wait-primary probe (currently a job-list call) with the authoritative reach-the-primary status call. Success is an authoritative reply from the primary; on success, print the primary's identity. A standby/hint answer is not success — the probe follows the redirect or keeps polling until the primary answers or the deadline elapses. Preserve the existing timeout-argument handling, monotonic deadline, poll interval, log-quieting, and success/failure exit semantics.
  • Observable: against a healthy cluster wait-primary returns and names the primary; a malformed timeout still errors; the existing exit-code behavior is unchanged; and it does not succeed from a standby-only answer.
  • Requirements: 9.1, 9.2, 9.3, 9.4
  • Boundary: cmd_wait_primary
  • Depends: 3.2

• 5. Validation (live cluster)

• 5.1 End-to-end status + authoritative wait-primary on a live cluster

  • Run against a live 3-node cluster: ctrlstatus on a healthy cluster shows the correct primary, peers, and version+checksum; querying via a standby still reports the primary with the "answered by " framing; the version+checksum match across nodes with identical config; and the authoritative wait-primary returns immediately on a healthy cluster, blocks until the NEW primary itself answers after a primary kill (without false-succeeding on a standby's stale primary view), and times out when no primary serves.
  • Observable: the live run shows a correct status (primary/peers/checksum) including via a standby, and the authoritative wait-primary names the new primary after a kill without a false success; manual environment prerequisite (live cluster), same status as other live-E2E tasks.
  • Requirements: 2.3, 5.3, 7.1, 7.2, 8.1, 9.1, 9.2, 9.3, 9.4
  • Depends: 4.1, 4.2

Implementation Notes

• All new C types are generated from the schema (task 1.1) — no hand-written struct headers. Build regenerates controller.capnp.h/.c.
• Task 2.2 is the one HA-critical change (request-handling path): keep it scoped to the single read-only opcode and prove no other opcode's gating widened.
• The authoritative redirect (3.2) is what makes wait-primary trustworthy (no stale-standby false success) — it is the resolution of the design-review decision; do not collapse it back into a plain any-node read for wait-primary.
• 1.1 LEARNING (affects later tasks): the $C.mapname(...) capnp types are NOT auto-generated in this schema — the generated controller.capnp.c references HAND-AUTHORED C typedefs in src/include/typemap.h (same convention as xce_clusterconf_t). So xce_ctrlstatus_t/xce_ctrlpeer_t (and the xce_reply_t.data.ctrlstatus union member) live in typemap.h, verified field-for-field against the regenerated codec. OPCode_ctrlstatus = 40, RequestReply_data_ctrlstatus = 12. configChecksum is xce_data_t*. The handler/libxce tasks build/encode against these typemap.h types.
• 2.1 LEARNING + SCHEMA AMENDMENT: capnp Data in THIS schema maps to the project's STRUCTURED xce_data_t union (table-switched), NOT raw bytes — encode_Data_ptr can't serialize a SHA-256 blob. So configChecksum @6 was changed Data → Text (a 64-char lowercase-hex string of the full 32-byte config_sha256); typemap member is char*; freed in ctrlstatus_free_fields. Ordinal @6 unchanged, no released decoder yet → safe. Handler lives in NEW src/controller/ctrlstatus.c (ctrlstatus_fill/ctrlstatus_free_fields/handle_ctrlstatus), proto in src/controller/include/cluster.h, routed at cluster.c opcode switch, test src/controller/testsuite/test_ctrlstatus.c (registered in controller_test.c, built via controller/meson.build). primary_host read under ha->lock; peer name/host strings are BORROWED from myconf (not freed). The libxce client (3.x) decodes configChecksum as Text.
• 3.1 LEARNING: the ctx-side reply union xce_ctx_t.reply.u (in src/include/xcectx.h) is SEPARATE from xce_reply_t.data (typemap.h) — the decode path needs BOTH a ctrlstatus member there AND a case OPCode_ctrlstatus in xcectx.c handle_reply (mirror clusterconf). Free: clusterconf's xce_free_clusterconf macro is a LATENT-BROKEN alias (undefined symbol, never expanded); use the REAL generated free — xce_free_ctrlstatus(s) → free_CtrlStatus(s), xcecli_free_ctrlstatus(s) → xce_free_ctrlstatus(s) (any TU expanding it must include the generated controller.capnp.h). xcecli_ctrlstatus is is_write=false; NULL ctx → -EINVAL.
• 3.2: xcecli_ctrlstatus_primary = bounded loop (cap failover.retry.max_retries) reusing xcecli_ctrlstatus + xce_ctx_capture_redirect_hint (map primaryName→addr via peers[], else master rotation); authoritative iff responderRole == CTRL_PRIMARY (== 0; ctrl_role_t PRIMARY=0/STANDBY=1/INIT=2/ELECTING=3 in ha.h — REVALIDATION TRIGGER: a role-enum reorder breaks the pinned wire ordinal XCECLI_CTRLSTATUS_ROLE_PRIMARY=0). On exhaustion → -E_XCE_NOT_READY, *status untouched (never a hint). Pure helpers xcecli_ctrlstatus_is_authoritative/_primary_addr unit-tested; live standby→primary redirect is 5.1.