myxce/.kiro/specs/controller-status/tasks.md

Implementation Plan

TDD is mandatory (project rule): add the failing C test in testsuite/ (or a piped xce_test behavioral script) before the change — red → green → refactor. The work runs along a single request path (schema → controller handler → dispatch gate → libxce client → test command → wait-primary), so tasks are sequential by design; no (P) markers. The one HA-critical change is the dispatch-gate allow-list (task 2.2).

• 1. Foundation: status reply schema + codegen

• 1.1 Add the ctrlstatus opcode and the status reply payload to the schema

  • Add a new request opcode ctrlstatus (next free ordinal) and a new reply-union member (next free ordinal), plus a lean status payload struct carrying: cluster name, primary name, responder name and role, term, loaded config version and content checksum, committed config version, commit and apply positions, fenced flag, and a bounded list of configured controllers (name, host, port) with a count.
  • Keep the payload focused on identity/health only — it must NOT carry the resource/queue/worker inventory.
  • Observable: the build regenerates the C types for the new payload and opcode (the generated status type, its reply-union tag, and a free helper exist) and the schema compiles clean.
  • Requirements: 1.1, 1.4, 2.1, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 6.1, 6.2, 6.3, 6.4

• 2. Controller: status handler + any-role dispatch

• 2.1 Implement the controller status handler

  • Add a handler that snapshots the responding node's local HA and config state into the status reply: responder name and role; current primary (taken under the HA lock; empty when none is known/elected); term; loaded config version + content checksum; committed config version; commit and apply positions; fenced state (catch-up or unrecoverable-fence); and the configured controllers list with count. It must be read-only (no state mutation) and answer in every role.
  • Route the new opcode to this handler in the controller request dispatch switch.
  • TDD: unit tests assert the field values for a PRIMARY (primary == self, role == primary, populated peers/version/checksum), a STANDBY with a known primary (reports that primary, role == standby), and a node with no known primary (empty primary → "unknown"); and assert the handler mutates no state.
  • Observable: the unit tests pass for the primary, standby, and unknown-primary cases, and a no-mutation assertion holds.
  • Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1, 6.2, 6.3, 6.4
  • Boundary: handle_ctrlstatus

• 2.2 Allow the status query through the HA dispatch gate in every role

  • Add the status opcode to the dispatch-gate allow-lists for primary (including the no-quorum branch), standby, electing, and initializing — so a status read always answers (including when fenced or without quorum), rather than being rejected as standby/not-ready. Scope the change strictly to this read-only opcode.
  • TDD: an integration test confirms the status request is answered (not rejected) on a standby, an electing node, an initializing node, and a primary without quorum; and a guard test confirms no other opcode's gating changed.
  • Observable: the gate test shows the status query answered in all four role conditions and unchanged behavior for at least one previously primary-only opcode.
  • Requirements: 7.1, 7.2
  • Boundary: ha_dispatch_gate
  • Depends: 2.1

• 3. libxce client API

• 3.1 Add the failover-aware any-node status call

  • Add the opcode-only request sender, the send-and-decode call, and the failover-aware client read (retryable, non-write) that returns whatever reachable node answers, plus the free helper. On no reachable controller, return a typed failure without a partial/empty success.
  • TDD: a round-trip test decodes every status field from a handler reply; a no-reachable-controller test returns a typed failure.
  • Observable: the client decodes all fields from a live/loopback handler reply and surfaces a typed error when no controller is reachable.
  • Requirements: 1.1, 7.1, 7.2, 7.3
  • Boundary: libxce client
  • Depends: 1.1, 2.1, 2.2

• 3.2 Add the authoritative reach-the-primary status call

  • Add a client call that returns only an authoritative reply (responder role == primary). When the first answer is from a non-primary, follow the redirect — target the reported primary via the reply's peer list, otherwise rotate the configured controller list — until the primary answers or the failover budget is exhausted.
  • TDD: a test where the first reachable node is a standby confirms the call follows the hint to the primary and returns a primary-role reply; a no-serving-primary test returns failure (never a standby hint as the authoritative result).
  • Observable: the authoritative call returns a reply whose responder role is primary even when a standby answered first, and fails (does not return a hint) when no primary serves.
  • Requirements: 2.3, 9.1, 9.2
  • Boundary: libxce client
  • Depends: 3.1

• 4. Test client command + wait-primary integration

• 4.1 Add the ctrlstatus client command

  • Add a client command that issues the any-node status call and prints the fields in human-readable form: the current primary, the answering node framed as "answered by ()" distinct from the primary, the configured controllers and count, the loaded config version and committed config version, the content checksum as hex, term, commit/apply positions, fenced state, and cluster name; render an empty primary as "unknown". The command exit status reflects success or failure of the query.
  • Observable: piping ctrlstatus through the client prints the status fields (with "answered by" and "unknown" rendering) and exits zero on success / non-zero with the error printed on failure.
  • Requirements: 1.3, 3.3, 5.1, 5.2, 8.1, 8.2, 8.3
  • Boundary: cmd_ctrlstatus
  • Depends: 3.1

• 4.2 Switch wait-primary to the authoritative status probe

  • Replace the wait-primary probe (currently a job-list call) with the authoritative reach-the-primary status call. Success is an authoritative reply from the primary; on success, print the primary's identity. A standby/hint answer is not success — the probe follows the redirect or keeps polling until the primary answers or the deadline elapses. Preserve the existing timeout-argument handling, monotonic deadline, poll interval, log-quieting, and success/failure exit semantics.
  • Observable: against a healthy cluster wait-primary returns and names the primary; a malformed timeout still errors; the existing exit-code behavior is unchanged; and it does not succeed from a standby-only answer.
  • Requirements: 9.1, 9.2, 9.3, 9.4
  • Boundary: cmd_wait_primary
  • Depends: 3.2

• 5. Validation (live cluster)

• 5.1 End-to-end status + authoritative wait-primary on a live cluster

  • Run against a live 3-node cluster: ctrlstatus on a healthy cluster shows the correct primary, peers, and version+checksum; querying via a standby still reports the primary with the "answered by " framing; the version+checksum match across nodes with identical config; and the authoritative wait-primary returns immediately on a healthy cluster, blocks until the NEW primary itself answers after a primary kill (without false-succeeding on a standby's stale primary view), and times out when no primary serves.
  • Observable: the live run shows a correct status (primary/peers/checksum) including via a standby, and the authoritative wait-primary names the new primary after a kill without a false success; manual environment prerequisite (live cluster), same status as other live-E2E tasks.
  • Requirements: 2.3, 5.3, 7.1, 7.2, 8.1, 9.1, 9.2, 9.3, 9.4
  • Depends: 4.1, 4.2

• 6. Delta: off-primary committed fidelity + responder timestamp (Req 10, 11)

  Sequential, single request path (schema → typemap → ctrlstatus_fill → render → E2E); no (P). Read-only reporting only — the live-fresh-committed heartbeat change is OUT of boundary (route to ha-heartbeat-liveness / ha-repl-hardening).

• 6.1 Add the responder snapshot timestamp field to the schema + hand-authored typemap

  • Append a responder snapshot-time field to the status payload at the next free ordinal (after the peer count), carrying the responder's wall-clock at sample time as epoch milliseconds (append-only — never reorder existing field ordinals). Add the matching field to the hand-authored xce_ctrlstatus_t typedef in src/include/typemap.h and verify it field-for-field against the regenerated codec (per the 1.1/2.1 learning: $C.mapname types are NOT auto-generated here).
  • Observable: the build regenerates controller.capnp.h/.c clean, the typemap matches the codec, and a codec round-trip carries the new field's value unchanged.
  • Requirements: 11.1
  • Boundary: schema + typemap

• 6.2 Source off-primary committed values + stamp the timestamp in ctrlstatus_fill

  • TDD (red first, in test_ctrlstatus.c): a STANDBY whose learned committed position is non-zero while its local committed marker is zero and it has applied entries reports the LEARNED committed LSN and committed config version (not zero) and the real applied position (no applied-greater-than-committed inversion); a STANDBY that has not yet learned any committed position reports committed LSN and committed config version as zero/unknown; a PRIMARY reports its own authoritative local committed LSN and committed config tag (unchanged); and the responder snapshot timestamp is non-zero and within tolerance of a test-captured wall-clock sample. Assert the handler still mutates no state.
  • Change ctrlstatus_fill (src/controller/ctrlstatus.c): role-branch the committed sourcing — when the responder is primary, keep its own committed LSN / committed config tag; otherwise report the learned committed LSN / learned committed config tag. Stamp the responder snapshot timestamp from the wall clock (CLOCK_REALTIME, NOT the monotonic clock) at snapshot time. All additions are atomic/aligned loads + a clock read — no state mutation.
  • Observable: the new unit tests go red → green; the standby case reports the learned non-zero committed value instead of zero; the no-mutation assertion holds.
  • Requirements: 10.1, 10.2, 10.3, 10.4, 11.1
  • Boundary: ctrlstatus_fill
  • Depends: 6.1

• 6.3 Render the responder snapshot timestamp in the ctrlstatus client command

  • Add one printed line to cmd_ctrlstatus (src/controller/testsuite/test.c) rendering the responder snapshot timestamp as a local YYYY-MM-DD HH:MM:SS,mmm string in the same form the controller logger uses, so it can be compared directly against ctrl-N.log. Confirm and reuse (or match) the exact wall-clock formatter in src/include/xcelog.h. The committed lines need no rendering change — task 6.2 only changed their values.
  • Observable: piping ctrlstatus through the client prints a responder-timestamp line in the log timestamp format alongside the other status fields.
  • Requirements: 11.2, 11.3
  • Boundary: cmd_ctrlstatus
  • Depends: 6.1, 6.2

• 6.4 End-to-end validation on a live cluster (off-primary committed + log correlation)

  • Run against a live 3-node cluster: submit jobs so the primary advances its committed position, then query a STANDBY. Assert the standby's reported committed LSN is non-zero (tracking the primary within the documented join-refresh caveat) rather than zero while it has applied entries, and that the printed responder timestamp matches that standby's own ctrl-N.log line for the same query.
  • Observable: the live run shows a non-zero off-primary committed LSN (no zero-while-applied inversion) and a responder timestamp that lines up with the standby's log; manual environment prerequisite (live cluster), same status as the other live-E2E task (5.1).
  • Requirements: 10.1, 10.3, 11.2, 11.3
  • Depends: 6.2, 6.3

Implementation Notes (5.1 E2E outcome)

• 5.1 ran live on a 3-node /xce-deploy-ha-style cluster (ext2/3 fs). PASSED in-scope: ctrlstatus shows the primary self-reporting (Primary: ctrl-1 on the primary), standbys honestly report Primary: unknown with answered by ctrl-2 (standby) (the dispatch-gate lets standbys answer — Req 7); peers + count correct; configChecksum = the real 64-hex SHA-256 (matched the deploy tag prefix) and is IDENTICAL across all 3 nodes with identical config (Req 5.3); configVersion==committedConfigVersion; authoritative wait-primary names the primary in <1s on a serveable cluster (Req 9.1/9.2).
• 5.1 FOUND + FIXED a real 2.1 bug: ha->primary_host is VESTIGIAL (never written anywhere in the controller), and voted_for_name is the last-vote (often self), so neither names the current primary off-primary. Corrected: PRIMARY self-reports its own name; STANDBY/ELECTING/INIT report empty → "unknown" (the authoritative client path reaches the real primary by rotation). Commit ade0081.
• 5.1 UPSTREAM FINDING (out of boundary, NOT controller-status): with a DEAD node FIRST in the masters list, the libxce client failover returns EINVAL instead of advancing to the live nodes — so the authoritative wait-primary (and ANY client op) can't reach the primary while a dead node leads the list. CONFIRMED pre-existing: plain joblist fails IDENTICALLY (xce_joblist: Invalid argument) in the same condition; with a live-only masters list wait-primary succeeds in 46ms. This spec consumes failover unchanged (per Boundary/Adjacent expectations) → route to libxce-failover, do NOT patch here. The durability soak avoided it by restarting killed nodes (no dead-first-master during waits).

Implementation Notes

• All new C types are generated from the schema (task 1.1) — no hand-written struct headers. Build regenerates controller.capnp.h/.c.
• Task 2.2 is the one HA-critical change (request-handling path): keep it scoped to the single read-only opcode and prove no other opcode's gating widened.
• The authoritative redirect (3.2) is what makes wait-primary trustworthy (no stale-standby false success) — it is the resolution of the design-review decision; do not collapse it back into a plain any-node read for wait-primary.
• 1.1 LEARNING (affects later tasks): the $C.mapname(...) capnp types are NOT auto-generated in this schema — the generated controller.capnp.c references HAND-AUTHORED C typedefs in src/include/typemap.h (same convention as xce_clusterconf_t). So xce_ctrlstatus_t/xce_ctrlpeer_t (and the xce_reply_t.data.ctrlstatus union member) live in typemap.h, verified field-for-field against the regenerated codec. OPCode_ctrlstatus = 40, RequestReply_data_ctrlstatus = 12. configChecksum is xce_data_t*. The handler/libxce tasks build/encode against these typemap.h types.
• 2.1 LEARNING + SCHEMA AMENDMENT: capnp Data in THIS schema maps to the project's STRUCTURED xce_data_t union (table-switched), NOT raw bytes — encode_Data_ptr can't serialize a SHA-256 blob. So configChecksum @6 was changed Data → Text (a 64-char lowercase-hex string of the full 32-byte config_sha256); typemap member is char*; freed in ctrlstatus_free_fields. Ordinal @6 unchanged, no released decoder yet → safe. Handler lives in NEW src/controller/ctrlstatus.c (ctrlstatus_fill/ctrlstatus_free_fields/handle_ctrlstatus), proto in src/controller/include/cluster.h, routed at cluster.c opcode switch, test src/controller/testsuite/test_ctrlstatus.c (registered in controller_test.c, built via controller/meson.build). primary_host read under ha->lock; peer name/host strings are BORROWED from myconf (not freed). The libxce client (3.x) decodes configChecksum as Text.
• 3.1 LEARNING: the ctx-side reply union xce_ctx_t.reply.u (in src/include/xcectx.h) is SEPARATE from xce_reply_t.data (typemap.h) — the decode path needs BOTH a ctrlstatus member there AND a case OPCode_ctrlstatus in xcectx.c handle_reply (mirror clusterconf). Free: clusterconf's xce_free_clusterconf macro is a LATENT-BROKEN alias (undefined symbol, never expanded); use the REAL generated free — xce_free_ctrlstatus(s) → free_CtrlStatus(s), xcecli_free_ctrlstatus(s) → xce_free_ctrlstatus(s) (any TU expanding it must include the generated controller.capnp.h). xcecli_ctrlstatus is is_write=false; NULL ctx → -EINVAL.
• 3.2: xcecli_ctrlstatus_primary = bounded loop (cap failover.retry.max_retries) reusing xcecli_ctrlstatus + xce_ctx_capture_redirect_hint (map primaryName→addr via peers[], else master rotation); authoritative iff responderRole == CTRL_PRIMARY (== 0; ctrl_role_t PRIMARY=0/STANDBY=1/INIT=2/ELECTING=3 in ha.h — REVALIDATION TRIGGER: a role-enum reorder breaks the pinned wire ordinal XCECLI_CTRLSTATUS_ROLE_PRIMARY=0). On exhaustion → -E_XCE_NOT_READY, *status untouched (never a hint). Pure helpers xcecli_ctrlstatus_is_authoritative/_primary_addr unit-tested; live standby→primary redirect is 5.1.
• 6.1: responderTimeMs @13: UInt64 appended to CtrlStatus (after peerCount @12, append-only) + matching uint64_t responder_time_ms in the hand-authored xce_ctrlstatus_t (src/include/typemap.h); codec wire offset 48, round-trips. Commit 65ddafe.
• 6.2: ctrlstatus_fill role-branches committed sourcing — PRIMARY → own persist.committed_lsn/committed_config_tag (atomic acquire); else → ha->last_known_committed_lsn (atomic acquire) + ha->last_known_committed_config_tag (plain aligned read, NO ha->lock per design — ha.h:172 is non-atomic, written under lock at ha_convergence.c:377). appliedLsn unchanged. Timestamp stamped from CLOCK_REALTIME (NOT monotonic). Read-only. STALENESS CAVEAT (in-boundary): last_known_committed_lsn refreshes at CtrlHello join/RDHUP, not per heartbeat — off-primary committed can lag; live-fresh = OUT of boundary → ha-heartbeat-liveness/ha-repl-hardening. Commit 3799bd6.
• 6.3 LEARNING: the controller log timestamp formatter is NOT in xcelog.h (an 85-line thin wrapper that just #include "tlog.h"); the YYYY-MM-DD HH:MM:SS,mmm prefix is assembled inline in vendored tinylog (tlog.c _tlog_root_default_format ~393/398, via my_tlog_format in src/lib/utils.c:221/228) — no caller-facing helper, so it was REPLICATED byte-equivalently (localtime_r + strftime "%Y-%m-%d %H:%M:%S" + ,%03u of ms%1000) in a new header-only src/controller/testsuite/ctrlstatus_fmt.h (ctrlstatus_format_time_ms), shared by test.c (cmd_ctrlstatus prints responderTime: after fenced) and test_ctrlstatus.c (TZ=UTC-pinned exact-string unit test). LOCAL time on both sides — comparable to ctrl-N.log under the same TZ.
• 6.4 VERIFIED 2026-07-02 (previously HAND-OFF, blocked on overlayfs — unblocked by ha-commit-durability 6.1 confirming ~/tmp is a genuine ext4 fs in this sandbox): deployed a 3-node cluster to ~/tmp/xce-ctrlstatus-deploy via src/tools/deploy-ha.sh, submitted jobs against the full masters list, then ran ctrlstatus pointed directly at standby ctrl-2 (set masters localhost:9992). Result: answered by ctrl-2 (standby), committedLsn: 4 / appliedLsn: 4 — non-zero, matching, no inversion (the exact 2026-06-20 bug symptom does not reproduce). responderTime: 2026-07-02 14:18:19,689 is a live, current wall-clock value consistent with the session timeline. NOTE on the log-correlation half of the Observable: ctrlstatus_fill (ctrlstatus.c) has no logging call for the query itself (pure snapshot-fill, no INFO/TRACE line), so there is no literal ctrl-N.log line to string-match against — verified the weaker-but-sufficient form instead (responderTime is live/plausible, not frozen/stale/wrong-TZ). PASS. Found and logged (unrelated, low-priority): bug-1400 (xce_test cmd_jobsub SEGVs on malformed positional args instead of required getopt flags — test-client input validation gap, not a production bug) and bug-1401 (ctrlstatus reply decode leaks ~120 bytes per call in decode_CtrlStatus_ptr, ASAN-only visibility, ~103 allocs/12KB observed over a dozen calls in this session).