myxce/.kiro/specs/controller-status/tasks.md

# Implementation Plan

> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor.
> The work runs along a single request path (schema → controller handler → dispatch gate → libxce client → test command → wait-primary), so tasks are sequential by design; no `(P)` markers. The one HA-critical change is the dispatch-gate allow-list (task 2.2).

- [x] 1. Foundation: status reply schema + codegen
- [x] 1.1 Add the `ctrlstatus` opcode and the status reply payload to the schema
  - Add a new request opcode `ctrlstatus` (next free ordinal) and a new reply-union member (next free ordinal), plus a lean status payload struct carrying: cluster name, primary name, responder name and role, term, loaded config version and content checksum, committed config version, commit and apply positions, fenced flag, and a bounded list of configured controllers (name, host, port) with a count.
  - Keep the payload focused on identity/health only — it must NOT carry the resource/queue/worker inventory.
  - Observable: the build regenerates the C types for the new payload and opcode (the generated status type, its reply-union tag, and a free helper exist) and the schema compiles clean.
  - _Requirements: 1.1, 1.4, 2.1, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 6.1, 6.2, 6.3, 6.4_

- [x] 2. Controller: status handler + any-role dispatch
- [x] 2.1 Implement the controller status handler
  - Add a handler that snapshots the responding node's local HA and config state into the status reply: responder name and role; current primary (taken under the HA lock; empty when none is known/elected); term; loaded config version + content checksum; committed config version; commit and apply positions; fenced state (catch-up or unrecoverable-fence); and the configured controllers list with count. It must be read-only (no state mutation) and answer in every role.
  - Route the new opcode to this handler in the controller request dispatch switch.
  - TDD: unit tests assert the field values for a PRIMARY (primary == self, role == primary, populated peers/version/checksum), a STANDBY with a known primary (reports that primary, role == standby), and a node with no known primary (empty primary → "unknown"); and assert the handler mutates no state.
  - Observable: the unit tests pass for the primary, standby, and unknown-primary cases, and a no-mutation assertion holds.
  - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1, 6.2, 6.3, 6.4_
  - _Boundary: handle_ctrlstatus_
- [x] 2.2 Allow the status query through the HA dispatch gate in every role
  - Add the status opcode to the dispatch-gate allow-lists for primary (including the no-quorum branch), standby, electing, and initializing — so a status read always answers (including when fenced or without quorum), rather than being rejected as standby/not-ready. Scope the change strictly to this read-only opcode.
  - TDD: an integration test confirms the status request is answered (not rejected) on a standby, an electing node, an initializing node, and a primary without quorum; and a guard test confirms no other opcode's gating changed.
  - Observable: the gate test shows the status query answered in all four role conditions and unchanged behavior for at least one previously primary-only opcode.
  - _Requirements: 7.1, 7.2_
  - _Boundary: ha_dispatch_gate_
  - _Depends: 2.1_

- [x] 3. libxce client API
- [x] 3.1 Add the failover-aware any-node status call
  - Add the opcode-only request sender, the send-and-decode call, and the failover-aware client read (retryable, non-write) that returns whatever reachable node answers, plus the free helper. On no reachable controller, return a typed failure without a partial/empty success.
  - TDD: a round-trip test decodes every status field from a handler reply; a no-reachable-controller test returns a typed failure.
  - Observable: the client decodes all fields from a live/loopback handler reply and surfaces a typed error when no controller is reachable.
  - _Requirements: 1.1, 7.1, 7.2, 7.3_
  - _Boundary: libxce client_
  - _Depends: 1.1, 2.1, 2.2_
- [x] 3.2 Add the authoritative reach-the-primary status call
  - Add a client call that returns only an authoritative reply (responder role == primary). When the first answer is from a non-primary, follow the redirect — target the reported primary via the reply's peer list, otherwise rotate the configured controller list — until the primary answers or the failover budget is exhausted.
  - TDD: a test where the first reachable node is a standby confirms the call follows the hint to the primary and returns a primary-role reply; a no-serving-primary test returns failure (never a standby hint as the authoritative result).
  - Observable: the authoritative call returns a reply whose responder role is primary even when a standby answered first, and fails (does not return a hint) when no primary serves.
  - _Requirements: 2.3, 9.1, 9.2_
  - _Boundary: libxce client_
  - _Depends: 3.1_

- [x] 4. Test client command + wait-primary integration
- [x] 4.1 Add the `ctrlstatus` client command
  - Add a client command that issues the any-node status call and prints the fields in human-readable form: the current primary, the answering node framed as "answered by <node> (<role>)" distinct from the primary, the configured controllers and count, the loaded config version and committed config version, the content checksum as hex, term, commit/apply positions, fenced state, and cluster name; render an empty primary as "unknown". The command exit status reflects success or failure of the query.
  - Observable: piping `ctrlstatus` through the client prints the status fields (with "answered by" and "unknown" rendering) and exits zero on success / non-zero with the error printed on failure.
  - _Requirements: 1.3, 3.3, 5.1, 5.2, 8.1, 8.2, 8.3_
  - _Boundary: cmd_ctrlstatus_
  - _Depends: 3.1_
- [x] 4.2 Switch wait-primary to the authoritative status probe
  - Replace the wait-primary probe (currently a job-list call) with the authoritative reach-the-primary status call. Success is an authoritative reply from the primary; on success, print the primary's identity. A standby/hint answer is not success — the probe follows the redirect or keeps polling until the primary answers or the deadline elapses. Preserve the existing timeout-argument handling, monotonic deadline, poll interval, log-quieting, and success/failure exit semantics.
  - Observable: against a healthy cluster wait-primary returns and names the primary; a malformed timeout still errors; the existing exit-code behavior is unchanged; and it does not succeed from a standby-only answer.
  - _Requirements: 9.1, 9.2, 9.3, 9.4_
  - _Boundary: cmd_wait_primary_
  - _Depends: 3.2_

- [x] 5. Validation (live cluster)
- [x] 5.1 End-to-end status + authoritative wait-primary on a live cluster
  - Run against a live 3-node cluster: `ctrlstatus` on a healthy cluster shows the correct primary, peers, and version+checksum; querying via a standby still reports the primary with the "answered by <standby>" framing; the version+checksum match across nodes with identical config; and the authoritative wait-primary returns immediately on a healthy cluster, blocks until the NEW primary itself answers after a primary kill (without false-succeeding on a standby's stale primary view), and times out when no primary serves.
  - Observable: the live run shows a correct status (primary/peers/checksum) including via a standby, and the authoritative wait-primary names the new primary after a kill without a false success; manual environment prerequisite (live cluster), same status as other live-E2E tasks.
  - _Requirements: 2.3, 5.3, 7.1, 7.2, 8.1, 9.1, 9.2, 9.3, 9.4_
  - _Depends: 4.1, 4.2_

- [x] 6. Delta: off-primary committed fidelity + responder timestamp (Req 10, 11)
  > Sequential, single request path (schema → typemap → `ctrlstatus_fill` → render → E2E); no `(P)`. Read-only reporting only — the live-fresh-committed heartbeat change is OUT of boundary (route to ha-heartbeat-liveness / ha-repl-hardening).
- [x] 6.1 Add the responder snapshot timestamp field to the schema + hand-authored typemap
  - Append a responder snapshot-time field to the status payload at the next free ordinal (after the peer count), carrying the responder's wall-clock at sample time as epoch milliseconds (append-only — never reorder existing field ordinals). Add the matching field to the hand-authored `xce_ctrlstatus_t` typedef in `src/include/typemap.h` and verify it field-for-field against the regenerated codec (per the 1.1/2.1 learning: `$C.mapname` types are NOT auto-generated here).
  - Observable: the build regenerates `controller.capnp.h/.c` clean, the typemap matches the codec, and a codec round-trip carries the new field's value unchanged.
  - _Requirements: 11.1_
  - _Boundary: schema + typemap_
- [x] 6.2 Source off-primary committed values + stamp the timestamp in `ctrlstatus_fill`
  - TDD (red first, in `test_ctrlstatus.c`): a STANDBY whose learned committed position is non-zero while its local committed marker is zero and it has applied entries reports the LEARNED committed LSN and committed config version (not zero) and the real applied position (no applied-greater-than-committed inversion); a STANDBY that has not yet learned any committed position reports committed LSN and committed config version as zero/unknown; a PRIMARY reports its own authoritative local committed LSN and committed config tag (unchanged); and the responder snapshot timestamp is non-zero and within tolerance of a test-captured wall-clock sample. Assert the handler still mutates no state.
  - Change `ctrlstatus_fill` (`src/controller/ctrlstatus.c`): role-branch the committed sourcing — when the responder is primary, keep its own committed LSN / committed config tag; otherwise report the learned committed LSN / learned committed config tag. Stamp the responder snapshot timestamp from the wall clock (`CLOCK_REALTIME`, NOT the monotonic clock) at snapshot time. All additions are atomic/aligned loads + a clock read — no state mutation.
  - Observable: the new unit tests go red → green; the standby case reports the learned non-zero committed value instead of zero; the no-mutation assertion holds.
  - _Requirements: 10.1, 10.2, 10.3, 10.4, 11.1_
  - _Boundary: ctrlstatus_fill_
  - _Depends: 6.1_
- [x] 6.3 Render the responder snapshot timestamp in the `ctrlstatus` client command
  - Add one printed line to `cmd_ctrlstatus` (`src/controller/testsuite/test.c`) rendering the responder snapshot timestamp as a local `YYYY-MM-DD HH:MM:SS,mmm` string in the same form the controller logger uses, so it can be compared directly against `ctrl-N.log`. Confirm and reuse (or match) the exact wall-clock formatter in `src/include/xcelog.h`. The committed lines need no rendering change — task 6.2 only changed their values.
  - Observable: piping `ctrlstatus` through the client prints a responder-timestamp line in the log timestamp format alongside the other status fields.
  - _Requirements: 11.2, 11.3_
  - _Boundary: cmd_ctrlstatus_
  - _Depends: 6.1, 6.2_
- [x] 6.4 End-to-end validation on a live cluster (off-primary committed + log correlation)  <!-- VERIFIED 2026-07-02 on a real-fs (~/tmp, ext4) 3-node deploy — see Implementation Notes -->
  - Run against a live 3-node cluster: submit jobs so the primary advances its committed position, then query a STANDBY. Assert the standby's reported committed LSN is non-zero (tracking the primary within the documented join-refresh caveat) rather than zero while it has applied entries, and that the printed responder timestamp matches that standby's own `ctrl-N.log` line for the same query.
  - Observable: the live run shows a non-zero off-primary committed LSN (no zero-while-applied inversion) and a responder timestamp that lines up with the standby's log; manual environment prerequisite (live cluster), same status as the other live-E2E task (5.1).
  - _Requirements: 10.1, 10.3, 11.2, 11.3_
  - _Depends: 6.2, 6.3_

## 7. Delta: primary election term + time (Req 12)
> Sequential — the election-time stamp touches the same HA promotion/heartbeat control flow as tasks 2.2/6.2 (ha.c); no `(P)`. Read-only reporting only: this task must not change how elections are conducted, how term advances, or how/when role transitions to primary — every write here is an additional observational stamp at an already-decided instant. `electedTerm` needs no new field (design.md Locked Decision 1); it is a CLI-side derivation added in 7.5, not new state.

- [x] 7.1 Add the election-time schema fields and controller state
  - Add a wall-clock election-time field to the status reply payload (next free ordinal, append-only) and to the heartbeat message the primary already sends to standbys (next free ordinal).
  - Add the corresponding piece of HA state that holds the currently-known primary's election time, alongside the existing learned-committed-position state.
  - Add the matching fields to the hand-authored C structs backing these two payloads (this schema's `$C.mapname` types are NOT auto-generated — the generated codec references hand-written typedefs in `typemap.h`, per tasks 1.1/2.1/6.1's established convention) and verify each new field field-for-field against the regenerated codec.
  - Observable: the build regenerates the reply and heartbeat codecs clean with the new fields present, the hand-authored C structs match the codec field-for-field, a codec round-trip carries each new field's value unchanged, and the new HA state field exists and defaults to zero/unknown before any election.
  - _Requirements: 12.1_
  - _Boundary: schema + typemap + ha_state_t_

- [x] 7.2 Stamp the election time when a node becomes primary
  - TDD (red first): a promotion via the non-election-protocol path and a promotion via the leader-election path both record a non-zero election-time at the moment the node commits to the primary role; a promotion attempt that is refused (fenced, ineligible, or otherwise rejected) records NO election-time and leaves any prior value untouched.
  - Stamp the wall-clock election time exactly once, at the point each promotion path irrevocably commits to the primary role — after any eligibility/refusal checks that path performs, never before.
  - Observable: the new tests go red → green; a refused promotion attempt (either path) leaves the election-time state unchanged, proving the stamp is not placed before the refusal checks.
  - _Requirements: 12.1, 12.5_
  - _Boundary: promotion (ha_do_promote, raft_cb_become_leader)_
  - _Depends: 7.1_

- [x] 7.3 Carry the primary's election time to standbys over the heartbeat
  - TDD (red first): a standby that processes a heartbeat carrying the primary's election time stores it as its own learned election-time and leaves every other learned/produced HA field (committed position, role, term, applied position) untouched (reconcile-free).
  - The primary includes its own election time in every heartbeat it sends; a standby that receives a heartbeat stores the carried election time as its learned value.
  - Observable: the reconcile-free test passes — only the learned election-time changes, nothing else moves; a live/loopback heartbeat round-trip shows the standby's learned election-time matching the primary's.
  - _Requirements: 12.2_
  - _Boundary: heartbeat (send + receive)_
  - _Depends: 7.1_

- [x] 7.4 Source the election time in the status handler
  - TDD (red first): the status handler reports a non-zero election-time equal to the responder's own learned value (self-stamped on a primary, heartbeat-learned on a standby); a standby seeded with a learned election-time (simulating an already-processed heartbeat) reports that learned value, not zero; the handler still mutates no state.
  - Read the learned election-time into the status reply unconditionally (no role-based branching, no "unknown" gating in the handler itself — that display decision belongs to the client command in 7.5, mirroring how the handler already reports raw term unconditionally regardless of whether a primary is currently known).
  - Observable: the new unit tests go red → green; a standby seeded with a learned election-time reports that value instead of zero; the no-mutation assertion holds.
  - _Requirements: 12.1, 12.2, 12.5_
  - _Boundary: ctrlstatus_fill_
  - _Depends: 7.1_

- [x] 7.5 Render the election term and time in the `ctrlstatus` client command
  - Print the primary's election term and election time next to the primary's identity in the client command's output, rendering the election time on the same clock/format already used for the responder snapshot timestamp; render both as "unknown" whenever the primary identity itself is reported as unknown.
  - Observable: piping `ctrlstatus` through the client prints the election term and a log-comparable election-time string alongside the existing fields when a primary is known, and prints "unknown" for both when it is not.
  - _Requirements: 12.1, 12.3, 12.4, 12.6_
  - _Boundary: cmd_ctrlstatus_
  - _Depends: 7.4_

- [x] 7.6 End-to-end validation on a live cluster
  - Run against a live 3-node cluster: query the primary right after a fresh election and confirm its printed election term matches its current term and its election time is close to "now"; query a standby and confirm it reports the SAME election term and election time as the primary (not "unknown", not stale) once at least one heartbeat has been exchanged; kill the primary, wait for a new election, and confirm the surviving standby's reported election term/time updates to the new primary's values within about one heartbeat interval.
  - Observable: the live run shows matching election term/time between the primary and a standby, and shows the standby's values converge to the new primary's after a kill within one heartbeat interval; manual environment prerequisite (live cluster), same status as the other live-E2E tasks (5.1, 6.4).
  - _Requirements: 12.1, 12.2, 12.4, 12.6_
  - _Depends: 7.2, 7.3, 7.4, 7.5_

## Implementation Notes (7.6 E2E outcome, Req 12 delta)
- First live run FOUND + FIXED a real bug (bug-1818, `.wolf/buglog.json`): `cmd_ctrlstatus` gated `electedTerm`/`electedAt` on `NULLSTR(status->primaryName)` (mirroring the `Primary:` line, per the original design.md Locked Decision 3) — but `primaryName` is unconditionally empty on every non-primary responder in this build (the pre-existing, vestigial-`primary_host` behavior from Req 2/`ade0081`), so the two new fields printed "unknown" on every standby, forever, despite the underlying `elected_at_ms` being correctly heartbeat-learned and non-stale the whole time (`term`/`committedLsn`/`appliedLsn`, sourced the same unconditional way but not primaryName-gated, correctly displayed and converged on the same standby query). All unit tests passed throughout — the bug lived entirely in the untested CLI print-gate, exactly the class of defect only a live cluster run could catch.
- Fix (commit `80839661`): gate on `status->elected_at_ms == 0` instead — necessary and sufficient, since that field only ever becomes nonzero via a real completed election (either promotion road) or genuine heartbeat-learning from an actual primary, unlike `term` which can bump from mere candidacy activity. `ctrlstatus_fill` itself was untouched (stays unconditional/no-role-branch, task 7.4's contract intact). design.md's Locked Decision 3 (+ dependent traceability/service-contract text) corrected in the same commit to document this live-E2E finding, mirroring the spec's Req 10/11 delta convention.
- Retry PASSED cleanly on a fresh 3-node deploy: primary ctrl-1 self-reported `electedTerm: 2` / `electedAt: 2026-09-16 09:12:02,485` (byte-for-byte matching its own `raft_cb_become_leader` log line); standby ctrl-2/ctrl-3 both reported the IDENTICAL values after ≥1 heartbeat interval (previously "unknown"). After killing ctrl-1, ctrl-2 was elected (term=3, `wait-primary` in 2253ms); the SAME surviving standby (ctrl-3, queried both before and after the kill) converged from `electedTerm: 2` to `electedTerm: 3` matching the new primary exactly, within ~18s (well under the ~5-heartbeat-interval budget). Review independently cross-checked every reported value/timestamp against the raw daemon logs still on disk, not just the transcript.

## Implementation Notes (5.1 E2E outcome)
- 5.1 ran live on a 3-node `/xce-deploy-ha`-style cluster (ext2/3 fs). PASSED in-scope: `ctrlstatus` shows the primary self-reporting (`Primary: ctrl-1` on the primary), standbys honestly report `Primary: unknown` with `answered by ctrl-2 (standby)` (the dispatch-gate lets standbys answer — Req 7); peers + count correct; `configChecksum` = the real 64-hex SHA-256 (matched the deploy tag prefix) and is IDENTICAL across all 3 nodes with identical config (Req 5.3); `configVersion`==`committedConfigVersion`; authoritative `wait-primary` names the primary in <1s on a serveable cluster (Req 9.1/9.2).
- 5.1 FOUND + FIXED a real 2.1 bug: `ha->primary_host` is VESTIGIAL (never written anywhere in the controller), and `voted_for_name` is the last-vote (often self), so neither names the current primary off-primary. Corrected: PRIMARY self-reports its own name; STANDBY/ELECTING/INIT report empty → "unknown" (the authoritative client path reaches the real primary by rotation). Commit ade0081.
- 5.1 UPSTREAM FINDING (out of boundary, NOT controller-status): with a DEAD node FIRST in the masters list, the libxce client failover returns `EINVAL` instead of advancing to the live nodes — so the authoritative `wait-primary` (and ANY client op) can't reach the primary while a dead node leads the list. CONFIRMED pre-existing: plain `joblist` fails IDENTICALLY (`xce_joblist: Invalid argument`) in the same condition; with a live-only masters list `wait-primary` succeeds in 46ms. This spec consumes failover unchanged (per Boundary/Adjacent expectations) → route to libxce-failover, do NOT patch here. The durability soak avoided it by restarting killed nodes (no dead-first-master during waits).

## Implementation Notes
- All new C types are generated from the schema (task 1.1) — no hand-written struct headers. Build regenerates `controller.capnp.h/.c`.
- Task 2.2 is the one HA-critical change (request-handling path): keep it scoped to the single read-only opcode and prove no other opcode's gating widened.
- The authoritative redirect (3.2) is what makes wait-primary trustworthy (no stale-standby false success) — it is the resolution of the design-review decision; do not collapse it back into a plain any-node read for wait-primary.
- 1.1 LEARNING (affects later tasks): the `$C.mapname(...)` capnp types are NOT auto-generated in this schema — the generated `controller.capnp.c` references HAND-AUTHORED C typedefs in `src/include/typemap.h` (same convention as `xce_clusterconf_t`). So `xce_ctrlstatus_t`/`xce_ctrlpeer_t` (and the `xce_reply_t.data.ctrlstatus` union member) live in `typemap.h`, verified field-for-field against the regenerated codec. `OPCode_ctrlstatus = 40`, `RequestReply_data_ctrlstatus = 12`. `configChecksum` is `xce_data_t*`. The handler/libxce tasks build/encode against these typemap.h types.
- 2.1 LEARNING + SCHEMA AMENDMENT: capnp `Data` in THIS schema maps to the project's STRUCTURED `xce_data_t` union (table-switched), NOT raw bytes — `encode_Data_ptr` can't serialize a SHA-256 blob. So `configChecksum @6` was changed `Data → Text` (a 64-char lowercase-hex string of the full 32-byte `config_sha256`); typemap member is `char*`; freed in `ctrlstatus_free_fields`. Ordinal @6 unchanged, no released decoder yet → safe. Handler lives in NEW `src/controller/ctrlstatus.c` (`ctrlstatus_fill`/`ctrlstatus_free_fields`/`handle_ctrlstatus`), proto in `src/controller/include/cluster.h`, routed at `cluster.c` opcode switch, test `src/controller/testsuite/test_ctrlstatus.c` (registered in `controller_test.c`, built via `controller/meson.build`). `primary_host` read under `ha->lock`; peer name/host strings are BORROWED from myconf (not freed). The libxce client (3.x) decodes `configChecksum` as Text.
- 3.1 LEARNING: the ctx-side reply union `xce_ctx_t.reply.u` (in `src/include/xcectx.h`) is SEPARATE from `xce_reply_t.data` (typemap.h) — the decode path needs BOTH a `ctrlstatus` member there AND a `case OPCode_ctrlstatus` in `xcectx.c` handle_reply (mirror clusterconf). Free: clusterconf's `xce_free_clusterconf` macro is a LATENT-BROKEN alias (undefined symbol, never expanded); use the REAL generated free — `xce_free_ctrlstatus(s) → free_CtrlStatus(s)`, `xcecli_free_ctrlstatus(s) → xce_free_ctrlstatus(s)` (any TU expanding it must include the generated `controller.capnp.h`). `xcecli_ctrlstatus` is is_write=false; NULL ctx → `-EINVAL`.
- 3.2: `xcecli_ctrlstatus_primary` = bounded loop (cap `failover.retry.max_retries`) reusing `xcecli_ctrlstatus` + `xce_ctx_capture_redirect_hint` (map `primaryName`→addr via `peers[]`, else master rotation); authoritative iff `responderRole == CTRL_PRIMARY` (== 0; `ctrl_role_t` PRIMARY=0/STANDBY=1/INIT=2/ELECTING=3 in `ha.h` — REVALIDATION TRIGGER: a role-enum reorder breaks the pinned wire ordinal `XCECLI_CTRLSTATUS_ROLE_PRIMARY=0`). On exhaustion → `-E_XCE_NOT_READY`, `*status` untouched (never a hint). Pure helpers `xcecli_ctrlstatus_is_authoritative`/`_primary_addr` unit-tested; live standby→primary redirect is 5.1.
- 6.1: `responderTimeMs @13: UInt64` appended to `CtrlStatus` (after `peerCount @12`, append-only) + matching `uint64_t responder_time_ms` in the hand-authored `xce_ctrlstatus_t` (`src/include/typemap.h`); codec wire offset 48, round-trips. Commit 65ddafe.
- 6.2: `ctrlstatus_fill` role-branches committed sourcing — PRIMARY → own `persist.committed_lsn`/`committed_config_tag` (atomic acquire); else → `ha->last_known_committed_lsn` (atomic acquire) + `ha->last_known_committed_config_tag` (plain aligned read, NO `ha->lock` per design — `ha.h:172` is non-atomic, written under lock at `ha_convergence.c:377`). `appliedLsn` unchanged. Timestamp stamped from `CLOCK_REALTIME` (NOT monotonic). Read-only. STALENESS CAVEAT (in-boundary): `last_known_committed_lsn` refreshes at CtrlHello join/RDHUP, not per heartbeat — off-primary committed can lag; live-fresh = OUT of boundary → ha-heartbeat-liveness/ha-repl-hardening. Commit 3799bd6.
- 6.3 LEARNING: the controller log timestamp formatter is NOT in `xcelog.h` (an 85-line thin wrapper that just `#include "tlog.h"`); the `YYYY-MM-DD HH:MM:SS,mmm` prefix is assembled inline in vendored tinylog (`tlog.c` `_tlog_root_default_format` ~393/398, via `my_tlog_format` in `src/lib/utils.c:221/228`) — no caller-facing helper, so it was REPLICATED byte-equivalently (`localtime_r` + `strftime "%Y-%m-%d %H:%M:%S"` + `,%03u` of `ms%1000`) in a new header-only `src/controller/testsuite/ctrlstatus_fmt.h` (`ctrlstatus_format_time_ms`), shared by `test.c` (`cmd_ctrlstatus` prints `responderTime:` after `fenced`) and `test_ctrlstatus.c` (TZ=UTC-pinned exact-string unit test). LOCAL time on both sides — comparable to `ctrl-N.log` under the same TZ.
- 8.6 FIXED 2026-09-16 (bug-1819, found live during 8.5's optional attempt): `ctrlstatus_fill`'s primary-side peer-status scan stopped at the first `replicator_t.standbys[]` slot matching a peer's `host:port` — after a kill+rejoin, the stale (ep==NULL) slot from before the kill can have a lower index than the new live slot the reconnect landed in, so the peer was reported `dead` PERMANENTLY even after fully reconnecting and catching up. Fixed by scanning all `MAX_STANDBYS` slots and preferring any `ep != NULL` match over an `ep == NULL` one. TDD RED→GREEN (1216 Tests: 2 Failures → 0 Failures), independently reproduced by the reviewer including an exact RED replay. Two independent live re-checks (implementer + reviewer, separate deploy dirs/ports) both confirmed a killed-then-restarted standby now converges to `ok` on every poll instead of sticking at `dead`. Neither live run happened to construct the literal two-distinct-slots scenario (an unrelated, already-tracked bug-1474 angle-a symptom consumed the cluster's one spare slot both times) — the two new unit tests are the deterministic proof of the exact fix mechanism; the live runs are corroborating, not substituting. design.md's Locked Decision 3 corrected in the same pass to document the fix (matches this spec's established as-built-drift-correction convention).

- 8.4 VERIFIED 2026-09-16: two independent live 3-node-cluster runs (implementer + reviewer, separate deploy dirs/ports) both confirmed the full Req 13 scenario cleanly with ZERO bugs found (unlike 5.1/6.4/7.6, which each found+fixed a real bug) — healthy primary reports all 3 controllers `ok`; killing a standby flips it to `dead` on the primary's view while the other two stay `ok`; querying the surviving standby directly shows `ok` for itself and `unknown` for every other peer, including the still-alive primary, confirming the asymmetric-visibility design decision (Locked Decision 5) holds live. The tasks 8.1-8.3 implementation required no live-only fixes.

- 8.2 LEARNING: the new `replicator_t.lock` acquisition in `ctrlstatus_fill`'s peer-building loop is held around the whole per-peer loop (calloc + self/primary/non-primary branching for all configured peers), slightly wider than `repl_active_standby_count`'s own lock-and-scan shape that design.md's Constraints-checked note cites as the precedent. Still bounded/safe (n ≤ `MAX_HA_CANDIDATES`, inner scan ≤ `MAX_STANDBYS`, no I/O) and reviewer-approved as non-blocking, but a future touch to this loop should narrow the lock to wrap only the inner `MAX_STANDBYS` registry scan per non-self peer, matching the cited precedent exactly.

- 6.4 VERIFIED 2026-07-02 (previously HAND-OFF, blocked on overlayfs — unblocked by ha-commit-durability 6.1 confirming `~/tmp` is a genuine ext4 fs in this sandbox): deployed a 3-node cluster to `~/tmp/xce-ctrlstatus-deploy` via `src/tools/deploy-ha.sh`, submitted jobs against the full masters list, then ran `ctrlstatus` pointed directly at standby ctrl-2 (`set masters localhost:9992`). Result: `answered by ctrl-2 (standby)`, `committedLsn: 4` / `appliedLsn: 4` — non-zero, matching, no inversion (the exact 2026-06-20 bug symptom does not reproduce). `responderTime: 2026-07-02 14:18:19,689` is a live, current wall-clock value consistent with the session timeline. NOTE on the log-correlation half of the Observable: `ctrlstatus_fill` (ctrlstatus.c) has no logging call for the query itself (pure snapshot-fill, no INFO/TRACE line), so there is no literal `ctrl-N.log` line to string-match against — verified the weaker-but-sufficient form instead (responderTime is live/plausible, not frozen/stale/wrong-TZ). PASS. Found and logged (unrelated, low-priority): bug-1400 (xce_test cmd_jobsub SEGVs on malformed positional args instead of required getopt flags — test-client input validation gap, not a production bug) and bug-1401 (ctrlstatus reply decode leaks ~120 bytes per call in `decode_CtrlStatus_ptr`, ASAN-only visibility, ~103 allocs/12KB observed over a dozen calls in this session).

## 8. Delta: per-controller connection status (Req 13)
> Sequential, single request path (schema → typemap → `ctrlstatus_fill` → render → E2E); no `(P)`. Read-only reporting only — this delta must not change how standby connections are established, monitored, evicted, or torn down (Req 13.5); it only reads the existing replication connection registry (`replicator_t.standbys[]`) that already tracks each standby's live connection and catch-up state for replication/quorum purposes.

- [x] 8.1 Add the peer connection-status field to the schema + hand-authored typemap
  - Add a `status` field to the `CtrlPeer` reply struct (next free ordinal, append-only — after `port`), carrying a small fixed-value code (0=unknown, 1=ok, 2=catching-up, 3=dead), and the matching field to the hand-authored `xce_ctrlpeer_t` typedef in `src/include/typemap.h`, verified field-for-field against the regenerated codec.
  - Observable: the build regenerates the reply codec clean with the new field present, the typemap matches the codec, and a codec round-trip carries the field's value unchanged (defaulting to 0/unknown when unset).
  - _Requirements: 13.1_
  - _Boundary: schema + typemap_

- [x] 8.2 Source each configured controller's connection status in the status handler
  - Confirm the exact field path from the controller's top-level cluster state to the replication connection registry (the research item carried from design) before wiring the read below.
  - TDD (red first, in `test_ctrlstatus.c`): a PRIMARY responder with three peer connection states (one live and caught up, one live and catching up, one with no live connection) reports ok/catching-up/dead respectively for those three peers, and ok for its own entry; a STANDBY responder reports ok for its own entry and unknown for every other configured peer, including the primary; a fixture whose own identity matches no configured peer (misconfiguration case) still produces a defined, non-crashing result via the normal non-self sourcing (no entry force-corrected to ok) and logs a one-time warning; the existing no-mutation assertion is extended to cover whatever connection-registry state this task reads.
  - Implement the sourcing: a peer matching the responder's own identity always reports ok; when the responder is currently primary, every other peer's status is sourced by matching it against the live replication connection registry (no live connection → dead; live but not yet caught up → catching-up; live and caught up → ok); when the responder is not primary, every other peer's status is reported as unknown, since only the primary has direct connection visibility into every other node. The read stays bounded (a lock-protected scan of a small, fixed-size in-memory registry, no network I/O or unbounded wait) and mutates no state.
  - Observable: the new unit tests go red → green; the primary, standby, and misconfiguration-fallback cases each report the values above; the no-mutation assertion holds.
  - _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5_
  - _Boundary: ctrlstatus_fill_
  - _Depends: 8.1_

- [x] 8.3 Render each configured controller's connection status in the `ctrlstatus` client command
  - Add a status-code-to-string renderer covering all four values plus a safe fallback for an unrecognized code (mirroring the existing responder-role renderer's convention), and extend the client command's existing per-controller line to print the rendered status alongside that controller's name and address.
  - Observable: piping `ctrlstatus` through the client prints each configured controller's connection status alongside its existing name:address line.
  - _Requirements: 13.1, 13.6_
  - _Boundary: cmd_ctrlstatus_
  - _Depends: 8.2_

- [x] 8.4 End-to-end validation on a live cluster
  - Run against a live 3-node cluster: query `ctrlstatus` on the primary of a healthy cluster and confirm all three configured controllers (including itself) show ok; kill one standby and re-query the primary, confirming that standby now shows dead while the other two still show ok; query a surviving standby directly and confirm its own entry shows ok while every other entry, including the primary, shows unknown — proving the asymmetric-visibility design decision holds live, not just in a unit test.
  - Observable: the live run shows the ok/dead transition on the primary's view and the unknown-for-others view on the standby's view exactly as described; manual environment prerequisite (live cluster), same status as the other live-E2E tasks (5.1, 6.4, 7.6).
  - _Requirements: 13.1, 13.2, 13.3, 13.4, 13.6_
  - _Depends: 8.2, 8.3_

- [ ]* 8.5 (Optional) Observe transient catching-up status on a live cluster
  - If the project's E2E tooling can hold a restarted standby in catch-up long enough to query it, restart a previously-killed standby and query the primary shortly after, confirming that peer transiently shows catching-up before settling to ok.
  - Observable: a live run captures at least one `ctrlstatus` query showing catching-up for a rejoining standby before it converges to ok.
  - _Requirements: 13.1_
  - _Depends: 8.4_

- [x] 8.6 Fix: primary-side peer-status matching must prefer a live registry slot over a stale disconnected one
  - Bug found live during task 8.5's attempt (read-only observation, no code touched at the time): `ctrlstatus_fill`'s primary-side peer-status loop (`src/controller/ctrlstatus.c`) scans `replicator_t.standbys[]` for a `host`/`port` match and stops at the FIRST match by slot index. When a standby is killed, its slot is freed (`ep` cleared) but keeps its `host`/`port` populated (existing bug-1474-documented "guard-holding slot" behavior, unrelated to this delta); when that standby reconnects, `repl_add_standby`'s `ep`-identity idempotency check cannot reuse the old slot (the old `ep` is gone) and allocates a NEW slot instead — so two slots now share the same configured peer's `host:port`, one stale (`ep == NULL`) and one live (`ep != NULL`). Because the stale slot has the lower index, `ctrlstatus_fill`'s first-match-wins scan reports that peer as `dead` PERMANENTLY, even minutes after it has genuinely reconnected and fully caught up — live-reproduced and log-cross-confirmed (see task 8.5's Status Report: primary's own log shows `acked` converging to the target LSN at a specific timestamp, while `ctrlstatus` kept reporting `dead` from restart through +5m45s). This is distinct from bug-1474's already-fixed angle (`on_standby_connected`'s live-duplicate-channel eviction, a different check) — this defect is specific to `ctrlstatus_fill`'s own status-derivation scan and violates Req 13.3's "true, currently-known connection status" guarantee for any standby that has ever been killed and rejoined.
  - TDD (red first, in `test_ctrlstatus.c`): a fixture with TWO `replicator_t.standbys[]` slots matching the SAME configured peer's `host:port` — a lower-index slot with `ep == NULL` (stale) and a higher-index slot with `ep != NULL && !in_catch_up` (live, caught up) — must report that peer's status as `ok` (sourced from the live slot), not `dead` (the stale slot's would-be status); a sibling case with the live slot `in_catch_up` must report `catching-up`, not `dead`.
  - Fix: change the matching loop to scan ALL `MAX_STANDBYS` slots for a given configured peer (not stop at the first match) and prefer any slot with `ep != NULL` over a slot with `ep == NULL` when multiple slots match the same `host:port`; if multiple live slots somehow match (shouldn't normally happen), any one of them is an acceptable source since they'd all be observing the same live peer. Keep the read bounded and lock-scoped exactly as task 8.2 left it (no scope regression).
  - Observable: the new unit test goes red → green; a live re-run of task 8.5's kill-then-restart-under-backlog scenario shows the rejoined standby transition away from a permanent `dead` (ideally observing `catching-up` before settling to `ok`, satisfying 8.5's original optional Observable as a side effect of this fix).
  - _Requirements: 13.1, 13.3_
  - _Boundary: ctrlstatus_fill_
  - _Depends: 8.2_