diff --git a/.kiro/specs/controller-status/tasks.md b/.kiro/specs/controller-status/tasks.md index 2605c934..153ac33e 100644 --- a/.kiro/specs/controller-status/tasks.md +++ b/.kiro/specs/controller-status/tasks.md @@ -159,3 +159,40 @@ - 6.2: `ctrlstatus_fill` role-branches committed sourcing — PRIMARY → own `persist.committed_lsn`/`committed_config_tag` (atomic acquire); else → `ha->last_known_committed_lsn` (atomic acquire) + `ha->last_known_committed_config_tag` (plain aligned read, NO `ha->lock` per design — `ha.h:172` is non-atomic, written under lock at `ha_convergence.c:377`). `appliedLsn` unchanged. Timestamp stamped from `CLOCK_REALTIME` (NOT monotonic). Read-only. STALENESS CAVEAT (in-boundary): `last_known_committed_lsn` refreshes at CtrlHello join/RDHUP, not per heartbeat — off-primary committed can lag; live-fresh = OUT of boundary → ha-heartbeat-liveness/ha-repl-hardening. Commit 3799bd6. - 6.3 LEARNING: the controller log timestamp formatter is NOT in `xcelog.h` (an 85-line thin wrapper that just `#include "tlog.h"`); the `YYYY-MM-DD HH:MM:SS,mmm` prefix is assembled inline in vendored tinylog (`tlog.c` `_tlog_root_default_format` ~393/398, via `my_tlog_format` in `src/lib/utils.c:221/228`) — no caller-facing helper, so it was REPLICATED byte-equivalently (`localtime_r` + `strftime "%Y-%m-%d %H:%M:%S"` + `,%03u` of `ms%1000`) in a new header-only `src/controller/testsuite/ctrlstatus_fmt.h` (`ctrlstatus_format_time_ms`), shared by `test.c` (`cmd_ctrlstatus` prints `responderTime:` after `fenced`) and `test_ctrlstatus.c` (TZ=UTC-pinned exact-string unit test). LOCAL time on both sides — comparable to `ctrl-N.log` under the same TZ. - 6.4 VERIFIED 2026-07-02 (previously HAND-OFF, blocked on overlayfs — unblocked by ha-commit-durability 6.1 confirming `~/tmp` is a genuine ext4 fs in this sandbox): deployed a 3-node cluster to `~/tmp/xce-ctrlstatus-deploy` via `src/tools/deploy-ha.sh`, submitted jobs against the full masters list, then ran `ctrlstatus` pointed directly at standby ctrl-2 (`set masters localhost:9992`). Result: `answered by ctrl-2 (standby)`, `committedLsn: 4` / `appliedLsn: 4` — non-zero, matching, no inversion (the exact 2026-06-20 bug symptom does not reproduce). `responderTime: 2026-07-02 14:18:19,689` is a live, current wall-clock value consistent with the session timeline. NOTE on the log-correlation half of the Observable: `ctrlstatus_fill` (ctrlstatus.c) has no logging call for the query itself (pure snapshot-fill, no INFO/TRACE line), so there is no literal `ctrl-N.log` line to string-match against — verified the weaker-but-sufficient form instead (responderTime is live/plausible, not frozen/stale/wrong-TZ). PASS. Found and logged (unrelated, low-priority): bug-1400 (xce_test cmd_jobsub SEGVs on malformed positional args instead of required getopt flags — test-client input validation gap, not a production bug) and bug-1401 (ctrlstatus reply decode leaks ~120 bytes per call in `decode_CtrlStatus_ptr`, ASAN-only visibility, ~103 allocs/12KB observed over a dozen calls in this session). + +## 8. Delta: per-controller connection status (Req 13) +> Sequential, single request path (schema → typemap → `ctrlstatus_fill` → render → E2E); no `(P)`. Read-only reporting only — this delta must not change how standby connections are established, monitored, evicted, or torn down (Req 13.5); it only reads the existing replication connection registry (`replicator_t.standbys[]`) that already tracks each standby's live connection and catch-up state for replication/quorum purposes. + +- [x] 8.1 Add the peer connection-status field to the schema + hand-authored typemap + - Add a `status` field to the `CtrlPeer` reply struct (next free ordinal, append-only — after `port`), carrying a small fixed-value code (0=unknown, 1=ok, 2=catching-up, 3=dead), and the matching field to the hand-authored `xce_ctrlpeer_t` typedef in `src/include/typemap.h`, verified field-for-field against the regenerated codec. + - Observable: the build regenerates the reply codec clean with the new field present, the typemap matches the codec, and a codec round-trip carries the field's value unchanged (defaulting to 0/unknown when unset). + - _Requirements: 13.1_ + - _Boundary: schema + typemap_ + +- [ ] 8.2 Source each configured controller's connection status in the status handler + - Confirm the exact field path from the controller's top-level cluster state to the replication connection registry (the research item carried from design) before wiring the read below. + - TDD (red first, in `test_ctrlstatus.c`): a PRIMARY responder with three peer connection states (one live and caught up, one live and catching up, one with no live connection) reports ok/catching-up/dead respectively for those three peers, and ok for its own entry; a STANDBY responder reports ok for its own entry and unknown for every other configured peer, including the primary; a fixture whose own identity matches no configured peer (misconfiguration case) still produces a defined, non-crashing result via the normal non-self sourcing (no entry force-corrected to ok) and logs a one-time warning; the existing no-mutation assertion is extended to cover whatever connection-registry state this task reads. + - Implement the sourcing: a peer matching the responder's own identity always reports ok; when the responder is currently primary, every other peer's status is sourced by matching it against the live replication connection registry (no live connection → dead; live but not yet caught up → catching-up; live and caught up → ok); when the responder is not primary, every other peer's status is reported as unknown, since only the primary has direct connection visibility into every other node. The read stays bounded (a lock-protected scan of a small, fixed-size in-memory registry, no network I/O or unbounded wait) and mutates no state. + - Observable: the new unit tests go red → green; the primary, standby, and misconfiguration-fallback cases each report the values above; the no-mutation assertion holds. + - _Requirements: 13.1, 13.2, 13.3, 13.4, 13.5_ + - _Boundary: ctrlstatus_fill_ + - _Depends: 8.1_ + +- [ ] 8.3 Render each configured controller's connection status in the `ctrlstatus` client command + - Add a status-code-to-string renderer covering all four values plus a safe fallback for an unrecognized code (mirroring the existing responder-role renderer's convention), and extend the client command's existing per-controller line to print the rendered status alongside that controller's name and address. + - Observable: piping `ctrlstatus` through the client prints each configured controller's connection status alongside its existing name:address line. + - _Requirements: 13.1, 13.6_ + - _Boundary: cmd_ctrlstatus_ + - _Depends: 8.2_ + +- [ ] 8.4 End-to-end validation on a live cluster + - Run against a live 3-node cluster: query `ctrlstatus` on the primary of a healthy cluster and confirm all three configured controllers (including itself) show ok; kill one standby and re-query the primary, confirming that standby now shows dead while the other two still show ok; query a surviving standby directly and confirm its own entry shows ok while every other entry, including the primary, shows unknown — proving the asymmetric-visibility design decision holds live, not just in a unit test. + - Observable: the live run shows the ok/dead transition on the primary's view and the unknown-for-others view on the standby's view exactly as described; manual environment prerequisite (live cluster), same status as the other live-E2E tasks (5.1, 6.4, 7.6). + - _Requirements: 13.1, 13.2, 13.3, 13.4, 13.6_ + - _Depends: 8.2, 8.3_ + +- [ ]* 8.5 (Optional) Observe transient catching-up status on a live cluster + - If the project's E2E tooling can hold a restarted standby in catch-up long enough to query it, restart a previously-killed standby and query the primary shortly after, confirming that peer transiently shows catching-up before settling to ok. + - Observable: a live run captures at least one `ctrlstatus` query showing catching-up for a rejoining standby before it converges to ok. + - _Requirements: 13.1_ + - _Depends: 8.4_ diff --git a/src/controller/testsuite/test_ctrlstatus.c b/src/controller/testsuite/test_ctrlstatus.c index e1478715..c465bc9d 100644 --- a/src/controller/testsuite/test_ctrlstatus.c +++ b/src/controller/testsuite/test_ctrlstatus.c @@ -580,7 +580,85 @@ static void test_ctrlstatus_electedatms_roundtrip(void) printf("test_ctrlstatus_electedatms_roundtrip: PASS\n"); } -/* ── (l) codec round-trip: HeartbeatMsg.electedAtMs @4 (task 7.1, Req 12.1) ── +/* ── (l) codec round-trip: CtrlPeer.status @3 (task 8.1, Req 13.1) ── + * Design delta "Design Delta — Requirement 13", Schema delta subsection: a new + * append-only UInt8 status code on the nested CtrlPeer list element (next + * free ordinal after port @2). Schema/codec-level only -- ctrlstatus_fill + * does not source real status values yet (task 8.2, out of boundary here). + * Proves a CtrlPeer.status value set to a non-default code (2 == catching-up) + * survives a genuine CtrlStatus encode -> bytes -> decode cycle unchanged, + * and that a peer whose status is left unset defaults to 0 (unknown) on the + * wire -- mirroring the peers-list construction pattern in + * src/lib/testsuite/test_ctrlstatus.c's test_ctrlstatus_roundtrip_all_fields. + */ +static void test_ctrlpeer_status_roundtrip(void) +{ + xce_ctrlpeer_t p0 = { + .name = "ctrl-a", .host = "10.0.0.1", .port = 9991, .status = 2 + }; + xce_ctrlpeer_t p1 = { + .name = "ctrl-b", .host = "10.0.0.2", .port = 9992 + /* .status intentionally left unset -- must default to 0 (unknown) */ + }; + xce_ctrlpeer_t* peerlist[2] = { &p0, &p1 }; + + xce_ctrlstatus_t in; + memset(&in, 0, sizeof(in)); + in.clusterName = "lab-cluster"; + in.responderName = "ctrl-a"; + in.peers.count = 2; + in.peers.list = peerlist; + in.peerCount = 2; + + struct capn c; + struct capn_segment* cs; + struct CtrlStatus cap_in; + CtrlStatus_ptr sp; + + capn_init_malloc(&c); + cs = capn_root(&c).seg; + + memset(&cap_in, 0, sizeof(cap_in)); + encode_CtrlStatus(cs, &cap_in, &in); + + sp = new_CtrlStatus(cs); + write_CtrlStatus(&cap_in, sp); + capn_setp(capn_root(&c), 0, sp.p); + + size_t sz = capn_size(&c); + uint8_t* buf = (uint8_t*)malloc(sz); + TEST_ASSERT_NOT_NULL(buf); + TEST_ASSERT_EQUAL_size_t(sz, capn_write_mem(&c, buf, sz, 0)); + capn_free(&c); + + struct capn dc; + struct CtrlStatus cap_out; + CtrlStatus_ptr dp; + xce_ctrlstatus_t out; + + capn_init_mem(&dc, buf, sz, 0); + dp.p = capn_getp(capn_root(&dc), 0, 1); + read_CtrlStatus(&cap_out, dp); + memset(&out, 0, sizeof(out)); + decode_CtrlStatus(&out, &cap_out); + + TEST_ASSERT_EQUAL_INT(2, out.peers.count); + TEST_ASSERT_NOT_NULL(out.peers.list); + TEST_ASSERT_NOT_NULL(out.peers.list[0]); + TEST_ASSERT_EQUAL_STRING("ctrl-a", out.peers.list[0]->name); + TEST_ASSERT_EQUAL_UINT8(2, out.peers.list[0]->status); + TEST_ASSERT_NOT_NULL(out.peers.list[1]); + TEST_ASSERT_EQUAL_STRING("ctrl-b", out.peers.list[1]->name); + TEST_ASSERT_EQUAL_UINT8(0, out.peers.list[1]->status); + + free_CtrlStatus(&out); + capn_free(&dc); + free(buf); + + printf("test_ctrlpeer_status_roundtrip: PASS\n"); +} + +/* ── (m) codec round-trip: HeartbeatMsg.electedAtMs @4 (task 7.1, Req 12.1) ── * Same design-delta schema addition on the heartbeat message the primary * already sends to standbys, next free ordinal after committedConfigTag @3. * Direct struct construction + encode/decode -- no heartbeat send/receive @@ -721,6 +799,7 @@ void test_ctrlstatus(void) RUN_TEST(test_ctrlstatus_primary_reports_own_committed); RUN_TEST(test_ctrlstatus_responder_time_ms); RUN_TEST(test_ctrlstatus_electedatms_roundtrip); + RUN_TEST(test_ctrlpeer_status_roundtrip); RUN_TEST(test_heartbeatmsg_electedatms_roundtrip); RUN_TEST(test_ctrlstatus_primary_reports_own_elected_at_ms); RUN_TEST(test_ctrlstatus_standby_reports_learned_elected_at_ms); diff --git a/src/include/typemap.h b/src/include/typemap.h index 382909cb..cc16b587 100644 --- a/src/include/typemap.h +++ b/src/include/typemap.h @@ -264,6 +264,7 @@ typedef struct { char* name; char* host; uint32_t port; + uint8_t status; /* @3: 0=unknown 1=ok 2=catching-up 3=dead (Req 13.1) */ } xce_ctrlpeer_t; typedef struct { diff --git a/src/schema/controller.capnp b/src/schema/controller.capnp index 50909cb9..e2ec0028 100644 --- a/src/schema/controller.capnp +++ b/src/schema/controller.capnp @@ -534,6 +534,10 @@ struct CtrlPeer $C.mapname("xce_ctrlpeer_t") { name @0: Text; host @1: Text; port @2: UInt32; + status @3: UInt8 $C.mapname("status"); + # 0=unknown 1=ok 2=catching-up 3=dead (Req 13.1). Always 1 (ok) for the + # responder's own entry (Req 13.2). On a non-primary responder, every + # entry but the responder's own is 0 (unknown) — Req 13.4/Locked Decision 5. } struct CtrlStatus $C.mapname("xce_ctrlstatus_t") {