feat(controller-status): source per-controller connection status in ctrlstatus_fill (task 8.2)
Populates CtrlPeer.status (added in task 8.1) inside ctrlstatus_fill's peer-building loop: self always reports ok; a primary responder sources every other peer from the live replication connection registry (replicator_t.standbys[], host+port matched, reusing repl_active_standby_count's ep!=NULL && !in_catch_up predicate) -- dead/catching-up/ok; a non-primary responder reports unknown for every non-self peer. A self-identity mismatch (pre-existing misconfiguration) logs one WARN and falls through to the normal non-self sourcing rather than forcing any peer to ok. Read-only: a bounded lock-protected scan, no state mutation. TDD: 3 new tests (primary-sourced-from-registry, standby-unknown-except- self, self-mismatch-falls-through) plus an extended no-mutation assertion; RED->GREEN independently reproduced by the reviewer. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
parent
1b278c0e9a
commit
ab5dd91537
3 changed files with 288 additions and 1 deletions
|
|
@ -158,6 +158,8 @@
|
|||
- 6.1: `responderTimeMs @13: UInt64` appended to `CtrlStatus` (after `peerCount @12`, append-only) + matching `uint64_t responder_time_ms` in the hand-authored `xce_ctrlstatus_t` (`src/include/typemap.h`); codec wire offset 48, round-trips. Commit 65ddafe.
|
||||
- 6.2: `ctrlstatus_fill` role-branches committed sourcing — PRIMARY → own `persist.committed_lsn`/`committed_config_tag` (atomic acquire); else → `ha->last_known_committed_lsn` (atomic acquire) + `ha->last_known_committed_config_tag` (plain aligned read, NO `ha->lock` per design — `ha.h:172` is non-atomic, written under lock at `ha_convergence.c:377`). `appliedLsn` unchanged. Timestamp stamped from `CLOCK_REALTIME` (NOT monotonic). Read-only. STALENESS CAVEAT (in-boundary): `last_known_committed_lsn` refreshes at CtrlHello join/RDHUP, not per heartbeat — off-primary committed can lag; live-fresh = OUT of boundary → ha-heartbeat-liveness/ha-repl-hardening. Commit 3799bd6.
|
||||
- 6.3 LEARNING: the controller log timestamp formatter is NOT in `xcelog.h` (an 85-line thin wrapper that just `#include "tlog.h"`); the `YYYY-MM-DD HH:MM:SS,mmm` prefix is assembled inline in vendored tinylog (`tlog.c` `_tlog_root_default_format` ~393/398, via `my_tlog_format` in `src/lib/utils.c:221/228`) — no caller-facing helper, so it was REPLICATED byte-equivalently (`localtime_r` + `strftime "%Y-%m-%d %H:%M:%S"` + `,%03u` of `ms%1000`) in a new header-only `src/controller/testsuite/ctrlstatus_fmt.h` (`ctrlstatus_format_time_ms`), shared by `test.c` (`cmd_ctrlstatus` prints `responderTime:` after `fenced`) and `test_ctrlstatus.c` (TZ=UTC-pinned exact-string unit test). LOCAL time on both sides — comparable to `ctrl-N.log` under the same TZ.
|
||||
- 8.2 LEARNING: the new `replicator_t.lock` acquisition in `ctrlstatus_fill`'s peer-building loop is held around the whole per-peer loop (calloc + self/primary/non-primary branching for all configured peers), slightly wider than `repl_active_standby_count`'s own lock-and-scan shape that design.md's Constraints-checked note cites as the precedent. Still bounded/safe (n ≤ `MAX_HA_CANDIDATES`, inner scan ≤ `MAX_STANDBYS`, no I/O) and reviewer-approved as non-blocking, but a future touch to this loop should narrow the lock to wrap only the inner `MAX_STANDBYS` registry scan per non-self peer, matching the cited precedent exactly.
|
||||
|
||||
- 6.4 VERIFIED 2026-07-02 (previously HAND-OFF, blocked on overlayfs — unblocked by ha-commit-durability 6.1 confirming `~/tmp` is a genuine ext4 fs in this sandbox): deployed a 3-node cluster to `~/tmp/xce-ctrlstatus-deploy` via `src/tools/deploy-ha.sh`, submitted jobs against the full masters list, then ran `ctrlstatus` pointed directly at standby ctrl-2 (`set masters localhost:9992`). Result: `answered by ctrl-2 (standby)`, `committedLsn: 4` / `appliedLsn: 4` — non-zero, matching, no inversion (the exact 2026-06-20 bug symptom does not reproduce). `responderTime: 2026-07-02 14:18:19,689` is a live, current wall-clock value consistent with the session timeline. NOTE on the log-correlation half of the Observable: `ctrlstatus_fill` (ctrlstatus.c) has no logging call for the query itself (pure snapshot-fill, no INFO/TRACE line), so there is no literal `ctrl-N.log` line to string-match against — verified the weaker-but-sufficient form instead (responderTime is live/plausible, not frozen/stale/wrong-TZ). PASS. Found and logged (unrelated, low-priority): bug-1400 (xce_test cmd_jobsub SEGVs on malformed positional args instead of required getopt flags — test-client input validation gap, not a production bug) and bug-1401 (ctrlstatus reply decode leaks ~120 bytes per call in `decode_CtrlStatus_ptr`, ASAN-only visibility, ~103 allocs/12KB observed over a dozen calls in this session).
|
||||
|
||||
## 8. Delta: per-controller connection status (Req 13)
|
||||
|
|
@ -169,7 +171,7 @@
|
|||
- _Requirements: 13.1_
|
||||
- _Boundary: schema + typemap_
|
||||
|
||||
- [ ] 8.2 Source each configured controller's connection status in the status handler
|
||||
- [x] 8.2 Source each configured controller's connection status in the status handler
|
||||
- Confirm the exact field path from the controller's top-level cluster state to the replication connection registry (the research item carried from design) before wiring the read below.
|
||||
- TDD (red first, in `test_ctrlstatus.c`): a PRIMARY responder with three peer connection states (one live and caught up, one live and catching up, one with no live connection) reports ok/catching-up/dead respectively for those three peers, and ok for its own entry; a STANDBY responder reports ok for its own entry and unknown for every other configured peer, including the primary; a fixture whose own identity matches no configured peer (misconfiguration case) still produces a defined, non-crashing result via the normal non-self sourcing (no entry force-corrected to ok) and logs a one-time warning; the existing no-mutation assertion is extended to cover whatever connection-registry state this task reads.
|
||||
- Implement the sourcing: a peer matching the responder's own identity always reports ok; when the responder is currently primary, every other peer's status is sourced by matching it against the live replication connection registry (no live connection → dead; live but not yet caught up → catching-up; live and caught up → ok); when the responder is not primary, every other peer's status is reported as unknown, since only the primary has direct connection visibility into every other node. The read stays bounded (a lock-protected scan of a small, fixed-size in-memory registry, no network I/O or unbounded wait) and mutates no state.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue