docs(controller-status): mark task 8.4 complete -- live E2E validation (Req 13 delta)
Two independent live 3-node-cluster runs both confirmed the full Requirement 13 scenario (healthy all-ok, dead-on-kill, unknown-from- standby-view) with zero bugs found. No source changes -- pure validation task. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
parent
64b18d5c34
commit
464489d972
1 changed files with 3 additions and 1 deletions
|
|
@ -158,6 +158,8 @@
|
|||
- 6.1: `responderTimeMs @13: UInt64` appended to `CtrlStatus` (after `peerCount @12`, append-only) + matching `uint64_t responder_time_ms` in the hand-authored `xce_ctrlstatus_t` (`src/include/typemap.h`); codec wire offset 48, round-trips. Commit 65ddafe.
|
||||
- 6.2: `ctrlstatus_fill` role-branches committed sourcing — PRIMARY → own `persist.committed_lsn`/`committed_config_tag` (atomic acquire); else → `ha->last_known_committed_lsn` (atomic acquire) + `ha->last_known_committed_config_tag` (plain aligned read, NO `ha->lock` per design — `ha.h:172` is non-atomic, written under lock at `ha_convergence.c:377`). `appliedLsn` unchanged. Timestamp stamped from `CLOCK_REALTIME` (NOT monotonic). Read-only. STALENESS CAVEAT (in-boundary): `last_known_committed_lsn` refreshes at CtrlHello join/RDHUP, not per heartbeat — off-primary committed can lag; live-fresh = OUT of boundary → ha-heartbeat-liveness/ha-repl-hardening. Commit 3799bd6.
|
||||
- 6.3 LEARNING: the controller log timestamp formatter is NOT in `xcelog.h` (an 85-line thin wrapper that just `#include "tlog.h"`); the `YYYY-MM-DD HH:MM:SS,mmm` prefix is assembled inline in vendored tinylog (`tlog.c` `_tlog_root_default_format` ~393/398, via `my_tlog_format` in `src/lib/utils.c:221/228`) — no caller-facing helper, so it was REPLICATED byte-equivalently (`localtime_r` + `strftime "%Y-%m-%d %H:%M:%S"` + `,%03u` of `ms%1000`) in a new header-only `src/controller/testsuite/ctrlstatus_fmt.h` (`ctrlstatus_format_time_ms`), shared by `test.c` (`cmd_ctrlstatus` prints `responderTime:` after `fenced`) and `test_ctrlstatus.c` (TZ=UTC-pinned exact-string unit test). LOCAL time on both sides — comparable to `ctrl-N.log` under the same TZ.
|
||||
- 8.4 VERIFIED 2026-09-16: two independent live 3-node-cluster runs (implementer + reviewer, separate deploy dirs/ports) both confirmed the full Req 13 scenario cleanly with ZERO bugs found (unlike 5.1/6.4/7.6, which each found+fixed a real bug) — healthy primary reports all 3 controllers `ok`; killing a standby flips it to `dead` on the primary's view while the other two stay `ok`; querying the surviving standby directly shows `ok` for itself and `unknown` for every other peer, including the still-alive primary, confirming the asymmetric-visibility design decision (Locked Decision 5) holds live. The tasks 8.1-8.3 implementation required no live-only fixes.
|
||||
|
||||
- 8.2 LEARNING: the new `replicator_t.lock` acquisition in `ctrlstatus_fill`'s peer-building loop is held around the whole per-peer loop (calloc + self/primary/non-primary branching for all configured peers), slightly wider than `repl_active_standby_count`'s own lock-and-scan shape that design.md's Constraints-checked note cites as the precedent. Still bounded/safe (n ≤ `MAX_HA_CANDIDATES`, inner scan ≤ `MAX_STANDBYS`, no I/O) and reviewer-approved as non-blocking, but a future touch to this loop should narrow the lock to wrap only the inner `MAX_STANDBYS` registry scan per non-self peer, matching the cited precedent exactly.
|
||||
|
||||
- 6.4 VERIFIED 2026-07-02 (previously HAND-OFF, blocked on overlayfs — unblocked by ha-commit-durability 6.1 confirming `~/tmp` is a genuine ext4 fs in this sandbox): deployed a 3-node cluster to `~/tmp/xce-ctrlstatus-deploy` via `src/tools/deploy-ha.sh`, submitted jobs against the full masters list, then ran `ctrlstatus` pointed directly at standby ctrl-2 (`set masters localhost:9992`). Result: `answered by ctrl-2 (standby)`, `committedLsn: 4` / `appliedLsn: 4` — non-zero, matching, no inversion (the exact 2026-06-20 bug symptom does not reproduce). `responderTime: 2026-07-02 14:18:19,689` is a live, current wall-clock value consistent with the session timeline. NOTE on the log-correlation half of the Observable: `ctrlstatus_fill` (ctrlstatus.c) has no logging call for the query itself (pure snapshot-fill, no INFO/TRACE line), so there is no literal `ctrl-N.log` line to string-match against — verified the weaker-but-sufficient form instead (responderTime is live/plausible, not frozen/stale/wrong-TZ). PASS. Found and logged (unrelated, low-priority): bug-1400 (xce_test cmd_jobsub SEGVs on malformed positional args instead of required getopt flags — test-client input validation gap, not a production bug) and bug-1401 (ctrlstatus reply decode leaks ~120 bytes per call in `decode_CtrlStatus_ptr`, ASAN-only visibility, ~103 allocs/12KB observed over a dozen calls in this session).
|
||||
|
|
@ -187,7 +189,7 @@
|
|||
- _Boundary: cmd_ctrlstatus_
|
||||
- _Depends: 8.2_
|
||||
|
||||
- [ ] 8.4 End-to-end validation on a live cluster
|
||||
- [x] 8.4 End-to-end validation on a live cluster
|
||||
- Run against a live 3-node cluster: query `ctrlstatus` on the primary of a healthy cluster and confirm all three configured controllers (including itself) show ok; kill one standby and re-query the primary, confirming that standby now shows dead while the other two still show ok; query a surviving standby directly and confirm its own entry shows ok while every other entry, including the primary, shows unknown — proving the asymmetric-visibility design decision holds live, not just in a unit test.
|
||||
- Observable: the live run shows the ok/dead transition on the primary's view and the unknown-for-others view on the standby's view exactly as described; manual environment prerequisite (live cluster), same status as the other live-E2E tasks (5.1, 6.4, 7.6).
|
||||
- _Requirements: 13.1, 13.2, 13.3, 13.4, 13.6_
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue