From 4388d99306ab18a9c3fa78242613640467522a6f Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 20 Jun 2026 09:30:44 +0000 Subject: [PATCH] docs(controller-status): mark 6.4 pending live-cluster validation (manual env) 6.1-6.3 committed + unit-green. 6.4 (off-primary committed + log correlation E2E) needs a real-fs /xce-deploy-ha run; sandbox overlayfs fdatasync errno=5 (bug-795) breaks the commit path. Hand-off recipe recorded in Implementation Notes. Co-Authored-By: Claude Opus 4.8 (1M context) --- .kiro/specs/controller-status/tasks.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.kiro/specs/controller-status/tasks.md b/.kiro/specs/controller-status/tasks.md index 1a11f853..e0ed32e0 100644 --- a/.kiro/specs/controller-status/tasks.md +++ b/.kiro/specs/controller-status/tasks.md @@ -88,6 +88,7 @@ - Observable: the live run shows a non-zero off-primary committed LSN (no zero-while-applied inversion) and a responder timestamp that lines up with the standby's log; manual environment prerequisite (live cluster), same status as the other live-E2E task (5.1). - _Requirements: 10.1, 10.3, 11.2, 11.3_ - _Depends: 6.2, 6.3_ + - _Blocked: live-cluster manual-env prerequisite — must run on a real-fs `/xce-deploy-ha` deployment (e.g. /tmp/xce-ha-deploy). This sandbox is overlayfs: `fdatasync errno=5` (bug-795) breaks the primary's commit path, so the commit→learn durability flow 6.4 exercises cannot run here. 6.1–6.3 are committed + unit-green; hand-off recipe in Implementation Notes._ ## Implementation Notes (5.1 E2E outcome) - 5.1 ran live on a 3-node `/xce-deploy-ha`-style cluster (ext2/3 fs). PASSED in-scope: `ctrlstatus` shows the primary self-reporting (`Primary: ctrl-1` on the primary), standbys honestly report `Primary: unknown` with `answered by ctrl-2 (standby)` (the dispatch-gate lets standbys answer — Req 7); peers + count correct; `configChecksum` = the real 64-hex SHA-256 (matched the deploy tag prefix) and is IDENTICAL across all 3 nodes with identical config (Req 5.3); `configVersion`==`committedConfigVersion`; authoritative `wait-primary` names the primary in <1s on a serveable cluster (Req 9.1/9.2). @@ -105,3 +106,4 @@ - 6.1: `responderTimeMs @13: UInt64` appended to `CtrlStatus` (after `peerCount @12`, append-only) + matching `uint64_t responder_time_ms` in the hand-authored `xce_ctrlstatus_t` (`src/include/typemap.h`); codec wire offset 48, round-trips. Commit 65ddafe. - 6.2: `ctrlstatus_fill` role-branches committed sourcing — PRIMARY → own `persist.committed_lsn`/`committed_config_tag` (atomic acquire); else → `ha->last_known_committed_lsn` (atomic acquire) + `ha->last_known_committed_config_tag` (plain aligned read, NO `ha->lock` per design — `ha.h:172` is non-atomic, written under lock at `ha_convergence.c:377`). `appliedLsn` unchanged. Timestamp stamped from `CLOCK_REALTIME` (NOT monotonic). Read-only. STALENESS CAVEAT (in-boundary): `last_known_committed_lsn` refreshes at CtrlHello join/RDHUP, not per heartbeat — off-primary committed can lag; live-fresh = OUT of boundary → ha-heartbeat-liveness/ha-repl-hardening. Commit 3799bd6. - 6.3 LEARNING: the controller log timestamp formatter is NOT in `xcelog.h` (an 85-line thin wrapper that just `#include "tlog.h"`); the `YYYY-MM-DD HH:MM:SS,mmm` prefix is assembled inline in vendored tinylog (`tlog.c` `_tlog_root_default_format` ~393/398, via `my_tlog_format` in `src/lib/utils.c:221/228`) — no caller-facing helper, so it was REPLICATED byte-equivalently (`localtime_r` + `strftime "%Y-%m-%d %H:%M:%S"` + `,%03u` of `ms%1000`) in a new header-only `src/controller/testsuite/ctrlstatus_fmt.h` (`ctrlstatus_format_time_ms`), shared by `test.c` (`cmd_ctrlstatus` prints `responderTime:` after `fenced`) and `test_ctrlstatus.c` (TZ=UTC-pinned exact-string unit test). LOCAL time on both sides — comparable to `ctrl-N.log` under the same TZ. +- 6.4 HAND-OFF (live-cluster manual validation — run on a real-fs deploy, NOT the overlayfs sandbox): on a `/xce-deploy-ha` 3-node cluster, (1) `jobsub` a few jobs so the primary advances committedLsn; (2) `ctrlstatus` against a STANDBY (the any-node hint view — e.g. point the client at a standby, or use the non-authoritative path) and confirm its `committedLsn` is NON-ZERO (tracking the primary within the join-refresh caveat) instead of the old `0`-while-`appliedLsn`-nonzero inversion from the 2026-06-20 bug report; (3) confirm the new `responderTime:` line matches that standby's own `ctrl-N.log` timestamp for the same query (same TZ). PASS = standby committedLsn no longer 0 while applied>0, and responderTime lines up with the log. This is the live confirmation of the 2026-06-20 finding that opened this delta.