From 196fa66387c2d040e0d771c719e9caf0fc2b8e3a Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Wed, 16 Sep 2026 09:23:34 +0000 Subject: [PATCH] docs(controller-status): mark task 7.6 complete, record live-E2E outcome Requirement 12 delta (primary election term/time) is now fully implemented: tasks 7.1-7.6 all complete, including the live 3-node failover validation that found and confirmed the fix for bug-1818. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC --- .kiro/specs/controller-status/tasks.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.kiro/specs/controller-status/tasks.md b/.kiro/specs/controller-status/tasks.md index 2cd58bff..2605c934 100644 --- a/.kiro/specs/controller-status/tasks.md +++ b/.kiro/specs/controller-status/tasks.md @@ -131,12 +131,17 @@ - _Boundary: cmd_ctrlstatus_ - _Depends: 7.4_ -- [ ] 7.6 End-to-end validation on a live cluster +- [x] 7.6 End-to-end validation on a live cluster - Run against a live 3-node cluster: query the primary right after a fresh election and confirm its printed election term matches its current term and its election time is close to "now"; query a standby and confirm it reports the SAME election term and election time as the primary (not "unknown", not stale) once at least one heartbeat has been exchanged; kill the primary, wait for a new election, and confirm the surviving standby's reported election term/time updates to the new primary's values within about one heartbeat interval. - Observable: the live run shows matching election term/time between the primary and a standby, and shows the standby's values converge to the new primary's after a kill within one heartbeat interval; manual environment prerequisite (live cluster), same status as the other live-E2E tasks (5.1, 6.4). - _Requirements: 12.1, 12.2, 12.4, 12.6_ - _Depends: 7.2, 7.3, 7.4, 7.5_ +## Implementation Notes (7.6 E2E outcome, Req 12 delta) +- First live run FOUND + FIXED a real bug (bug-1818, `.wolf/buglog.json`): `cmd_ctrlstatus` gated `electedTerm`/`electedAt` on `NULLSTR(status->primaryName)` (mirroring the `Primary:` line, per the original design.md Locked Decision 3) — but `primaryName` is unconditionally empty on every non-primary responder in this build (the pre-existing, vestigial-`primary_host` behavior from Req 2/`ade0081`), so the two new fields printed "unknown" on every standby, forever, despite the underlying `elected_at_ms` being correctly heartbeat-learned and non-stale the whole time (`term`/`committedLsn`/`appliedLsn`, sourced the same unconditional way but not primaryName-gated, correctly displayed and converged on the same standby query). All unit tests passed throughout — the bug lived entirely in the untested CLI print-gate, exactly the class of defect only a live cluster run could catch. +- Fix (commit `80839661`): gate on `status->elected_at_ms == 0` instead — necessary and sufficient, since that field only ever becomes nonzero via a real completed election (either promotion road) or genuine heartbeat-learning from an actual primary, unlike `term` which can bump from mere candidacy activity. `ctrlstatus_fill` itself was untouched (stays unconditional/no-role-branch, task 7.4's contract intact). design.md's Locked Decision 3 (+ dependent traceability/service-contract text) corrected in the same commit to document this live-E2E finding, mirroring the spec's Req 10/11 delta convention. +- Retry PASSED cleanly on a fresh 3-node deploy: primary ctrl-1 self-reported `electedTerm: 2` / `electedAt: 2026-09-16 09:12:02,485` (byte-for-byte matching its own `raft_cb_become_leader` log line); standby ctrl-2/ctrl-3 both reported the IDENTICAL values after ≥1 heartbeat interval (previously "unknown"). After killing ctrl-1, ctrl-2 was elected (term=3, `wait-primary` in 2253ms); the SAME surviving standby (ctrl-3, queried both before and after the kill) converged from `electedTerm: 2` to `electedTerm: 3` matching the new primary exactly, within ~18s (well under the ~5-heartbeat-interval budget). Review independently cross-checked every reported value/timestamp against the raw daemon logs still on disk, not just the transcript. + ## Implementation Notes (5.1 E2E outcome) - 5.1 ran live on a 3-node `/xce-deploy-ha`-style cluster (ext2/3 fs). PASSED in-scope: `ctrlstatus` shows the primary self-reporting (`Primary: ctrl-1` on the primary), standbys honestly report `Primary: unknown` with `answered by ctrl-2 (standby)` (the dispatch-gate lets standbys answer — Req 7); peers + count correct; `configChecksum` = the real 64-hex SHA-256 (matched the deploy tag prefix) and is IDENTICAL across all 3 nodes with identical config (Req 5.3); `configVersion`==`committedConfigVersion`; authoritative `wait-primary` names the primary in <1s on a serveable cluster (Req 9.1/9.2). - 5.1 FOUND + FIXED a real 2.1 bug: `ha->primary_host` is VESTIGIAL (never written anywhere in the controller), and `voted_for_name` is the last-vote (often self), so neither names the current primary off-primary. Corrected: PRIMARY self-reports its own name; STANDBY/ELECTING/INIT report empty → "unknown" (the authoritative client path reaches the real primary by rotation). Commit ade0081.