fix(controller-status): ctrlstatus_fill must prefer a live registry slot over a stale disconnected one (task 8.6, bug-1819)
Found live during task 8.5's optional catching-up-observation attempt: a killed-then-reconnected standby can end up occupying a NEW replicator_t.standbys[] slot while its OLD slot (host/port still populated, ep == NULL) also matches the same configured peer. ctrlstatus_fill's primary-side peer-status loop stopped at the first host:port match, so it permanently reported that peer dead even minutes after it had genuinely reconnected and fully caught up -- violating Requirement 13.3's "true, currently-known connection status" guarantee. Fix: scan all MAX_STANDBYS slots for a given peer and prefer any slot with ep != NULL over a slot with ep == NULL; fall back to dead only when no matching slot is live. Read-only, lock scope unchanged from task 8.2. TDD RED->GREEN (1216 Tests: 2 Failures -> 0), independently reproduced by the reviewer (exact RED replay) plus two independent live kill/backlog/restart/poll cycles on separate fresh clusters, both confirming the previously-stuck standby now converges to ok. design.md's Locked Decision 3 corrected to document the fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
parent
464489d972
commit
6c437e2bed
5 changed files with 790 additions and 440 deletions
|
|
@ -426,7 +426,7 @@ User enhancement request: `ctrlstatus`'s "Controllers (N):" list should addition
|
|||
### Locked design decisions
|
||||
1. **Wire type: `CtrlPeer.status @3: UInt8`**, a small fixed-value code — not `Text`. Mirrors the existing `CtrlStatus.responderRole @3: UInt8` precedent exactly (backed by a C enum + a `ctrlstatus_..._str()` string renderer at the client layer), rather than introducing a new stringly-typed convention for this one field.
|
||||
2. **Status values, in ordinal order:** `0 = unknown`, `1 = ok`, `2 = catching-up`, `3 = dead`. `unknown` is `0` (the default/zero value) so a peer entry that nothing ever sets defaults to the safe, honest "can't tell" state rather than an implicitly-true `ok`.
|
||||
3. **Primary-side sourcing: match by host+port, not by name or position.** `standby_conn_t` (the primary's live connection-tracking slot) stores `host`/`port` but not a configured peer name or a stable index into `peers_cfg.candidates[]` — so `ctrlstatus_fill`, when the responder is primary, must scan `replicator_t.standbys[]` for a slot whose `host`/`port` match each configured peer's `hostname`/`port`. No match (or a matched slot with `ep == NULL`) → `dead`; a match with `ep != NULL && in_catch_up` → `catching-up`; a match with `ep != NULL && !in_catch_up` → `ok`. This mirrors `repl_active_standby_count`'s own existing `ep != NULL && !in_catch_up` predicate (`repl.c`) — reuse that exact condition rather than inventing a new one.
|
||||
3. **Primary-side sourcing: match by host+port, not by name or position.** `standby_conn_t` (the primary's live connection-tracking slot) stores `host`/`port` but not a configured peer name or a stable index into `peers_cfg.candidates[]` — so `ctrlstatus_fill`, when the responder is primary, must scan `replicator_t.standbys[]` for a slot whose `host`/`port` match each configured peer's `hostname`/`port`. No match (or a matched slot with `ep == NULL`) → `dead`; a match with `ep != NULL && in_catch_up` → `catching-up`; a match with `ep != NULL && !in_catch_up` → `ok`. This mirrors `repl_active_standby_count`'s own existing `ep != NULL && !in_catch_up` predicate (`repl.c`) — reuse that exact condition rather than inventing a new one. **Correction (task 8.6, 2026-09-16, live-E2E finding):** the scan must NOT stop at the first `host`/`port` match. A standby that is killed and later reconnects can end up occupying a NEW `replicator_t.standbys[]` slot while its OLD slot (host/port still populated, `ep == NULL`) also still matches the same configured peer — `repl_add_standby`'s `ep`-identity idempotency check cannot reuse the old slot once its `ep` is gone. A first-match-wins scan then reports that peer `dead` PERMANENTLY, even after it has genuinely reconnected and fully caught up (live-reproduced: `ctrlstatus` stayed `dead` for 5+ minutes after a rejoin the primary's own log showed as fully caught up), violating Req 13.3's "true, currently-known" guarantee. The corrected algorithm scans ALL `MAX_STANDBYS` slots for a given peer and prefers ANY slot with `ep != NULL` (deriving `ok`/`catching-up` from that live slot) over a slot with `ep == NULL`; only falls back to `dead` when no matching slot has `ep != NULL`. See `.wolf/buglog.json` bug-1819.
|
||||
4. **The responder always reports its own entry as `ok`** (Req 13.2) — matched by comparing each configured peer's name against `st->responderName`, the same identity `ctrlstatus_fill` already establishes for itself.
|
||||
5. **Non-primary responder: only self is determinable; every other peer — including the primary — reports `unknown`.** This was an open question at requirements time (whether a standby could additionally report its own primary's status via the liveness signal it already tracks for failover); design-time investigation closes it: that liveness signal (`ha->last_activity`, `liveness_poll_fn`) is a bare elapsed-time scalar with no peer identity attached, and this codebase has no other mechanism (analogous to the vestigial `ha->primary_host`) that would let a standby reliably attribute it to one specific configured peer. Building one would be materially larger than this delta (a genuinely new identity-carrying signal on the heartbeat/connection-accept path) and is explicitly out of boundary here — a future spec's concern, not silently absorbed into this one.
|
||||
6. **Read-only, no role-branch beyond what Decision 3/5 already requires**: `ctrlstatus_fill` performs only reads (of `peers_cfg`, and — only when the responder is primary — of `replicator_t.standbys[]` under its existing lock); no new state is written, no existing connection-management behavior changes.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue