Found live during task 8.5's optional catching-up-observation attempt:
a killed-then-reconnected standby can end up occupying a NEW
replicator_t.standbys[] slot while its OLD slot (host/port still
populated, ep == NULL) also matches the same configured peer.
ctrlstatus_fill's primary-side peer-status loop stopped at the first
host:port match, so it permanently reported that peer dead even
minutes after it had genuinely reconnected and fully caught up --
violating Requirement 13.3's "true, currently-known connection status"
guarantee.
Fix: scan all MAX_STANDBYS slots for a given peer and prefer any slot
with ep != NULL over a slot with ep == NULL; fall back to dead only
when no matching slot is live. Read-only, lock scope unchanged from
task 8.2.
TDD RED->GREEN (1216 Tests: 2 Failures -> 0), independently reproduced
by the reviewer (exact RED replay) plus two independent live
kill/backlog/restart/poll cycles on separate fresh clusters, both
confirming the previously-stuck standby now converges to ok.
design.md's Locked Decision 3 corrected to document the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Two independent live 3-node-cluster runs both confirmed the full
Requirement 13 scenario (healthy all-ok, dead-on-kill, unknown-from-
standby-view) with zero bugs found. No source changes -- pure
validation task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds ctrlstatus_peer_status_str (mirrors ctrlstatus_role_str's
convention: 0=unknown/1=ok/2=catching-up/3=dead, safe default fallback
for any other code) and extends cmd_ctrlstatus's per-controller print
line to render it alongside each configured controller's existing
name:address, completing the CtrlPeer.status wire field added in task
8.1 and sourced in task 8.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Populates CtrlPeer.status (added in task 8.1) inside ctrlstatus_fill's
peer-building loop: self always reports ok; a primary responder sources
every other peer from the live replication connection registry
(replicator_t.standbys[], host+port matched, reusing
repl_active_standby_count's ep!=NULL && !in_catch_up predicate) --
dead/catching-up/ok; a non-primary responder reports unknown for every
non-self peer. A self-identity mismatch (pre-existing misconfiguration)
logs one WARN and falls through to the normal non-self sourcing rather
than forcing any peer to ok. Read-only: a bounded lock-protected scan,
no state mutation.
TDD: 3 new tests (primary-sourced-from-registry, standby-unknown-except-
self, self-mismatch-falls-through) plus an extended no-mutation
assertion; RED->GREEN independently reproduced by the reviewer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds CtrlPeer.status (UInt8, next free ordinal @3 after port) plus the
matching xce_ctrlpeer_t.status typemap field, verified field-for-field
against the regenerated codec with a genuine encode/decode round-trip
test. No sourcing/rendering logic yet (task 8.2/8.3) -- field defaults
to 0 (unknown) until wired.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Requirement 12 delta (primary election term/time) is now fully
implemented: tasks 7.1-7.6 all complete, including the live 3-node
failover validation that found and confirmed the fix for bug-1818.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
cmd_ctrlstatus (src/harness/test.c) now prints electedTerm/electedAt
right after the Primary: line, gated on the same NULLSTR(primaryName)
"unknown" check that line already uses. electedTerm prints the existing
term field verbatim (no new wire field, per design's Locked Decision 1);
electedAt reuses the existing ctrlstatus_format_time_ms formatter
already used for responderTime, falling back to "unknown" on a NULL
format result. No leakage of a stale term/elected_at_ms when no primary
is known.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
ctrlstatus_fill now unconditionally reads ha->last_known_elected_at_ms
into the reply's elected_at_ms field -- one atomic-acquire load, no
role-branch, mirroring how the handler already reports raw term
unconditionally. Serves both a self-stamped primary and a
heartbeat-learned standby from the same field. "Unknown" display
gating is deliberately left to the client command (task 7.5), not
this handler.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
The primary echoes its own last_known_elected_at_ms on every outgoing
heartbeat (threaded through xce_ha_heartbeat_req's existing scalar-arg
plumbing, alongside term/committed_lsn/committed_config_tag); a standby
processing a received heartbeat (ha_consume_heartbeat_committed, the sole
production call site) stores it unconditionally into its own
last_known_elected_at_ms, reconcile-free -- no other HA field moves.
ha_observe_heartbeat_committed itself is untouched; the new store sits
next to its call site, not inside it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Stamps ha_state_t.last_known_elected_at_ms (CLOCK_REALTIME, epoch-ms) in
both ha_do_promote (non-raft road) and raft_cb_become_leader (raft road),
placed at each road's genuine, final commit point to CTRL_PRIMARY -- past
every early-return refusal path in raft_cb_become_leader (first-boot
phase2 failure, the Live Apply Resource Fence re-check, the promotion
eligibility gate, config-serve-fenced, upgrade-materialization-fenced).
Purely observational: does not change whether/when a promotion succeeds
or is refused. Three new tests (2 positive, 1 negative) prove placement:
a refused promotion leaves the field at its pre-attempt value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds CtrlStatus.electedAtMs@14 and HeartbeatMsg.electedAtMs@4 to the
capnp schema, their hand-authored typemap.h counterparts, and a new
ha_state_t.last_known_elected_at_ms field alongside last_known_committed_lsn.
Schema/state plumbing only -- nothing stamps or consumes these fields yet
(tasks 7.2-7.5). Codec round-trip tests confirm both new fields survive
encode/decode unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds requirements, design, research, and tasks for reporting the term
and wall-clock time the current primary was elected, including
standby-side learned fidelity (mirroring the Req 10 pattern), following
the existing controller-status spec's delta convention (Req 10/11).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
controller-status, ha-commit-engine, ha-heartbeat-committed-propagation,
ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and
raft-ha-core-safety each had every numbered sub-task marked [x] but their
group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic
gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real
unchecked sub-tasks in any of the eight files. No task content changed.
ha-commit-durability: closes the "pending confirmation under simultaneous
total outage" caveat. Reproduced cold-restart-all (kill all 3 members +
restart) 5 times against a live 3-node cluster on genuine ext4 (not the
sandbox's overlayfs) — zero committed-job loss across 40 rolling-failover
rounds + 5 cold-restart-all cycles. Confirms bug-795 was an overlayfs
fdatasync artifact, not a real durability gap; no production fix needed.
Commit Durability is now proven for both rolling single-node failover and
simultaneous total-cluster outage (src/controller/CLAUDE.md updated).
Live-validated and marked complete against a real 3-node cluster:
- controller-status 6.4: off-primary committed fidelity + responder
timestamp, confirmed via a real failover
- ha-heartbeat-committed-propagation 4.1: standby committedLsn tracks the
primary via heartbeat within the 3s interval
- ha-wal-replication 5.4: standby WAL replication verified via committedLsn
match (job-facing ops are correctly primary-only by design; standbys
return E_XCE_STANDBY, not a bug)
- libxce-failover 6.2: live 3-node failover E2E, jobsub raced against a
primary kill resolves to exactly one job, no duplication
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Add ctrlstatus_fmt.h (ctrlstatus_format_time_ms): renders responder_time_ms as
local YYYY-MM-DD HH:MM:SS,mmm, byte-equivalent to the controller log prefix
(tlog/my_tlog_format) so a captured ctrlstatus lines up against ctrl-N.log.
cmd_ctrlstatus prints a responderTime line; TZ-pinned exact-string unit test.
Req 11.2, 11.3. ctrlstatus group passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Role-branch the committed sourcing: PRIMARY reports its own authoritative
persist.committed_lsn / committed_config_tag; off-primary reports the LEARNED
ha->last_known_committed_lsn / last_known_committed_config_tag instead of the
primary-only raw markers that read 0 on a standby (fixes the applied>committed
inversion). Stamp responder_time_ms from CLOCK_REALTIME for log correlation.
Read-only (Req 10.1-10.4, 11.1). 4 new TDD tests; controller suite 110/0/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append-only UInt64 epoch-millis field for the responder snapshot timestamp
(Req 11.1, wire-format setup). Hand-authored xce_ctrlstatus_t typemap gains a
matching responder_time_ms member; codec round-trips it. Controller suite 1/1 OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
