/kiro-validate-impl: full suite green (controller 1210/0, lib 368/0,
harness 173/0), live 3-node smoke boot confirms bug-1818's fix holds,
12/12 requirement groups covered, cross-task integration traced at the
source level, dependency direction clean, zero blocked tasks. Prior
2026-07-17 validate_impl record preserved under prior_validate_impl.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Feature-level /kiro-validate-impl found the delta's File Structure Plan
omitted 5 as-built files task 7.3 actually needed (request.h/request.c
-- the real heartbeat_msg_t struct-builder -- plus 3 cascading
signature-conformance test files). Documents the as-built reality and
the already-fixed cmd_ctrlstatus gate text; also notes the pre-existing,
deliberate lack of a checked-in unit test for cmd_ctrlstatus's print
layer (the class of gap bug-1818 exploited) as a known, accepted scope
boundary for future readers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
phase -> implementation-complete; delta.impl_status records tasks
7.1-7.6 done, including the bug-1818 find-and-fix from live E2E.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Requirement 12 delta (primary election term/time) is now fully
implemented: tasks 7.1-7.6 all complete, including the live 3-node
failover validation that found and confirmed the fix for bug-1818.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Live 3-node E2E validation (task 7.6) found that cmd_ctrlstatus's
electedTerm/electedAt lines (task 7.5) gated on NULLSTR(primaryName),
which is unconditionally empty on every non-primary responder in this
build (ctrlstatus_fill only self-reports the primary's own name --
ha->primary_host is vestigial, a pre-existing Req 2 behavior). This made
electedTerm/electedAt print "unknown" on every standby, forever, even
though elected_at_ms was correctly, non-zero, and convergently populated
the whole time (term/committedLsn/appliedLsn, sourced the same
unconditional way, DID correctly display and converge on the same query).
Fix: gate on status->elected_at_ms == 0 instead -- the field stays
exactly 0 until a real promotion stamps it (either road) or a heartbeat
carries a learned value from an actual primary, unlike term (which can
legitimately bump from mere vote/candidacy activity before any primary
is elected). ctrlstatus_fill itself is untouched; this is purely a
cmd_ctrlstatus display-layer correction.
Also corrects design.md's "Design Delta -- Requirement 12" Locked
Decision 3 (and its dependent traceability/service-contract text) to
document this live-E2E finding, mirroring the spec's existing delta
convention (Req 10/11's own live-E2E correction).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
cmd_ctrlstatus (src/harness/test.c) now prints electedTerm/electedAt
right after the Primary: line, gated on the same NULLSTR(primaryName)
"unknown" check that line already uses. electedTerm prints the existing
term field verbatim (no new wire field, per design's Locked Decision 1);
electedAt reuses the existing ctrlstatus_format_time_ms formatter
already used for responderTime, falling back to "unknown" on a NULL
format result. No leakage of a stale term/elected_at_ms when no primary
is known.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
ctrlstatus_fill now unconditionally reads ha->last_known_elected_at_ms
into the reply's elected_at_ms field -- one atomic-acquire load, no
role-branch, mirroring how the handler already reports raw term
unconditionally. Serves both a self-stamped primary and a
heartbeat-learned standby from the same field. "Unknown" display
gating is deliberately left to the client command (task 7.5), not
this handler.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
The primary echoes its own last_known_elected_at_ms on every outgoing
heartbeat (threaded through xce_ha_heartbeat_req's existing scalar-arg
plumbing, alongside term/committed_lsn/committed_config_tag); a standby
processing a received heartbeat (ha_consume_heartbeat_committed, the sole
production call site) stores it unconditionally into its own
last_known_elected_at_ms, reconcile-free -- no other HA field moves.
ha_observe_heartbeat_committed itself is untouched; the new store sits
next to its call site, not inside it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Stamps ha_state_t.last_known_elected_at_ms (CLOCK_REALTIME, epoch-ms) in
both ha_do_promote (non-raft road) and raft_cb_become_leader (raft road),
placed at each road's genuine, final commit point to CTRL_PRIMARY -- past
every early-return refusal path in raft_cb_become_leader (first-boot
phase2 failure, the Live Apply Resource Fence re-check, the promotion
eligibility gate, config-serve-fenced, upgrade-materialization-fenced).
Purely observational: does not change whether/when a promotion succeeds
or is refused. Three new tests (2 positive, 1 negative) prove placement:
a refused promotion leaves the field at its pre-attempt value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds CtrlStatus.electedAtMs@14 and HeartbeatMsg.electedAtMs@4 to the
capnp schema, their hand-authored typemap.h counterparts, and a new
ha_state_t.last_known_elected_at_ms field alongside last_known_committed_lsn.
Schema/state plumbing only -- nothing stamps or consumes these fields yet
(tasks 7.2-7.5). Codec round-trip tests confirm both new fields survive
encode/decode unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds requirements, design, research, and tasks for reporting the term
and wall-clock time the current primary was elected, including
standby-side learned fidelity (mirroring the Req 10 pattern), following
the existing controller-status spec's delta convention (Req 10/11).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
All tasks (1.1-6.4) were already [x] with no prior validate_impl
record. Fixed stale delta.impl_status (still said task 6.4 blocked on
sandbox overlayfs despite tasks.md recording it VERIFIED live since
2026-07-02).
Full suite 9/9 green, 36/36 requirement acceptance criteria mapped,
cross-task wire contract verified field-for-field end to end (schema
-> typemap -> handler -> dispatch -> libxce client -> CLI), two live
smoke runs (3-round sequential HA soak + targeted 3-node deploy
exercising Req 10/11 directly) both PASS. One transient test failure
during the run traced to a concurrent second session competing for
the same build directory, not a regression -- reproduced clean 3x in
isolation.
controller-status, ha-commit-engine, ha-heartbeat-committed-propagation,
ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and
raft-ha-core-safety each had every numbered sub-task marked [x] but their
group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic
gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real
unchecked sub-tasks in any of the eight files. No task content changed.
ha-commit-durability: closes the "pending confirmation under simultaneous
total outage" caveat. Reproduced cold-restart-all (kill all 3 members +
restart) 5 times against a live 3-node cluster on genuine ext4 (not the
sandbox's overlayfs) — zero committed-job loss across 40 rolling-failover
rounds + 5 cold-restart-all cycles. Confirms bug-795 was an overlayfs
fdatasync artifact, not a real durability gap; no production fix needed.
Commit Durability is now proven for both rolling single-node failover and
simultaneous total-cluster outage (src/controller/CLAUDE.md updated).
Live-validated and marked complete against a real 3-node cluster:
- controller-status 6.4: off-primary committed fidelity + responder
timestamp, confirmed via a real failover
- ha-heartbeat-committed-propagation 4.1: standby committedLsn tracks the
primary via heartbeat within the 3s interval
- ha-wal-replication 5.4: standby WAL replication verified via committedLsn
match (job-facing ops are correctly primary-only by design; standbys
return E_XCE_STANDBY, not a bug)
- libxce-failover 6.2: live 3-node failover E2E, jobsub raced against a
primary kill resolves to exactly one job, no duplication
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Add ctrlstatus_fmt.h (ctrlstatus_format_time_ms): renders responder_time_ms as
local YYYY-MM-DD HH:MM:SS,mmm, byte-equivalent to the controller log prefix
(tlog/my_tlog_format) so a captured ctrlstatus lines up against ctrl-N.log.
cmd_ctrlstatus prints a responderTime line; TZ-pinned exact-string unit test.
Req 11.2, 11.3. ctrlstatus group passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Role-branch the committed sourcing: PRIMARY reports its own authoritative
persist.committed_lsn / committed_config_tag; off-primary reports the LEARNED
ha->last_known_committed_lsn / last_known_committed_config_tag instead of the
primary-only raw markers that read 0 on a standby (fixes the applied>committed
inversion). Stamp responder_time_ms from CLOCK_REALTIME for log correlation.
Read-only (Req 10.1-10.4, 11.1). 4 new TDD tests; controller suite 110/0/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append-only UInt64 epoch-millis field for the responder snapshot timestamp
(Req 11.1, wire-format setup). Hand-authored xce_ctrlstatus_t typemap gains a
matching responder_time_ms member; codec round-trips it. Controller suite 1/1 OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EARS requirements for a controller status query (XCE lsid analog): primary identity
(unknown when unelected), answered-by node+role (distinct from cluster view),
configured peers+count, committed config version + content checksum (catch same-version
divergence), term/commit-apply/fenced/cluster-name, failover-aware delivery, test-client
command, and wait-primary switching its probe to it. C tier only.
