Rongsong Shen
7512dc83f0
libxce-failover, raft-sim-conformance, ha-commit-durability, ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote, ha-elect-ctx-lifecycle, and ha-entity-snapshots all had 100% of tasks checked but had never run a formal /kiro-validate-impl closure pass (spec.json stuck at phase "tasks-generated", some with stale ready_for_implementation flags). Each now has a fresh validate_impl GO decision backed by an ASAN build, full standalone test-suite run, and a live 10-round sequential HA soak with zero committed-job loss.
32 lines
8.7 KiB
JSON
32 lines
8.7 KiB
JSON
{
|
|
"feature_name": "ha-elect-ctx-lifecycle",
|
|
"created_at": "2026-06-16T03:10:00Z",
|
|
"updated_at": "2026-07-18T09:56:00Z",
|
|
"language": "en",
|
|
"phase": "implementation-complete",
|
|
"approvals": {
|
|
"requirements": {
|
|
"generated": true,
|
|
"approved": true
|
|
},
|
|
"design": {
|
|
"generated": true,
|
|
"approved": true
|
|
},
|
|
"tasks": {
|
|
"generated": true,
|
|
"approved": true
|
|
}
|
|
},
|
|
"ready_for_implementation": true,
|
|
"validate_impl": {
|
|
"decision": "GO",
|
|
"validated_at": "2026-07-18T09:56:00Z",
|
|
"note": "Cross-task/feature-level validation of a spec whose 13/13 tasks.md leaf checkboxes (1.1, 2.1-2.4, 3.1-3.5) had been complete since before 2026-07-11, but whose spec.json phase was never advanced past tasks-generated (same administrative-oversight pattern as several sibling specs validated earlier today: ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote). This is the LAST of 9 Group-1 validate-impl closure passes run this session. Ran immediately after ha-prevote (GO) -- extra weight was given to the fact that ha-prevote's own soak already independently exercised this spec's fix (commit 6529dcf) via test_many_prevote_rounds_no_carrier_leak, giving this pass a useful cross-check without treating it as sufficient on its own.\n\nCROSS-TASK CONSISTENCY / DESIGN ALIGNMENT: confirmed live on-disk source matches design.md's File Structure Plan exactly, field-for-field. src/include/netio.h:103/110 -- on_fini is declared immediately adjacent to on_connected with the documented cross-reference comment (as specified). src/lib/netio.c:1112-1113 -- endpoint_fini calls ep->on_fini(ep) immediately before FREEUP(ep->data) (line 1116), matching the designed firing point exactly. src/controller/ha.h:126 -- _Atomic int refcount added to ha_elect_ctx_t with a doc comment that reproduces the design's bias-reference rationale verbatim. src/controller/ha.c: ha_elect_ctx_release (line 150, decrement + pthread_mutex_destroy/sem_destroy/free at zero) and vote_ep_on_fini (line 180, reads vctx from ep->data, releases vctx->ctx) are the single choke points design.md specifies; both raft_cb_send_vote_request (line 282 increment, line 293 hook wire) and raft_cb_send_pre_vote_request (line 421 increment, line 432 hook wire) apply the identical increment+wire pattern -- Requirement 3's 'uniform coverage' claim holds by direct source inspection, not just test titles. Supersession release sites present at lines 7106-7108 and 7376-7378 (atomic_exchange_explicit retrieving the outgoing pointer, then ha_elect_ctx_release on it) matching the design's 'exchange instead of store' mechanism for both ha_start_election_async and ha_start_prevote_async. Two additional release call sites (7423, 7442) cover error/abort paths per Requirement 2.4. Dependency direction and Boundary Commitments honored: no change to endpoint_fini's existing teardown mechanics beyond the one conditional hook call, no change to PreVote/election/failover semantics, the unrelated bug-1298 endpoint refcnt mechanism is untouched and not conflated with the new carrier refcount. No Revalidation Trigger has fired (no second on_fini consumer exists, no new vote endpoint capture site was added outside the two documented ones).\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan) -- zero compiler/linker errors, xce_controller/controller_unit_test/xce_test binaries confirmed with session-fresh mtimes. Standalone controller_unit_test binary run directly (authoritative over meson's aggregate line per this project's convention): 972 Tests, 0 Failures, 0 Ignored, OK, exit 0 -- zero regressions across the full suite, including existing vote-tally/quorum/freshness-gate tests (bug-339/bug-340 vote-freshness fixes unaffected). LEAK-SPECIFIC CHECK (this spec's core deliverable is a leak fix, so this is the load-bearing evidence): re-ran test_ha_elect_ctx_lifecycle.c in isolation with ASAN_OPTIONS=detect_leaks=1 (leak detection is off by project-wide default per cerebrum.md convention; this spec's own tests are written to enable it explicitly for exactly this purpose) -- 5 Tests, 0 Failures, 0 Ignored, OK, exit 0: test_release_at_refcount_one_frees_no_crash, test_release_at_refcount_two_leaves_valid, test_many_election_rounds_no_carrier_leak (20 rounds x 2 votes), test_many_prevote_rounds_no_carrier_leak (20 rounds x 2 votes), test_carrier_survives_supersession_with_outstanding_endpoint -- zero leaked ha_elect_ctx_t allocations across 40 real-election + 40 PreVote vote endpoints, and no premature-free UAF. TBD/TODO/FIXME/HACK/XXX grep on the spec's boundary files (ha.c, ha.h, netio.h, netio.c, test_ha_elect_ctx_lifecycle.c): one match (ha.c:1844, 'TODO(ha-term-safety task 2.x)') -- belongs to a different spec's stub, not this one's boundary. Secrets grep: clean.\n\nRUNTIME LIVENESS: ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh session binaries) against a real ext4 deploy dir. FIRST ATTEMPT FAILED at round 1: 'no new primary elected after kill (no serving primary at term>2)'. Root-caused via the controller logs, not assumed: cold-start election was unusually slow this run (ctrl-1 reached CTRL_PRIMARY term=2 only ~10.1s after process start), and the driver's round-1 kill fired ~1.7s after that -- before ctrl-1's just-started config-push handshake to the two standbys ('ctrlhello sent' at the same timestamp as promotion) could complete. Both standbys were still at tag=0 (no local config) at kill time, so per ha-config-bootstrap's own Req 2.8 guard they correctly refused to start PreVote/RDHUP re-election ('no local config (tag=0, n_peers=3) -- voter only, not starting PreVote/RDHUP re-election') rather than risk electing an unconfigured primary. This is a test-harness timing race (round 1 has no guard waiting for initial standby config receipt before killing the primary) -- not a code defect in this spec's boundary (no code this spec owns touches config propagation or the ha-config-bootstrap tag gate). RETRY: same command, fresh deploy dir, immediately after -- PASS: 10/10 rounds, 36 jobs submitted, zero job loss, zero committed-prefix divergence, all 3 nodes (ctrl-1/ctrl-2/ctrl-3) served as primary at least once. Grepped both the failed and the passing run's driver + all per-node controller logs for bug-1447's exact signature (AddressSanitizer, DEADLYSIGNAL, SEGV, ssl3_read_bytes, handle_ssl_handshake, io_worker) and other crash signatures (Segmentation fault, core dumped, Aborted, double free, heap-buffer-overflow, use-after-free): zero matches in either run. No leftover xce_controller processes after either run (driver's own cleanup on FAIL, and clean exit on PASS, both confirmed via ps).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory, not reproduced here. The round-1 config-push/kill-timing race is a test-harness gap in run-ha-soak.sh's sequential mode (no wait for initial 3-of-3 config convergence before the first kill), not owned by this spec -- flagged here as a known_limitation for whoever next touches the driver or ha-config-bootstrap, not adopted as a blocker for this spec's own GO. No new bugs found or fixed during this validation pass; no code changes were made -- this was a read/build/test/soak-only pass.",
|
|
"known_limitations": [
|
|
"Test-harness timing race in .claude/skills/xce-test-ha/run-ha-soak.sh sequential mode: round 1's kill is not gated on the standbys having received their initial config push, so on a slow cold-start election (observed here: ~10s to CTRL_PRIMARY) round 1 can fire before config propagates, hitting ha-config-bootstrap's Req 2.8 voter-only guard and failing the round with 'no serving primary at term>2'. Reproduced once in this session, did not reproduce on an immediate retry (10/10 clean). Not owned by this spec (no code this spec touches affects config propagation or the tag=0 guard) -- flagged for whoever next touches run-ha-soak.sh or ha-config-bootstrap; a fix would add a short 3-of-3-config-received wait before round 1's first kill.",
|
|
"bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker. Did not reproduce in either soak run in this pass (failed or passing). No action item for this spec.",
|
|
"Requirement 4.3 names both sequential AND chaotic /xce-test-ha regression modes. Per the explicit validation-scope instruction for this pass, only the sequential 10-round soak was (re-)run; chaotic mode was not independently re-run for this spec. This mirrors the evidence scope of the sibling ha-prevote validation (also sequential-only) and is a low-risk gap given this is a pure memory-lifetime fix with zero semantic surface on election/failover outcomes (chaotic mode's specific hazard class -- double-fault commit-prefix divergence timing -- is orthogonal to carrier refcounting)."
|
|
]
|
|
}
|
|
}
|