Rongsong Shen
7512dc83f0
libxce-failover, raft-sim-conformance, ha-commit-durability, ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote, ha-elect-ctx-lifecycle, and ha-entity-snapshots all had 100% of tasks checked but had never run a formal /kiro-validate-impl closure pass (spec.json stuck at phase "tasks-generated", some with stale ready_for_implementation flags). Each now has a fresh validate_impl GO decision backed by an ASAN build, full standalone test-suite run, and a live 10-round sequential HA soak with zero committed-job loss.
36 lines
8.1 KiB
JSON
36 lines
8.1 KiB
JSON
{
|
|
"feature_name": "ha-commit-durability",
|
|
"created_at": "2026-06-17T23:40:42Z",
|
|
"updated_at": "2026-07-18T00:55:00Z",
|
|
"language": "en",
|
|
"phase": "implementation-complete",
|
|
"reopened_at": "2026-06-19T00:00:00Z",
|
|
"reopened_reason": "bug-795: total-cluster-outage (cold-restart-all) committed-job loss surfaced by the xce-test-e2e-primitives soak; Phase 6 added (6.1 reproduce-on-real-fs gates the rest).",
|
|
"approvals": {
|
|
"requirements": {
|
|
"generated": true,
|
|
"approved": true
|
|
},
|
|
"design": {
|
|
"generated": true,
|
|
"approved": true
|
|
},
|
|
"tasks": {
|
|
"generated": true,
|
|
"approved": true
|
|
}
|
|
},
|
|
"ready_for_implementation": true,
|
|
"blocked_on": [],
|
|
"validate_impl": {
|
|
"decision": "GO",
|
|
"validated_at": "2026-07-18T00:55:00Z",
|
|
"note": "Prior state: all 28 tasks (1-6.4) were [x] in tasks.md but phase was stuck at \"tasks-generated\" with no validate_impl record (administrative oversight, same pattern as ha-entity-snapshots). Per-task Implementation Notes in tasks.md already document extensive self-review of each task (including a corrective REWORK note for the never-self-terminate invariant and a cross-task fix, bug-760, for an adjacent test regression) -- this pass focuses on cross-task/feature-level checks.\n\nMECHANICAL: Full ASAN build clean (nix develop + xce-build.sh controller asan), zero compiler errors. Meson test summary Ok:3 Fail:0. Standalone controller_unit_test binary (authoritative per this project's own meson-Fail:0-masks-Unity-subgroups convention): 0 \":FAIL:\" occurrences, 791 individual \": PASS\" test lines, all green (process exit code alone is not gating -- pre-existing unrelated ASAN leak-report noise at exit is a documented quirk, not a functional failure). TBD/TODO grep on the feature's owned files (persist.c, persist.h, commit.c, ha.c, cluster.c, test_commit_durability.c, test-commit-durability.sh) found only pre-existing, unrelated items not introduced by this spec: cluster.c's 3 TODOs are an unrelated config-generation-tool note and a config-group FIXME, and ha.c:1844 is the same pre-existing ha-term-safety-owned stub TODO already flagged as unrelated by the ha-entity-snapshots validate_impl pass. Secrets grep clean.\n\nSTALENESS RE-CHECK (the key judgment call): this spec's own live evidence (deterministic RED->GREEN E2E and rolling-soak zero-loss) was captured 2026-06-18/2026-06-19, and the dedicated cold-restart-all real-fs revalidation (task 6.1-6.4) on 2026-07-02 -- both materially predate today. In the interim, persist.c/ha.c/cluster.c/commit.c (the exact files this spec owns) received substantial further edits: bug-1433/1434 (2026-07-11, added persist_committed_lsn_advance_atomic -- a CAS-safe committed_lsn advance, and removed a force-flush call reachable from a never-blocks vtask class), bug-1417/1419 (vote-index mapping, WAL replay-tally dedup), the job-array-single-wal-record series (new WAL_JOB_ARRAY_SUBMIT record type + apply/replay changes), and bug-1445 (2026-07-17, first-boot promotion-check-vs-own-first-replay race in raft_cb_become_leader -- the SAME function this spec's promotion-eligibility gate lives in). Given this density of churn in exactly the owned files, the old live evidence alone was judged insufficient without a fresh check. Code-level verification: majority_candidate_locked (commit.c:135) still caps the self contribution at persist_durable_lsn (unchanged); persist_committed_lsn_advance_atomic (persist.c:5881) and its two call sites explicitly preserve \"committed_lsn never advances past durable_lsn\" in their own code comments, i.e. the later CAS fix was written to respect this spec's invariant rather than bypass it; bug-1445's new check is an orthogonal job-array live-apply-fence re-check added alongside, not a modification of, this spec's recovery-truncated/durable-floor promotion gate.\n\nFRESH LIVE RE-VALIDATION (this session, genuine ext4 -- confirmed /home/claude, backing device /dev/nvme0n1p2 ext4, NOT the container's overlay /): (a) --deterministic strace mode returned a false RED (3/3 probes \"0 fdatasync in-window\"). Diagnosed, not accepted at face value: this sandbox's date is BusyBox date, which does not support GNU date's %N (nanoseconds) -- confirmed by direct invocation, it emits the literal unexpanded string \"%N\", breaking the script's sub-second window arithmetic. Cross-checked strace's own (unaffected) microsecond timestamps against the controller's committed_lsn advance log lines for all 3 probes: fdatasync consistently PRECEDED the commit-advance/ack by 16-43ms in every probe -- i.e. durable-before-ack genuinely holds on the current build; the script's verdict was a test-harness false positive, not a product regression. Logged as bug-1446 (test-harness only). (b) 4-round rolling SIGKILL+restart soak plus a full cold-restart-all cycle on the same ext4 deploy: 8/8 PASS, 0 job loss -- all 20 committed jobs survived both rolling failover and cold-restart-all. This directly reconfirms Req 1.5 and the Phase-6 total-outage guarantee on today's code, not just June/early-July code.\n\nINTEGRATION: durable_lsn -> majority_candidate_locked self-cap -> committed_lsn -> commit barrier -> client reply chain verified intact end to end (code read + the live soak's observed ordering above). recovery-verdict -> promotion-gate -> freshness-floor chain verified intact (persist_recovery_verdict/persist_durable_floor still consumed by ha.c's promotion gate and cluster.c's vote/PreVote freshness comparisons). No boundary spillover: the only two touches to raft_cb_become_leader since this spec landed (bug-1433/1434's force-flush removal, bug-1445's first-boot fence re-check) are both additive/orthogonal, not modifications of this spec's owned gate logic. src/controller/CLAUDE.md's \"Commit Durability\" guarantee note (Protocol Verification Checklist) is current and accurate, and is explicitly cross-referenced by the newer bug-1434 code comments.\n\nCOVERAGE: all 3 requirements sections (1.1-1.6 Durable Commit, 2.1-2.6 Recovery Integrity and Leadership Fence, 3.1-3.3 Deterministic Durability Verification) map through design.md's Requirements Traceability table to a completed task; no gaps. File Structure Plan matches actual files (persist.c/.h, commit.c, ha.c, cluster.c, config.h, main.c, test_commit_durability.c, test-commit-durability.sh) with no drift. Blocked tasks: none.",
|
|
"known_limitations": [
|
|
"bug-1396 (open, low-priority, unconfirmed): an extra/duplicate committed job appeared after one cold-restart-all cycle in the original 2026-07-02 5-run validation -- the opposite symptom of the loss this spec targets (over-counting, not loss), not yet root-caused. Does not block this GO (it is not a durability-invariant violation) but remains open for a future investigation.",
|
|
"persist_apply_pump_run_sync (flagged by bug-1433, not introduced by this spec): still reachable from raft_cb_become_leader on a VTASK_LIVENESS (never-blocks) vtask and performs bounded pread() -- lower risk than the fdatasync this spec's own group-commit flusher correctly keeps off that path, but strictly still blocking I/O. Tracked as its own follow-up in bug-1433, adjacent to but outside this spec's boundary.",
|
|
"bug-1446 (this session, test-harness only): src/tools/test-commit-durability.sh's --deterministic mode produces a false RED in any environment whose `date` lacks GNU date's %N support (e.g. BusyBox date, as in this sandbox) because its submit-to-ack window arithmetic silently breaks on the unexpanded literal \"%N\". The soak mode used for this session's live re-validation is unaffected and remains the authoritative live check in such environments. A future fix should make the script detect/require GNU date or use an alternative sub-second time source, and fail loud on an incompatible date rather than emitting a false verdict.",
|
|
"The group-commit batching mechanism's interaction with the newer job-array-single-wal-record feature (multi-entry array WAL records) was confirmed consistent at the shared committed_lsn/durable_lsn-invariant level (code read) but was not independently re-exercised by a dedicated array-specific durability soak in this pass; job-array-single-wal-record carries its own validate_impl track for that."
|
|
]
|
|
}
|
|
}
|