15 KiB
Jul 13 → Jul 18 Progress Report
Overview
Six days, 57 commits. Two threads dominated: job-array-single-wal-record landed as a complete feature (single-WAL-record array job submission, compact scheduler/API-server push, live apply fence, full regression GO — then reopened once for user-directed residual-gap closure and re-validated GO), and a backlog of 9 HA/Raft specs that had 100%-complete tasks but had never been through a formal /kiro-validate-impl closure pass got worked through systematically, surfacing one genuine production crash bug (bug-1447) that got root-caused and fixed. ha-consistency-invariants (the umbrella Raft+CP contract spec) closed out — all 4 sub-specs now complete. The controller test suite's Unity harness got a suite-wide correctness fix (83 files). Net: 9 bugs fixed (1435, 1436, 1438, 1442, 1443, 1444, 1445, 1447, 1449), 9 specs validated GO in one day (Jul 18), 1 feature spec (job-array-single-wal-record) shipped end-to-end.
Jul 13: Election liveness cleanup + ha-consistency-invariants closes out
| Bug | Manifestation | Fix |
|---|---|---|
| 1435 | 40-round soak FAIL at round 36/40 — verify_no_loss got 0/111 jobs confirmed; freshly-elected primary answered E_XCE_STANDBY |
New ha_state_t.raft_lock (recursive mutex) serializes every entry point into raft_state_t on the election/vote path (510b566) |
| 1436 | 40-round soak FAIL at round 36/40 — xce_joblist: connection reset by peer right after wait-primary confirmed the new primary serving in 46ms |
XCECLI_FAILOVER_CALL_COUNT macro, a count-aware sibling of XCECLI_FAILOVER_CALL for xce_get-family calls (40c2351) |
ha-consistency-invariants (the umbrella Raft+CP contract spec covering ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery): Criterion 2.9 (the Facts-Apply invariant, added 2026-06-25) marked satisfied — closed by raft-apply-decoupling's fact-apply-after-commit split plus decision D1 (block-but-durable quorum-loss policy). All 4 sub-specs now complete; the umbrella itself needs no separate task list by design (895a204).
Also: xce-test-ha cross-run collision guard added to the harness (b321963).
Jul 14-17: job-array-single-wal-record — feature build-out
The largest single thread this period — 24 commits. Single WAL record per array-job submission instead of N individual records, with N-way expansion on apply.
Build-out (Jul 14-15): WAL_JOB_ARRAY_SUBMIT record type + compact capnp variant → jobj.c hardening → single-record append/commit → wire into handle_jobsubmit → restore_job_array primitive → wire array expansion into both live apply and apply_wal_entry → make the apply pump respect apply failures → Live Apply Resource Fence (self-healing, gates promotion on fence state) → compact scheduler-side and API-server-side (Go) local expansion of the push, so downstream consumers only see the compact form, not N expanded records on the wire.
Bugs surfaced during build-out:
- bug-1438: array-submit test got a WAL type mismatch then hung all 3 threads on
futex_do_wait— a WAL-header-offset helper copied from a zero-payload-only test assumed no payload; fixed by passing the entry's start offset explicitly instead of back-computing it (jap_read_wal_hdr_at/jap_read_wal_entry_at). - bug-1442: AddressSanitizer double-free on
TestWireDispatch_OnePushProducesNRecordsAndLogsOnce— Go'sgo_push_handlerdeferredfree_Requestwhile the datamsg-case free path double-freed the same buffer; fixed and verified. - bug-1437, 1439, 1440, 1441 also surfaced (accounts-leak test finding, a controller.capnp.h build break, two Go cgo-in-test failures) but the user explicitly scoped those sessions to build/report only — fix paths were documented but not applied; these remain open, low-priority.
Full regression GO recorded 2026-07-16 (1cfdac4/2b0a37a), backfilling bug-1444's fix into the record (2a76940).
Reopened (575f928, user-directed) for three residual gaps left open by the first GO:
- bug-1443:
restore_job_array's bulk-insert pass could fail partway through, leaving a partial-visibility window — closed with new purely-additivekvmgr_reserve()/xzarray_reserve()primitives (dcf9f39). - bug-1444:
replay_wal's job-count tally was blind to array submissions, causing anAPPLY_DIVERGENTrecovery mismatch — fixed (16747aa). - Task 4.2's boot-replay fence gap: Live Apply Resource Fence extended to cover boot replay, not just live apply (
ea9d3d8). - A
jobArraydataset leak on the scheduler receive path, found and fixed the next day (53bd686).
Fresh /kiro-validate-impl GO recorded 2026-07-17 (9cf61c9), closing stale notes from the first pass.
Jul 17: bug-1445, first GO passes for ha-term-safety/controller-status, Unity harness fix
- nix: repinned
librttito fix amakeInstanceleak-on-init-failure (8c7d26a). - bug-1445: a node's very first promotion-eligibility check in a fresh process ran before that node's own first
persist_recover()/replay_wal()call, racing the Live Apply Resource Fence. Fixed by makingraft_cb_become_leader()'s first-boot path re-checkpersist_apply_resource_fenced()immediately aftercluster_init_phase2()completes (ff01db3). - ha-term-safety and controller-status both got their first-ever
/kiro-validate-implpass — both GO (84fde22,c08a348). - Unity test harness: nested
UNITY_BEGIN()/UNITY_END()calls across 83controller/testsuite/*.cfiles were maskingcontroller_unit_test's real exit code and mis-attributing failing tests to the wrong source file. Suite-wide fix (UnitySetTestFile(__FILE__)per group function) verified via a single consolidated 972-test, 0-failure summary block matching the ground-truthRUN_TESTcount (a81f2a2).
Jul 18: Group 1 validate-impl batch — 9 specs closed, 1 real bug found and fixed
A backlog audit found 9 HA/Raft specs with 100% of tasks checked but spec.json stuck at phase: "tasks-generated" — never run through a formal validation closure. Ran /kiro-validate-impl on all 9, each with a fresh ASAN build, full standalone test-suite run, and a live 10-round sequential HA soak:
| Spec | Result | Notes |
|---|---|---|
| raft-sim-conformance | GO | TLA+ conformance harness; 293/293 lib tests, corrected a stale buglog entry (bug-1095) |
| libxce-failover | GO | ready_for_implementation flag was stale since 2026-06-18, not a real block |
| ha-entity-snapshots | GO | Traced and confirmed a 2026-07-02 task 7.1 blocker was genuinely resolved (real fix chain, not a checkbox) |
| ha-commit-durability | GO | Re-verified rather than trusting stale evidence given later churn; found bug-1446 (test-harness-only, BusyBox date lacks %N) |
| raft-drive-async | NO-GO → GO | Live soak crashed round 9 — see bug-1447 below; re-validated GO after the fix |
| ha-wal-replication | GO | 972/972 tests, 10/10 soak |
| ha-heartbeat-committed-propagation | GO | Needed 2 attempts — first agent run died mid-soak to a transient harness API-connection error, clean retry |
| ha-prevote | GO | Term climbed 2→13 cleanly across the soak, zero election-storm churn — exactly PreVote's purpose |
| ha-elect-ctx-lifecycle | GO | Found bug-1449 (harness-only, see below), closed via retry |
bug-1447 (the real one): raft-drive-async's soak crashed at round 9 — an ASAN SEGV deep inside OpenSSL, triggered during a reconnect-offload burst. Initial hypothesis (concurrent SSL_do_handshake from two io_worker threads) was wrong and retracted — iomgr_start() only ever creates one io_worker thread per iomgr_t. Root cause, confirmed via a targeted regression test: a same-batch use-after-free — on_standby_connected() evicts a stale duplicate peer channel mid-batch, but epoll_ctl(DEL) only stops future epoll_wait() calls, not an event for that same endpoint already sitting in the current batch's array. Fixed by ref-counting every endpoint in a batch before dispatch and releasing only after the whole batch completes (mirrors the existing bug-1298 idiom). TDD RED/GREEN confirmed, 2 independent clean 10-round soaks, full suites clean (294/294 + 972/972).
bug-1449 (harness-only): run-ha-soak.sh had no wait for initial 3-of-3 config convergence before round 1's first kill — a slow cold-start election could let the kill land before config-push reached the standbys, producing a false "no primary elected" failure. Fixed by adding wait_for_initial_config_convergence(), mirroring the script's existing reconvergence-wait pattern.
Bug Tally: Jul 13-18
Fixed (9)
1435, 1436, 1438, 1442, 1443, 1444, 1445, 1447, 1449
Known residual / open (low-priority, environment- or scope-limited)
- 1437: accounts-leak test finding — user scoped session to build/report only, no fix applied
- 1439:
controller.capnp.hbuild break — build/report-only session, no fix applied - 1440, 1441: Go
cgo-in-test failures inapiserver/backend— build/report-only sessions; fix paths documented (removeimport "C"from the test file; add the missingexterndeclaration) but not applied
Commits (Jul 13-18)
8f614b6 chore: update OpenWolf session state (Group 1 validate-impl batch + bug-1447/1449)
426728b fix(netio): bug-1447 same-batch endpoint use-after-free in io_worker dispatch
4d2c903 fix(test-harness): bug-1449 wait for initial config convergence before round-1 kill
7512dc8 docs(specs): kiro-validate-impl GO closure for 8 HA/raft specs
781b0f7 chore: update OpenWolf session state + cerebrum learning (Unity test-harness fix)
a81f2a2 fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide
e6956ec chore: update OpenWolf session state
6592f9c chore: update OpenWolf session state + cerebrum learnings (meson parallel-timeout false-positive, /tmp/xce_* cleanup)
3f54ef3 chore: update OpenWolf session state (ha-term-safety + controller-status validation)
c08a348 docs(controller-status): record first /kiro-validate-impl pass -- GO
84fde22 docs(ha-term-safety): record first /kiro-validate-impl pass -- GO
1a0960b chore: update OpenWolf session state (bug-1445 fix)
ff01db3 fix: bug-1445 -- first-boot promotion check races its own first replay
5ddc20f docs: log bug-1445 for job-array-single-wal-record's first-boot promotion gap
fb701c2 chore: update OpenWolf session state (job-array-single-wal-record validation pass)
9cf61c9 docs(job-array-single-wal-record): record fresh /kiro-validate-impl GO, close stale notes
ccee18b chore: update OpenWolf session state
2cfb454 docs(job-array-single-wal-record): mark task 1.3's rtti leak note FIXED
8c7d26a chore(nix): repin librtti to fix makeInstance leak-on-init-failure
9176164 chore: update OpenWolf session state (anatomy, cerebrum, memory, token ledger)
b469b17 docs(job-array-single-wal-record): sync spec docs with phase-7 residual-gap closure
53bd686 fix(job-array-single-wal-record): free jobArray dataset on scheduler receive path
ea8d9d3 fix(job-array-single-wal-record): extend Live Apply Resource Fence to boot replay (task 4.2's gap)
5a81362 docs: mark bug-1443 FIXED (task 7.1, commit dcf9f39)
dcf9f39 fix(job-array-single-wal-record): close bug-1443's bulk-insert partial-visibility gap
575f928 plan(job-array-single-wal-record): reopen spec for residual gap closure
85ce5da docs: track recurrences of pre-existing bugs 1437/1438 during re-validation
2a76940 docs(job-array-single-wal-record): backfill bug-1444 for task 6.0a's replay_wal fix
2b0a37a docs(job-array-single-wal-record): record /kiro-validate-impl GO decision
1cfdac4 test(job-array-single-wal-record): full regression pass - GO
c613849 test(job-array-single-wal-record): chaotic-failover array retry scenario
0d8986a test(job-array-single-wal-record): large-array HA soak scenario
16747aa fix(job-array-single-wal-record): replay_wal's job-count tally was blind to array submissions
b7b90e9 test(job-array-single-wal-record): end-to-end integration test for the compact push
9e0d87f feat(job-array-single-wal-record): API-server-side (Go) local expansion of the compact push
1cbd7df fix(job-array-single-wal-record): wire dispatcher was silently dropping compact array pushes
d168471 feat(job-array-single-wal-record): scheduler-side local expansion of the compact push
e0cec75 feat(job-array-single-wal-record): build the compact API-server push
5e7a4aa feat(job-array-single-wal-record): build the compact scheduler push
1888498 test(job-array-single-wal-record): end-to-end fault-injection test for the live fence
8cb8c25 feat(job-array-single-wal-record): gate promotion on the live apply fence
b47cc66 feat(job-array-single-wal-record): add live, self-healing apply resource fence
cc7e99c feat(job-array-single-wal-record): make apply pump respect apply failures
107e383 test(job-array-single-wal-record): close remaining array expansion gaps
b0c7cfe feat(job-array-single-wal-record): wire array expansion into apply_wal_entry
0c4573b feat(job-array-single-wal-record): wire array expansion into live apply
7c755dc feat(job-array-single-wal-record): implement restore_job_array primitive
78eb983 test(job-array-single-wal-record): close remaining array append/commit gaps
d38ca72 feat(job-array-single-wal-record): wire array submissions into handle_jobsubmit
7c4afbb feat(job-array-single-wal-record): implement single-record array append/commit
76862bd feat(job-array-single-wal-record): harden jobj.c job construction primitives
bc4e70b feat(job-array-single-wal-record): add compact array push capnp variant
2e62e73 feat(job-array-single-wal-record): add WAL_JOB_ARRAY_SUBMIT record type
895a204 docs(spec): ha-consistency-invariants — Criterion 2.9 now satisfied
b321963 chore(wolf): session trail + xce-test-ha cross-run collision guard
40c2351 fix(client): bug-1436 count-aware failover macro for xce_get-family calls
510b566 fix(ha): bug-1435 raft_lock serializes concurrent raft_state_t access
Current State
| Area | Status |
|---|---|
| job-array-single-wal-record | COMPLETE — fresh GO 2026-07-17, all residual gaps closed |
| ha-consistency-invariants | COMPLETE — umbrella + all 4 sub-specs (commit-engine, barrier-rollout, single-leader-role, convergence-recovery) |
| ha-term-safety | GO (first validate-impl pass, 2026-07-17) |
| controller-status | GO (first validate-impl pass, 2026-07-17) |
| raft-sim-conformance | GO (2026-07-18) |
| libxce-failover | GO (2026-07-18) |
| ha-entity-snapshots | GO (2026-07-18) |
| ha-commit-durability | GO (2026-07-18) |
| raft-drive-async | GO (2026-07-18, after bug-1447 fix — was NO-GO same day) |
| ha-wal-replication | GO (2026-07-18) |
| ha-heartbeat-committed-propagation | GO (2026-07-18) |
| ha-prevote | GO (2026-07-18) |
| ha-elect-ctx-lifecycle | GO (2026-07-18) |
| HA sequential soak (xce-test-ha) | Round-1 config-convergence race fixed (bug-1449); reliable across all 9 validation passes today |
| netio.c io_worker dispatch | Same-batch endpoint UAF fixed (bug-1447), regression test added |
| Controller test suite | 972/972 clean, Unity harness attribution fixed suite-wide |
| admin-config-tool | Unchanged — spec finalized, pending implementation |
| scheduler-framework | Unchanged — requirements-only, pending design |
| config-upgrade-snapshot | Unchanged — requirements-only |
| Known open (low-priority) | bug-1437 (test-only), bug-1439 (capnp build break), bug-1440/1441 (Go cgo-in-test) — all build/report-only sessions, fix paths documented not applied |