From 6e51a97465ef3d05b5011f509a0818a21624f3b6 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 18 Jul 2026 13:30:19 +0000 Subject: [PATCH] docs: add Jul 13-18 progress report job-array-single-wal-record shipped end-to-end, ha-consistency-invariants umbrella closed out, Unity test-harness suite-wide fix, and today's Group 1 kiro-validate-impl batch (9 specs) which surfaced and fixed a real same-batch endpoint UAF crash (bug-1447) plus a harness timing race (bug-1449). --- reports/jul-13-jul-18.md | 175 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 175 insertions(+) create mode 100644 reports/jul-13-jul-18.md diff --git a/reports/jul-13-jul-18.md b/reports/jul-13-jul-18.md new file mode 100644 index 0000000..7828e87 --- /dev/null +++ b/reports/jul-13-jul-18.md @@ -0,0 +1,175 @@ +# Jul 13 → Jul 18 Progress Report + +## Overview + +Six days, 57 commits. Two threads dominated: **job-array-single-wal-record** landed as a complete feature (single-WAL-record array job submission, compact scheduler/API-server push, live apply fence, full regression GO — then reopened once for user-directed residual-gap closure and re-validated GO), and a **backlog of 9 HA/Raft specs that had 100%-complete tasks but had never been through a formal `/kiro-validate-impl` closure pass** got worked through systematically, surfacing one genuine production crash bug (bug-1447) that got root-caused and fixed. **ha-consistency-invariants** (the umbrella Raft+CP contract spec) closed out — all 4 sub-specs now complete. The controller test suite's Unity harness got a suite-wide correctness fix (83 files). Net: **9 bugs fixed** (1435, 1436, 1438, 1442, 1443, 1444, 1445, 1447, 1449), **9 specs validated GO** in one day (Jul 18), **1 feature spec** (job-array-single-wal-record) shipped end-to-end. + +--- + +## Jul 13: Election liveness cleanup + ha-consistency-invariants closes out + +| Bug | Manifestation | Fix | +|-----|---------------|-----| +| **1435** | 40-round soak FAIL at round 36/40 — `verify_no_loss` got 0/111 jobs confirmed; freshly-elected primary answered `E_XCE_STANDBY` | New `ha_state_t.raft_lock` (recursive mutex) serializes every entry point into `raft_state_t` on the election/vote path (`510b566`) | +| **1436** | 40-round soak FAIL at round 36/40 — `xce_joblist`: connection reset by peer right after wait-primary confirmed the new primary serving in 46ms | `XCECLI_FAILOVER_CALL_COUNT` macro, a count-aware sibling of `XCECLI_FAILOVER_CALL` for `xce_get`-family calls (`40c2351`) | + +**ha-consistency-invariants** (the umbrella Raft+CP contract spec covering ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery): Criterion 2.9 (the Facts-Apply invariant, added 2026-06-25) marked **satisfied** — closed by raft-apply-decoupling's fact-apply-after-commit split plus decision D1 (block-but-durable quorum-loss policy). All 4 sub-specs now complete; the umbrella itself needs no separate task list by design (`895a204`). + +Also: `xce-test-ha` cross-run collision guard added to the harness (`b321963`). + +--- + +## Jul 14-17: job-array-single-wal-record — feature build-out + +The largest single thread this period — 24 commits. Single WAL record per array-job submission instead of N individual records, with N-way expansion on apply. + +**Build-out** (Jul 14-15): `WAL_JOB_ARRAY_SUBMIT` record type + compact capnp variant → `jobj.c` hardening → single-record append/commit → wire into `handle_jobsubmit` → `restore_job_array` primitive → wire array expansion into both live apply and `apply_wal_entry` → make the apply pump respect apply failures → **Live Apply Resource Fence** (self-healing, gates promotion on fence state) → compact scheduler-side and API-server-side (Go) local expansion of the push, so downstream consumers only see the compact form, not N expanded records on the wire. + +**Bugs surfaced during build-out**: +- **bug-1438**: array-submit test got a WAL type mismatch then hung all 3 threads on `futex_do_wait` — a WAL-header-offset helper copied from a zero-payload-only test assumed no payload; fixed by passing the entry's start offset explicitly instead of back-computing it (`jap_read_wal_hdr_at`/`jap_read_wal_entry_at`). +- **bug-1442**: AddressSanitizer double-free on `TestWireDispatch_OnePushProducesNRecordsAndLogsOnce` — Go's `go_push_handler` deferred `free_Request` while the datamsg-case free path double-freed the same buffer; fixed and verified. +- bug-1437, 1439, 1440, 1441 also surfaced (accounts-leak test finding, a controller.capnp.h build break, two Go cgo-in-test failures) but the user explicitly scoped those sessions to build/report only — fix paths were documented but **not applied**; these remain open, low-priority. + +**Full regression GO** recorded 2026-07-16 (`1cfdac4`/`2b0a37a`), backfilling bug-1444's fix into the record (`2a76940`). + +**Reopened** (`575f928`, user-directed) for three residual gaps left open by the first GO: +- **bug-1443**: `restore_job_array`'s bulk-insert pass could fail partway through, leaving a partial-visibility window — closed with new purely-additive `kvmgr_reserve()`/`xzarray_reserve()` primitives (`dcf9f39`). +- **bug-1444**: `replay_wal`'s job-count tally was blind to array submissions, causing an `APPLY_DIVERGENT` recovery mismatch — fixed (`16747aa`). +- Task 4.2's boot-replay fence gap: Live Apply Resource Fence extended to cover boot replay, not just live apply (`ea9d3d8`). +- A `jobArray` dataset leak on the scheduler receive path, found and fixed the next day (`53bd686`). + +Fresh `/kiro-validate-impl` GO recorded 2026-07-17 (`9cf61c9`), closing stale notes from the first pass. + +--- + +## Jul 17: bug-1445, first GO passes for ha-term-safety/controller-status, Unity harness fix + +- **nix**: repinned `librtti` to fix a `makeInstance` leak-on-init-failure (`8c7d26a`). +- **bug-1445**: a node's very first promotion-eligibility check in a fresh process ran *before* that node's own first `persist_recover()`/`replay_wal()` call, racing the Live Apply Resource Fence. Fixed by making `raft_cb_become_leader()`'s first-boot path re-check `persist_apply_resource_fenced()` immediately after `cluster_init_phase2()` completes (`ff01db3`). +- **ha-term-safety** and **controller-status** both got their first-ever `/kiro-validate-impl` pass — both **GO** (`84fde22`, `c08a348`). +- **Unity test harness**: nested `UNITY_BEGIN()`/`UNITY_END()` calls across 83 `controller/testsuite/*.c` files were masking `controller_unit_test`'s real exit code and mis-attributing failing tests to the wrong source file. Suite-wide fix (`UnitySetTestFile(__FILE__)` per group function) verified via a single consolidated 972-test, 0-failure summary block matching the ground-truth `RUN_TEST` count (`a81f2a2`). + +--- + +## Jul 18: Group 1 validate-impl batch — 9 specs closed, 1 real bug found and fixed + +A backlog audit found 9 HA/Raft specs with 100% of tasks checked but `spec.json` stuck at `phase: "tasks-generated"` — never run through a formal validation closure. Ran `/kiro-validate-impl` on all 9, each with a fresh ASAN build, full standalone test-suite run, and a live 10-round sequential HA soak: + +| Spec | Result | Notes | +|------|--------|-------| +| raft-sim-conformance | GO | TLA+ conformance harness; 293/293 lib tests, corrected a stale buglog entry (bug-1095) | +| libxce-failover | GO | `ready_for_implementation` flag was stale since 2026-06-18, not a real block | +| ha-entity-snapshots | GO | Traced and confirmed a 2026-07-02 task 7.1 blocker was genuinely resolved (real fix chain, not a checkbox) | +| ha-commit-durability | GO | Re-verified rather than trusting stale evidence given later churn; found bug-1446 (test-harness-only, BusyBox `date` lacks `%N`) | +| **raft-drive-async** | **NO-GO → GO** | Live soak crashed round 9 — see bug-1447 below; re-validated GO after the fix | +| ha-wal-replication | GO | 972/972 tests, 10/10 soak | +| ha-heartbeat-committed-propagation | GO | Needed 2 attempts — first agent run died mid-soak to a transient harness API-connection error, clean retry | +| ha-prevote | GO | Term climbed 2→13 cleanly across the soak, zero election-storm churn — exactly PreVote's purpose | +| ha-elect-ctx-lifecycle | GO | Found bug-1449 (harness-only, see below), closed via retry | + +**bug-1447** (the real one): raft-drive-async's soak crashed at round 9 — an ASAN SEGV deep inside OpenSSL, triggered during a reconnect-offload burst. Initial hypothesis (concurrent `SSL_do_handshake` from two `io_worker` threads) was **wrong and retracted** — `iomgr_start()` only ever creates one `io_worker` thread per `iomgr_t`. Root cause, confirmed via a targeted regression test: a **same-batch use-after-free** — `on_standby_connected()` evicts a stale duplicate peer channel mid-batch, but `epoll_ctl(DEL)` only stops *future* `epoll_wait()` calls, not an event for that same endpoint already sitting in the *current* batch's array. Fixed by ref-counting every endpoint in a batch before dispatch and releasing only after the whole batch completes (mirrors the existing bug-1298 idiom). TDD RED/GREEN confirmed, 2 independent clean 10-round soaks, full suites clean (294/294 + 972/972). + +**bug-1449** (harness-only): `run-ha-soak.sh` had no wait for initial 3-of-3 config convergence before round 1's first kill — a slow cold-start election could let the kill land before config-push reached the standbys, producing a false "no primary elected" failure. Fixed by adding `wait_for_initial_config_convergence()`, mirroring the script's existing reconvergence-wait pattern. + +--- + +## Bug Tally: Jul 13-18 + +### Fixed (9) +1435, 1436, 1438, 1442, 1443, 1444, 1445, 1447, 1449 + +### Known residual / open (low-priority, environment- or scope-limited) +- **1437**: accounts-leak test finding — user scoped session to build/report only, no fix applied +- **1439**: `controller.capnp.h` build break — build/report-only session, no fix applied +- **1440, 1441**: Go `cgo`-in-test failures in `apiserver/backend` — build/report-only sessions; fix paths documented (remove `import "C"` from the test file; add the missing `extern` declaration) but not applied + +--- + +## Commits (Jul 13-18) + +``` +8f614b6 chore: update OpenWolf session state (Group 1 validate-impl batch + bug-1447/1449) +426728b fix(netio): bug-1447 same-batch endpoint use-after-free in io_worker dispatch +4d2c903 fix(test-harness): bug-1449 wait for initial config convergence before round-1 kill +7512dc8 docs(specs): kiro-validate-impl GO closure for 8 HA/raft specs +781b0f7 chore: update OpenWolf session state + cerebrum learning (Unity test-harness fix) +a81f2a2 fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide +e6956ec chore: update OpenWolf session state +6592f9c chore: update OpenWolf session state + cerebrum learnings (meson parallel-timeout false-positive, /tmp/xce_* cleanup) +3f54ef3 chore: update OpenWolf session state (ha-term-safety + controller-status validation) +c08a348 docs(controller-status): record first /kiro-validate-impl pass -- GO +84fde22 docs(ha-term-safety): record first /kiro-validate-impl pass -- GO +1a0960b chore: update OpenWolf session state (bug-1445 fix) +ff01db3 fix: bug-1445 -- first-boot promotion check races its own first replay +5ddc20f docs: log bug-1445 for job-array-single-wal-record's first-boot promotion gap +fb701c2 chore: update OpenWolf session state (job-array-single-wal-record validation pass) +9cf61c9 docs(job-array-single-wal-record): record fresh /kiro-validate-impl GO, close stale notes +ccee18b chore: update OpenWolf session state +2cfb454 docs(job-array-single-wal-record): mark task 1.3's rtti leak note FIXED +8c7d26a chore(nix): repin librtti to fix makeInstance leak-on-init-failure +9176164 chore: update OpenWolf session state (anatomy, cerebrum, memory, token ledger) +b469b17 docs(job-array-single-wal-record): sync spec docs with phase-7 residual-gap closure +53bd686 fix(job-array-single-wal-record): free jobArray dataset on scheduler receive path +ea8d9d3 fix(job-array-single-wal-record): extend Live Apply Resource Fence to boot replay (task 4.2's gap) +5a81362 docs: mark bug-1443 FIXED (task 7.1, commit dcf9f39) +dcf9f39 fix(job-array-single-wal-record): close bug-1443's bulk-insert partial-visibility gap +575f928 plan(job-array-single-wal-record): reopen spec for residual gap closure +85ce5da docs: track recurrences of pre-existing bugs 1437/1438 during re-validation +2a76940 docs(job-array-single-wal-record): backfill bug-1444 for task 6.0a's replay_wal fix +2b0a37a docs(job-array-single-wal-record): record /kiro-validate-impl GO decision +1cfdac4 test(job-array-single-wal-record): full regression pass - GO +c613849 test(job-array-single-wal-record): chaotic-failover array retry scenario +0d8986a test(job-array-single-wal-record): large-array HA soak scenario +16747aa fix(job-array-single-wal-record): replay_wal's job-count tally was blind to array submissions +b7b90e9 test(job-array-single-wal-record): end-to-end integration test for the compact push +9e0d87f feat(job-array-single-wal-record): API-server-side (Go) local expansion of the compact push +1cbd7df fix(job-array-single-wal-record): wire dispatcher was silently dropping compact array pushes +d168471 feat(job-array-single-wal-record): scheduler-side local expansion of the compact push +e0cec75 feat(job-array-single-wal-record): build the compact API-server push +5e7a4aa feat(job-array-single-wal-record): build the compact scheduler push +1888498 test(job-array-single-wal-record): end-to-end fault-injection test for the live fence +8cb8c25 feat(job-array-single-wal-record): gate promotion on the live apply fence +b47cc66 feat(job-array-single-wal-record): add live, self-healing apply resource fence +cc7e99c feat(job-array-single-wal-record): make apply pump respect apply failures +107e383 test(job-array-single-wal-record): close remaining array expansion gaps +b0c7cfe feat(job-array-single-wal-record): wire array expansion into apply_wal_entry +0c4573b feat(job-array-single-wal-record): wire array expansion into live apply +7c755dc feat(job-array-single-wal-record): implement restore_job_array primitive +78eb983 test(job-array-single-wal-record): close remaining array append/commit gaps +d38ca72 feat(job-array-single-wal-record): wire array submissions into handle_jobsubmit +7c4afbb feat(job-array-single-wal-record): implement single-record array append/commit +76862bd feat(job-array-single-wal-record): harden jobj.c job construction primitives +bc4e70b feat(job-array-single-wal-record): add compact array push capnp variant +2e62e73 feat(job-array-single-wal-record): add WAL_JOB_ARRAY_SUBMIT record type +895a204 docs(spec): ha-consistency-invariants — Criterion 2.9 now satisfied +b321963 chore(wolf): session trail + xce-test-ha cross-run collision guard +40c2351 fix(client): bug-1436 count-aware failover macro for xce_get-family calls +510b566 fix(ha): bug-1435 raft_lock serializes concurrent raft_state_t access +``` + +--- + +## Current State + +| Area | Status | +|------|--------| +| job-array-single-wal-record | **COMPLETE** — fresh GO 2026-07-17, all residual gaps closed | +| ha-consistency-invariants | **COMPLETE** — umbrella + all 4 sub-specs (commit-engine, barrier-rollout, single-leader-role, convergence-recovery) | +| ha-term-safety | GO (first validate-impl pass, 2026-07-17) | +| controller-status | GO (first validate-impl pass, 2026-07-17) | +| raft-sim-conformance | GO (2026-07-18) | +| libxce-failover | GO (2026-07-18) | +| ha-entity-snapshots | GO (2026-07-18) | +| ha-commit-durability | GO (2026-07-18) | +| raft-drive-async | GO (2026-07-18, after bug-1447 fix — was NO-GO same day) | +| ha-wal-replication | GO (2026-07-18) | +| ha-heartbeat-committed-propagation | GO (2026-07-18) | +| ha-prevote | GO (2026-07-18) | +| ha-elect-ctx-lifecycle | GO (2026-07-18) | +| HA sequential soak (xce-test-ha) | Round-1 config-convergence race fixed (bug-1449); reliable across all 9 validation passes today | +| netio.c io_worker dispatch | Same-batch endpoint UAF fixed (bug-1447), regression test added | +| Controller test suite | 972/972 clean, Unity harness attribution fixed suite-wide | +| admin-config-tool | Unchanged — spec finalized, pending implementation | +| scheduler-framework | Unchanged — requirements-only, pending design | +| config-upgrade-snapshot | Unchanged — requirements-only | +| Known open (low-priority) | bug-1437 (test-only), bug-1439 (capnp build break), bug-1440/1441 (Go cgo-in-test) — all build/report-only sessions, fix paths documented not applied |