lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

update wolf and project status

Browse source
This commit is contained in:
Rongsong Shen 2026-07-18 14:22:02 +00:00
parent 6e51a97465
commit aeb6e37ce7
6 changed files with 198 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

175
reports/jul13-jul18-2026.md Normal file
View file

@ -0,0 +1,175 @@
# Jul 13 → Jul 18 Progress Report
## Overview
Six days, 57 commits. Two threads dominated: **job-array-single-wal-record** landed as a complete feature (single-WAL-record array job submission, compact scheduler/API-server push, live apply fence, full regression GO — then reopened once for user-directed residual-gap closure and re-validated GO), and a **backlog of 9 HA/Raft specs that had 100%-complete tasks but had never been through a formal `/kiro-validate-impl` closure pass** got worked through systematically, surfacing one genuine production crash bug (bug-1447) that got root-caused and fixed. **ha-consistency-invariants** (the umbrella Raft+CP contract spec) closed out — all 4 sub-specs now complete. The controller test suite's Unity harness got a suite-wide correctness fix (83 files). Net: **9 bugs fixed** (1435, 1436, 1438, 1442, 1443, 1444, 1445, 1447, 1449), **9 specs validated GO** in one day (Jul 18), **1 feature spec** (job-array-single-wal-record) shipped end-to-end.
---
## Jul 13: Election liveness cleanup + ha-consistency-invariants closes out
| Bug | Manifestation | Fix |
|-----|---------------|-----|
| **1435** | 40-round soak FAIL at round 36/40 — `verify_no_loss` got 0/111 jobs confirmed; freshly-elected primary answered `E_XCE_STANDBY` | New `ha_state_t.raft_lock` (recursive mutex) serializes every entry point into `raft_state_t` on the election/vote path (`510b566`) |
| **1436** | 40-round soak FAIL at round 36/40 — `xce_joblist`: connection reset by peer right after wait-primary confirmed the new primary serving in 46ms | `XCECLI_FAILOVER_CALL_COUNT` macro, a count-aware sibling of `XCECLI_FAILOVER_CALL` for `xce_get`-family calls (`40c2351`) |
**ha-consistency-invariants** (the umbrella Raft+CP contract spec covering ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery): Criterion 2.9 (the Facts-Apply invariant, added 2026-06-25) marked **satisfied** — closed by raft-apply-decoupling's fact-apply-after-commit split plus decision D1 (block-but-durable quorum-loss policy). All 4 sub-specs now complete; the umbrella itself needs no separate task list by design (`895a204`).
Also: `xce-test-ha` cross-run collision guard added to the harness (`b321963`).
---
## Jul 14-17: job-array-single-wal-record — feature build-out
The largest single thread this period — 24 commits. Single WAL record per array-job submission instead of N individual records, with N-way expansion on apply.
**Build-out** (Jul 14-15): `WAL_JOB_ARRAY_SUBMIT` record type + compact capnp variant → `jobj.c` hardening → single-record append/commit → wire into `handle_jobsubmit` → `restore_job_array` primitive → wire array expansion into both live apply and `apply_wal_entry` → make the apply pump respect apply failures → **Live Apply Resource Fence** (self-healing, gates promotion on fence state) → compact scheduler-side and API-server-side (Go) local expansion of the push, so downstream consumers only see the compact form, not N expanded records on the wire.
**Bugs surfaced during build-out**:
- **bug-1438**: array-submit test got a WAL type mismatch then hung all 3 threads on `futex_do_wait` — a WAL-header-offset helper copied from a zero-payload-only test assumed no payload; fixed by passing the entry's start offset explicitly instead of back-computing it (`jap_read_wal_hdr_at`/`jap_read_wal_entry_at`).
- **bug-1442**: AddressSanitizer double-free on `TestWireDispatch_OnePushProducesNRecordsAndLogsOnce` — Go's `go_push_handler` deferred `free_Request` while the datamsg-case free path double-freed the same buffer; fixed and verified.
- bug-1437, 1439, 1440, 1441 also surfaced (accounts-leak test finding, a controller.capnp.h build break, two Go cgo-in-test failures) but the user explicitly scoped those sessions to build/report only — fix paths were documented but **not applied**; these remain open, low-priority.
**Full regression GO** recorded 2026-07-16 (`1cfdac4`/`2b0a37a`), backfilling bug-1444's fix into the record (`2a76940`).
**Reopened** (`575f928`, user-directed) for three residual gaps left open by the first GO:
- **bug-1443**: `restore_job_array`'s bulk-insert pass could fail partway through, leaving a partial-visibility window — closed with new purely-additive `kvmgr_reserve()`/`xzarray_reserve()` primitives (`dcf9f39`).
- **bug-1444**: `replay_wal`'s job-count tally was blind to array submissions, causing an `APPLY_DIVERGENT` recovery mismatch — fixed (`16747aa`).
- Task 4.2's boot-replay fence gap: Live Apply Resource Fence extended to cover boot replay, not just live apply (`ea9d3d8`).
- A `jobArray` dataset leak on the scheduler receive path, found and fixed the next day (`53bd686`).
Fresh `/kiro-validate-impl` GO recorded 2026-07-17 (`9cf61c9`), closing stale notes from the first pass.
---
## Jul 17: bug-1445, first GO passes for ha-term-safety/controller-status, Unity harness fix
- **nix**: repinned `librtti` to fix a `makeInstance` leak-on-init-failure (`8c7d26a`).
- **bug-1445**: a node's very first promotion-eligibility check in a fresh process ran *before* that node's own first `persist_recover()`/`replay_wal()` call, racing the Live Apply Resource Fence. Fixed by making `raft_cb_become_leader()`'s first-boot path re-check `persist_apply_resource_fenced()` immediately after `cluster_init_phase2()` completes (`ff01db3`).
- **ha-term-safety** and **controller-status** both got their first-ever `/kiro-validate-impl` pass — both **GO** (`84fde22`, `c08a348`).
- **Unity test harness**: nested `UNITY_BEGIN()`/`UNITY_END()` calls across 83 `controller/testsuite/*.c` files were masking `controller_unit_test`'s real exit code and mis-attributing failing tests to the wrong source file. Suite-wide fix (`UnitySetTestFile(__FILE__)` per group function) verified via a single consolidated 972-test, 0-failure summary block matching the ground-truth `RUN_TEST` count (`a81f2a2`).
---
## Jul 18: Group 1 validate-impl batch — 9 specs closed, 1 real bug found and fixed
A backlog audit found 9 HA/Raft specs with 100% of tasks checked but `spec.json` stuck at `phase: "tasks-generated"` — never run through a formal validation closure. Ran `/kiro-validate-impl` on all 9, each with a fresh ASAN build, full standalone test-suite run, and a live 10-round sequential HA soak:
| Spec | Result | Notes |
|------|--------|-------|
| raft-sim-conformance | GO | TLA+ conformance harness; 293/293 lib tests, corrected a stale buglog entry (bug-1095) |
| libxce-failover | GO | `ready_for_implementation` flag was stale since 2026-06-18, not a real block |
| ha-entity-snapshots | GO | Traced and confirmed a 2026-07-02 task 7.1 blocker was genuinely resolved (real fix chain, not a checkbox) |
| ha-commit-durability | GO | Re-verified rather than trusting stale evidence given later churn; found bug-1446 (test-harness-only, BusyBox `date` lacks `%N`) |
| **raft-drive-async** | **NO-GO → GO** | Live soak crashed round 9 — see bug-1447 below; re-validated GO after the fix |
| ha-wal-replication | GO | 972/972 tests, 10/10 soak |
| ha-heartbeat-committed-propagation | GO | Needed 2 attempts — first agent run died mid-soak to a transient harness API-connection error, clean retry |
| ha-prevote | GO | Term climbed 2→13 cleanly across the soak, zero election-storm churn — exactly PreVote's purpose |
| ha-elect-ctx-lifecycle | GO | Found bug-1449 (harness-only, see below), closed via retry |
**bug-1447** (the real one): raft-drive-async's soak crashed at round 9 — an ASAN SEGV deep inside OpenSSL, triggered during a reconnect-offload burst. Initial hypothesis (concurrent `SSL_do_handshake` from two `io_worker` threads) was **wrong and retracted** — `iomgr_start()` only ever creates one `io_worker` thread per `iomgr_t`. Root cause, confirmed via a targeted regression test: a **same-batch use-after-free** — `on_standby_connected()` evicts a stale duplicate peer channel mid-batch, but `epoll_ctl(DEL)` only stops *future* `epoll_wait()` calls, not an event for that same endpoint already sitting in the *current* batch's array. Fixed by ref-counting every endpoint in a batch before dispatch and releasing only after the whole batch completes (mirrors the existing bug-1298 idiom). TDD RED/GREEN confirmed, 2 independent clean 10-round soaks, full suites clean (294/294 + 972/972).
**bug-1449** (harness-only): `run-ha-soak.sh` had no wait for initial 3-of-3 config convergence before round 1's first kill — a slow cold-start election could let the kill land before config-push reached the standbys, producing a false "no primary elected" failure. Fixed by adding `wait_for_initial_config_convergence()`, mirroring the script's existing reconvergence-wait pattern.
---
## Bug Tally: Jul 13-18
### Fixed (9)
1435, 1436, 1438, 1442, 1443, 1444, 1445, 1447, 1449
### Known residual / open (low-priority, environment- or scope-limited)
- **1437**: accounts-leak test finding — user scoped session to build/report only, no fix applied
- **1439**: `controller.capnp.h` build break — build/report-only session, no fix applied
- **1440, 1441**: Go `cgo`-in-test failures in `apiserver/backend` — build/report-only sessions; fix paths documented (remove `import "C"` from the test file; add the missing `extern` declaration) but not applied
---
## Commits (Jul 13-18)
```
8f614b6 chore: update OpenWolf session state (Group 1 validate-impl batch + bug-1447/1449)
426728b fix(netio): bug-1447 same-batch endpoint use-after-free in io_worker dispatch
4d2c903 fix(test-harness): bug-1449 wait for initial config convergence before round-1 kill
7512dc8 docs(specs): kiro-validate-impl GO closure for 8 HA/raft specs
781b0f7 chore: update OpenWolf session state + cerebrum learning (Unity test-harness fix)
a81f2a2 fix(controller-tests): make Unity test-file attribution and counters consistent suite-wide
e6956ec chore: update OpenWolf session state
6592f9c chore: update OpenWolf session state + cerebrum learnings (meson parallel-timeout false-positive, /tmp/xce_* cleanup)
3f54ef3 chore: update OpenWolf session state (ha-term-safety + controller-status validation)
c08a348 docs(controller-status): record first /kiro-validate-impl pass -- GO
84fde22 docs(ha-term-safety): record first /kiro-validate-impl pass -- GO
1a0960b chore: update OpenWolf session state (bug-1445 fix)
ff01db3 fix: bug-1445 -- first-boot promotion check races its own first replay
5ddc20f docs: log bug-1445 for job-array-single-wal-record's first-boot promotion gap
fb701c2 chore: update OpenWolf session state (job-array-single-wal-record validation pass)
9cf61c9 docs(job-array-single-wal-record): record fresh /kiro-validate-impl GO, close stale notes
ccee18b chore: update OpenWolf session state
2cfb454 docs(job-array-single-wal-record): mark task 1.3's rtti leak note FIXED
8c7d26a chore(nix): repin librtti to fix makeInstance leak-on-init-failure
9176164 chore: update OpenWolf session state (anatomy, cerebrum, memory, token ledger)
b469b17 docs(job-array-single-wal-record): sync spec docs with phase-7 residual-gap closure
53bd686 fix(job-array-single-wal-record): free jobArray dataset on scheduler receive path
ea8d9d3 fix(job-array-single-wal-record): extend Live Apply Resource Fence to boot replay (task 4.2's gap)
5a81362 docs: mark bug-1443 FIXED (task 7.1, commit dcf9f39)
dcf9f39 fix(job-array-single-wal-record): close bug-1443's bulk-insert partial-visibility gap
575f928 plan(job-array-single-wal-record): reopen spec for residual gap closure
85ce5da docs: track recurrences of pre-existing bugs 1437/1438 during re-validation
2a76940 docs(job-array-single-wal-record): backfill bug-1444 for task 6.0a's replay_wal fix
2b0a37a docs(job-array-single-wal-record): record /kiro-validate-impl GO decision
1cfdac4 test(job-array-single-wal-record): full regression pass - GO
c613849 test(job-array-single-wal-record): chaotic-failover array retry scenario
0d8986a test(job-array-single-wal-record): large-array HA soak scenario
16747aa fix(job-array-single-wal-record): replay_wal's job-count tally was blind to array submissions
b7b90e9 test(job-array-single-wal-record): end-to-end integration test for the compact push
9e0d87f feat(job-array-single-wal-record): API-server-side (Go) local expansion of the compact push
1cbd7df fix(job-array-single-wal-record): wire dispatcher was silently dropping compact array pushes
d168471 feat(job-array-single-wal-record): scheduler-side local expansion of the compact push
e0cec75 feat(job-array-single-wal-record): build the compact API-server push
5e7a4aa feat(job-array-single-wal-record): build the compact scheduler push
1888498 test(job-array-single-wal-record): end-to-end fault-injection test for the live fence
8cb8c25 feat(job-array-single-wal-record): gate promotion on the live apply fence
b47cc66 feat(job-array-single-wal-record): add live, self-healing apply resource fence
cc7e99c feat(job-array-single-wal-record): make apply pump respect apply failures
107e383 test(job-array-single-wal-record): close remaining array expansion gaps
b0c7cfe feat(job-array-single-wal-record): wire array expansion into apply_wal_entry
0c4573b feat(job-array-single-wal-record): wire array expansion into live apply
7c755dc feat(job-array-single-wal-record): implement restore_job_array primitive
78eb983 test(job-array-single-wal-record): close remaining array append/commit gaps
d38ca72 feat(job-array-single-wal-record): wire array submissions into handle_jobsubmit
7c4afbb feat(job-array-single-wal-record): implement single-record array append/commit
76862bd feat(job-array-single-wal-record): harden jobj.c job construction primitives
bc4e70b feat(job-array-single-wal-record): add compact array push capnp variant
2e62e73 feat(job-array-single-wal-record): add WAL_JOB_ARRAY_SUBMIT record type
895a204 docs(spec): ha-consistency-invariants — Criterion 2.9 now satisfied
b321963 chore(wolf): session trail + xce-test-ha cross-run collision guard
40c2351 fix(client): bug-1436 count-aware failover macro for xce_get-family calls
510b566 fix(ha): bug-1435 raft_lock serializes concurrent raft_state_t access
```
---
## Current State
| Area | Status |
|------|--------|
| job-array-single-wal-record | **COMPLETE** — fresh GO 2026-07-17, all residual gaps closed |
| ha-consistency-invariants | **COMPLETE** — umbrella + all 4 sub-specs (commit-engine, barrier-rollout, single-leader-role, convergence-recovery) |
| ha-term-safety | GO (first validate-impl pass, 2026-07-17) |
| controller-status | GO (first validate-impl pass, 2026-07-17) |
| raft-sim-conformance | GO (2026-07-18) |
| libxce-failover | GO (2026-07-18) |
| ha-entity-snapshots | GO (2026-07-18) |
| ha-commit-durability | GO (2026-07-18) |
| raft-drive-async | GO (2026-07-18, after bug-1447 fix — was NO-GO same day) |
| ha-wal-replication | GO (2026-07-18) |
| ha-heartbeat-committed-propagation | GO (2026-07-18) |
| ha-prevote | GO (2026-07-18) |
| ha-elect-ctx-lifecycle | GO (2026-07-18) |
| HA sequential soak (xce-test-ha) | Round-1 config-convergence race fixed (bug-1449); reliable across all 9 validation passes today |
| netio.c io_worker dispatch | Same-batch endpoint UAF fixed (bug-1447), regression test added |
| Controller test suite | 972/972 clean, Unity harness attribution fixed suite-wide |
| admin-config-tool | Unchanged — spec finalized, pending implementation |
| scheduler-framework | Unchanged — requirements-only, pending design |
| config-upgrade-snapshot | Unchanged — requirements-only |
| Known open (low-priority) | bug-1437 (test-only), bug-1439 (capnp build break), bug-1440/1441 (Go cgo-in-test) — all build/report-only sessions, fix paths documented not applied |