lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

61
.kiro/specs/resval-id-decoupling/research.md
View file

@ -240,3 +240,64 @@ Mark tasks approved, record R5-2 (SHOULD) and the missing R6-1 test as known acc
- **RN-1**: Confirm `XCE_SCHED_OP_UPDATE_RESVALS` (`scheduler.c:814`) is a live path (who sends `schedUpdateResvals` capnp field @8, and with which resources) before fixing the hardcoded slot 0 — it may be stub/dead code.
- **RN-2**: Verify no out-of-tree or generated consumer (e.g., `src/build/lib/controller.capnp.c`) still references `ValueWithIdx` codecs before deleting the schema struct; a clean `ninja -C src/build` after removal settles this.
- **RN-3**: EARS-R2-4 (resource add/remove mid-run) has no test harness today; determine whether the scheduler testsuite can drive `sched_handle_resources` followed by a worker push without a live controller.
---
## Gap Re-Audit (2026-06-18)
> POST-IMPLEMENTATION GAP AUDIT. Tasks 24/24 `[x]`, `spec.json.phase: implemented`.
> Independent re-verification of R1–R6 against current `persistent_ha` code via codegraph + grep + verbatim reads. Supersedes the 2026-06-09 validation pass; records two status changes since then.
### Changes since the 2026-06-09 pass
1. **Process gap CLOSED.** `spec.json` now shows `phase: implemented` and `approvals.tasks.approved: true` (was `false`/`tasks-generated`). The prior "tasks unapproved" finding no longer applies.
2. **Adjacent slot-0 defect FIXED.** `XCE_SCHED_OP_UPDATE_RESVALS` (per-host running update) at `src/scheduler/src/scheduler.c:816-834` now resolves `rv->name` via `key_find(sched->resmgr, ...)` and stores with `zarray_put(&w->resvals, r->idx, rv->val)` (line 831) — the hardcoded `zarray_put(..., 0, ...)` flagged on 2026-06-09 is gone. The last index-style hazard in the resval surface is closed. (Belongs to `scheduler-controller-msg-gap` scope, but worth recording: the defect class this spec targets is now eliminated across ALL resval paths A/B/C and both running-update handlers.)
### Requirement-to-Asset Map
| Req | EARS | Asset(s) — file:line | Tag |
|-----|------|----------------------|-----|
| R1 — name-keyed shared resvals in initial config | R1-1 | `src/schema/controller.capnp:241` `resvals @6: List(NameValue)`; `src/controller/dataex.c:381` `collect_resval(c->resmgr, &c->sharedinst, l)`; `src/controller/sched.c` cast to `xce_name_val_t**` | MET |
| | R1-2 | `src/scheduler/src/scheduler.c:587-595` `sched_init_sharedInst`: `key_find(sched->resmgr, nv->name)` → `zarray_put(&sched->sharedInst, r->idx, val)` | MET |
| | R1-3 | `scheduler.c:583-592`: NULL nv/name skipped; unknown name → `WARN` + `continue`, `sharedInst` untouched | MET |
| | R1-4 | Same name→`r->idx` resolution as Path C (`scheduler.c:797-812`, `UPDATE_RESVALS_FULL`) | MET (by inspection; no equivalence test — see R6-1) |
| R2 — name-keyed per-worker resvals to scheduler | R2-1 | `controller.capnp:140` `resvals @5: List(NameValue)`; `dataex.c:374` `collect_resvals_workers(c->resmgr, l)`; `sched.c:477` `collect_resval(c->resmgr, &w->host.resvals, ...)` | MET |
| | R2-2 | `src/scheduler/src/schedobj.c:274-292` `make_sched_worker`: name lookup → `zarray_put(&worker->resvals, r->idx, val)` (line 291) | MET |
| | R2-3 | `schedobj.c:279-288`: NULL skip; unknown name → `WARN` + skip; worker construction continues | MET |
| | R2-4 | Resolution runs against live `sched->resmgr` on every receipt; no idx cached across messages (`schedobj.c:274-292`) | MET (by design; no mid-run add/remove test — RN-3) |
| R3 — name-keyed worker resvals to API server | R3-1 | `src/controller/apiserver.c` `apiserver_worker_change`: same `collect_resval` + `xce_name_val_t**` path as scheduler push | MET |
| | R3-2 | `src/frontend/apiserver/backend/convert.go:100-105` iterates `worker.resvals_to_sched` (`unsafe.Slice`), `convert.go:283` `cNameValToWorkerResval` (name-keyed, nil-checked); `convert_pure.go:210` `ResourceValues: r.resvals`; `models/hosts.go:94` `ResourceValues []ResourceDistributionInfo` | MET |
| R4 — Path C unchanged | R4-1 | `src/scheduler/src/handler.c` (`Data_data_resvals` → `XCE_SCHED_OP_UPDATE_RESVALS_FULL`) and `scheduler.c:797-814` untouched, still `xce_resval_t` name-keyed | MET |
| | R4-2 | `xce_conf4sched_t.resvals` (`typemap.h`) and `xce_worker_t.resvals_to_sched` (`types.h`) are `xce_name_val_t**`; codec auto-generated; `free_name_val_list` (`src/lib/utils.c:388`); all 7 controller free sites converted (codegraph: 7 callers across apiserver.c, dataex.c, opget.c, reply.c, sched.c) | MET |
| R5 — remove index-keyed call sites | R5-1 | Zero cross-subsystem index-keyed transfers remain (grep: `collect_resval_with_idx`/`free_resval_with_idx_list` = 0 matches in src/ excl. build/); both receivers resolve by name | MET |
| | R5-2 | `xce_val_with_idx_t` typedef (`src/include/xce/types.h:38-41`) and capnp `struct ValueWithIdx` (`controller.capnp:178`) still present — zero non-generated users, **no scoping comment** | Missing (SHOULD-level) |
| R6 — no regression / correctness under divergent indices | R6-1 | Correct by code inspection (name resolution is order-independent at `scheduler.c:587` and `schedobj.c:283`); scheduler testsuite (`src/scheduler/testsuite/test.c`) has **zero** references to `sched_init_sharedInst` / `make_sched_worker` / divergent resval order | Missing (test only; code MET) |
| | R6-2 | Same-order behavior preserved; controller/scheduler suites passed at task 8.2 | MET |
| | R6-3 | `calc_wanted_*` read `sharedInst` / `worker->resvals` via scheduler-side `r->idx`; all writers (`sched_init_sharedInst`, `make_sched_worker`, both UPDATE_RESVALS handlers) now key by `r->idx` | MET |
**Score: 21/23 EARS MET. 2 Missing — both non-MUST (R5-2 SHOULD; R6-1 test-only, code present).** All MUST-level acceptance criteria are demonstrably satisfied.
### Residual Gaps
- **R5-2 (SHOULD):** Dead `xce_val_with_idx_t` / `struct ValueWithIdx` linger without the "removed or restricted to intra-subsystem use, with a comment explaining its scope" that EARS-R5-2 requests. Invites accidental reuse of the very index-keyed shape this feature eliminated.
- **R6-1 (test gap):** The feature's entire raison d'être — correctness when controller and scheduler register resources in *different* orders — has no automated test. Code is order-independent by construction, but the invariant is unguarded against regression.
- **Carry-forward note:** RN-3 (EARS-R2-4 mid-run resource add/remove) still has no harness. Lower priority than R6-1.
### Options for Open Work
**Option A — Close residuals in-spec (Recommended).**
Delete `xce_val_with_idx_t` (types.h:38-41) + `struct ValueWithIdx` (controller.capnp:178) [or add the EARS-R5-2 scoping comment if retained for tooling], and add one table-driven scheduler testsuite case feeding `sched_init_sharedInst` / `make_sched_worker` a name-val list against a resmgr populated in reversed order, asserting `sharedInst` / `worker->resvals` slot contents.
- Effort: **S** (~2 files + 1 test). Risk: **Low**.
- Caveat: deleting from `types.h` is a shared-header edit — requires a clean `nix develop --command ninja -C src/build` to confirm no generated/out-of-tree consumer references `ValueWithIdx` (RN-2).
**Option B — Defer to a cleanup/test sweep across HA specs.**
Bundle R5-2 + R6-1 + RN-3 with similar dead-type/test debt in adjacent specs.
- Effort: **S-M**. Risk: **Low**. Cost: residue stays detached from this spec's history; may be dropped if the sweep stalls.
**Option C — Accept and close as-is.**
Record R5-2 and the missing R6-1 test as known accepted debt; all MUST criteria met, slot-0 hazard already fixed, spec.json fully approved.
- Effort: **none**. Risk: **Low-Medium** — dead type invites reuse; the feature's core scenario remains unverified by automation.
### Verdict
Feature is **functionally complete and CLOSED for all MUST requirements** (R1-1..R1-3, R2-1..R2-4, R3-1/R3-2, R4-1/R4-2, R5-1, R6-2/R6-3). Two non-blocking residuals remain (R5-2 dead-type cleanup; R6-1 divergent-order test). Recommend **Option A** to fully satisfy the SHOULD clause and lock the core invariant under test. Effort **S**, Risk **Low**.