lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

96
.kiro/specs/scheduler-controller-msg-gap/research.md
View file

@ -206,3 +206,99 @@ All 19 NotifyData event tags:
- **R-2:** Confirm the notify-driven `schedUpdateJob` actually reaches `sched_update_existing_job()`: the dequeue `XCE_SCHED_OP_UPDATE_JOB` case calls `sched_update_jobs()` (bulk path); verify `sched_update_jobs` performs update-in-place for existing jobids (design Component 5 said it should) rather than duplicating jobs.
- **R-3:** Is there a planned controller-side emitter for `Data_data_resources`/`Data_data_resvals` pushes to the scheduler (e.g., post-failover config resync in the persistent_ha work)? If not, decide whether Req 1's handlers stay forward-looking or get an emitter spec.
- **R-4:** Whether `makeInstance(&resobj_type, r)` deep-copies all of `xce_resource_t` (name string included) — determines whether resource row payloads can be freed immediately after `makeResource` returns.
---
## Gap Re-Audit (2026-06-18)
> Post-implementation audit. tasks.md = 36/36 `[x]` (9.9 gated), phase=implemented. Method: every EARS AC re-verified against current on-disk source via codegraph (handler.c, scheduler.c, schedobj.c, plugin.c, helper.c) + grep. This pass tracks the 2026-06-09 G-1..G-7 findings: **G-1 and G-2 are now FIXED; G-3, G-4, G-5, G-6, G-7 remain OPEN; one NEW finding (G-8) supersedes the 2026-06-09 R-2 caveat.**
### 0. Delta vs 2026-06-09 audit
| Prior gap | 2026-06-18 status | Evidence |
|---|---|---|
| G-1 UPDATE_RESVALS clobbered worker index 0 | **FIXED** | scheduler.c:826-831 now resolves `rv->name`→`key_find(sched->resmgr)`→`r->idx`, `zarray_put(&w->resvals, r->idx, rv->val)`; unknown resource WARN+skip. Matches schedobj.c:283-291. |
| G-2 UPDATE_RESOURCES could not update existing resource | **FIXED** | scheduler.c:775-792: on `makeResource()`==NULL (EEXIST), `key_find` the existing resource and copy `valtype/ordtype/flags/interval`; WARN on update and on lookup-miss. "Add-or-update" now satisfied for mutable scalar fields. |
| G-3 killer/dequeue never free new-op payloads (leak) | **OPEN** | `sched_pend_update_killer` (scheduler.c:311-320) still has empty `break;` for UPDATE_QUEUE / UPDATE_RESOURCES / UPDATE_RESVALS_FULL / UPDATE_RESVALS / JOB_RUSAGE. Dequeue cases (scheduler.c:767-847) free nothing. Ownership IS transferred (handler.c:297-316 nullifies `notify->data.*`; sched_handle_dataset:130-145 nullifies rows) ⇒ confirmed per-message leak. |
| G-4 job update copies only `state` | **OPEN (worse — see G-8)** | `sched_update_existing_job` (scheduler.c:707-725) still copies only `job->state`; priority/queue/resreq untouched. AND it is never reached (G-8). |
| G-5 rusage not stored on job | **OPEN** | `xce_sched_job_t` (sched/types.h:80-96) has NO rusage field. JOB_RUSAGE case (scheduler.c:836-847) calls `plugin_job_rusage_update()` then drops+leaks the payload. UB-6.1 "store the usage data on the job object" unmet. |
| G-6 test debt (9.2/9.3/9.4/9.6/9.7 `[x]` but absent; 9.5 stub; 9.8 TEST_PASS) | **OPEN** | testsuite/test.c still defines a LOCAL stub `sched_update_existing_queue` (test.c:25-34) — real schedobj.c impl never linked. Only real new tests: 9.1 notify-event-name (test.c:399-423) + enum compile check. No tests for any enqueue handler, dequeue case, `sched_update_existing_job`, `sched_op_name`, leaks. |
| G-7 doc drift: `xce_sched_op_name` in lib/helper per design/tasks 1.1,1.3 | **OPEN** | Implemented as static `sched_op_name()` in scheduler.c:685-705, NOT `xce_sched_op_name()` in lib/helper.c. design.md Component 8 + tasks.md 1.1/1.3 (both `[x]`) still describe the wrong location/name. |
### 1. NEW finding
- **G-8 (correctness, HIGH — supersedes R-2): `sched_update_existing_job()` is dead code; notify-driven job updates take the bulk CREATE path.** `grep -rn sched_update_existing_job src/` returns ONLY its definition (scheduler.c:707) — zero callers. The `XCE_SCHED_OP_UPDATE_JOB` dequeue case (scheduler.c:755-758) routes to `sched_update_jobs()` (scheduler.c:629-642), which for every job calls `make_sched_job()` (a fresh `calloc`) + `sched_add_job()` (`key_add` into `sched->jobs`). For an existing jobid this either duplicate-keys or replaces the live `xce_sched_job_t` with a new object — discarding `allocs`/`accounts`/`schedobj` runtime state and leaking the old node. There is no `-ENOENT` for an unknown jobid (UB-3.2 unmet) and no in-place mutation (UB-3.1 unmet). Req 3 is effectively NOT closed at the dequeue layer despite the handler-side wiring and the `[x]` on tasks 5.6 / 3.1 / 3.2.
### 2. Requirement-to-Asset map (tagged)
| Req / AC | Asset (file:line) | Tag |
|---|---|---|
| 1 / UB-1.1 resources add-or-update | handler.c:59-80; scheduler.c:770-796 | CLOSED |
| 1 / UB-1.2 resvals→sharedInst by idx | handler.c:82-103; scheduler.c:797-815 | CLOSED |
| 1 / UB-1.3 skip NULL rows | scheduler.c:774, 802-804 | CLOSED |
| 1 / UB-1.4 -ENOMEM, free partial | handler.c:68-71,91-94 | CLOSED |
| 2 / UB-2.1, UB-2.2 dead pushmsg gone | handler.c:333-362 (no pushmsg case); grep: no `sched_handle_jobpush`/`workerpush` | CLOSED |
| 3 / UB-3.1 update mutable fields | scheduler.c:707-725 (`state` only) AND unreachable | OPEN (G-4 + G-8) |
| 3 / UB-3.2 -ENOENT unknown jobid | scheduler.c:716-718 — never reached; live path `sched_update_jobs` create-anyway | OPEN (G-8) |
| 3 / UB-3.3 -EINVAL NULL payload | handler.c:181-183 | CLOSED (enqueue layer) |
| 4 / UB-4.1/4.2/4.3 queue update | schedobj.c:485-511; scheduler.c:767-769 | CLOSED |
| 5 / UB-5.1 per-worker resvals | scheduler.c:816-835 (idx fix) | CLOSED |
| 5 / UB-5.2 skip unknown host | scheduler.c:824-825 | CLOSED |
| 5 / UB-5.3 zero values→0 | handler.c:236-238 | CLOSED |
| 6 / UB-6.1 store rusage on job | scheduler.c:836-847 (plugin only; not stored) | OPEN (G-5) |
| 6 / UB-6.2 -ENOENT unknown jobid | scheduler.c:840-841 | CLOSED |
| 6 / UB-6.3 -EINVAL NULL rusage | handler.c:258-260 | CLOSED |
| 7 / UB-7.1 WARN op+errno | scheduler.c:855-858 | CLOSED |
| 7 / UB-7.2 unknown notify→-EINVAL | handler.c:318-320 | CLOSED |
| 7 / UB-7.3 unknown body_which→-EINVAL | handler.c:350-352 | CLOSED |
| 8 / UB-8.1 op name in req log | handler.c:331 | CLOSED |
| 8 / UB-8.2 dataset table name | handler.c:109 | CLOSED |
| 8 / UB-8.3 notify event name | handler.c:279 | CLOSED |
| 8 / UB-8.4 UNKNOWN(%d) fallback | helper.c (xce_notify_event_name) + scheduler.c:703 | CLOSED |
| 8 / UB-8.5 symbolic op name in WARN | scheduler.c:685-705 `sched_op_name` (static, not lib helper) | CLOSED-with-deviation (G-7) |
**Score: 17 / 23 ACs fully closed.** OPEN: UB-3.1, UB-3.2 (G-8/G-4), UB-6.1 (G-5). Cross-cutting non-AC defects: G-3 (leak), G-6 (test debt), G-7 (doc drift). Of the 8 top-level requirements, **Req 2, 4, 5, 7, 8 are fully CLOSED; Req 1 CLOSED; Req 3 OPEN; Req 6 PARTIAL (delivery yes, storage no).**
### 3. Residual / deferred gaps (priority order)
1. **G-8 (HIGH, correctness)** — notify `schedUpdateJob` hits the create path; `sched_update_existing_job` is dead code. Corrupts/duplicates live job state; no -ENOENT. Req 3 unmet.
2. **G-3 (MED, leak)** — five new ops never freed despite ownership transfer. Unbounded slow leak in a long-lived daemon; aggravated by G-8 (old job node also leaked on overwrite).
3. **G-5 (MED, requirement shortfall)** — rusage never stored on job (no field exists). UB-6.1 unmet.
4. **G-4 (LOW, requirement shortfall)** — even once reachable, job update applies only `state`; priority/queue/resreq dropped.
5. **G-6 (MED, test debt / tracking integrity)** — tasks 9.2/9.3/9.4/9.6/9.7 `[x]` but no tests in test.c; 9.5 tests a local stub, not real schedobj.c; 9.8 is TEST_PASS+comment. Checkboxes overstate coverage; none of the dequeue correctness fixes (G-1/G-2) are tested.
6. **G-7 (LOW, doc drift)** — design.md Component 8 + tasks 1.1/1.3 name `xce_sched_op_name` in lib/helper; reality is static `sched_op_name` in scheduler.c.
### 4. Integration risks
- Req 1 handlers remain forward-looking: no controller emitter pushes `Data_data_resources`/`Data_data_resvals` to the scheduler today (controller/sched.c pushes only schedjobs/schedworkers; opget.c serves them only as get-responses). G-1/G-2 fixes are correct but untriggered in production — regression-prone because untested AND unexercised. (Prior R-3 still open.)
- G-8 + G-3 interact: overwriting a live job via `sched_add_job` leaks the prior node's `schedobj`/`allocs`/`accounts`. Surfaces under ASAN only when a notify update arrives for an already-loaded job — exactly the persistent_ha post-failover resync scenario.
### 5. Options for open work (G-8, G-3, G-5, G-4, G-6, G-7)
**Option A — Minimal correctness patch.**
- G-8: in dequeue `XCE_SCHED_OP_UPDATE_JOB`, branch on `njobs==1 && sched_get_job(jobid)!=NULL` → `sched_update_existing_job()`; else keep `sched_update_jobs()`. OR route the notify-only op to a distinct `XCE_SCHED_OP_*` so the single-job update never enters the bulk path.
- G-4: copy priority/queue/resreq (+resreq realloc) into `sched_update_existing_job`, mirroring `sched_update_existing_queue` (schedobj.c:498-506).
- G-3: add per-op `free_*` in `sched_pend_update_killer` (reuse src/lib/request.c deep-free helpers — confirm exports per prior R-1 against ASAN).
- G-7: edit design.md/tasks.md to record the `sched_op_name` location/name deviation.
- Defer G-5, G-6.
- Effort **S–M**, Risk **Medium** (ownership/free work must pass ASAN; double-free risk).
**Option B — Close everything incl. requirement + tests.**
- All of A, plus G-5 (add `xce_job_rusages_t* rusages` to `xce_sched_job_t` + free in `kill_sched_job` + store in JOB_RUSAGE case) and G-6 (new `testsuite/test_msg_handlers.c` linking scheduler objects + capnp headers — the friction that derailed Task 9 originally — replacing the local-stub 9.5 and TEST_PASS 9.8 with real assertions; un-check 9.2-9.8 until real).
- Effort **M–L (~5-7 days)**, Risk **Medium-High** (new test binary + meson wiring; struct lifetime changes).
**Option C — Hybrid (recommended).**
- Phase 1: A's correctness/leak fixes (G-8, G-4, G-3) with a failing test first where linkable; G-7 doc sync.
- Phase 2: G-6 dedicated test program for enqueue+dequeue paths incl. the G-1/G-2 regressions.
- Phase 3 (needs decision): G-5 — add job-side rusage storage, OR formally amend UB-6.1 to "deliver to plugins only" (zero code). Do not leave requirement and code disagreeing.
- Effort **M**, Risk **Medium**; staged so correctness lands first and tracking becomes honest before requirement closure.
### 6. Recommendation
Do NOT treat the spec as truly `implemented`. Five top-level requirements (2,4,5,7,8) and Req 1 are genuinely closed; **Req 3 is broken at the dequeue layer (G-8) and Req 6 stores nothing (G-5)**. Adopt Option C. Re-open/annotate tasks 5.6 (job update wiring), 9.2-9.8 (test debt), and 1.1/1.3 (doc drift). G-8 and G-3 are defects against the approved spec and should block a clean `implemented` sign-off.
### 7. Research still needed
- **R-1 (carry-over):** Confirm exported `free_*` deep-free helpers in src/lib/request.c for `xce_queue_t` / `xce_resval_list_t` / `xce_job_rusages_t` / `xce_resource_t*[]` and validate under ASAN before wiring the killer (G-3).
- **R-3 (carry-over):** Planned controller emitter for `Data_data_resources`/`Data_data_resvals` to scheduler (persistent_ha resync)? Determines whether Req 1 stays forward-looking.
- **R-4 (carry-over):** Does `makeResource`/`makeInstance` deep-copy `xce_resource_t` (name incl.)? Determines if resource rows can be freed right after enqueue (G-3 scope).