lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

57
.kiro/specs/worker-join-refactor/research.md
View file

@ -287,3 +287,60 @@ Skip a dedicated build; rely on the next `/xce-build` from any active spec (e.g.
## Research Needed
- None for this spec. Single carry-forward verification item: green `/xce-build controller` run on current HEAD (worker.c/join.c unmodified since 93ed282, but HA-branch changes to ha.c/raft.c in the working tree share the controller test binary — confirm no link/test interference).
---
## Gap Re-Audit (2026-06-18)
**Mode:** POST-IMPLEMENTATION GAP AUDIT (spec.json phase=implemented; tasks 12/12 [x]).
**Method:** EARS requirement → on-disk code verification via codegraph + targeted Read. Cites file:line at current HEAD.
**Scope reminder (roadmap line 26):** this spec covers **attribute/endpoint update on rejoin only**. The job-state resync (running/finished jobs re-reported on rejoin) is the SEPARATE, not-yet-started spec **worker-rejoin-resync** (planned, no effort). That is explicitly NOT a gap of THIS spec — see Boundary note below.
### Requirement-to-Asset Map
| Req | Statement (abbrev) | Asset (file:line) | Status |
|-----|--------------------|-------------------|--------|
| 1.1 | `key_find` before create-or-update | `handle_worker_join` brief READ lock + `key_find` — join.c:91-95 | **CLOSED** |
| 1.2 | Branch on lookup result (new vs exists) | `if (w==NULL) init_worker … else rejoin_worker` — join.c:101-113 | **CLOSED** |
| 1.3 | Reject on non-ENOENT `key_find` error | `if (rc<0 && rc!=-ENOENT){ … idx=rc; }` — join.c:97-100 | **CLOSED** |
| 2.1 | Create new entry via `makeWorker` | `init_worker` builds tmp `xce_worker_t`, calls `makeWorker(c->workermgr,&tmp)` — worker.c:339-354 | **CLOSED** |
| 2.2 | On create fail: log, set status, no `on_worker_join` | NULL→`ERROR`+`return -errno`; caller sets `reply.status=idx`, `if(idx<0) return idx` before `on_worker_join` — worker.c:351-353, join.c:104-105/138-147 | **CLOSED** |
| 2.3 | Assign new idx to `priv->u.idx` | `priv->u.idx = idx;` before `on_worker_join` — join.c:146 | **CLOSED** |
| 3.1 | Update `hostattrs` on rejoin | `rejoin_worker` frees+`makeString` hosttype/hostmodel, copies numeric attrs — worker.c:376-386 | **CLOSED** |
| 3.2 | Update `ep` on rejoin | `apply_join_state` sets `w->ep = ep` (worker.c:295), invoked by rejoin path — worker.c:388 | **CLOSED** |
| 3.3 | Preserve admin status bits | `admin_status = w->host.status & XCE_WORKER_ADMIN_MASK` (worker.c:374) restored `w->host.status |= admin_status` (worker.c:390); mask = LOCKEDU\|CLOSED\|HIDDEN\|CLOSED_BY_ADMIN — worker.c:275-277 | **CLOSED** |
| 4.1 | Clear `XCE_WORKER_UNAVAIL` | `w->host.status &= ~XCE_WORKER_UNAVAIL;` — worker.c:294 | **CLOSED** |
| 4.2 | Apply hconfig overrides (type/model/mxj slots/dispatch window/thresholds/bool res) | `apply_join_state` worker.c:297-334: hosttype/model "!"-guarded override, `set_host_slots(mxj, fallback maxcpus)`, dispatch_window dup, thresholds dup, `mark_host_res` loop | **CLOSED** |
| 4.3 | Update host-attr resource values (type,model,cpuf,maxcpus,maxmem,maxswap,maxtmp,ndisks,slots) | `add_hostattrs_resvals` (worker.c:223-250, 8 attrs) + `set_host_slots` (slots) — called worker.c:307,312 | **CLOSED** |
| 4.4 | Reject if `find_matched_hostconfig`==NULL, no workermgr mutation | fail-fast before any lock: `hconfig==NULL → reply.status=-errno/-EINVAL; return` — join.c:77-84 | **CLOSED** |
| 5.1 | BEFORE before mutation, AFTER after | `HOOK_BEFORE_EVENT` at join.c:87-88 (before key_find); `HOOK_AFTER_EVENT` at join.c:120-121 (after create/update, outside lock) | **CLOSED** |
| 5.2 | `on_worker_join` after successful join (both paths) | `on_worker_join(c, ep)` — join.c:147 (reached for both init and rejoin when idx>=0) | **CLOSED** |
| 5.3 | Critical section as short as current | No outer workermgr WRITE lock; only brief READ lock for key_find (join.c:91-95); mutation under per-worker WRITE lock in init/rejoin (worker.c:356-358, 372-392) | **CLOSED** |
| 5.4 | Existing tests pass; `add_worker` removed | Controller `add_worker` (old join path) deleted from worker.c & worker.h:11-14 (only `init_worker`/`rejoin_worker` declared); new `test_worker_join.c` registered in controller_test.c:37,166. NOTE: residual unrelated `add_worker` symbols exist — `add_worker_to_xlist` (misc.c:11) and worker-daemon `add_worker` (xcecd_worker_utils.c:12); both are distinct functions, not the refactored one | **CLOSED (verify build)** |
**Score: 16/16 acceptance criteria CLOSED in code.** All 12 tasks have on-disk evidence. Design's planned architecture (explicit branch, no outer write lock, static `apply_join_state`, `init_worker`/`rejoin_worker`) matches the implementation exactly.
### Implementation deltas vs design (not gaps — improvements/additions)
1. **`validate_host` gate added** (join.c:65-72) — rejects unknown hosts with `-EPERM` before hostconfig lookup. Not in the original design; a correct hardening, no requirement conflict.
2. **`persist_commit_barrier` added** (join.c:129-136) — the success reply is gated on the join WAL entry being majority-committed (ha-commit-barrier-rollout Req 1.1-1.4). This is the cross-spec commit-durability integration; barrier failure becomes the error reply and skips `on_worker_join`, preserving Req 2.2/5.2 semantics. Landed after this spec's design was written; consistent with it.
3. **`apply_join_state` linkage** is `static` in production, non-`static` under `#ifdef XCE_TESTING` (worker.c:283-289) — enables the unit test to call it directly while keeping production encapsulation. Matches design intent.
4. **`reference(w)`/`dereference(w)` discipline** around the found worker (join.c:93-94, 109) — refcount safety across the lock release between key_find and rejoin. Correct; not specified but required given the read-lock-then-release pattern.
### Residual / Deferred Gaps
**No open requirement gaps for this spec.** All criteria are met in code. Residual items are verification-only or out-of-boundary:
- **R1 (verification-only, carried from prior audit): green build/test on current HEAD.** worker.c/join.c are unmodified by HA-branch work, but join.c now depends on `persist_commit_barrier` and the controller test binary is shared with HA-branch changes (ha.c/raft.c) in the working tree. Action: run `/xce-build controller` and confirm `test_worker_join` (7 cases) plus `test_commit_barrier_reply` pass and no link interference. Effort: S (~1 build). Risk: Low.
- **R2 (latent, MEDIUM — was flagged in prior audit body): `apply_join_state` resource-update failures are swallowed.** `update_resvals` returns 0 unconditionally (worker.c:133-146; logs via `PERROR` but discards rc), and `mark_host_res` rc in the `apply_join_state` loop (worker.c:333-334) is ignored. So Req 4.3 "update all resource values" can partially fail silently and the join still reports success. This MATCHES pre-refactor behavior (Error Handling table, design.md:423 — "log with PERROR; continue"), so it is **not a regression and not a spec gap** — it is a pre-existing design-accepted behavior. Logged here only as a known limitation.
- **R3 (out-of-boundary — DO NOT count as this spec's gap): worker job-state resync on rejoin.** `rejoin_worker` updates attributes/endpoint/resources only. It does NOT recover running/finished job facts that were lost during the disconnect/failover window. `xce_join_req` carries only `hostattrs_t` (no job state). This is owned by the separate **worker-rejoin-resync** spec (roadmap line 26: "planned long ago, no effort yet"; dependencies: ha-consistency-invariants commit barrier + idempotency guards). Roadmap explicitly records this spec as "Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only)." **Boundary is clean and intentional.**
### Options for the one actionable open item (R1)
| Option | Description | Effort | Risk |
|--------|-------------|--------|------|
| **A (recommended)** | Run `/xce-build controller` on current HEAD; confirm `test_worker_join` + `test_commit_barrier_reply` green and no link interference with HA-branch ha.c/raft.c changes. Record result; close R1. | S (~1 build cycle) | Low |
| **B** | Defer until next controller build for any reason; treat R1 as a standing CI gate rather than a spec action. | None now | Low (build drift risk if join.c's `persist_commit_barrier` dep changes upstream) |
| **C** | Additionally harden R2: make `update_resvals`/`mark_host_res` propagate rc so a join with failed resource setup fails loud (per fail-loud preference). Re-scopes Req 4.3 error semantics — would need a spec amendment since current behavior is design-accepted. | M (touches worker.c shared helpers + tests; affects `restore_worker`/`mark_worker_unavail` callers) | Medium (behavior change to a shared path; out of original scope) |
**Recommendation: Option A.** The spec is functionally complete (16/16 criteria CLOSED); only a build/test confirmation on the integrated HEAD remains. R2 is a pre-existing accepted behavior — pursue Option C only if a separate fail-loud hardening spec is opened. R3 belongs to worker-rejoin-resync.