111 lines
14 KiB
Markdown
111 lines
14 KiB
Markdown
# XCE Project Status — Aug 08 – Aug 15, 2026
|
||
|
||
## TL;DR
|
||
|
||
**component-failover-adoption: implementation-complete and GO-validated — but not yet committed.** The week opened with scheduler-core documentation cleanup (carried over from last week), then a worker CLI/config cleanup pass, then the bulk of the week (Aug 11–13) was the 21-task component-failover-adoption spec: unifying worker/scheduler/apiserver's dual connect paths into one candidate-list pipeline, adding rejoin-on-failover for all three components, and an apiserver backend-health circuit breaker. Live E2E validation during that work found two real production bugs (bug-1548/bug-1549: apiserver never rejoined a new primary during an idle/heartbeat-only failover, so job pushes silently stopped reaching it). `/kiro-validate-impl` caught this as an in-scope gap and returned **NO-GO** on Aug 15 at 13:10 UTC; a same-day Phase 6 remediation (a monotonic `reconnect_gen` counter) fixed it, live-reverified, and the decision flipped to **GO** at 14:20 UTC. That fix, plus a new `xceadmin` cluster-config dump feature and a batch of build-tooling fixes, are sitting validated-but-uncommitted in the working tree as of end of week — 57 modified/untracked files, no commits since Aug 13.
|
||
|
||
## Commits by day
|
||
|
||
### Aug 08 (Sat) — scheduler-core documentation cleanup (carried over from prior week)
|
||
|
||
5 commits, docs-only:
|
||
- `docs:` add scheduler recording/simulation/REPL to deployment guide
|
||
- `docs(scheduler-core):` sync CLAUDE.md for core/driver/picker/sim/REPL architecture
|
||
- `docs(scheduler-core):` commit accumulated brief/requirements/roadmap edits
|
||
- `docs(scheduler-core):` backfill `_Verified` notes for tasks 2.1–2.6
|
||
- `docs(scheduler-core):` commit design.md/research.md, sync design.md with as-built code
|
||
|
||
### Aug 09 (Sun) — no commits
|
||
|
||
### Aug 10 (Mon) — worker CLI/config cleanup + probeConfigTag fix
|
||
|
||
11 commits:
|
||
- **Worker cleanup:** removed the config-file mechanism and deprecated CLI flags entirely (`5e4baa27`, `f0f46f87`, `b3124767`), fixed `xcecd -h` exiting before root/config checks (`9ffb1d51`, **bug-1509** class), default logdir to `/tmp` when `-l` omitted (`e959ee61`), added `-v` verbose flag (`0df1cf5f`)
|
||
- **fix(xceadmin)/feat(controller):** `OPCode_probeConfigTag` handler with Request-envelope reply, and `probe_config` updated to match (`90d7dd82`, `b00e97ff` — **bug-1506, bug-1507**)
|
||
- `feat(scheduler):` working default record-dir, improved REPL output, job ID formatting (`c705cada`)
|
||
|
||
### Aug 11 (Tue) — component-failover-adoption: foundation + worker + scheduler connect unification
|
||
|
||
11 commits — Task groups 1–3 begin:
|
||
- **Group 1 (Foundation):** reconciled `N_MAX_MASTERS`/`MAX_HA_CANDIDATES` (`fe8b4b86`), added the candidate-list→client-config conversion helper (`210faa2b`)
|
||
- **Group 2 (Worker):** unified worker's dual connect paths into the single pipeline (`ce9f734a`), fixed startup ordering + added rejoin retry-on-failure (`331fa024`), validated thread isolation and fixed rejoin bugs found by live E2E (`619b119e`)
|
||
- **Group 3 (Scheduler):** unified scheduler's dual connect paths (`a9f283dc`), added decision-publish retry-until-success-or-shutdown (`7c260e0c`), validated backpressure + fixed candidate-name diagnostics (`b7df8d60`), added decision-redelivery queue coverage (`d383d043`)
|
||
|
||
### Aug 12 (Wed) — component-failover-adoption: apiserver connect unification + rejoin + circuit breaker
|
||
|
||
7 commits — Task group 4:
|
||
- Routed apiserver's candidate parsing through the real C parser (`8b1e13a5`)
|
||
- Unified apiserver's dual connect paths — found and fixed a genuine use-after-free on `ctx->conf->masters[i]` during failover rotation (`e6359039`, **bug-1532**)
|
||
- Added apiserver's rejoin logic with correctly-placed cgo registration (`2a205291`)
|
||
- Wired rejoin to trigger from the idle-period heartbeat path (`ecf5eddc`)
|
||
- Added the backend-health circuit breaker (`c8325e75`)
|
||
- Closed an apiserver concurrency coverage gap (`79273a7f`)
|
||
- Group 5 begins: full-cluster E2E reconnect-after-failover test passes (`6cb8f22b`) — **this run first surfaced bug-1548/bug-1549** (apiserver silently fails to rejoin on an idle/heartbeat-only failover)
|
||
|
||
### Aug 13 (Thu) — component-failover-adoption closes; resource/config bugfixes; buglog integrity hardening
|
||
|
||
15 commits (6 feature/fix, 9 wolf-buglog checkpoints):
|
||
- Task 5.2 doubly-confirms bug-1548 in an isolated heartbeat-only scenario (`04d41d29`); task 5.3 validates the circuit breaker live (`51589b68`); task 5.4 closes the spec on a full regression pass (`d170623b`) — bug-1548/1549 were (incorrectly, see below) treated as out-of-scope follow-up items at this point
|
||
- `chore(wolf):` buglog.json integrity guard, dedupe rule, auto-checkpoint hook (`ba4e5fd2`)
|
||
- `fix(controller,xceadmin):` zero-slot scheduling block, misleading resval logs, byte-unit display (`e3b3ffaa` — **bug-1555, bug-1556**)
|
||
- `refactor(controller,xceadmin):` dropped generic `Resource.unit`, hardcoded bytes for maxmem (`5bf123cc`)
|
||
|
||
### Aug 14–15 (Fri–Sat) — validation reopened, Phase 6 remediation, tooling fixes — all uncommitted
|
||
|
||
No commits landed, but substantial validated work sits in the working tree:
|
||
- `/kiro-validate-impl` reopened component-failover-adoption: **NO-GO** (Aug 15, 13:10 UTC) — bug-1548 lives inside task 4.4's own `ApiserverBackendBridge` boundary against an already-existing acceptance criterion, so it doesn't qualify as legitimately out-of-scope
|
||
- **Phase 6 remediation** (tasks 6.1/6.2): added a monotonic `reconnect_gen` counter to `endpoint_t` (`src/include/netio.h`, `src/lib/xcectx.c`), bumped on every successful transport reconnect; `heartbeat.go`'s `runHeartbeat()` replaced its failure-to-success transition detector with a per-tick generation comparison. New regression tests in `heartbeat_test.go`/`heartbeat_rejoin_test.go`. Live E2E re-run of the exact task-5.2 scenario passed: rejoin fired 22s after a primary kill (12s after election) with zero write traffic in the preceding 5.5 minutes; a post-recovery job submit correctly appeared via `GET /xce/v1/jobs`
|
||
- Decision flipped to **GO** (14:20 UTC)
|
||
- **bug-1558** fixed: `make test-backend` now self-derives `CGO_CFLAGS`/`CGO_LDFLAGS` via `pkg-config`
|
||
- **bug-1559** fixed: `xce-build.sh`'s scheduler case now runs `meson test --suite scheduler` (all 41 registered targets, not just one binary by path)
|
||
- **New, separate uncommitted feature:** `xceadmin` cluster-config dump/pretty-printer (`build.py`: `dump_cluster_config` + formatters, ~236 new lines) and a `parse_size_or_number` helper in `mapping.py` for size-string threshold parsing, with new tests (`test_cli.py`, `test_mapping.py`, untracked `test_config_dump.py`)
|
||
|
||
## Bugs opened/fixed this window
|
||
|
||
Named/significant bugs (55 total opened this window, `bug-1506`–`bug-1560`; most of the unlisted ones are terse review-catches — "significant refactor of X", "incorrect value in code" — logged during component-failover-adoption's two-round review process):
|
||
|
||
| Bug | Status | Summary |
|
||
|---|---|---|
|
||
| bug-1506, bug-1507 | Fixed (`90d7dd82`, `b00e97ff`) | `probeConfigTag` missing Request-envelope handling; SIGABRT in REPL breakpoint inspection |
|
||
| bug-1508 | **Open** | `handle_fetchfile()` — possible job-ownership/path-traversal gap found during doc audit; flagged, needs a security-reviewer pass |
|
||
| bug-1509 | Fixed | `extralib` flake dependency removed; ported trivial linked-list code local to autoscaler |
|
||
| bug-1514, bug-1515 | Fixed | design.md/requirements.md defects in component-failover-adoption's scheduler-backpressure description, found by `/kiro-validate-design` and a deep re-review |
|
||
| bug-1519 | **Open** | AddressSanitizer "joining already joined thread" abort under `unshare -r`; worked around, not root-caused |
|
||
| bug-1532 | Fixed (`e6359039`) | apiserver `tryConnect()` use-after-free on `ctx->conf->masters[i]` during failover rotation |
|
||
| bug-1542 | Fixed | Illegal `import "C"` in a `_test.go` file; exposed the needed constant via a plain-Go package var instead |
|
||
| **bug-1548** | Fixed (uncommitted, Phase 6) | apiserver never rejoined a new primary on an idle/heartbeat-only failover — root cause: `XCECLI_FF_CALL`'s silent inline reconnect was invisible to the old failure→success transition detector |
|
||
| **bug-1549** | Fixed (uncommitted, Phase 6) | Downstream symptom of bug-1548 — job pushes never reached apiserver's local store after failover |
|
||
| bug-1553 | **Open** | `controller_unit_test` segfaults intermittently (~1 in 3 runs); crash site not yet isolated |
|
||
| bug-1554 | **Open, deferred by user** | `maxswap`/`maxtmp` report hardcoded stub values instead of real hardware detection |
|
||
| bug-1555, bug-1556 | Fixed (`e3b3ffaa`) | Misleading `update_resval` error logs; zero-slot scheduling block on the documented `hosts.yaml` auto-detect default |
|
||
| bug-1557 | **Open** | `xce_test worker join` defaults `maxmem=64` (bytes, not MB) |
|
||
| bug-1558, bug-1559 | Fixed (uncommitted) | `make test-backend` CGO flags now self-derived via `pkg-config`; `xce-build.sh` scheduler suite now runs all 41 test targets |
|
||
| bug-1560 | **Open, low priority** | `test_recordstart_symlink_escape_replies_error` fails under sandbox root — confirmed pre-existing, environment-specific, not a regression |
|
||
|
||
## Spec status changes
|
||
|
||
- **component-failover-adoption** — tasks 1.1–5.4 (21 tasks) implemented and initially marked complete (Aug 13). Reopened same week by its own `/kiro-validate-impl` run: **NO-GO** (Aug 15, 13:10 UTC) → Phase 6 remediation (2 tasks) → **GO** (Aug 15, 14:20 UTC). **Implementation and validation are done; the fix itself is not yet committed to git.**
|
||
- **scheduler-core** — carried into this window only for Aug 08's documentation cleanup; no further status change.
|
||
- Three new specs created this week, all still at brief/pre-requirements stage, not yet started: **component-rejoin-resync** and **worker-rejoin-resync** (both born from this week's bug-1548/1549 investigation — full-state resend cost on rejoin), and **scheduler-sim-diagnostics** (limit/pend-reason visibility in the REPL; not yet even `spec.json`-initialized).
|
||
|
||
## Stats
|
||
|
||
- **49 commits** in the window (39 feature/fix/docs, 10 wolf-infra/checkpoint)
|
||
- **~124 files changed** across the week's commits (109,317 insertions / 18,170 deletions, dominated by component-failover-adoption)
|
||
- **55 bugs opened** (bug-1506–bug-1560), **48 fixed**, **7 still open** (bug-1508, 1519, 1553, 1554, 1557, 1560, plus bug-1548/1549 fixed-but-uncommitted)
|
||
- **1 spec closed and reopened same week**, remediated, now GO
|
||
- **3 new specs** opened at brief stage
|
||
- **0 commits on Aug 14–15** despite validated GO work landing — 57 files (modified + untracked) sitting uncommitted at week's end
|
||
|
||
## Open threads / to-do heading into next session
|
||
|
||
1. ~~**Commit the Phase 6 failover fix**~~ — **Done (2026-08-15, `d479b70`).** Verified via a full `lib asan` rebuild (all libxce tests incl. the new `test_netio_reconnect_gen` pass) and `make test-backend` (apiserver backend + auth suites pass) before committing. Scoped to `netio.h`, `xcectx.c`, `lib/meson.build`, `lib/testsuite/test.c`, `lib/testsuite/test_netio_reconnect_gen.c`, `heartbeat.go`, `clib.go`, `heartbeat_test.go`, `heartbeat_rejoin_test.go`, `spec.json`, `tasks.md`. Note: `design.md`/`requirements.md`/`research.md` were deliberately left out — they've been stale since the spec's very first commit (`4ad31d77`, never updated across all 21+2 tasks), so syncing them is a separate, broader doc-catch-up task, not part of the Phase 6 diff.
|
||
2. ~~**Commit or explicitly shelve the xceadmin config-dump feature**~~ — **Done (2026-08-15, `3afe3d5`).** Verified via `xce-build.sh xceadmin`'s full pinned out-of-tree install + pytest run (256 passed, 3 skipped) before committing. Scoped to `build.py`, `cli.py`, `errors.py`, `mapping.py`, `hosts.schema.json`, `queues.schema.json` (the `SizeOrNumber` schema type is tightly coupled to `mapping.py`'s new `parse_size_or_number`, so it had to travel with it), plus the three test files. **New discovery while scoping this:** `resources.yaml`'s diff also contains unrelated changes — new resource catalog entries (`r15m`, `ut`, `pg`, `ls`, `it`, `swp`, `io`) that the worker (`resource_sync.c`) already registers but the shipped default `resources.yaml` never declared, plus `mem`/`tmp` byte-unit description fixes matching `node_information.c`'s uncommitted change (worker-reported `mem`/`tmp` now actually returns bytes, not MB) and `xcecd.c`'s `maxswap`/`maxtmp` stub-value unit fix (bug-1554's stub, now byte-scaled). These three files (`resources.yaml`, `worker/src/node_information.c`, `worker/src/xcecd.c`) were deliberately left out of both commits so far — added as new item 10 below.
|
||
3. **`.wolf/dashboard-token`** is untracked, holds a live-looking token, and is not covered by `.gitignore` — add a gitignore rule before it lands in history via a broad `git add`.
|
||
4. **Triage the remaining uncommitted files** not obviously tied to items 1, 2, or 10 (`deploy-ha.sh`, `xce-build.sh`, `controller/testsuite/test.c` — the `xce_test` detailed-help-text feature, unrelated to bug-1557's one-line default fix it happens to share a file with).
|
||
5. **bug-1553** (intermittent controller test segfault) is the most concerning open item — a real memory-safety bug in the shared test binary, unisolated. Needs a bisect/gdb session.
|
||
6. **bug-1508** (possible `handle_fetchfile()` ownership/path-traversal gap) — flagged during a doc audit, never given a security-reviewer pass.
|
||
7. **bug-1557** — one-line fix (`maxmem` default should be `64 * 1024 * 1024`, not `64`), still open.
|
||
8. **bug-1554** — swap/tmp-disk detection deferred by explicit user request; needs a design discussion when picked back up.
|
||
9. Decide which of the three new brief-stage specs (component-rejoin-resync, worker-rejoin-resync, scheduler-sim-diagnostics) to move into `/kiro-spec-requirements` next.
|
||
10. **New (found while scoping item 2):** commit the worker-side byte-unit completion + resource-catalog sync (`resources.yaml`, `src/worker/src/node_information.c`, `src/worker/src/xcecd.c`) — finishes what `5bf123cc`/`e3b3ffaa` started (mem/tmp now genuinely report bytes, not just documented as bytes) and adds the 7 missing built-in resource entries (`r15m`, `ut`, `pg`, `ls`, `it`, `swp`, `io`) the worker already reports but the catalog never declared. Worth its own scoped commit with a build+test pass first, same as items 1 and 2.
|