121 lines
17 KiB
Markdown
121 lines
17 KiB
Markdown
# XCE Project Status — Aug 30 – Sep 05, 2026
|
||
|
||
> Note: the calendar-default window (Aug 24–29) is already covered by
|
||
> `aug23-aug29-2026.md`; this report covers the actual gap — the last
|
||
> completed day before this report was generated.
|
||
|
||
## TL;DR
|
||
|
||
The single densest spec-throughput week on record: six full specs went
|
||
requirements-to-implementation-complete inside seven days —
|
||
`worker-rejoin-resync`, `scheduler-sim-repl-verbosity`,
|
||
`eliminate-frontend-makefile`, `scheduler-sim-step-verbosity`,
|
||
`scheduler-sim-reconcile-terminal-guard`, and — closing the week — the
|
||
flagship `scheduler-allocation` feature (lazy iterator-driven allocation,
|
||
parallel-job combinatorial planning, and the unsatisfiable-resreq cache),
|
||
confirmed via full regression (`58e6a855`) and an HA soak (`ecfb0a3c`) on
|
||
Saturday. Sunday and Monday cleared a dense backlog of live-deployment bugs
|
||
(`bug-1429`, `bug-1560`, `bug-1727`, `bug-1734`–`bug-1739`) and closed both
|
||
of last week's spec-metadata carry-over threads. The week's one unresolved
|
||
thread, `bug-1714` (the scheduler UAF crash), had two more candidate causes
|
||
ruled out but is still not root-caused after 11 sessions. Most consequential
|
||
finding of the week: `scheduler-allocation`'s own closing commit documents
|
||
that `parjob_allocator` — the entire multi-host allocation path just built
|
||
and soak-validated — has **zero live production path** today, because
|
||
`parse_resreq_primary()` hardcodes every job's task count to 1 (`bug-1770`);
|
||
the feature is soak-clean but not yet "done" for real clusters.
|
||
|
||
## Commits by day
|
||
|
||
### Sun 08-30 — live-bug sweep + closing last week's metadata carry-overs
|
||
|
||
- **`0e96a69f`** — `bug-1429` (`controller_unit_test` 300s TIMEOUT, open since Jul 10, last week's oldest unresolved item): `persist_stop()` now runs before `commit_engine_fini()` in 21 test fixtures — test-fixture-only fix, no production code change.
|
||
- **`dfcd345d`** — `bug-1560`: the symlink-escape test was asserting the wrong error string, not a real regression.
|
||
- **`b0476a84`, `5c2c6664`, `d8a37fa2`** — `bug-1734` (merge redelivered job state instead of discarding it), `bug-1736` (capture `memory_swap`/`tasks_current`, compute `peak_cpu_time`), `bug-1735` (job memory/CPU usage always read 0 in scheduler replay) — a related trio of worker/controller resource-usage tracking gaps.
|
||
- **`38437e57`** — scheduler no longer hands an orphaned duplicate job to the picker.
|
||
- **`ff9de2e0`** — `bug-924`: prune stale `/tmp` test-fixture dirs in `xce-build.sh`.
|
||
- **`a02d4777`** — `bug-1561`: fixed flaky `test_conf_trim_keeps_only_n_versions`/`test_conf_get_saved_tag_no_config`.
|
||
- **`f59890ce`** — **closes last week's open thread #2 (half)**: `scheduler-snapshot-job-allocation`'s `spec.json` phase synced to `implementation-complete`, matching work already done.
|
||
- **`ccef5610`** — new `xce-changes` skill for component-grouped `CHANGES.md` generation.
|
||
- **`63261097`** — commits last week's `CHANGES.md` and `aug23-aug29-2026.md` report.
|
||
|
||
### Mon 08-31 — memory reservation-vs-usage reconciliation + a live 3-bug HA sweep
|
||
|
||
- **`87a2a46b`, `f4103e0e`, `a6a0bdf7`** — scheduler now reconciles memory reservation against real usage and charges unreserved jobs' real memory too, with a doc explaining why admission control deliberately doesn't query live usage.
|
||
- **`fc559948`** — `bug-1727`: `info worker`/`info resource` was double-counting used capacity (`avail + used` instead of `avail` alone, since `avail` already included it).
|
||
- **`0e78f863`, `026fe97b`** — a live 3-node HA cluster + 50-element array-job investigation surfaced and fixed `bug-1737` (scheduler could never learn a job finished successfully — EXIT vs. DONE confusion), `bug-1738` (a separate terminal-state regression), and `bug-1739` (phantom RUN jobs in `JOBS_PUSH` merge).
|
||
|
||
### Tue 09-01 — `worker-rejoin-resync`: Foundation through Integration in one day
|
||
|
||
- **`1ff57ef6`→`80d5a062`** (~20 commits) — wire protocol, worker reconciling status, reconciliation round lifecycle, idempotent outcome application, job-state query/answer, config-field wiring into xceadmin, and dispatch exclusion of a reconciling host — tasks 1.1 through 5 all landed the same day this large spec's requirements/design/tasks were also written (`f2ff45f1`, `6c908d50`).
|
||
- **`8a2e22f4`** — **closes last week's open thread #2 (other half)**: `scheduler-dispatch-commit`'s stale 13/18 task count synced to the true 18/18; also adds the parked `commit-batch-window` spec (deliberately not being pursued — logged as a future cross-cutting WAL-commit-batching facility).
|
||
- **`343639fc`, `728496be`, `eb06f16d`** — three more fixes: apiserver job-control panic on int/int64 mismatch (`bug-1742`, Gitea #45), hosts-list slot usage counts never populated (`bug-1743`, Gitea #46), and mem-reservation reconciliation wired into the live rusage event path (`bug-1740`).
|
||
|
||
### Wed 09-02 — `worker-rejoin-resync` closes out; `scheduler-sim-repl-verbosity` built end to end
|
||
|
||
- **`613b80e9`→`a1dc2396`** — `worker-rejoin-resync`'s remaining tasks (bounded pagination, burst/failover recovery, gating periodic reporting on `resync_pending`) land and all group headers get checked off — spec reaches `implementation-complete`.
|
||
- **`f165e3a7`→`e6c67019`** (~20 commits) — the entire `scheduler-sim-repl-verbosity` feature (job-id range compression, round-boundary decision summaries, mode-aware event echo, dynamic REPL prompt, resource formatting/table rendering, discoverable help/break types) built and regression-verified in a single day, fixing `bug-1761` (`repl_cmd_delete()` rejecting invalid ids) along the way.
|
||
- **`00fc0a43`** — `xce-test-ha` soak gains worker-rejoin-resync coverage in both sequential and chaotic modes.
|
||
|
||
### Thu 09-03 — busiest day: two more specs closed, a design pivot, and a hermetic build
|
||
|
||
- **`6c3ff201`→`375c49ec`** — `eliminate-frontend-makefile` built and closed end to end: Meson-based CGO test target, `xce-build.sh` migrated off Make, 11 doc files + 3 scripts + 2 C testsuite files + 2 Go source files updated, live-reference sweep clean, `src/frontend/Makefile` deleted.
|
||
- **`6a2aa47f`→`9dcffce8`** — `scheduler-sim-step-verbosity` (preamble deduplication, fixing `bug-1762`/`bug-1763`) and `scheduler-sim-reconcile-terminal-guard` (terminal-state guard on job reconciliation, fixing `bug-1764`) both go from spec to `implementation-complete`.
|
||
- **`0f7699de`, `177e9b29`** — `scheduler-primary-lifecycle` marked **superseded** by new spec `xce-node-manager` (a general per-node component-supervisor design, design-generated, tasks not yet drafted); `bug-1760`'s reactive-rejoin deadlock requirements carry forward into it.
|
||
- **`68bc86bc`** — hermetic Nix build: vendors all Go modules and blocks PyPI/goproxy network access during build — the source of this week's outsized diff (see Stats).
|
||
- **`df4b386a`, `8557b869`, `1abab949`, `b14e53f5`** — spec-metadata reconciliation sweep across `worker-rejoin-resync`, `xce-test-rc-file`, and 3 more specs.
|
||
- **`4a224e8c`, `651b5138`** — AppImage build simplified to `nix build .#xce.appImage`; the `xce-build` skill now dispatches builds via `Bash run_in_background` instead of the `Agent` subagent tool.
|
||
|
||
### Fri 09-04 — `scheduler-allocation`: Foundation through cache-wiring in one day
|
||
|
||
- **`d5d2c1ca`→`5c022d0b`** (~14 commits) — the shared plan/host-iterator contracts, resreq fingerprinting, the unsatisfiable-resreq cache, seqjob's rewrap onto shared iterators, parjob's combinatorial plan generation and per-bucket allocation loop, and cache/dispatch wiring — tasks 1.1 through 5.1 of a 6-phase, 16-subtask spec, all in one day.
|
||
- **`88b0ac99`** — `bug-1760` fixed: post-failover rejoin detection gap closed by adding passive, socket-free broken-connection detection (`xcecli_is_connection_broken()`) alongside the existing reactive path, live-verified against a real 3-node failover with zero jobs in flight.
|
||
- **`1fc08d74`, `d912d33e`** — `bug-1714` forensics: ruled out the controller-side dual-push-path race and two more candidates; all known scheduler- and controller-side memory-ownership candidates are now exhausted with the stale second reader still unfound.
|
||
- **`0b1e3107`** — removed an external "extralib" dependency, replaced with an in-tree `xcelist.c`/`xcelist.h` primitive, shrinking `flake.lock`.
|
||
|
||
### Sat 09-05 — `scheduler-allocation` closes the feature
|
||
|
||
- **`e8c2087d`** — cache cleared per scheduling session (task 5.2).
|
||
- **`c009f108`** — end-to-end multi-host parallel job proof through the real dispatch path (task 5.3).
|
||
- **`58e6a855`** — full existing scheduler regression pass, zero unexpected assertion changes (task 6.1).
|
||
- **`ecfb0a3c`** — HA soak regression, zero new job loss/divergence/stall (task 6.2) — **feature complete**, with a standing caveat (see TL;DR and Open threads).
|
||
|
||
## Bugs opened/fixed this window
|
||
|
||
- **40 genuinely new bugs** (bug-1734 – bug-1774, one intentional ID gap at bug-1749), **22 fully fixed**, **3 partially fixed** (bug-1736, bug-1743, bug-1747 — each split into a fixed slice and an explicitly deferred remainder), **16 still open** at window end. One additional entry, `bug-1727`, was carried over from last week's open list and fixed this window (Mon, `fc559948`).
|
||
- **Carry-over status from last week's report**: `bug-1429` — **fixed** (Sun, `0e96a69f`), closing the oldest open item. `bug-1714` — **still open**; two more candidates ruled out (controller dual-push race, redelivery-storm volume), all known static leads now exhausted. `bug-1630` housekeeping note — closed (fix field backfilled, re-verified zero collisions). `bug-1632`, `bug-1723`, `bug-1726`, `bug-1728` — no movement this window, still open.
|
||
- Highest-severity fixes: **bug-1760** (HA availability — every job submitted after a controller failover could stay PEND forever until the scheduler was manually restarted), **bug-1737/1738/1739** (a live 3-node array-job run showing phantom RUN states and a scheduler that could never learn a job finished), **bug-1740** (memory-reservation reconciliation dead-wired to nothing).
|
||
- ⚠️ **bug-1770**, discovered during `scheduler-allocation` task 5.3 (Sep 4): `parse_resreq_primary()` hardcodes every job's `minTasks`/`maxTasks` to 1, so `parjob_allocator` — the multi-host allocation path this window's flagship feature just built — has no live production path. Its `fix` field records a resolution dated **2026-09-06 (outside this window)**, by a separate feature; as of this window's close (Sep 5) it was still open. Not counted as fixed in this report's tallies. See Open threads.
|
||
- **Integrity check**: of 41 unique bug IDs touched in-window, 1 (`bug-1727`) falls below the prior report's ID floor — a legitimate carry-over fix, not a duplicate-ID defect. The buglog counter otherwise stayed clean (40 new sequential-range IDs, one deliberate gap at bug-1749).
|
||
|
||
## Spec status changes
|
||
|
||
- **`scheduler-allocation`** — 0 → `implementation-complete` this window (all 16 subtasks, `spec.json` still reads `tasks-generated`/`ready_for_implementation: true` — a phase-sync pass is due). Standing caveat: soak-validated but not production-effective until `bug-1770` lands.
|
||
- **`worker-rejoin-resync`** — `implementation-complete`, all tasks and group headers checked (Foundation through Validation, ~19 subtasks), spanning Tue–Wed.
|
||
- **`scheduler-sim-repl-verbosity`** — `implementation-complete`, 10 major tasks / 23 subtasks, built and regression-verified in one day (Wed).
|
||
- **`eliminate-frontend-makefile`** — tasks.md shows 14/14 complete and its own closing commit (`375c49ec`) declares the feature complete, but `spec.json` still reads phase `tasks-generated` / `ready_for_implementation: false` — a metadata/reality mismatch matching the pattern fixed for two other specs this same window.
|
||
- **`scheduler-sim-step-verbosity`** and **`scheduler-sim-reconcile-terminal-guard`** — both `implementation-complete` (Thu), fixing bug-1762/1763 and bug-1764 respectively.
|
||
- **`scheduler-dispatch-commit`** — task-count metadata lag from last week's report fixed (`8a2e22f4`): now correctly reads 18/18 matching its already-`implementation-complete` phase.
|
||
- **`scheduler-snapshot-job-allocation`** — phase synced to `implementation-complete` (Sun), closing the other half of last week's metadata carry-over.
|
||
- **`scheduler-primary-lifecycle`** — superseded by new spec **`xce-node-manager`** (design-generated; requirements/design approved, tasks not yet drafted) after a design pivot: node-level component supervision, not controller-internal scheduler spawning.
|
||
- **`scheduler-sim-diagnostics`** — new draft spec added (requirements only, unapproved).
|
||
- **`commit-batch-window`** — new spec created, explicitly parked/not pursued.
|
||
|
||
## Stats
|
||
|
||
- **160 commits** — **130 real** (feature/fix/docs/test), **30 `.wolf` bookkeeping**.
|
||
- **3,565 non-`.wolf` files changed**, **+1,479,878 / −1,710** (non-wolf headline; raw incl. `.wolf`: +1,487,194 / −4,224). This figure is dominated by a single commit (`68bc86bc`, hermetic Nix build) that vendored ~3,177 third-party Go source files (~1.45M lines) — excluding that one-time vendor dump, the week's real hand-written churn is a small fraction of the headline number.
|
||
- Busiest areas by files touched: `src/frontend` (3,188, ~3,177 of which are the vendor dump), `src/autoscaler` (151, also vendored), `src/scheduler` (78), `.kiro` (55), `src/controller` (35), `src/worker` (12), `src/lib` (10).
|
||
- **6 specs shipped end-to-end** this window (`scheduler-allocation`, `worker-rejoin-resync`, `scheduler-sim-repl-verbosity`, `eliminate-frontend-makefile`, `scheduler-sim-step-verbosity`, `scheduler-sim-reconcile-terminal-guard`); **2 more** had metadata-only carry-over fixes (`scheduler-dispatch-commit`, `scheduler-snapshot-job-allocation`); **1 design pivot** (`scheduler-primary-lifecycle` → `xce-node-manager`); **3 new specs** opened.
|
||
- **11 sessions** logged in `.wolf/memory.md` for Aug 30 – Sep 4 (no Sep 5 session flushed yet at report time); busiest logged days Sep 1 (396 actions, worker-rejoin-resync Foundation-to-Integration) and Sep 2 (331 actions, worker-rejoin-resync close-out + scheduler-sim-repl-verbosity).
|
||
- Commits landed on all 7 days; peak day **Fri 09-04** for `scheduler-allocation`'s Foundation-through-cache-wiring push.
|
||
|
||
## Open threads / to-do heading into next session
|
||
|
||
1. **`bug-1714` still unresolved after 11 sessions.** All known scheduler-side memory-ownership candidates and the controller-side dual-push-path race are now ruled out; the stale second reader for `job_submit_data_t` remains unfound. Next step needs either a fresh live gdb-watchpoint campaign specifically instrumenting `core->jobs`'s own lifecycle (snapshot/recording code, REPL inspection commands, picker-plugin job-reference caching — none checked yet), or an explicit project-owner decision to accept this as a monitored low-frequency risk rather than continue speculative re-chasing.
|
||
2. **`scheduler-allocation` is soak-validated but not production-effective.** `parjob_allocator`'s entire multi-host allocation path (tasks 4.1–4.3, 5.1–5.2) has zero live production path because `parse_resreq_primary()` (`src/lib/resreq.c:431-432`) hardcodes `minTasks=maxTasks=1` regardless of what a job actually requests, and the resreq grammar's own `span[ptile=N]` syntax is parsed then discarded. Logged as `bug-1770` (with sibling `bug-1771`, an unrelated fcfs NULL-queue-deref found while building the same test). This needs a dedicated, high-priority follow-up spec before the feature has any real-cluster effect.
|
||
3. **`eliminate-frontend-makefile`'s `spec.json` metadata lag** — tasks.md and its own closing commit say the feature is done; phase still reads `tasks-generated` and `ready_for_implementation: false`. Same class of fix already applied to `scheduler-dispatch-commit` and `scheduler-snapshot-job-allocation` this window — needs the same treatment.
|
||
4. **`scheduler-allocation`'s own `spec.json`** likewise needs its phase synced to `implementation-complete` now that all 16 subtasks and the HA soak are done.
|
||
5. **Four lower-priority open bugs with zero movement this window**: `bug-1632`, `bug-1723`, `bug-1726`, `bug-1728` — still awaiting triage.
|
||
6. **`xce-node-manager` needs `/kiro-spec-tasks`** — design is approved (GO, all findings fixed in place) but tasks aren't drafted yet; it's the successor to superseded `scheduler-primary-lifecycle` and carries forward `bug-1760`'s now-fixed rejoin-detection work as validated precedent.
|
||
7. **A cluster of discovery-only bugs from this week's spec reviews remain open and unowned by any spec's current boundary**: `bug-1766`–`bug-1769`, `bug-1771`–`bug-1774` (compound/alter-resreq rollback and resource-accounting gaps, a heap-OOB type confusion, snapshot/schedobj duplication patterns) — worth a triage pass to decide which need dedicated follow-up specs.
|