XCE Project Status — Aug 30 – Sep 05, 2026

Note: the calendar-default window (Aug 24–29) is already covered by aug23-aug29-2026.md; this report covers the actual gap — the last completed day before this report was generated.

TL;DR

The single densest spec-throughput week on record: six full specs went requirements-to-implementation-complete inside seven days — worker-rejoin-resync, scheduler-sim-repl-verbosity, eliminate-frontend-makefile, scheduler-sim-step-verbosity, scheduler-sim-reconcile-terminal-guard, and — closing the week — the flagship scheduler-allocation feature (lazy iterator-driven allocation, parallel-job combinatorial planning, and the unsatisfiable-resreq cache), confirmed via full regression (58e6a855) and an HA soak (ecfb0a3c) on Saturday. Sunday and Monday cleared a dense backlog of live-deployment bugs (bug-1429, bug-1560, bug-1727, bug-1734–bug-1739) and closed both of last week's spec-metadata carry-over threads. The week's one unresolved thread, bug-1714 (the scheduler UAF crash), had two more candidate causes ruled out but is still not root-caused after 11 sessions. Most consequential finding of the week: scheduler-allocation's own closing commit documents that parjob_allocator — the entire multi-host allocation path just built and soak-validated — has zero live production path today, because parse_resreq_primary() hardcodes every job's task count to 1 (bug-1770); the feature is soak-clean but not yet "done" for real clusters.

Commits by day

Sun 08-30 — live-bug sweep + closing last week's metadata carry-overs

0e96a69f — bug-1429 (controller_unit_test 300s TIMEOUT, open since Jul 10, last week's oldest unresolved item): persist_stop() now runs before commit_engine_fini() in 21 test fixtures — test-fixture-only fix, no production code change.
dfcd345d — bug-1560: the symlink-escape test was asserting the wrong error string, not a real regression.
b0476a84, 5c2c6664, d8a37fa2 — bug-1734 (merge redelivered job state instead of discarding it), bug-1736 (capture memory_swap/tasks_current, compute peak_cpu_time), bug-1735 (job memory/CPU usage always read 0 in scheduler replay) — a related trio of worker/controller resource-usage tracking gaps.
38437e57 — scheduler no longer hands an orphaned duplicate job to the picker.
ff9de2e0 — bug-924: prune stale /tmp test-fixture dirs in xce-build.sh.
a02d4777 — bug-1561: fixed flaky test_conf_trim_keeps_only_n_versions/test_conf_get_saved_tag_no_config.
f59890ce — closes last week's open thread #2 (half): scheduler-snapshot-job-allocation's spec.json phase synced to implementation-complete, matching work already done.
ccef5610 — new xce-changes skill for component-grouped CHANGES.md generation.
63261097 — commits last week's CHANGES.md and aug23-aug29-2026.md report.

Mon 08-31 — memory reservation-vs-usage reconciliation + a live 3-bug HA sweep

87a2a46b, f4103e0e, a6a0bdf7 — scheduler now reconciles memory reservation against real usage and charges unreserved jobs' real memory too, with a doc explaining why admission control deliberately doesn't query live usage.
fc559948 — bug-1727: info worker/info resource was double-counting used capacity (avail + used instead of avail alone, since avail already included it).
0e78f863, 026fe97b — a live 3-node HA cluster + 50-element array-job investigation surfaced and fixed bug-1737 (scheduler could never learn a job finished successfully — EXIT vs. DONE confusion), bug-1738 (a separate terminal-state regression), and bug-1739 (phantom RUN jobs in JOBS_PUSH merge).

Tue 09-01 — `worker-rejoin-resync`: Foundation through Integration in one day

1ff57ef6→80d5a062 (~20 commits) — wire protocol, worker reconciling status, reconciliation round lifecycle, idempotent outcome application, job-state query/answer, config-field wiring into xceadmin, and dispatch exclusion of a reconciling host — tasks 1.1 through 5 all landed the same day this large spec's requirements/design/tasks were also written (f2ff45f1, 6c908d50).
8a2e22f4 — closes last week's open thread #2 (other half): scheduler-dispatch-commit's stale 13/18 task count synced to the true 18/18; also adds the parked commit-batch-window spec (deliberately not being pursued — logged as a future cross-cutting WAL-commit-batching facility).
343639fc, 728496be, eb06f16d — three more fixes: apiserver job-control panic on int/int64 mismatch (bug-1742, Gitea #45), hosts-list slot usage counts never populated (bug-1743, Gitea #46), and mem-reservation reconciliation wired into the live rusage event path (bug-1740).

Wed 09-02 — `worker-rejoin-resync` closes out; `scheduler-sim-repl-verbosity` built end to end

613b80e9→a1dc2396 — worker-rejoin-resync's remaining tasks (bounded pagination, burst/failover recovery, gating periodic reporting on resync_pending) land and all group headers get checked off — spec reaches implementation-complete.
f165e3a7→e6c67019 (~20 commits) — the entire scheduler-sim-repl-verbosity feature (job-id range compression, round-boundary decision summaries, mode-aware event echo, dynamic REPL prompt, resource formatting/table rendering, discoverable help/break types) built and regression-verified in a single day, fixing bug-1761 (repl_cmd_delete() rejecting invalid ids) along the way.
00fc0a43 — xce-test-ha soak gains worker-rejoin-resync coverage in both sequential and chaotic modes.

Thu 09-03 — busiest day: two more specs closed, a design pivot, and a hermetic build

6c3ff201→375c49ec — eliminate-frontend-makefile built and closed end to end: Meson-based CGO test target, xce-build.sh migrated off Make, 11 doc files + 3 scripts + 2 C testsuite files + 2 Go source files updated, live-reference sweep clean, src/frontend/Makefile deleted.
6a2aa47f→9dcffce8 — scheduler-sim-step-verbosity (preamble deduplication, fixing bug-1762/bug-1763) and scheduler-sim-reconcile-terminal-guard (terminal-state guard on job reconciliation, fixing bug-1764) both go from spec to implementation-complete.
0f7699de, 177e9b29 — scheduler-primary-lifecycle marked superseded by new spec xce-node-manager (a general per-node component-supervisor design, design-generated, tasks not yet drafted); bug-1760's reactive-rejoin deadlock requirements carry forward into it.
68bc86bc — hermetic Nix build: vendors all Go modules and blocks PyPI/goproxy network access during build — the source of this week's outsized diff (see Stats).
df4b386a, 8557b869, 1abab949, b14e53f5 — spec-metadata reconciliation sweep across worker-rejoin-resync, xce-test-rc-file, and 3 more specs.
4a224e8c, 651b5138 — AppImage build simplified to nix build .#xce.appImage; the xce-build skill now dispatches builds via Bash run_in_background instead of the Agent subagent tool.

Fri 09-04 — `scheduler-allocation`: Foundation through cache-wiring in one day

d5d2c1ca→5c022d0b (~14 commits) — the shared plan/host-iterator contracts, resreq fingerprinting, the unsatisfiable-resreq cache, seqjob's rewrap onto shared iterators, parjob's combinatorial plan generation and per-bucket allocation loop, and cache/dispatch wiring — tasks 1.1 through 5.1 of a 6-phase, 16-subtask spec, all in one day.
88b0ac99 — bug-1760 fixed: post-failover rejoin detection gap closed by adding passive, socket-free broken-connection detection (xcecli_is_connection_broken()) alongside the existing reactive path, live-verified against a real 3-node failover with zero jobs in flight.
1fc08d74, d912d33e — bug-1714 forensics: ruled out the controller-side dual-push-path race and two more candidates; all known scheduler- and controller-side memory-ownership candidates are now exhausted with the stale second reader still unfound.
0b1e3107 — removed an external "extralib" dependency, replaced with an in-tree xcelist.c/xcelist.h primitive, shrinking flake.lock.

Sat 09-05 — `scheduler-allocation` closes the feature

e8c2087d — cache cleared per scheduling session (task 5.2).
c009f108 — end-to-end multi-host parallel job proof through the real dispatch path (task 5.3).
58e6a855 — full existing scheduler regression pass, zero unexpected assertion changes (task 6.1).
ecfb0a3c — HA soak regression, zero new job loss/divergence/stall (task 6.2) — feature complete, with a standing caveat (see TL;DR and Open threads).

Bugs opened/fixed this window

40 genuinely new bugs (bug-1734 – bug-1774, one intentional ID gap at bug-1749), 22 fully fixed, 3 partially fixed (bug-1736, bug-1743, bug-1747 — each split into a fixed slice and an explicitly deferred remainder), 16 still open at window end. One additional entry, bug-1727, was carried over from last week's open list and fixed this window (Mon, fc559948).
Carry-over status from last week's report: bug-1429 — fixed (Sun, 0e96a69f), closing the oldest open item. bug-1714 — still open; two more candidates ruled out (controller dual-push race, redelivery-storm volume), all known static leads now exhausted. bug-1630 housekeeping note — closed (fix field backfilled, re-verified zero collisions). bug-1632, bug-1723, bug-1726, bug-1728 — no movement this window, still open.
Highest-severity fixes: bug-1760 (HA availability — every job submitted after a controller failover could stay PEND forever until the scheduler was manually restarted), bug-1737/1738/1739 (a live 3-node array-job run showing phantom RUN states and a scheduler that could never learn a job finished), bug-1740 (memory-reservation reconciliation dead-wired to nothing).
⚠️ bug-1770, discovered during scheduler-allocation task 5.3 (Sep 4): parse_resreq_primary() hardcodes every job's minTasks/maxTasks to 1, so parjob_allocator — the multi-host allocation path this window's flagship feature just built — has no live production path. Its fix field records a resolution dated 2026-09-06 (outside this window), by a separate feature; as of this window's close (Sep 5) it was still open. Not counted as fixed in this report's tallies. See Open threads.
Integrity check: of 41 unique bug IDs touched in-window, 1 (bug-1727) falls below the prior report's ID floor — a legitimate carry-over fix, not a duplicate-ID defect. The buglog counter otherwise stayed clean (40 new sequential-range IDs, one deliberate gap at bug-1749).

Spec status changes

scheduler-allocation — 0 → implementation-complete this window (all 16 subtasks, spec.json still reads tasks-generated/ready_for_implementation: true — a phase-sync pass is due). Standing caveat: soak-validated but not production-effective until bug-1770 lands.
worker-rejoin-resync — implementation-complete, all tasks and group headers checked (Foundation through Validation, ~19 subtasks), spanning Tue–Wed.
scheduler-sim-repl-verbosity — implementation-complete, 10 major tasks / 23 subtasks, built and regression-verified in one day (Wed).
eliminate-frontend-makefile — tasks.md shows 14/14 complete and its own closing commit (375c49ec) declares the feature complete, but spec.json still reads phase tasks-generated / ready_for_implementation: false — a metadata/reality mismatch matching the pattern fixed for two other specs this same window.
scheduler-sim-step-verbosity and scheduler-sim-reconcile-terminal-guard — both implementation-complete (Thu), fixing bug-1762/1763 and bug-1764 respectively.
scheduler-dispatch-commit — task-count metadata lag from last week's report fixed (8a2e22f4): now correctly reads 18/18 matching its already-implementation-complete phase.
scheduler-snapshot-job-allocation — phase synced to implementation-complete (Sun), closing the other half of last week's metadata carry-over.
scheduler-primary-lifecycle — superseded by new spec xce-node-manager (design-generated; requirements/design approved, tasks not yet drafted) after a design pivot: node-level component supervision, not controller-internal scheduler spawning.
scheduler-sim-diagnostics — new draft spec added (requirements only, unapproved).
commit-batch-window — new spec created, explicitly parked/not pursued.

Stats

160 commits — 130 real (feature/fix/docs/test), 30 .wolf bookkeeping.
3,565 non-.wolf files changed, +1,479,878 / −1,710 (non-wolf headline; raw incl. .wolf: +1,487,194 / −4,224). This figure is dominated by a single commit (68bc86bc, hermetic Nix build) that vendored ~3,177 third-party Go source files (~1.45M lines) — excluding that one-time vendor dump, the week's real hand-written churn is a small fraction of the headline number.
Busiest areas by files touched: src/frontend (3,188, ~3,177 of which are the vendor dump), src/autoscaler (151, also vendored), src/scheduler (78), .kiro (55), src/controller (35), src/worker (12), src/lib (10).
6 specs shipped end-to-end this window (scheduler-allocation, worker-rejoin-resync, scheduler-sim-repl-verbosity, eliminate-frontend-makefile, scheduler-sim-step-verbosity, scheduler-sim-reconcile-terminal-guard); 2 more had metadata-only carry-over fixes (scheduler-dispatch-commit, scheduler-snapshot-job-allocation); 1 design pivot (scheduler-primary-lifecycle → xce-node-manager); 3 new specs opened.
11 sessions logged in .wolf/memory.md for Aug 30 – Sep 4 (no Sep 5 session flushed yet at report time); busiest logged days Sep 1 (396 actions, worker-rejoin-resync Foundation-to-Integration) and Sep 2 (331 actions, worker-rejoin-resync close-out + scheduler-sim-repl-verbosity).
Commits landed on all 7 days; peak day Fri 09-04 for scheduler-allocation's Foundation-through-cache-wiring push.

Open threads / to-do heading into next session

bug-1714 still unresolved after 11 sessions. All known scheduler-side memory-ownership candidates and the controller-side dual-push-path race are now ruled out; the stale second reader for job_submit_data_t remains unfound. Next step needs either a fresh live gdb-watchpoint campaign specifically instrumenting core->jobs's own lifecycle (snapshot/recording code, REPL inspection commands, picker-plugin job-reference caching — none checked yet), or an explicit project-owner decision to accept this as a monitored low-frequency risk rather than continue speculative re-chasing.
scheduler-allocation is soak-validated but not production-effective. parjob_allocator's entire multi-host allocation path (tasks 4.1–4.3, 5.1–5.2) has zero live production path because parse_resreq_primary() (src/lib/resreq.c:431-432) hardcodes minTasks=maxTasks=1 regardless of what a job actually requests, and the resreq grammar's own span[ptile=N] syntax is parsed then discarded. Logged as bug-1770 (with sibling bug-1771, an unrelated fcfs NULL-queue-deref found while building the same test). This needs a dedicated, high-priority follow-up spec before the feature has any real-cluster effect.
eliminate-frontend-makefile's spec.json metadata lag — tasks.md and its own closing commit say the feature is done; phase still reads tasks-generated and ready_for_implementation: false. Same class of fix already applied to scheduler-dispatch-commit and scheduler-snapshot-job-allocation this window — needs the same treatment.
scheduler-allocation's own spec.json likewise needs its phase synced to implementation-complete now that all 16 subtasks and the HA soak are done.
Four lower-priority open bugs with zero movement this window: bug-1632, bug-1723, bug-1726, bug-1728 — still awaiting triage.
xce-node-manager needs /kiro-spec-tasks — design is approved (GO, all findings fixed in place) but tasks aren't drafted yet; it's the successor to superseded scheduler-primary-lifecycle and carries forward bug-1760's now-fixed rejoin-detection work as validated precedent.
A cluster of discovery-only bugs from this week's spec reviews remain open and unowned by any spec's current boundary: bug-1766–bug-1769, bug-1771–bug-1774 (compound/alter-resreq rollback and resource-accounting gaps, a heap-OOB type confusion, snapshot/schedobj duplication patterns) — worth a triage pass to decide which need dedicated follow-up specs.

17 KiB Raw Blame History Unescape Escape