myxce/reports/sep06-sep12-2026.md

XCE Project Status — Sep 06 – Sep 12, 2026

Note: the calendar-default window (Aug 31–Sep 05) is already covered by aug30-sep05-2026.md; this report covers the actual gap — the last completed day before this report was generated.

TL;DR

A CLI-quality week bookended by two scheduler-side finishes: Sunday closed out resreq-maxtasks-wiring (real multi-host task-count wiring, proven end-to-end and HA-soak clean), and the middle of the week was almost entirely xce-test-cli-ux — a large, self-contained rebuild of xce_test's interactive shell (interrupt-safe RPCs, confirmation gates, tab completion, JSON output, flag parsing) that surfaced its own cluster of testsuite bugs (bug-1791–1797). Monday/Tuesday cleared the last of the gitea#46 limit-plugin cascade and closed the from-scratch build race (bug-1798) for good. Saturday's session — driven by a live scheduler-replay debugging conversation — fixed a real resource over-commit (bug-1799/1801), added human-scaled resource-unit display (bug-1800), and, after live user feedback, revised scheduler-sim-eager-reconciliation's own design to make captured jobs' job->state transition synchronously (bug-1804) rather than deferred. Last week's flagged flagship gap, bug-1770 (multi-host allocation had no live production path), is now fixed.

Commits by day

Sun 09-06 — resreq-maxtasks-wiring to done, plus last week's report

• 799a71ec..5c87f1cc (12 commits) — built the real task-count derivation helper, wired it into fresh job-ingest and snapshot/replay-restore ingest, closed a worker launch-type coverage gap, then validated with a real multi-host end-to-end harness (plain multi-task job + array-job element, both genuinely split across hosts) and a full regression + HA soak pass. This is the fix for last week's flagged bug-1770 (multi-host allocation had zero live production path).
• a6b735b6 — added aug30-sep05-2026.md and CHANGES.md for the prior window.

Mon 09-07 — gitea#46 limit-plugin cascade, apiserver job-control fixes

• 22cfe616..924ac32c — fixed the limit-plugin usergroup/scope/error-code cascade (gitea#46) end-to-end: named-usergroup resolution, default-scope semantics, kvmgr not-found-vs-error confusion (twice), an RTTI/pointer-indirection type-confusion bug, and finally live-verified all 7 limits.yaml limits matching expectations (bug-1784–1789).
• 099dbdab, 5ab344e1 — apiserver: fixed a job-control union discriminant bug + added human-readable job IDs (gitea#45 reopened), and propagated submit user as job execuser (bug-1780).
• 0d3f925c, ef3c4670 — scheduler: retired reconciled jobs from picker candidates and excluded terminal jobs from the assigned-jobs listing (bug-1759/1778/1779).
• a35f597e, e395eaa3 — worker: real maxswap/maxtmp hardware detection (bug-1554); frontend: fixed host job-state/EXEC_HOST columns to count real slots.
• 850d186d, 028d8f6a — build: xce-build.sh now self-wraps into nix develop and caps its run-queue length via a live-computed --jobs flag.

Tue 09-08 — bug-1790 residual gap, leak fixes, scheduler global-limit refactor

• 9caf1424..a6d96705 — fixed bug-1790 (global limits on per-host resources reading the wrong "want"), live-verified with a real global mem limit, then refactored job-total limit computation into the limit plugin itself and fixed a regression that briefly broke slots-based global enforcement along the way.
• 4b20590c, c832706e — fixed two pre-existing ASAN leaks (host_account/job_account test fixtures, bug-1621/1723) and a wrong resInUse precondition in a mem-reservation test (bug-1758).

Wed 09-09 — xce-test-cli-ux: foundation and core (bulk of the week)

• 0748f10c..96746af3 (16 commits) — built the flag-descriptor parsing/rejection core, typo suggestions, SIGINT/EINTR handling, an interruptible-call heap-ownership adapter with per-connection locking, a pty-driven black-box CLI test harness, tab completion, in-session source with cycle detection, a JSON list-output helper library, a destructive-command confirmation gate wired to five commands, flagspec migration for every flag-accepting command, and consolidated help generation. Along the way, fixed bug-1557 (join's maxmem default) and bug-1711 (display/text defects).
• Testsuite bugs surfaced and fixed same-day: bug-1791 (cross-thread pthread mutex unlock, EPERM), bug-1792 (tab-completion test silent-input-mode gap).

Thu 09-10 — xce-test-cli-ux: integration, validation, and the build-race root-cause fix

• a98fad24..38595099 — routed diagnostic/warning output to stderr uniformly, then verified Ctrl-C cancellation, the confirmation gate, source's interaction with it, and help/completion/context-switching end-to-end (tasks 4.1–4.4). Found and deliberately deferred four more testsuite bugs (bug-1794–1797: a cmd_jobrun ASAN abort, a leave_ctx() use-after-free, misleading perror() diagnostics, a queue-name leak) — all out of boundary for their discovering (validation-only) tasks.
• 654c6e8c, c1c6c9a2, f09fbdb3 — root-caused and closed the long-standing intermittent from-scratch build failure (fatal error: controller.capnp.h): a missing-dependency-edge race in the capnp codegen custom_target, fixed at the meson.build level and extended to controller/worker/scheduler and all four PEG parser generators (bug-1798).

Fri–Sat 09-11/09-12 — live replay debugging: resource over-commit, unit display, design revision

• Live scheduler-replay session surfaced and fixed, same-day (all TDD, RED confirmed before GREEN): bug-1799/1801 (sim replay resInUse over-commit — a worker's used slots could exceed its own capacity after a resync replay; sched_account_jobs now skips RECONCILING/UNAVAIL workers like core_reset_workers already does), bug-1800 (bytes-valued CONSUMABLE resources now render human-scaled units), and bug-1802/1803 (a break ANY breakpoint kind was added, then reverted after live feedback it over-fired inside busy sessions, replaced with a prescan-based "next non-dummy session" hint in info summary).
• 25fb5c11 (Sat) — bundled commit landing all of the above plus bug-1804: a user-directed design-fork resolution revising scheduler-sim-eager-reconciliation so a captured job's job->state is set to RUN synchronously at dispatch capture (mirroring normal-mode's sched_commit_decision, bug-1624) instead of deferring until its separately-recorded JOB_STATE_CHANGE event replays. Full scheduler suite verified 77/77 green before commit.

Bugs opened/fixed this window

• 29 genuinely new bugs (bug-1775 – bug-1804, one intentional ID gap at bug-1783), 24 fixed, 5 deliberately deferred/still open (bug-1777, bug-1793, bug-1796, bug-1797 — all explicitly out-of-boundary for the validation-only tasks that found them; bug-1799 itself stays logged as "not yet fixed" by this project's own convention of routing its actual fix through a separate follow-up ID, bug-1801, which is fixed).
• Carry-over status from last week's report: bug-1770 — fixed (Sun, resreq-maxtasks-wiring), closing the flagship gap flagged last week (multi-host allocation now has a real production path). bug-1621/bug-1723/bug-1758 — fixed (Tue). bug-1714 — still open, no movement this window (last touched Sep 4). bug-1632, bug-1726, bug-1728 — no movement, still open.
• Highest-severity fixes: bug-1799/1801 (resource-accounting over-commit — a worker could show more slots in use than it physically has), bug-1798 (root-caused a long-standing intermittent from-scratch build failure), bug-1784–1789 (gitea#46's full limit-plugin cascade, 6 stacked bugs found and fixed in sequence).
• Integrity check: 29 unique new bug IDs, sequential except one deliberate gap (bug-1783); 0 duplicate IDs in the buglog overall.

Spec status changes

• resreq-maxtasks-wiring — implementation-complete, all subtasks done (Sun), HA-soak and full-suite verified with no regression. Closes bug-1770.
• scheduler-sim-eager-reconciliation — revision 4 landed (Sat, 25fb5c11): synchronous job-state-at-capture per bug-1804/user direction. Phase remains implementation-complete, 12/12 tasks.
• xce-test-cli-ux — all 23 real subtasks across all 4 task groups (Foundation → Validation) are checked done, but spec.json still reads phase tasks-generated (ready_for_implementation: true) — the same metadata-lag pattern flagged for three other specs in the prior two reports. A phase-sync pass is due.

Stats

• 73 commits — 62 real (feature/fix/refactor/test/docs), 11 .wolf bookkeeping.
• 287 non-.wolf files changed, +22,190 / −1,889 (non-wolf headline; raw incl. .wolf: +30,044 / −7,591).
• Busiest areas by files touched: src/scheduler (47), src/controller (28), .kiro (16), src/lib (10), src/worker (7), src/frontend (7), .wolf (7), .claude (5).
• By spec scope: xce-test-cli-ux (25 commits), resreq-maxtasks-wiring (13), scheduler (11, unscoped fixes), build (5).
• 2 specs shipped end-to-end this window (resreq-maxtasks-wiring, and scheduler-sim-eager-reconciliation's revision-4 patch); 1 spec functionally complete pending metadata sync (xce-test-cli-ux).
• 8 sessions logged in .wolf/memory.md touching this window (Sep 7 ×5, Sep 9, Sep 11, Sep 12); busiest logged day Sep 7 (184 actions, gitea#46 cascade + apiserver fixes).
• Commits landed on all 7 days; peak day Wed 09-09 (16 commits, xce-test-cli-ux foundation-to-core).

Open threads / to-do heading into next session

1. xce-test-cli-ux's spec.json metadata lag — tasks.md shows all 23 real subtasks complete across all 4 groups, but phase still reads tasks-generated. Same class of fix already applied to three other specs in recent weeks (scheduler-dispatch-commit, scheduler-snapshot-job-allocation, and implicitly eliminate-frontend-makefile) — needs the same treatment: sync phase to implementation-complete.
2. Four deliberately-deferred testsuite bugs from xce-test-cli-ux's validation tasks remain open and unowned: bug-1796 (misleading perror() diagnostics on RPC-failure paths), bug-1797 (queue-name leak on an RPC-error path), bug-1793 (join's stale maxswap/maxtmp defaults — docs and code both need fixing together to avoid a new mismatch), bug-1777 (workerlist MaxSwap can disagree with the scheduler's own view). Worth a triage pass to decide which need a dedicated follow-up spec vs. quick fixes.
3. bug-1714 still unresolved, now over two weeks old with zero movement this window — the stale second reader for job_submit_data_t->project_name remains unfound. Needs either a fresh live gdb-watchpoint campaign or an explicit decision to accept it as a monitored low-frequency risk.
4. Three lower-priority open bugs with zero movement across two consecutive windows: bug-1632, bug-1726, bug-1728 — still awaiting triage.
5. gitea MCP server connectivity — noted in .wolf/STATUS.md as failing to connect (CONNECT_TIMEOUT) as of this window's close; buglog entries bug-1798 through bug-1804 have not yet been synced to Gitea issues via gitea-sync. Retry once the server is reachable.
6. scheduler-sim-eager-reconciliation's revision 4 is committed and test-verified (77/77), but has not yet been exercised against a live multi-round replay beyond the unit/integration fixtures added this window — worth a live-replay smoke pass next session to confirm the synchronous-state change behaves as expected outside the test harness.