Rongsong Shen
0480798583
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
70 lines
12 KiB
Markdown
70 lines
12 KiB
Markdown
# XCE Project Status — Sep 06 – Sep 12, 2026
|
||
|
||
> Note: the calendar-default window (Aug 31–Sep 05) is already covered by
|
||
> `aug30-sep05-2026.md`; this report covers the actual gap — the last
|
||
> completed day before this report was generated.
|
||
|
||
## TL;DR
|
||
|
||
A CLI-quality week bookended by two scheduler-side finishes: Sunday closed out `resreq-maxtasks-wiring` (real multi-host task-count wiring, proven end-to-end and HA-soak clean), and the middle of the week was almost entirely `xce-test-cli-ux` — a large, self-contained rebuild of `xce_test`'s interactive shell (interrupt-safe RPCs, confirmation gates, tab completion, JSON output, flag parsing) that surfaced its own cluster of testsuite bugs (bug-1791–1797). Monday/Tuesday cleared the last of the gitea#46 limit-plugin cascade and closed the from-scratch build race (bug-1798) for good. Saturday's session — driven by a live scheduler-replay debugging conversation — fixed a real resource over-commit (bug-1799/1801), added human-scaled resource-unit display (bug-1800), and, after live user feedback, revised `scheduler-sim-eager-reconciliation`'s own design to make captured jobs' `job->state` transition synchronously (bug-1804) rather than deferred. Last week's flagged flagship gap, bug-1770 (multi-host allocation had no live production path), is now fixed.
|
||
|
||
## Commits by day
|
||
|
||
### Sun 09-06 — resreq-maxtasks-wiring to done, plus last week's report
|
||
- **`799a71ec`..`5c87f1cc`** (12 commits) — built the real task-count derivation helper, wired it into fresh job-ingest and snapshot/replay-restore ingest, closed a worker launch-type coverage gap, then validated with a real multi-host end-to-end harness (plain multi-task job + array-job element, both genuinely split across hosts) and a full regression + HA soak pass. This is the fix for last week's flagged **bug-1770** (multi-host allocation had zero live production path).
|
||
- **`a6b735b6`** — added `aug30-sep05-2026.md` and `CHANGES.md` for the prior window.
|
||
|
||
### Mon 09-07 — gitea#46 limit-plugin cascade, apiserver job-control fixes
|
||
- **`22cfe616`..`924ac32c`** — fixed the limit-plugin usergroup/scope/error-code cascade (gitea#46) end-to-end: named-usergroup resolution, default-scope semantics, kvmgr not-found-vs-error confusion (twice), an RTTI/pointer-indirection type-confusion bug, and finally live-verified all 7 `limits.yaml` limits matching expectations (bug-1784–1789).
|
||
- **`099dbdab`, `5ab344e1`** — apiserver: fixed a job-control union discriminant bug + added human-readable job IDs (gitea#45 reopened), and propagated submit user as job execuser (bug-1780).
|
||
- **`0d3f925c`, `ef3c4670`** — scheduler: retired reconciled jobs from picker candidates and excluded terminal jobs from the assigned-jobs listing (bug-1759/1778/1779).
|
||
- **`a35f597e`, `e395eaa3`** — worker: real maxswap/maxtmp hardware detection (bug-1554); frontend: fixed host job-state/EXEC_HOST columns to count real slots.
|
||
- **`850d186d`, `028d8f6a`** — build: `xce-build.sh` now self-wraps into `nix develop` and caps its run-queue length via a live-computed `--jobs` flag.
|
||
|
||
### Tue 09-08 — bug-1790 residual gap, leak fixes, scheduler global-limit refactor
|
||
- **`9caf1424`..`a6d96705`** — fixed bug-1790 (global limits on per-host resources reading the wrong "want"), live-verified with a real global mem limit, then refactored job-total limit computation into the limit plugin itself and fixed a regression that briefly broke slots-based global enforcement along the way.
|
||
- **`4b20590c`, `c832706e`** — fixed two pre-existing ASAN leaks (host_account/job_account test fixtures, bug-1621/1723) and a wrong resInUse precondition in a mem-reservation test (bug-1758).
|
||
|
||
### Wed 09-09 — xce-test-cli-ux: foundation and core (bulk of the week)
|
||
- **`0748f10c`..`96746af3`** (16 commits) — built the flag-descriptor parsing/rejection core, typo suggestions, SIGINT/EINTR handling, an interruptible-call heap-ownership adapter with per-connection locking, a pty-driven black-box CLI test harness, tab completion, in-session `source` with cycle detection, a JSON list-output helper library, a destructive-command confirmation gate wired to five commands, flagspec migration for every flag-accepting command, and consolidated help generation. Along the way, fixed bug-1557 (join's maxmem default) and bug-1711 (display/text defects).
|
||
- Testsuite bugs surfaced and fixed same-day: **bug-1791** (cross-thread pthread mutex unlock, EPERM), **bug-1792** (tab-completion test silent-input-mode gap).
|
||
|
||
### Thu 09-10 — xce-test-cli-ux: integration, validation, and the build-race root-cause fix
|
||
- **`a98fad24`..`38595099`** — routed diagnostic/warning output to stderr uniformly, then verified Ctrl-C cancellation, the confirmation gate, `source`'s interaction with it, and help/completion/context-switching end-to-end (tasks 4.1–4.4). Found and deliberately deferred four more testsuite bugs (bug-1794–1797: a `cmd_jobrun` ASAN abort, a `leave_ctx()` use-after-free, misleading `perror()` diagnostics, a queue-name leak) — all out of boundary for their discovering (validation-only) tasks.
|
||
- **`654c6e8c`, `c1c6c9a2`, `f09fbdb3`** — root-caused and closed the long-standing intermittent from-scratch build failure (`fatal error: controller.capnp.h`): a missing-dependency-edge race in the capnp codegen `custom_target`, fixed at the `meson.build` level and extended to controller/worker/scheduler and all four PEG parser generators (bug-1798).
|
||
|
||
### Fri–Sat 09-11/09-12 — live replay debugging: resource over-commit, unit display, design revision
|
||
- Live scheduler-replay session surfaced and fixed, same-day (all TDD, RED confirmed before GREEN): **bug-1799/1801** (sim replay resInUse over-commit — a worker's used slots could exceed its own capacity after a resync replay; `sched_account_jobs` now skips RECONCILING/UNAVAIL workers like `core_reset_workers` already does), **bug-1800** (bytes-valued CONSUMABLE resources now render human-scaled units), and **bug-1802/1803** (a `break ANY` breakpoint kind was added, then reverted after live feedback it over-fired inside busy sessions, replaced with a prescan-based "next non-dummy session" hint in `info summary`).
|
||
- **`25fb5c11`** (Sat) — bundled commit landing all of the above plus **bug-1804**: a user-directed design-fork resolution revising `scheduler-sim-eager-reconciliation` so a captured job's `job->state` is set to `RUN` synchronously at dispatch capture (mirroring normal-mode's `sched_commit_decision`, bug-1624) instead of deferring until its separately-recorded `JOB_STATE_CHANGE` event replays. Full scheduler suite verified 77/77 green before commit.
|
||
|
||
## Bugs opened/fixed this window
|
||
|
||
- **29 genuinely new bugs** (bug-1775 – bug-1804, one intentional ID gap at bug-1783), **24 fixed**, **5 deliberately deferred/still open** (bug-1777, bug-1793, bug-1796, bug-1797 — all explicitly out-of-boundary for the validation-only tasks that found them; bug-1799 itself stays logged as "not yet fixed" by this project's own convention of routing its actual fix through a separate follow-up ID, bug-1801, which is fixed).
|
||
- **Carry-over status from last week's report**: **bug-1770 — fixed** (Sun, `resreq-maxtasks-wiring`), closing the flagship gap flagged last week (multi-host allocation now has a real production path). `bug-1621`/`bug-1723`/`bug-1758` — fixed (Tue). `bug-1714` — still open, no movement this window (last touched Sep 4). `bug-1632`, `bug-1726`, `bug-1728` — no movement, still open.
|
||
- Highest-severity fixes: **bug-1799/1801** (resource-accounting over-commit — a worker could show more slots in use than it physically has), **bug-1798** (root-caused a long-standing intermittent from-scratch build failure), **bug-1784–1789** (gitea#46's full limit-plugin cascade, 6 stacked bugs found and fixed in sequence).
|
||
- **Integrity check**: 29 unique new bug IDs, sequential except one deliberate gap (bug-1783); 0 duplicate IDs in the buglog overall.
|
||
|
||
## Spec status changes
|
||
|
||
- **`resreq-maxtasks-wiring`** — `implementation-complete`, all subtasks done (Sun), HA-soak and full-suite verified with no regression. Closes bug-1770.
|
||
- **`scheduler-sim-eager-reconciliation`** — revision 4 landed (Sat, `25fb5c11`): synchronous job-state-at-capture per bug-1804/user direction. Phase remains `implementation-complete`, 12/12 tasks.
|
||
- **`xce-test-cli-ux`** — all 23 real subtasks across all 4 task groups (Foundation → Validation) are checked done, but `spec.json` still reads phase `tasks-generated` (`ready_for_implementation: true`) — the same metadata-lag pattern flagged for three other specs in the prior two reports. A phase-sync pass is due.
|
||
|
||
## Stats
|
||
|
||
- **73 commits** — **62 real** (feature/fix/refactor/test/docs), **11 `.wolf` bookkeeping**.
|
||
- **287 non-`.wolf` files changed**, **+22,190 / −1,889** (non-wolf headline; raw incl. `.wolf`: +30,044 / −7,591).
|
||
- Busiest areas by files touched: `src/scheduler` (47), `src/controller` (28), `.kiro` (16), `src/lib` (10), `src/worker` (7), `src/frontend` (7), `.wolf` (7), `.claude` (5).
|
||
- By spec scope: `xce-test-cli-ux` (25 commits), `resreq-maxtasks-wiring` (13), `scheduler` (11, unscoped fixes), `build` (5).
|
||
- **2 specs shipped end-to-end** this window (`resreq-maxtasks-wiring`, and `scheduler-sim-eager-reconciliation`'s revision-4 patch); **1 spec functionally complete pending metadata sync** (`xce-test-cli-ux`).
|
||
- **8 sessions** logged in `.wolf/memory.md` touching this window (Sep 7 ×5, Sep 9, Sep 11, Sep 12); busiest logged day Sep 7 (184 actions, gitea#46 cascade + apiserver fixes).
|
||
- Commits landed on all 7 days; peak day **Wed 09-09** (16 commits, `xce-test-cli-ux` foundation-to-core).
|
||
|
||
## Open threads / to-do heading into next session
|
||
|
||
1. **`xce-test-cli-ux`'s `spec.json` metadata lag** — tasks.md shows all 23 real subtasks complete across all 4 groups, but phase still reads `tasks-generated`. Same class of fix already applied to three other specs in recent weeks (`scheduler-dispatch-commit`, `scheduler-snapshot-job-allocation`, and implicitly `eliminate-frontend-makefile`) — needs the same treatment: sync phase to `implementation-complete`.
|
||
2. **Four deliberately-deferred testsuite bugs from `xce-test-cli-ux`'s validation tasks remain open and unowned**: bug-1796 (misleading `perror()` diagnostics on RPC-failure paths), bug-1797 (queue-name leak on an RPC-error path), bug-1793 (join's stale maxswap/maxtmp defaults — docs and code both need fixing together to avoid a new mismatch), bug-1777 (workerlist MaxSwap can disagree with the scheduler's own view). Worth a triage pass to decide which need a dedicated follow-up spec vs. quick fixes.
|
||
3. **`bug-1714` still unresolved**, now over two weeks old with zero movement this window — the stale second reader for `job_submit_data_t->project_name` remains unfound. Needs either a fresh live gdb-watchpoint campaign or an explicit decision to accept it as a monitored low-frequency risk.
|
||
4. **Three lower-priority open bugs with zero movement across two consecutive windows**: `bug-1632`, `bug-1726`, `bug-1728` — still awaiting triage.
|
||
5. **`gitea` MCP server connectivity** — noted in `.wolf/STATUS.md` as failing to connect (CONNECT_TIMEOUT) as of this window's close; buglog entries bug-1798 through bug-1804 have not yet been synced to Gitea issues via `gitea-sync`. Retry once the server is reachable.
|
||
6. **`scheduler-sim-eager-reconciliation`'s revision 4** is committed and test-verified (77/77), but has not yet been exercised against a live multi-round replay beyond the unit/integration fixtures added this window — worth a live-replay smoke pass next session to confirm the synchronous-state change behaves as expected outside the test harness.
|