Rongsong Shen
0c736c795b
New scheduler-sim-eager-reconciliation requirements spec; weekly_report skill; weekly status report for Aug 16-22. Also records cross-spec dependency notes on scheduler-allocation and scheduler-preemption reflecting how scheduler-dispatch-commit's redefined RUN-state timing affects their designs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
103 lines
11 KiB
Markdown
103 lines
11 KiB
Markdown
# XCE Project Status — Aug 16 – Aug 22, 2026
|
||
|
||
## TL;DR
|
||
|
||
The week opened by clearing the entire uncommitted backlog the Aug 08–15 report ended on, then ran four specs start-to-finish. **resource-value-verification** (Tue), **build-time-versioning** (Thu), **dirty-content-fingerprint** (Fri), and **scheduler-sim-decision-parity** (Sat) each went spec → implementation → tests inside a single day. Monday was a firefight: seven stacked bugs, one of which (**bug-1569**) was blocking *all* job dispatch. Every one of the prior week's 7 open carry-over bugs is now closed, and 64 new bugs (bug-1561–1624) were opened and fixed within the window. The one thing that got worse: `.wolf/buglog.json` picked up **39 duplicate IDs** from a restarted counter on Aug 15–17.
|
||
|
||
## Commits by day
|
||
|
||
### Aug 16 (Sun) — clearing the prior week's uncommitted backlog
|
||
|
||
- **`d479b707`** — component-failover-adoption Phase 6: closed apiserver's idle-period rejoin gap (bug-1548/bug-1549). This was open thread #1 from last week's report.
|
||
- **`3afe3d55`** — xceadmin `config dump` command + size-string thresholds (open thread #2).
|
||
- **`45400c3a`** — controller: closed 6 data-safety races, an apply-pump O(N²) regression, and a boot-time TOCTOU.
|
||
- **`8a6c339a`** — libxce bug-1425: `jobsubmit` now populates `*jobid` on idempotent-replay success.
|
||
- **`ce0ee78b`, `5a08dac5`, `570e464d`** — spec metadata reconciliation: promoted 3 new specs, closed 3 stalled specs whose follow-up work had already landed, closed **admin-config-tool** task 6.3 (now 27/27).
|
||
- **`69f44c2f`** — published the Aug 8–15 weekly report.
|
||
|
||
### Aug 17 (Mon) — HA + dispatch firefight (bug-1562 → bug-1569)
|
||
|
||
- **`846713ab`** — bug-1562 standby election freeze + bug-1563 flusher retry storm.
|
||
- **`986a3304`** — bug-1564: `persist_flush` now fires the durable-advance callback.
|
||
- **`7041d988`** — **bug-1569: three stacked crashes blocking ALL job dispatch.** The week's most severe fix.
|
||
- **`6e5c3a89`** — bug-1569 companion: worker periodic resval reports were tagged with the wrong host.
|
||
- **`aabddf60`** — bug-1568: worker resource values always displayed as 0.00.
|
||
- **`0c8e417b`** — bug-1567: sim-mode picker never learned about the initial snapshot's own jobs/queues.
|
||
- **`1c943e98`** — bug-1565: `--incremental` flag for `xce-build.sh`.
|
||
- **`0f1df499`** — bug-1566: standalone xceadmin packaging + deploy-ha usability.
|
||
- Started **resource-value-verification** (tasks 1.1, 1.2, 2.1).
|
||
|
||
### Aug 18 (Tue) — resource-value-verification, spec to done in one day
|
||
|
||
- Tasks 2.2 → 8.2 landed: resupdate per-name accept/reject verification, `xce-deploy-ha` apiserver step + `start-all.sh`, scheduler and apiserver resval trace logs, then the four-tier verification chain — worker join/injection (6.1), controller (6.2), scheduler (6.3), apiserver (6.4) — plus verdict aggregation (7.1), live E2E run (8.1), and a verdict-logic selftest (8.2).
|
||
- **`b406dd76`** — bug-1578 (CRITICAL, reviewer-discovered): `sched_worker_change()` use-after-free on worker disconnect, reproducible controller SIGSEGV.
|
||
- **`b673b7c9`** — bug-1579 closed for real via `xce_test`'s `!` shell-escape.
|
||
|
||
### Aug 19 (Wed) — scheduler-core task 12.5 closeout
|
||
|
||
- **`c9acd502`** — clock/resval/jobid-format fixes plus REPL live-operator gaps.
|
||
- **`a13068dc`** — extended the core-isolation `nm` audit to `core_events.c`/`core_session.c`/`core_clock.c` (Req 5 AC 1/2).
|
||
- **`b450a680`** — activated virtual-clock logging for sim/REPL sessions (Req 5 AC 3).
|
||
- **`e52c68f4`** — admission-gated `OPCode_schedAdmin`'s blocking wait (Req 5 AC 10).
|
||
- **`70fe9e46`** — bug-1589: `dbus_workers_update()` starved workers after the first DONE/EXITED.
|
||
- **`515f4ca0`** — bug-1588: `job.c` TRACE lines now print human-readable jobids.
|
||
- **`83e61eec`** — `xce-deploy-ha` gained real `xcecd` worker deployment alongside virtual.
|
||
|
||
### Aug 20 (Thu) — build-time-versioning, spec to done in one day (21 tasks)
|
||
|
||
- **`1ff9e66c`** spec → **`4b7cc786`** `build_time` meson option → full fan-out across every binary: controller/scheduler `--version` (`88649a04`), worker (`bc5c669b`), apiserver `-version` (`17e837b1`), xcecli via Cobra (`fb09afe6`), xceadmin build-info module + fallback stub (`e1cd9d25`, `7f5c87ce`).
|
||
- Build plumbing: `xce-build.sh` real commit id/build time (`bca63974`), nix build-phase timestamps (`d08fda76`), `go_build` ldflags injection (`18ae0cc3`, `071f3895`), Makefile ldflags (`45c9b026`).
|
||
- Three regression scripts: motivating-incident (`0eefc95f`), version/startup-log consistency (`79941a9f`), xceadmin end-to-end build/install/version cycle (`084a5a95`).
|
||
- Also **`ddee884c`** — bug-1600: `makeGroup()` freed an rtti `makeString()` pointer via raw `free()`, crashing the scheduler during controller failover.
|
||
- Also **`5147396d`** — controller job `start_time`/`finish_time`/`run_time` were never set by the apply-pump.
|
||
|
||
### Aug 21 (Fri) — dirty-content-fingerprint, spec to done in one day (21 tasks)
|
||
|
||
- **`d3c583b3`** spec → **`a860141b`** `dirty_fingerprint` meson option + C define → fingerprint surfaced in all six binaries (`fd62e26b`, `d3d9522f`, `a4c1b5ad`, `d65d18ad`, `2bf62a64`, `c79ccf0f`).
|
||
- Computation paths: nix via `self.narHash` (`fc5cba15`), `xce-build.sh` dev-loop (`c9d40ba1`), Makefile dev-loop (`aa3391a9`), `--version-only` composition (`7ebcba6d`).
|
||
- Three regression tests: dev-loop distinguishability (`a7d64c82`), startup-log-unchanged consistency (`a4785418`), Nix production-path fingerprint (`f4d2c123`).
|
||
|
||
### Aug 22 (Sat) — scheduler-sim-decision-parity, spec to done in one day (13 tasks)
|
||
|
||
- **`5f7e1e98`/`25e35bac`** documented the driver-equivalence invariant and scoped the fix; **`0b2c856d`** finalized design + tasks.
|
||
- Implementation: relocated resource accounting to core level (`3106c82e`), gave sim mode `driver.c`'s round-boundary split (`134380b8`), added reconciliation trigger/lookup (`81358b52`), reconstructed job allocation with safe ownership (`eccba841`), reported reconciled decisions through the existing channel (`0c237f60`), wired reconciliation into the round boundary (`12644094`).
|
||
- Validation: contention-parity proof (`b3190732`), regression suite for existing guarantees (`88bed529`), live verification against a real 24-round recording (`3c4091d8`), then **`ca01509f`** marked the feature complete.
|
||
- **`979cd24f`** — scheduler-sim-repl-verbosity spec approved (requirements/design/tasks).
|
||
- **`1c906e95`** — bug-1588 follow-up: worker job ids print via `fmt_jobid()`.
|
||
|
||
## Bugs opened/fixed this window
|
||
|
||
- **64 genuinely new bugs** (bug-1561 – bug-1624), **all 64 fixed** — zero left open at window end.
|
||
- **All 8 carry-over bugs** the prior report listed as open are now FIXED: bug-1508, 1519, 1548, 1549, 1553, 1554, 1557, 1560. That includes bug-1553, which last week's report flagged as "the most concerning open item" (intermittent controller test segfault).
|
||
- Highest-severity fixes: **bug-1569** (three stacked crashes blocking all job dispatch), **bug-1578** (CRITICAL controller UAF on worker disconnect), **bug-1600** (allocator-mismatch heap corruption during failover), **bug-1572** (scheduler double-free/dangling pointer on resvals delta).
|
||
- ⚠️ **`.wolf/buglog.json` integrity regressed.** The file grew 917 → 1004 entries, but **39 IDs are now duplicated**: entries appended on Aug 15 (8), Aug 16 (14), and Aug 17 (22) reused the already-taken `bug-875`–`bug-949` range. Example: `bug-908` is both a 2026-06-21 `raft_sim.c` guard-clause bug and a 2026-08-16 `config-upgrade-snapshot/spec.json` refactor note. This is exactly the counter-restart failure the OpenWolf bug-logging rule warns is "unrecoverable without a git-history merge."
|
||
|
||
## Spec status changes
|
||
|
||
- **resource-value-verification** — created and driven to `implementation-complete` on Aug 17–18. Task checkboxes read 15/23 despite the phase; the trailing 8 are unchecked.
|
||
- **build-time-versioning** — created and fully implemented Aug 20, **21/21 tasks**, but `spec.json` phase still reads `tasks-generated`. Metadata lag, not work lag.
|
||
- **dirty-content-fingerprint** — created and fully implemented Aug 21, **21/21 tasks**, same `tasks-generated` phase lag.
|
||
- **scheduler-sim-decision-parity** — created and completed Aug 22, **13/13 tasks**, phase `implementation-complete`.
|
||
- **scheduler-sim-repl-verbosity** — spec approved Aug 22, **0/15 tasks**, not started.
|
||
- **admin-config-tool** — closed Aug 16 at 27/27.
|
||
- **component-failover-adoption** — Phase 6 fix finally committed Aug 16, closing last week's #1 open thread; checkboxes read 22/27.
|
||
- Three stalled specs closed Aug 16 as already-superseded by landed work.
|
||
|
||
## Stats
|
||
|
||
- **192 commits** in the window — **103 feature/fix/docs**, 89 `.wolf` bookkeeping/checkpoint commits.
|
||
- **218 non-`.wolf` files changed**, **+28,657 / −2,268** lines (excluding `.wolf/`; including it the raw number is +288,983 / −22,630, dominated by `token-ledger.json` and `buglog.json` churn).
|
||
- Busiest areas: `src/scheduler` (55 files), `.kiro` (54), `src/controller` (25), `src/worker` (13), `src/frontend` (13).
|
||
- **4 specs implemented end-to-end**, 1 spec closed, 1 spec approved-not-started, 3 stalled specs retired.
|
||
- **19 sessions** logged in `.wolf/memory.md` across Aug 16–21.
|
||
- Commits landed on **all 7 days**; peak days Mon and Thu (35 each).
|
||
|
||
## Open threads / to-do heading into next session
|
||
|
||
1. **Repair the 39 duplicate buglog IDs.** The Aug 15–17 counter restart collided new entries with `bug-875`–`bug-949`. Needs a git-history-informed merge: pull the pre-collision file, identify which of each duplicate pair is the newcomer, and renumber the newcomers above the current max (1624+). The Stop hook validates duplicates, so this will keep flagging until fixed.
|
||
2. **Sync spec.json phase metadata** for build-time-versioning and dirty-content-fingerprint — both are 21/21 implemented but still read `tasks-generated`.
|
||
3. **Reconcile resource-value-verification's checkboxes** — phase says `implementation-complete`, tasks say 15/23. Either the last 8 tasks are done-but-unchecked or the phase is premature.
|
||
4. **scheduler-core is 49/59** and **component-failover-adoption is 22/27** — both sit at `implementation-complete` with unchecked tasks. Same reconcile-or-reopen question.
|
||
5. **Decide the next spec to start.** scheduler-sim-repl-verbosity is approved at 0/15. A new untracked spec, `scheduler-sim-eager-reconciliation`, appeared in the working tree and is not yet committed.
|
||
6. **Uncommitted at window end:** `src/scheduler/src/handler.c`, `src/scheduler/testsuite/test_handlers.c`, and untracked generated `src/lib/controller.capnp.{c,h}` — the generated pair should probably be gitignored rather than committed.
|
||
7. **`.wolf/dashboard-token`** — still untracked, still holding a live-looking token, still not covered by `.gitignore`. Carried over unresolved from last week's item #3.
|