myxce/reports/aug16-aug22-2026.md
Rongsong Shen 0c736c795b
chore(specs): add scheduler-sim-eager-reconciliation spec, weekly report, cross-spec sequencing notes
New scheduler-sim-eager-reconciliation requirements spec; weekly_report skill;
weekly status report for Aug 16-22. Also records cross-spec dependency notes
on scheduler-allocation and scheduler-preemption reflecting how
scheduler-dispatch-commit's redefined RUN-state timing affects their designs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
2026-08-24 12:56:35 +00:00

11 KiB
Raw Blame History

XCE Project Status — Aug 16 – Aug 22, 2026

TL;DR

The week opened by clearing the entire uncommitted backlog the Aug 08–15 report ended on, then ran four specs start-to-finish. resource-value-verification (Tue), build-time-versioning (Thu), dirty-content-fingerprint (Fri), and scheduler-sim-decision-parity (Sat) each went spec → implementation → tests inside a single day. Monday was a firefight: seven stacked bugs, one of which (bug-1569) was blocking all job dispatch. Every one of the prior week's 7 open carry-over bugs is now closed, and 64 new bugs (bug-1561–1624) were opened and fixed within the window. The one thing that got worse: .wolf/buglog.json picked up 39 duplicate IDs from a restarted counter on Aug 15–17.

Commits by day

Aug 16 (Sun) — clearing the prior week's uncommitted backlog

  • d479b707 — component-failover-adoption Phase 6: closed apiserver's idle-period rejoin gap (bug-1548/bug-1549). This was open thread #1 from last week's report.
  • 3afe3d55 — xceadmin config dump command + size-string thresholds (open thread #2).
  • 45400c3a — controller: closed 6 data-safety races, an apply-pump O(N²) regression, and a boot-time TOCTOU.
  • 8a6c339a — libxce bug-1425: jobsubmit now populates *jobid on idempotent-replay success.
  • ce0ee78b, 5a08dac5, 570e464d — spec metadata reconciliation: promoted 3 new specs, closed 3 stalled specs whose follow-up work had already landed, closed admin-config-tool task 6.3 (now 27/27).
  • 69f44c2f — published the Aug 8–15 weekly report.

Aug 17 (Mon) — HA + dispatch firefight (bug-1562 → bug-1569)

  • 846713ab — bug-1562 standby election freeze + bug-1563 flusher retry storm.
  • 986a3304 — bug-1564: persist_flush now fires the durable-advance callback.
  • 7041d988 — bug-1569: three stacked crashes blocking ALL job dispatch. The week's most severe fix.
  • 6e5c3a89 — bug-1569 companion: worker periodic resval reports were tagged with the wrong host.
  • aabddf60 — bug-1568: worker resource values always displayed as 0.00.
  • 0c8e417b — bug-1567: sim-mode picker never learned about the initial snapshot's own jobs/queues.
  • 1c943e98 — bug-1565: --incremental flag for xce-build.sh.
  • 0f1df499 — bug-1566: standalone xceadmin packaging + deploy-ha usability.
  • Started resource-value-verification (tasks 1.1, 1.2, 2.1).

Aug 18 (Tue) — resource-value-verification, spec to done in one day

  • Tasks 2.2 → 8.2 landed: resupdate per-name accept/reject verification, xce-deploy-ha apiserver step + start-all.sh, scheduler and apiserver resval trace logs, then the four-tier verification chain — worker join/injection (6.1), controller (6.2), scheduler (6.3), apiserver (6.4) — plus verdict aggregation (7.1), live E2E run (8.1), and a verdict-logic selftest (8.2).
  • b406dd76 — bug-1578 (CRITICAL, reviewer-discovered): sched_worker_change() use-after-free on worker disconnect, reproducible controller SIGSEGV.
  • b673b7c9 — bug-1579 closed for real via xce_test's ! shell-escape.

Aug 19 (Wed) — scheduler-core task 12.5 closeout

  • c9acd502 — clock/resval/jobid-format fixes plus REPL live-operator gaps.
  • a13068dc — extended the core-isolation nm audit to core_events.c/core_session.c/core_clock.c (Req 5 AC 1/2).
  • b450a680 — activated virtual-clock logging for sim/REPL sessions (Req 5 AC 3).
  • e52c68f4 — admission-gated OPCode_schedAdmin's blocking wait (Req 5 AC 10).
  • 70fe9e46 — bug-1589: dbus_workers_update() starved workers after the first DONE/EXITED.
  • 515f4ca0 — bug-1588: job.c TRACE lines now print human-readable jobids.
  • 83e61eec — xce-deploy-ha gained real xcecd worker deployment alongside virtual.

Aug 20 (Thu) — build-time-versioning, spec to done in one day (21 tasks)

  • 1ff9e66c spec → 4b7cc786 build_time meson option → full fan-out across every binary: controller/scheduler --version (88649a04), worker (bc5c669b), apiserver -version (17e837b1), xcecli via Cobra (fb09afe6), xceadmin build-info module + fallback stub (e1cd9d25, 7f5c87ce).
  • Build plumbing: xce-build.sh real commit id/build time (bca63974), nix build-phase timestamps (d08fda76), go_build ldflags injection (18ae0cc3, 071f3895), Makefile ldflags (45c9b026).
  • Three regression scripts: motivating-incident (0eefc95f), version/startup-log consistency (79941a9f), xceadmin end-to-end build/install/version cycle (084a5a95).
  • Also ddee884c — bug-1600: makeGroup() freed an rtti makeString() pointer via raw free(), crashing the scheduler during controller failover.
  • Also 5147396d — controller job start_time/finish_time/run_time were never set by the apply-pump.

Aug 21 (Fri) — dirty-content-fingerprint, spec to done in one day (21 tasks)

  • d3c583b3 spec → a860141b dirty_fingerprint meson option + C define → fingerprint surfaced in all six binaries (fd62e26b, d3d9522f, a4c1b5ad, d65d18ad, 2bf62a64, c79ccf0f).
  • Computation paths: nix via self.narHash (fc5cba15), xce-build.sh dev-loop (c9d40ba1), Makefile dev-loop (aa3391a9), --version-only composition (7ebcba6d).
  • Three regression tests: dev-loop distinguishability (a7d64c82), startup-log-unchanged consistency (a4785418), Nix production-path fingerprint (f4d2c123).

Aug 22 (Sat) — scheduler-sim-decision-parity, spec to done in one day (13 tasks)

  • 5f7e1e98/25e35bac documented the driver-equivalence invariant and scoped the fix; 0b2c856d finalized design + tasks.
  • Implementation: relocated resource accounting to core level (3106c82e), gave sim mode driver.c's round-boundary split (134380b8), added reconciliation trigger/lookup (81358b52), reconstructed job allocation with safe ownership (eccba841), reported reconciled decisions through the existing channel (0c237f60), wired reconciliation into the round boundary (12644094).
  • Validation: contention-parity proof (b3190732), regression suite for existing guarantees (88bed529), live verification against a real 24-round recording (3c4091d8), then ca01509f marked the feature complete.
  • 979cd24f — scheduler-sim-repl-verbosity spec approved (requirements/design/tasks).
  • 1c906e95 — bug-1588 follow-up: worker job ids print via fmt_jobid().

Bugs opened/fixed this window

  • 64 genuinely new bugs (bug-1561 – bug-1624), all 64 fixed — zero left open at window end.
  • All 8 carry-over bugs the prior report listed as open are now FIXED: bug-1508, 1519, 1548, 1549, 1553, 1554, 1557, 1560. That includes bug-1553, which last week's report flagged as "the most concerning open item" (intermittent controller test segfault).
  • Highest-severity fixes: bug-1569 (three stacked crashes blocking all job dispatch), bug-1578 (CRITICAL controller UAF on worker disconnect), bug-1600 (allocator-mismatch heap corruption during failover), bug-1572 (scheduler double-free/dangling pointer on resvals delta).
  • ⚠️ .wolf/buglog.json integrity regressed. The file grew 917 → 1004 entries, but 39 IDs are now duplicated: entries appended on Aug 15 (8), Aug 16 (14), and Aug 17 (22) reused the already-taken bug-875–bug-949 range. Example: bug-908 is both a 2026-06-21 raft_sim.c guard-clause bug and a 2026-08-16 config-upgrade-snapshot/spec.json refactor note. This is exactly the counter-restart failure the OpenWolf bug-logging rule warns is "unrecoverable without a git-history merge."

Spec status changes

  • resource-value-verification — created and driven to implementation-complete on Aug 17–18. Task checkboxes read 15/23 despite the phase; the trailing 8 are unchecked.
  • build-time-versioning — created and fully implemented Aug 20, 21/21 tasks, but spec.json phase still reads tasks-generated. Metadata lag, not work lag.
  • dirty-content-fingerprint — created and fully implemented Aug 21, 21/21 tasks, same tasks-generated phase lag.
  • scheduler-sim-decision-parity — created and completed Aug 22, 13/13 tasks, phase implementation-complete.
  • scheduler-sim-repl-verbosity — spec approved Aug 22, 0/15 tasks, not started.
  • admin-config-tool — closed Aug 16 at 27/27.
  • component-failover-adoption — Phase 6 fix finally committed Aug 16, closing last week's #1 open thread; checkboxes read 22/27.
  • Three stalled specs closed Aug 16 as already-superseded by landed work.

Stats

  • 192 commits in the window — 103 feature/fix/docs, 89 .wolf bookkeeping/checkpoint commits.
  • 218 non-.wolf files changed, +28,657 / −2,268 lines (excluding .wolf/; including it the raw number is +288,983 / −22,630, dominated by token-ledger.json and buglog.json churn).
  • Busiest areas: src/scheduler (55 files), .kiro (54), src/controller (25), src/worker (13), src/frontend (13).
  • 4 specs implemented end-to-end, 1 spec closed, 1 spec approved-not-started, 3 stalled specs retired.
  • 19 sessions logged in .wolf/memory.md across Aug 16–21.
  • Commits landed on all 7 days; peak days Mon and Thu (35 each).

Open threads / to-do heading into next session

  1. Repair the 39 duplicate buglog IDs. The Aug 15–17 counter restart collided new entries with bug-875–bug-949. Needs a git-history-informed merge: pull the pre-collision file, identify which of each duplicate pair is the newcomer, and renumber the newcomers above the current max (1624+). The Stop hook validates duplicates, so this will keep flagging until fixed.
  2. Sync spec.json phase metadata for build-time-versioning and dirty-content-fingerprint — both are 21/21 implemented but still read tasks-generated.
  3. Reconcile resource-value-verification's checkboxes — phase says implementation-complete, tasks say 15/23. Either the last 8 tasks are done-but-unchecked or the phase is premature.
  4. scheduler-core is 49/59 and component-failover-adoption is 22/27 — both sit at implementation-complete with unchecked tasks. Same reconcile-or-reopen question.
  5. Decide the next spec to start. scheduler-sim-repl-verbosity is approved at 0/15. A new untracked spec, scheduler-sim-eager-reconciliation, appeared in the working tree and is not yet committed.
  6. Uncommitted at window end: src/scheduler/src/handler.c, src/scheduler/testsuite/test_handlers.c, and untracked generated src/lib/controller.capnp.{c,h} — the generated pair should probably be gitignored rather than committed.
  7. .wolf/dashboard-token — still untracked, still holding a live-looking token, still not covered by .gitignore. Carried over unresolved from last week's item #3.