XCE Project Status — Aug 23 – Aug 29, 2026

Note: the calendar-default window (Aug 17–22) is already covered by aug16-aug22-2026.md; this report covers the actual gap — the last complete day before this report was generated (Aug 30 is in progress).

TL;DR

An unusually dense week: 97 real commits shipped two specs end-to-end (scheduler-dispatch-commit, xce-test-rc-file), drove scheduler-sim-eager-reconciliation and scheduler-snapshot-job-allocation to substantial completion, and closed out a systematic five-bug sweep of WAL-replay field-completeness gaps alongside several safety-critical HA correctness fixes (vote-denial on persist failure, stale-term re-election blocking, fail-loud disconnect handling). Last week's #1 open thread — 39 duplicate buglog IDs — was fully repaired on Monday. The week closed with a deep multi-session live-debugging investigation that fixed three confirmed production bugs (a scheduler resource leak, a controller heap-use-after-free that crashed 2 of 3 controllers under load, and a set of permanently-dead job-notification hooks that could leave a job's state stuck as RUN forever in the scheduler and apiserver) and made major forensic progress on a fourth, still-open scheduler crash (bug-1714) — proved it's a genuine cross-thread use-after-free, not the overflow first suspected, but the exact stale-pointer source remains unisolated.

Commits by day

Sun 08-23 — `scheduler-dispatch-commit` spec, start to finish in one day

5724d1e9→a9cfee5f (13 commits) — dispatch-failure classification, a durable WAL_JOB_DISPATCH fact type, scheduler transitions to RUN at commit time, controller-side retryable/terminal dispatch-failure resolution with bounded escalation, and full multi-round + terminal + retryable e2e coverage. Spec marked implementation-complete (f784e2bd).

Mon 08-24 — Gitea issue triage and buglog integrity repair

1c9cfc1f, e83d207a — reproduced and logged 4 Gitea issues (#40–42, #44) plus a gitea-sync bug found while doing it; corrected an earlier false-negative on #43 (bug-1678 — turned out reproducible).
db142d41→88c7c05d — fixed the 4 reproduced issues: dispatch-time resource allocation not persisted into WAL_JOB_DISPATCH (bug-1678), live job-status pushes to the apiserver never firing (bug-1679), jobs list -l/-o validating against the wrong struct (bug-1674), unmatched workers not falling back to a default HostConfig (bug-1673) — plus from_host threaded end-to-end through the wire format, CGO layer, and job-submit interning object (bug-1676).
755976fc — repaired last week's #1 open thread: renumbered 40 buglog entries that had collided with bug-875–bug-949 after the Aug 13–17 counter restart.
5c82ad0b — repaired last week's #2 open thread: synced spec.json phase metadata for build-time-versioning/dirty-content-fingerprint (both were 21/21 done but stuck on tasks-generated) and flipped 23 stale section-header checkboxes across 3 more specs whose numbered subtasks were already complete.

Tue 08-25 — peak day (46 commits): WAL-replay field-completeness sweep + HA correctness

955efecd→d0e5f85a — five-bug systematic sweep: WAL_JOB_START losing pidinfo/start_time (bug-1687), WAL_JOB_FINISH losing exit_info/finish_time — and exit_info was never set in production at all (bug-1689), WAL_JOB_RUSAGE losing run_usage (bug-1690), WAL_JOB_CKPT losing ckptinfo (bug-1691), WAL_JOB_PREASON losing pend_reasons (bug-1692) — all on restart replay.
e5e6cd18 — raft_cb_persist must deny the vote and roll back state on a persist failure (bug-1684): ha_persist_term()'s every open/write/fsync/rename failure was silently swallowed; the raft library always granted the vote regardless. Safety-critical.
3eba7c16 — fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693).
b4eb9ec8 — offload checkpoint_task_fn off the delaymgr thread (bug-1694, async-I/O blocking-violation).
ac46bfa6 — restored-but-stale raft term permanently blocked re-election after restart (bug-1695), found via the mandatory live HA chaotic soak for this session's controller audit campaign.
b040dfc0, bc3f76b4, 6400f96d — removed a double-free in the limit plugin's config update (bug-1685), invalidated stale builtin_fcfs picker cursors across calls (bug-1686), mirrored plugin_begin_session/plugin_end_session into the sim-mode driver (bug-1681).
9a230f08 — apiserver FilterRows routing non-bitmask operators through bitmask matching (bug-1680, found while verifying bug-1679).

Wed 08-26 — `xce-test-rc-file` spec

9de4db95→8fc86ef4 (7 commits) — rc-file path resolution with XDG precedence, --norc flag, pluggable rc-line-loading runner wired into xce_test startup, and full e2e coverage for discovery/loading/--norc/failure-handling, confirmed not to disturb existing automation.

Thu 08-27 — `scheduler-sim-eager-reconciliation` spec + a fix batch + `scheduler-snapshot-job-allocation` kickoff

e2bb98c8→446437bf (6 commits) — state-independent dispatch capture, resInUse deferral proof, capture-safety unit coverage, resource-contention/sim-verify/full regression proofs. 6/9 tasks done.
f886c809 — standby dispatch-gate rejections now redirect to a known primary (bug-1700, bug-1701) — closes a gap admin-config-tool's own design doc had flagged as a documented blocker.
ac1e312e, 0ebf3a81, 47532f0a, cfd7db8c — job.start_time never set due to a dispatch/WAL_JOB_START RUN-state collision (bug-1697); repeated WORKERS_PUSH with a NULL resval clobbering good values (bug-1688); first-boot promotion evaluating eligibility before its own WAL recovery ran (bug-1696); xceadmin retrying transient scheduler-admin failures after a redirect (bug-1713, bug-1715).
097e33ae→22353849 — scheduler-snapshot-job-allocation begins: dup_job_alloc deep-copy primitive, schedobj construction for restored jobs.

Fri 08-28 — `scheduler-snapshot-job-allocation` completion + a heap-overflow fix

6fa2d970→5948e079 — allocs populated on snapshot save, restored on load, full cross-instance round-trip proof, backward-compat for allocation-less snapshots, allocator-crash regression test, and the final invariant-audit follow-up closed. Spec docs mark it feature-complete.
00207503 — heap buffer overflow in make_job_account() for multi-host jobs (bug-1722): the function calloc'd room for exactly 1 host account pointer but its own loop wrote up to nhosts-1 — any job spanning more than one host overflowed the allocation.

Sat 08-29 — live-debugging investigation: 4 bugs found, 3 fixed, 1 deeply diagnosed

b1323f8d — scheduler leaked job->accounts/job->allocs on a JOB_UPDATE terminal transition (bug-1730).
42e1eeaa — heap-use-after-free in vtask_enq() (bug-1733): a task was published to the dispatch queue before its own owner field finished initializing; a concurrent worker could dequeue, run, and free it before the producer's own write executed. ASAN-confirmed, crashed 2 of 3 controllers under a live burst-array-job HA workload.
ebe691c3, 67009815 — apply_committed_entry() only ever fired the generic HOOK_DETAIL_JOB_UPDATE for job-state changes; HOOK_DETAIL_JOB_START/JOB_EXIT/JOB_FAIL were permanently dead code, leaving the Go apiserver's already-wired OnJobFinished handler starved of data — a finished (or failed) job's state could stay stuck as RUN in the scheduler's own view forever (bug-1731). Live-verified end to end on both the scheduler and apiserver sides.
83cb660b — xce-build.sh's worker target never built xcecd_virtual, so deploy tooling silently skipped generating the worker-start script in sandboxes with no real systemd manager (bug-1732).
347e996e, 14f95b06 — worker now distinguishes signal-killed job exits from genuine crashes in its logs; scheduler snapshot load re-derives worker/shared resource "used" totals instead of leaving them at zero until the first session.
6193fa7e — test_e2e_live_ha_push.py's standby-redirect test rewritten to assert the new success behavior (was asserting the old, now-fixed rejection behavior).
Not landed as a commit, but the week's deepest investigation: bug-1714 (a scheduler SIGSEGV in sched_get_project, first reported live days earlier) was reproduced twice more this session — once under ASAN (no shadow-memory violation fired), once via a scripted gdb watchpoint that caught the actual corrupting write: freed job_submit_data_t memory being recycled by malloc for an unrelated Cap'n Proto decode buffer on a different thread. The free itself was verified legitimate by static reading — the bug is an unidentified second/stale reader elsewhere. Two related side-bugs found during the chase (an orphaned duplicate job silently handed to the picker, and the dedup-race that made it reachable) were fixed and confirmed independent of bug-1714's own root cause, which remains open.

Bugs opened/fixed this window

69 genuinely new bugs (bug-1625 – bug-1733), 66 fixed across the window's 75 touched entries, 9 still open at window end. Zero ID collisions — the buglog counter stayed clean all week.
Both of last week's open threads are resolved: the 39 duplicate IDs were repaired Monday (755976fc); build-time-versioning/dirty-content-fingerprint's phase-metadata lag was fixed the same day (5c82ad0b).
Highest-severity fixes: bug-1722 (heap buffer overflow in make_job_account() for any multi-host job — memory corruption, not just a crash), bug-1733 (controller UAF crashing 2/3 nodes under live HA load), bug-1684 (raft would grant a vote even when its own persist call failed — HA safety-critical), bug-1695 (a restored-but-stale raft term could permanently block re-election after restart).
Still open at window end (carry-over risk into next week):
- bug-1714 — scheduler UAF crash, mechanism proven, stale-reader source not yet isolated (highest priority — this is the week's one unresolved thread with active investigation momentum).
- bug-1429 — controller_unit_test 300s TIMEOUT, deadlock-suspected, open since Jul 10, no owner.
- bug-1560 — test_recordstart_symlink_escape_replies_error, a known sandbox filesystem-permissions flake, open since Aug 15.
- bug-1630, bug-1632, bug-1723, bug-1726, bug-1727, bug-1728 — six newer, lower-priority findings (a pre-existing ASAN flake in the sim driver, a test-fixture memory leak, a flaky HA e2e test distinct from bug-1485, a REPL resource-display bug, and a false-ERROR log severity on worker join) — none blocking, none investigated further this week.
- Note on bug-1630 specifically: its own fix field is still empty in the buglog, but the underlying problem it describes (the 39 duplicate IDs) is verifiably resolved — 755976fc's renumbering left zero duplicate IDs in the current file (re-verified directly). The entry itself just never got marked fixed; worth a one-line buglog update, not further investigation.

Spec status changes

scheduler-dispatch-commit — went 0→implementation-complete in one day (Sun); spec.json still reads 13/18 tasks despite the phase — metadata lag, not work lag (same pattern as last week's build-time-versioning).
xce-test-rc-file — 7/10 tasks, phase tasks-generated.
scheduler-sim-eager-reconciliation — 6/9 tasks, phase tasks-generated.
scheduler-snapshot-job-allocation — driven to feature-complete per its own closing commit message, but spec.json phase still reads tasks-generated at 11/15 tasks — another phase/reality mismatch to sync.
scheduler-config-version-tag — new spec created (requirements/design/research/tasks all approved), 0/8 tasks — not started.
admin-config-tool — no new work, but its own design doc's last documented blocker (the standby-redirect gap) was closed this week by f886c809.

Stats

164 commits in the window — 97 real (feature/fix/docs/test), 67 .wolf bookkeeping/checkpoint commits.
341 non-.wolf files changed, +26,205 / −1,002 lines (excluding .wolf/; including it the raw figure is +220,722 / −1,049,394, dominated by buglog renumbering and token-ledger churn).
Busiest areas: src/controller (46 files), .kiro (42), src/scheduler (35), .wolf (25), src/frontend (16).
2 specs shipped end-to-end (scheduler-dispatch-commit, xce-test-rc-file), 2 more substantially advanced (scheduler-sim-eager-reconciliation, scheduler-snapshot-job-allocation), 1 new spec created.
18 sessions logged in .wolf/memory.md, single busiest day Aug 23 (383 logged actions).
Commits landed on all 7 days; peak day Tue 08-25 (46 commits — the WAL-replay + HA sweep).

Open threads / to-do heading into next session

Isolate bug-1714's stale-reader. The UAF mechanism is proven (freed job_submit_data_t recycled across threads); the actual second reader that still holds a pointer past the job's legitimate free is not yet found. Two leads already ruled unlikely or weak by this week's own static reading — needs either deeper tracing of the pending-update queue lifecycle or another live session with pointer-tracking instrumentation.
Sync scheduler-dispatch-commit and scheduler-snapshot-job-allocation's spec.json phase metadata — both report a phase behind their actual completion state, the same lag class fixed for two other specs this Monday.
Five lower-priority open bugs (1632, 1723, 1726, 1727, 1728) have no assigned follow-up yet — worth a triage pass to confirm none has quietly become higher-priority. (bug-1630 is housekeeping only — see the note above; just needs its buglog entry marked fixed.)
bug-1429 (controller test-suite deadlock-suspected timeout) has been open since Jul 10 with no owner — the oldest unresolved item in the log.

14 KiB Raw Blame History Unescape Escape