myxce/reports/aug08-aug15-2026.md

XCE Project Status — Aug 08 – Aug 15, 2026

TL;DR

component-failover-adoption: implementation-complete and GO-validated — but not yet committed. The week opened with scheduler-core documentation cleanup (carried over from last week), then a worker CLI/config cleanup pass, then the bulk of the week (Aug 11–13) was the 21-task component-failover-adoption spec: unifying worker/scheduler/apiserver's dual connect paths into one candidate-list pipeline, adding rejoin-on-failover for all three components, and an apiserver backend-health circuit breaker. Live E2E validation during that work found two real production bugs (bug-1548/bug-1549: apiserver never rejoined a new primary during an idle/heartbeat-only failover, so job pushes silently stopped reaching it). /kiro-validate-impl caught this as an in-scope gap and returned NO-GO on Aug 15 at 13:10 UTC; a same-day Phase 6 remediation (a monotonic reconnect_gen counter) fixed it, live-reverified, and the decision flipped to GO at 14:20 UTC. That fix, plus a new xceadmin cluster-config dump feature and a batch of build-tooling fixes, are sitting validated-but-uncommitted in the working tree as of end of week — 57 modified/untracked files, no commits since Aug 13.

Commits by day

Aug 08 (Sat) — scheduler-core documentation cleanup (carried over from prior week)

5 commits, docs-only:

• docs: add scheduler recording/simulation/REPL to deployment guide
• docs(scheduler-core): sync CLAUDE.md for core/driver/picker/sim/REPL architecture
• docs(scheduler-core): commit accumulated brief/requirements/roadmap edits
• docs(scheduler-core): backfill _Verified notes for tasks 2.1–2.6
• docs(scheduler-core): commit design.md/research.md, sync design.md with as-built code

Aug 09 (Sun) — no commits

Aug 10 (Mon) — worker CLI/config cleanup + probeConfigTag fix

11 commits:

• Worker cleanup: removed the config-file mechanism and deprecated CLI flags entirely (5e4baa27, f0f46f87, b3124767), fixed xcecd -h exiting before root/config checks (9ffb1d51, bug-1509 class), default logdir to /tmp when -l omitted (e959ee61), added -v verbose flag (0df1cf5f)
• fix(xceadmin)/feat(controller): OPCode_probeConfigTag handler with Request-envelope reply, and probe_config updated to match (90d7dd82, b00e97ff — bug-1506, bug-1507)
• feat(scheduler): working default record-dir, improved REPL output, job ID formatting (c705cada)

Aug 11 (Tue) — component-failover-adoption: foundation + worker + scheduler connect unification

11 commits — Task groups 1–3 begin:

• Group 1 (Foundation): reconciled N_MAX_MASTERS/MAX_HA_CANDIDATES (fe8b4b86), added the candidate-list→client-config conversion helper (210faa2b)
• Group 2 (Worker): unified worker's dual connect paths into the single pipeline (ce9f734a), fixed startup ordering + added rejoin retry-on-failure (331fa024), validated thread isolation and fixed rejoin bugs found by live E2E (619b119e)
• Group 3 (Scheduler): unified scheduler's dual connect paths (a9f283dc), added decision-publish retry-until-success-or-shutdown (7c260e0c), validated backpressure + fixed candidate-name diagnostics (b7df8d60), added decision-redelivery queue coverage (d383d043)

Aug 12 (Wed) — component-failover-adoption: apiserver connect unification + rejoin + circuit breaker

7 commits — Task group 4:

• Routed apiserver's candidate parsing through the real C parser (8b1e13a5)
• Unified apiserver's dual connect paths — found and fixed a genuine use-after-free on ctx->conf->masters[i] during failover rotation (e6359039, bug-1532)
• Added apiserver's rejoin logic with correctly-placed cgo registration (2a205291)
• Wired rejoin to trigger from the idle-period heartbeat path (ecf5eddc)
• Added the backend-health circuit breaker (c8325e75)
• Closed an apiserver concurrency coverage gap (79273a7f)
• Group 5 begins: full-cluster E2E reconnect-after-failover test passes (6cb8f22b) — this run first surfaced bug-1548/bug-1549 (apiserver silently fails to rejoin on an idle/heartbeat-only failover)

Aug 13 (Thu) — component-failover-adoption closes; resource/config bugfixes; buglog integrity hardening

15 commits (6 feature/fix, 9 wolf-buglog checkpoints):

• Task 5.2 doubly-confirms bug-1548 in an isolated heartbeat-only scenario (04d41d29); task 5.3 validates the circuit breaker live (51589b68); task 5.4 closes the spec on a full regression pass (d170623b) — bug-1548/1549 were (incorrectly, see below) treated as out-of-scope follow-up items at this point
• chore(wolf): buglog.json integrity guard, dedupe rule, auto-checkpoint hook (ba4e5fd2)
• fix(controller,xceadmin): zero-slot scheduling block, misleading resval logs, byte-unit display (e3b3ffaa — bug-1555, bug-1556)
• refactor(controller,xceadmin): dropped generic Resource.unit, hardcoded bytes for maxmem (5bf123cc)

Aug 14–15 (Fri–Sat) — validation reopened, Phase 6 remediation, tooling fixes — all uncommitted

No commits landed, but substantial validated work sits in the working tree:

• /kiro-validate-impl reopened component-failover-adoption: NO-GO (Aug 15, 13:10 UTC) — bug-1548 lives inside task 4.4's own ApiserverBackendBridge boundary against an already-existing acceptance criterion, so it doesn't qualify as legitimately out-of-scope
• Phase 6 remediation (tasks 6.1/6.2): added a monotonic reconnect_gen counter to endpoint_t (src/include/netio.h, src/lib/xcectx.c), bumped on every successful transport reconnect; heartbeat.go's runHeartbeat() replaced its failure-to-success transition detector with a per-tick generation comparison. New regression tests in heartbeat_test.go/heartbeat_rejoin_test.go. Live E2E re-run of the exact task-5.2 scenario passed: rejoin fired 22s after a primary kill (12s after election) with zero write traffic in the preceding 5.5 minutes; a post-recovery job submit correctly appeared via GET /xce/v1/jobs
• Decision flipped to GO (14:20 UTC)
• bug-1558 fixed: make test-backend now self-derives CGO_CFLAGS/CGO_LDFLAGS via pkg-config
• bug-1559 fixed: xce-build.sh's scheduler case now runs meson test --suite scheduler (all 41 registered targets, not just one binary by path)
• New, separate uncommitted feature: xceadmin cluster-config dump/pretty-printer (build.py: dump_cluster_config + formatters, ~236 new lines) and a parse_size_or_number helper in mapping.py for size-string threshold parsing, with new tests (test_cli.py, test_mapping.py, untracked test_config_dump.py)

Bugs opened/fixed this window

Named/significant bugs (55 total opened this window, bug-1506–bug-1560; most of the unlisted ones are terse review-catches — "significant refactor of X", "incorrect value in code" — logged during component-failover-adoption's two-round review process):

Bug	Status	Summary
bug-1506, bug-1507	Fixed (90d7dd82, b00e97ff)	probeConfigTag missing Request-envelope handling; SIGABRT in REPL breakpoint inspection
bug-1508	Open	handle_fetchfile() — possible job-ownership/path-traversal gap found during doc audit; flagged, needs a security-reviewer pass
bug-1509	Fixed	extralib flake dependency removed; ported trivial linked-list code local to autoscaler
bug-1514, bug-1515	Fixed	design.md/requirements.md defects in component-failover-adoption's scheduler-backpressure description, found by /kiro-validate-design and a deep re-review
bug-1519	Open	AddressSanitizer "joining already joined thread" abort under unshare -r; worked around, not root-caused
bug-1532	Fixed (e6359039)	apiserver tryConnect() use-after-free on ctx->conf->masters[i] during failover rotation
bug-1542	Fixed	Illegal import "C" in a _test.go file; exposed the needed constant via a plain-Go package var instead
bug-1548	Fixed (uncommitted, Phase 6)	apiserver never rejoined a new primary on an idle/heartbeat-only failover — root cause: XCECLI_FF_CALL's silent inline reconnect was invisible to the old failure→success transition detector
bug-1549	Fixed (uncommitted, Phase 6)	Downstream symptom of bug-1548 — job pushes never reached apiserver's local store after failover
bug-1553	Open	controller_unit_test segfaults intermittently (~1 in 3 runs); crash site not yet isolated
bug-1554	Open, deferred by user	maxswap/maxtmp report hardcoded stub values instead of real hardware detection
bug-1555, bug-1556	Fixed (e3b3ffaa)	Misleading update_resval error logs; zero-slot scheduling block on the documented hosts.yaml auto-detect default
bug-1557	Open	xce_test worker join defaults maxmem=64 (bytes, not MB)
bug-1558, bug-1559	Fixed (uncommitted)	make test-backend CGO flags now self-derived via pkg-config; xce-build.sh scheduler suite now runs all 41 test targets
bug-1560	Open, low priority	test_recordstart_symlink_escape_replies_error fails under sandbox root — confirmed pre-existing, environment-specific, not a regression

Spec status changes

• component-failover-adoption — tasks 1.1–5.4 (21 tasks) implemented and initially marked complete (Aug 13). Reopened same week by its own /kiro-validate-impl run: NO-GO (Aug 15, 13:10 UTC) → Phase 6 remediation (2 tasks) → GO (Aug 15, 14:20 UTC). Implementation and validation are done; the fix itself is not yet committed to git.
• scheduler-core — carried into this window only for Aug 08's documentation cleanup; no further status change.
• Three new specs created this week, all still at brief/pre-requirements stage, not yet started: component-rejoin-resync and worker-rejoin-resync (both born from this week's bug-1548/1549 investigation — full-state resend cost on rejoin), and scheduler-sim-diagnostics (limit/pend-reason visibility in the REPL; not yet even spec.json-initialized).

Stats

• 49 commits in the window (39 feature/fix/docs, 10 wolf-infra/checkpoint)
• ~124 files changed across the week's commits (109,317 insertions / 18,170 deletions, dominated by component-failover-adoption)
• 55 bugs opened (bug-1506–bug-1560), 48 fixed, 7 still open (bug-1508, 1519, 1553, 1554, 1557, 1560, plus bug-1548/1549 fixed-but-uncommitted)
• 1 spec closed and reopened same week, remediated, now GO
• 3 new specs opened at brief stage
• 0 commits on Aug 14–15 despite validated GO work landing — 57 files (modified + untracked) sitting uncommitted at week's end

Open threads / to-do heading into next session

1. Commit the Phase 6 failover fix — Done (2026-08-15, d479b70). Verified via a full lib asan rebuild (all libxce tests incl. the new test_netio_reconnect_gen pass) and make test-backend (apiserver backend + auth suites pass) before committing. Scoped to netio.h, xcectx.c, lib/meson.build, lib/testsuite/test.c, lib/testsuite/test_netio_reconnect_gen.c, heartbeat.go, clib.go, heartbeat_test.go, heartbeat_rejoin_test.go, spec.json, tasks.md. Note: design.md/requirements.md/research.md were deliberately left out — they've been stale since the spec's very first commit (4ad31d77, never updated across all 21+2 tasks), so syncing them is a separate, broader doc-catch-up task, not part of the Phase 6 diff.
2. Commit or explicitly shelve the xceadmin config-dump feature — Done (2026-08-15, 3afe3d5). Verified via xce-build.sh xceadmin's full pinned out-of-tree install + pytest run (256 passed, 3 skipped) before committing. Scoped to build.py, cli.py, errors.py, mapping.py, hosts.schema.json, queues.schema.json (the SizeOrNumber schema type is tightly coupled to mapping.py's new parse_size_or_number, so it had to travel with it), plus the three test files. New discovery while scoping this: resources.yaml's diff also contains unrelated changes — new resource catalog entries (r15m, ut, pg, ls, it, swp, io) that the worker (resource_sync.c) already registers but the shipped default resources.yaml never declared, plus mem/tmp byte-unit description fixes matching node_information.c's uncommitted change (worker-reported mem/tmp now actually returns bytes, not MB) and xcecd.c's maxswap/maxtmp stub-value unit fix (bug-1554's stub, now byte-scaled). These three files (resources.yaml, worker/src/node_information.c, worker/src/xcecd.c) were deliberately left out of both commits so far — added as new item 10 below.
3. .wolf/dashboard-token is untracked, holds a live-looking token, and is not covered by .gitignore — add a gitignore rule before it lands in history via a broad git add.
4. Triage the remaining uncommitted files not obviously tied to items 1, 2, or 10 (deploy-ha.sh, xce-build.sh, controller/testsuite/test.c — the xce_test detailed-help-text feature, unrelated to bug-1557's one-line default fix it happens to share a file with).
5. bug-1553 (intermittent controller test segfault) is the most concerning open item — a real memory-safety bug in the shared test binary, unisolated. Needs a bisect/gdb session.
6. bug-1508 (possible handle_fetchfile() ownership/path-traversal gap) — flagged during a doc audit, never given a security-reviewer pass.
7. bug-1557 — one-line fix (maxmem default should be 64 * 1024 * 1024, not 64), still open.
8. bug-1554 — swap/tmp-disk detection deferred by explicit user request; needs a design discussion when picked back up.
9. Decide which of the three new brief-stage specs (component-rejoin-resync, worker-rejoin-resync, scheduler-sim-diagnostics) to move into /kiro-spec-requirements next.
10. New (found while scoping item 2): commit the worker-side byte-unit completion + resource-catalog sync (resources.yaml, src/worker/src/node_information.c, src/worker/src/xcecd.c) — finishes what 5bf123cc/e3b3ffaa started (mem/tmp now genuinely report bytes, not just documented as bytes) and adds the 7 missing built-in resource entries (r15m, ut, pg, ls, it, swp, io) the worker already reports but the catalog never declared. Worth its own scoped commit with a build+test pass first, same as items 1 and 2.