myxce/.wolf/STATUS.md

STATUS

Single source of truth for resuming work. Read this FIRST when starting a session. Update at the end of every work phase so the next /clear resumes in 1 read.

---

✅ Done (2026-10-08, all pushed on scheduler-framework, HEAD f69390a3)

• scheduler-sim-decision-trace: implementation complete, feature validation GO (spec phase implementation-complete). The --sim --interactive REPL gains trace on all|job SEL... [level summary|full] [lines N] [file PATH], trace off, trace status and info trace <job|array>. Each traced decision prints selection, requirement branching, cluster-wide checks, plans/positions/host verdicts and the outcome, inline and to a trace file. Latest record per job is kept (64 MiB bound).
  • 27 tasks: 1.1–6.4, plus 2.4, added mid-run to split the trace sources under the 800-line cap. Each task was independently reviewed.
  • Sources: core side sched_trace.c/_scan.c/_capture.c/_outcome.c + sched_trace_internal.h + include/sched/trace.h (790 lines, split before growing). REPL side repl_trace.c/_emit.c/_store.c/_render.c/_render_outcome.c. Hooks via SCHED_TRACE(core, call). The live driver never traces.
  • Proof: test_trace_equivalence (off/summary/full/subset × step/continue × 2 replays) gives identical decisions, pend reasons, recordings, snapshots, verify output and accounting. Also test_repl_trace_e2e, test_trace_error_paths (7 injected faults), test_trace_bounds, and the trace-off baseline test_repl_trace_baseline.
  • Validation round 1 closed coverage gaps 1.5/7.3/8.1/8.7/10.4/11.1 and synced design.md. Two criteria, 10.3 (job->accounts) and 11.4 (live), are covered only indirectly.
• Bugs fixed and closed on Gitea (#348–#355):
  • bug-1923: sim dropped shared/per-worker resvals from snapshots. Invariant audit (scheduler dims 1/4) clean.
  • bug-1924: clock test failed when the PID contained 1999.
  • bug-1925: quit mid-session freed trace settings while the allocator ran.
  • bug-1926: break ... jobid= matched only raw decimal; now dashed, X[k], X[a-b] and raw all work, and an invalid id is rejected.
  • bug-1927: pend reasons were repeated per host per plan; add_job_preason_once dedups them; the bounds test went from 19 s to 2 s.
  • bug-1928: REPL iterator leaks; standalone ASAN --sim now exits 0 with leaks on.
  • bug-1929: continue rollup jobids= was empty or pend-only; it now lists every affected job (repl-verbosity Req 2.1/3.2). The baseline transcript changed only in 2 jobids= lines.
  • bug-1930: trace on job now accepts raw decimal ids.
• Verification: full all-components ASAN build 148/148 meson tests, xceadmin 373, Go ok (after bug-1927); scheduler suite 118/118.
• Earlier (2026-10-03..07, already pushed): HA fixes bug-1914/1915/1917–1922, bug-1736 run-usage exposure, harness fake-worker fixes, soak WAL-layout gate (Gitea #338–#346).

---

🚀 Next phase

Goal: None active. Pick from the open items below.

Open items (not yet actioned)

• Review + merge PR #347 (nexa/xce#347, rshen:scheduler-framework -> nexa/xce develop): it now also carries the decision-trace feature and bug-1923..1930 (63+ commits since 33437a39). master (fdfc42f1) is 2 commits ahead of develop and was not merged into the branch.
• Checks needing a host with systemd/dbus (this container has none): a running array element prints the right XCE_JOBINDEX; bug-1908's kept signal reaches a real (non-virtual) worker over IPC at RUNNING; real xcecd (cgroup-measured) run usage in xcecli jobs list -l end to end.
• Specs awaiting the next phase: design review -- xce-node-manager; requirements written -- job-status-detail-display, scheduler-preemption; initialized -- commit-batch-window, resource-distribution-mapping; brief -- component-rejoin-resync.
• Roadmap backlog (briefs only): application-profile → job-sandbox-execution → job-sandbox-multinode; job-time-window → job-dependency → job-dependency-insight; rbac-persistence; scheduler-alter-pend-reasons; controller-job-cleanup.
• Small, optional:
  • The fake worker reports finished jobs as unrecognized in its resync answer.
  • --truncated-fence is opt-in.
  • Decision-trace follow-ups:
    • capture's key_find/vget run without the RTTI read lock (safe in sim lock-step);
    • trace.h is at 790/800 lines;
    • the snapshot save silently skips a shared resval with no matching resource (snapshot.c ~446), while the per-worker side warns;
    • most REPL tests still run with detect_leaks=0 because their fixtures don't tear down.

Files to create / edit

• none pending

Open decisions

• REST job output for jobs with a custom out_file/err_file: the worker never writes those paths (always /tmp/job_<id>_task<N>.out), so since bug-1508 GET /jobs/:id/output for such a job returns a permission error from the worker. Either the apiserver's resolveOutputPath() (controllers/output.go) always targets the worker's own output file, or the worker starts honouring out_file. Waiting on the user.
• Bug records not mirrored to Gitea: ~400 older records plus bug-1885–bug-1908 (bug-1736, bug-1914/1915/1917-1922 and bug-1923–1930 are mirrored). A sync without --since/--ids would create an issue for each. Default: leave them off Gitea.

---

📁 Active architecture

• Stack: Go API/CLI tier (src/frontend/apiserver) fronting C daemons (src/controller, src/worker, src/scheduler); Meson-based build via src/tools/xce-build.sh, hermetic nix dev shell.
• Bug tracking: .wolf/bugs/*.md (records) → scripts/buglog.py (CLI + checks) → .wolf/buglog.json (generated index) → scripts/gitea-sync.py (Gitea issues in nexa/xce).
• Local-only, never committed: .claude/settings.json (hook paths rewritten for this machine), data/ (450 MB replays/logs).

---

🔧 Useful commands

python3 scripts/buglog.py list --open | new ... --binary <bin> | check <id> | status <id> fixed --commit <sha>
ln -sf ../../scripts/git-hooks/pre-commit .git/hooks/pre-commit   # once per clone
XCE_HA_ROUNDS=6 nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh --sequential <dir>
nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh --truncated-fence <dir>
python3 .claude/skills/xce-test-ha/wal-layout-check.py <deploy dir>   # WAL segment layout (bug-1920)
python3 scripts/gitea-sync.py --dry-run --bugs --since <date> [--ids bug-N,bug-M]   # dry-run first
bash src/tools/xce-build.sh scheduler --jobs 12 --incremental   # ASAN build dir is src/xce.build
ASAN_OPTIONS=detect_leaks=0 XCE_TRACE_E2E_DUMP=1 src/xce.build/scheduler/testsuite/sched_repl_trace_e2e_test   # dump traced transcripts