From 7d3c305bd619ff86110c6682f4932e2392739984 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Fri, 9 Oct 2026 06:34:28 +0000 Subject: [PATCH] docs(wolf): STATUS.md -- scheduler-sim-decision-trace complete (GO), bug-1923..1930 fixed Co-Authored-By: Claude Opus 5.5 (1M context) --- .wolf/STATUS.md | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-) diff --git a/.wolf/STATUS.md b/.wolf/STATUS.md index 156fa2b6..60eb597b 100644 --- a/.wolf/STATUS.md +++ b/.wolf/STATUS.md @@ -5,14 +5,24 @@ --- -## ✅ Done (2026-10-03..08, all pushed on `scheduler-framework`) +## ✅ Done (2026-10-08, all pushed on `scheduler-framework`, HEAD `f69390a3`) -- **HA fixes** (Gitea #338-#340): bug-1915 (a PreVote round is decided once; a leader promotes only with its own election's votes), bug-1919 (promotion fence released once a RECOVER_TRUNCATED node has caught up), bug-1920 (replicated WAL writes refused before this boot's `persist_recover`; catch-up streams from min(join position, standby report)). -- **bug-1914/1917/1918** (Gitea #341-#343): config-push barrier 2000 ms under the commit-wait admission reservation; `xce_ctx_reconnect` quiesces the old endpoint (follow-up `f26a2d45`: from its own task it skips the wait, and stale-generation tasks are dropped); refused inbound packets are no longer leaked. -- **bug-1736 exposure** (Gitea #344, `1cedd7f7`): capnp `Job.runUsage` carries the controller's run usage to the apiserver and into snapshots (checkpoints used to drop it); REST columns `max_mem`, `avg_mem`, `used_mem`, `cpu_time`, `peak_cpu`, `peak_cpu_time`, `peak_cpu_duration`, `mem_efficiency`, `avg_cpu_efficiency`, `peak_cpu_efficiency`; xcecli `-o` fields `MAX_MEM`, `AVG_MEM`, `MEM_EFFICIENCY`, `CPU_PEAK`, `CPU_PEAK_DURATION`, `CPU_EFFICIENCY`, `CPU_PEAK_EFFICIENCY` and an LSF-style usage block in `jobs list -l`. -- **Harness fake worker** (Gitea #345, #346): bug-1921 (handler notifications deferred to the delay-manager thread; re-join after a reconnect), bug-1922 (answers the worker-rejoin-resync job-state query). -- **Soak** (`5cf92335`): `wal-layout-check.py` gate before PASS (sequential/chaotic/truncated-fence) and the `--truncated-fence` mode (truncated boot -> refused promotion -> fence released -> leads). -- Verification across these: 127/127 suites; soaks sequential 4x20 + 50 + 20, chaotic 50, loaded 30 (40 spinners on 32 cores), truncated-fence -- all PASS; 80/80 live HA config pushes under load; live run-usage check (fake worker -> controller -> apiserver -> xcecli). +- **scheduler-sim-decision-trace: implementation complete, feature validation GO** (spec phase `implementation-complete`). The `--sim --interactive` REPL gains `trace on all|job SEL... [level summary|full] [lines N] [file PATH]`, `trace off`, `trace status` and `info trace `. Each traced decision prints selection, requirement branching, cluster-wide checks, plans/positions/host verdicts and the outcome, inline and to a trace file. Latest record per job is kept (64 MiB bound). + - 27 tasks: 1.1–6.4, plus 2.4, added mid-run to split the trace sources under the 800-line cap. Each task was independently reviewed. + - Sources: core side `sched_trace.c`/`_scan.c`/`_capture.c`/`_outcome.c` + `sched_trace_internal.h` + `include/sched/trace.h` (790 lines, split before growing). REPL side `repl_trace.c`/`_emit.c`/`_store.c`/`_render.c`/`_render_outcome.c`. Hooks via `SCHED_TRACE(core, call)`. The live driver never traces. + - Proof: `test_trace_equivalence` (off/summary/full/subset × step/continue × 2 replays) gives identical decisions, pend reasons, recordings, snapshots, verify output and accounting. Also `test_repl_trace_e2e`, `test_trace_error_paths` (7 injected faults), `test_trace_bounds`, and the trace-off baseline `test_repl_trace_baseline`. + - Validation round 1 closed coverage gaps 1.5/7.3/8.1/8.7/10.4/11.1 and synced design.md. Two criteria, 10.3 (job->accounts) and 11.4 (live), are covered only indirectly. +- **Bugs fixed and closed on Gitea (#348–#355):** + - bug-1923: sim dropped shared/per-worker resvals from snapshots. Invariant audit (scheduler dims 1/4) clean. + - bug-1924: clock test failed when the PID contained 1999. + - bug-1925: `quit` mid-session freed trace settings while the allocator ran. + - bug-1926: `break ... jobid=` matched only raw decimal; now dashed, `X[k]`, `X[a-b]` and raw all work, and an invalid id is rejected. + - bug-1927: pend reasons were repeated per host per plan; `add_job_preason_once` dedups them; the bounds test went from 19 s to 2 s. + - bug-1928: REPL iterator leaks; standalone ASAN `--sim` now exits 0 with leaks on. + - bug-1929: continue rollup `jobids=` was empty or pend-only; it now lists every affected job (repl-verbosity Req 2.1/3.2). The baseline transcript changed only in 2 `jobids=` lines. + - bug-1930: `trace on job` now accepts raw decimal ids. +- Verification: full all-components ASAN build 148/148 meson tests, xceadmin 373, Go ok (after bug-1927); scheduler suite 118/118. +- Earlier (2026-10-03..07, already pushed): HA fixes bug-1914/1915/1917–1922, bug-1736 run-usage exposure, harness fake-worker fixes, soak WAL-layout gate (Gitea #338–#346). --- @@ -21,18 +31,25 @@ **Goal:** _None active._ Pick from the open items below. ### Open items (not yet actioned) -- **Review + merge PR #347** (https://gitea.shenrs.eu/nexa/xce/pulls/347, `rshen:scheduler-framework` -> `nexa/xce` `develop`, the repo's first PR): 2,140 commits, fast-forward onto `develop`; most of the +2 M lines are vendored Go modules. `master` (`fdfc42f1`) is 2 commits ahead of `develop` and was not merged into the branch. +- **Review + merge PR #347** (https://gitea.shenrs.eu/nexa/xce/pulls/347, `rshen:scheduler-framework` -> `nexa/xce` `develop`): it now also carries the decision-trace feature and bug-1923..1930 (63+ commits since `33437a39`). `master` (`fdfc42f1`) is 2 commits ahead of `develop` and was not merged into the branch. - **Checks needing a host with systemd/dbus** (this container has none): a running array element prints the right `XCE_JOBINDEX`; bug-1908's kept signal reaches a real (non-virtual) worker over IPC at RUNNING; real `xcecd` (cgroup-measured) run usage in `xcecli jobs list -l` end to end. -- **Specs awaiting the next phase:** design review -- `xce-node-manager`; requirements written -- `job-status-detail-display`, `scheduler-preemption`, `scheduler-sim-decision-trace`; initialized -- `commit-batch-window`, `resource-distribution-mapping`; brief -- `component-rejoin-resync`. +- **Specs awaiting the next phase:** design review -- `xce-node-manager`; requirements written -- `job-status-detail-display`, `scheduler-preemption`; initialized -- `commit-batch-window`, `resource-distribution-mapping`; brief -- `component-rejoin-resync`. - **Roadmap backlog (briefs only):** application-profile → job-sandbox-execution → job-sandbox-multinode; job-time-window → job-dependency → job-dependency-insight; rbac-persistence; scheduler-alter-pend-reasons; controller-job-cleanup. -- Small, optional: the fake worker reports finished jobs as `unrecognized` in its resync answer (no finished-job retention); `--truncated-fence` is opt-in (could join a default soak, ~3 min). +- **Small, optional:** + - The fake worker reports finished jobs as `unrecognized` in its resync answer. + - `--truncated-fence` is opt-in. + - Decision-trace follow-ups: + - capture's `key_find`/`vget` run without the RTTI read lock (safe in sim lock-step); + - `trace.h` is at 790/800 lines; + - the snapshot save silently skips a shared resval with no matching resource (snapshot.c ~446), while the per-worker side warns; + - most REPL tests still run with `detect_leaks=0` because their fixtures don't tear down. ### Files to create / edit - _none pending_ ### Open decisions - **REST job output for jobs with a custom `out_file`/`err_file`**: the worker never writes those paths (always `/tmp/job__task.out`), so since bug-1508 `GET /jobs/:id/output` for such a job returns a permission error from the worker. Either the apiserver's `resolveOutputPath()` (controllers/output.go) always targets the worker's own output file, or the worker starts honouring `out_file`. Waiting on the user. -- **Bug records not mirrored to Gitea:** ~400 older records plus bug-1885–bug-1908 (bug-1736 and bug-1914/1915/1917-1922 are mirrored). A sync without `--since`/`--ids` would create an issue for each. Default: leave them off Gitea. +- **Bug records not mirrored to Gitea:** ~400 older records plus bug-1885–bug-1908 (bug-1736, bug-1914/1915/1917-1922 and bug-1923–1930 are mirrored). A sync without `--since`/`--ids` would create an issue for each. Default: leave them off Gitea. --- @@ -53,4 +70,6 @@ XCE_HA_ROUNDS=6 nix develop ./src --command bash .claude/skills/xce-test-ha/run- nix develop ./src --command bash .claude/skills/xce-test-ha/run-ha-soak.sh --truncated-fence python3 .claude/skills/xce-test-ha/wal-layout-check.py # WAL segment layout (bug-1920) python3 scripts/gitea-sync.py --dry-run --bugs --since [--ids bug-N,bug-M] # dry-run first +bash src/tools/xce-build.sh scheduler --jobs 12 --incremental # ASAN build dir is src/xce.build +ASAN_OPTIONS=detect_leaks=0 XCE_TRACE_E2E_DUMP=1 src/xce.build/scheduler/testsuite/sched_repl_trace_e2e_test # dump traced transcripts ```