From 1f932bf22d75bf56838cee99412c2bdd1e0d1582 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Wed, 30 Sep 2026 10:20:39 +0000 Subject: [PATCH] chore(scheduler-allocator-runner): implementation complete; log bug-1897 Feature validation GO (ASAN scheduler suite 96/96, smoke replay clean on the allocation path). Roadmap entry marked implemented with follow-ups. bug-1897 (found by the validation smoke run, not caused by this feature): xce_scheduler --sim segfaults in limit_job_election on a job whose snapshot resreq failed to parse (pre-bug-1888-fix replays). Co-Authored-By: Claude Opus 5.5 (1M context) --- .../scheduler-allocator-runner/spec.json | 4 +- .kiro/steering/roadmap.md | 2 +- .wolf/buglog.json | 27 ++++++++ .wolf/bugs/bug-1897.md | 62 +++++++++++++++++++ 4 files changed, 92 insertions(+), 3 deletions(-) create mode 100644 .wolf/bugs/bug-1897.md diff --git a/.kiro/specs/scheduler-allocator-runner/spec.json b/.kiro/specs/scheduler-allocator-runner/spec.json index 20f1093a..802c2158 100644 --- a/.kiro/specs/scheduler-allocator-runner/spec.json +++ b/.kiro/specs/scheduler-allocator-runner/spec.json @@ -1,9 +1,9 @@ { "feature_name": "scheduler-allocator-runner", "created_at": "2026-09-29T03:45:23Z", - "updated_at": "2026-09-30T07:34:49Z", + "updated_at": "2026-09-30T10:20:39Z", "language": "en", - "phase": "tasks-generated", + "phase": "implementation-complete", "approvals": { "requirements": { "generated": true, diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 42f4bc93..aff8f3e2 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -59,7 +59,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [x] ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks) - [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks) - [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.) -- [ ] scheduler-allocator-runner — NEW 2026-09-28 (`/kiro-discovery`, Path C). The framework owns `job_alloc_fn` through a new `xce_sched_allocator_iface_t` (accepts guard + plan iterator) and a single `sched_allocator_run()`; both `my_alloc()`s are deleted (seqjob/parjob), and `alloc_iter.h` `init` is retyped to `xce_sched_core_t*`. Behaviour must stay identical (bug-1882/1767/1768). Closes `issues/new_requirements.md` item 4. Brief: `.kiro/specs/scheduler-allocator-runner/brief.md`. Dependencies: scheduler-allocation, scheduler-core (both done). +- [x] scheduler-allocator-runner — NEW 2026-09-28 (`/kiro-discovery`, Path C). The framework owns `job_alloc_fn` through a new `xce_sched_allocator_iface_t` (accepts guard + plan iterator) and a single `sched_allocator_run()`; both `my_alloc()`s are deleted (seqjob/parjob), and `alloc_iter.h` `init` is retyped to `xce_sched_core_t*`. Behaviour must stay identical (bug-1882/1767/1768). Closes `issues/new_requirements.md` item 4. Brief: `.kiro/specs/scheduler-allocator-runner/brief.md`. Dependencies: scheduler-allocation, scheduler-core (both done). **IMPLEMENTED 2026-09-30** (11/11 tasks, e2266582..28cff179; validate-impl GO; scheduler ASAN suite 96/96): runner in `src/scheduler/src/alloc_runner.c`; decided along the way: the runner clears `alloc_req.host` before every candidate so every short host gets pend reasons (both allocators), and fixes bug-1893 (shared reasons were never recorded). Follow-ups: publish pend reasons only when changed; sim-mode parjob allocation parity fixture; bug-1892; bug-1897. - [ ] scheduler-alter-pend-reasons — NEW 2026-09-28 (`/kiro-discovery`, Path E). An ALTER-resreq job keeps a pend reason for EVERY alternative instead of only the last (`scheduler.c:424` FIXME). Wire shape (user decision): additive `alternatives @3: List(JobPendReason)` on `JobPendReason`, per-alternative attribution; carried end-to-end through controller job record, WAL preason fact, recordings and `job_preason_eq`. Also fixes the confirmed `obj->reason` overwrite leak (`scheduler.c:388`). Brief: `.kiro/specs/scheduler-alter-pend-reasons/brief.md`. Dependencies: libxce-alloc-preason-equality (HARD: extends its `job_preason_eq`); sequence after scheduler-allocator-runner (its "no pend-reason content change" rule). - [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; only profile-marked overridable attributes may be overridden; access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller resolves at submit and persists effective values on the job (wire additive, WAL/snapshot field-completeness). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. Worker capability advertisement + scheduler placement. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none. - [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation). diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 84a7939b..94a59553 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -16658,6 +16658,33 @@ ], "fix_commits": [], "record": ".wolf/bugs/bug-1896.md" + }, + { + "id": "bug-1897", + "status": "open", + "status_updated": "2026-09-30", + "status_note": "newly logged", + "component": "scheduler", + "severity": "crash", + "timestamp": "2026-09-30T10:19:59Z", + "last_seen": "2026-09-30T10:19:59Z", + "occurrences": 1, + "error_message": "xce_scheduler --sim segfaults in limit_job_election on a job whose snapshot resreq failed to parse\n\n**Observed:** replaying `data/scheduler-replay-20260928-064842.replay` in simulation mode crashes before any decision, 3/3 runs:\n```\n[ERROR] [xce_sched_core_load_snapshot] xce_sched_core_load_snapshot parse_resreq failed for jobid 4667606771186008064: Primary: ...\n==2723571==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x555555a7cb6e ... T3)\n #0 limit_job_election ../scheduler/src/plugins/limit.c:336\n #1 plugin_job_election ../scheduler/src/plugin.c:601\n #2 sched_allocator ../scheduler/src/sched_election.c:236\n```\nThe replay's snapshot stores resreqs in the debug-dump format that bug-1888 wrote before its fix. Since the bug-1887 fix, `parse_resreq` rejects that text, so the snapshot loader logs the error but keeps the job with no schedobj. The allocator thread's job election then reads `job->schedobj` as NULL in the limit plugin.\n\n**Expected:** a job whose resreq cannot be parsed from a snapshot is either dropped (with the error logged) or kept pending with a reason, and never reaches election with a NULL schedobj; the simulator must not crash on an old replay.\n\n**Impact:** crash (SIGSEGV) of `xce_scheduler --sim` on any replay whose snapshot has unparsable resreqs, including every replay captured before the bug-1888 fix. Suspected: the same path could crash a normal-mode scheduler loading such a snapshot (not tried).\n\n**Found by:** feature-level validation smoke run of spec `scheduler-allocator-runner` (2026-09-30); not caused by that feature (limit.c, snapshot_load.c, sched_election.c untouched by it).", + "file": "src/scheduler/src/plugins/limit.c, src/scheduler/src/snapshot_load.c", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "scheduler", + "sim", + "snapshot", + "limit" + ], + "related_bugs": [ + "bug-1887", + "bug-1888" + ], + "fix_commits": [], + "record": ".wolf/bugs/bug-1897.md" } ] } diff --git a/.wolf/bugs/bug-1897.md b/.wolf/bugs/bug-1897.md new file mode 100644 index 00000000..e930909a --- /dev/null +++ b/.wolf/bugs/bug-1897.md @@ -0,0 +1,62 @@ +--- +id: "bug-1897" +status: "open" +status_updated: "2026-09-30" +status_note: "newly logged" +component: "scheduler" +severity: "crash" +timestamp: "2026-09-30T10:19:59Z" +last_seen: "2026-09-30T10:19:59Z" +occurrences: 1 +files: ["src/scheduler/src/plugins/limit.c", "src/scheduler/src/snapshot_load.c"] +tags: ["scheduler", "sim", "snapshot", "limit"] +related: ["bug-1887", "bug-1888"] +--- + +# bug-1897: xce_scheduler --sim segfaults in limit_job_election on a job whose snapshot resreq failed to parse + +## Description +**Observed:** replaying `data/scheduler-replay-20260928-064842.replay` in simulation mode crashes before any decision, 3/3 runs: +``` +[ERROR] [xce_sched_core_load_snapshot] xce_sched_core_load_snapshot parse_resreq failed for jobid 4667606771186008064: Primary: ... +==2723571==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x555555a7cb6e ... T3) + #0 limit_job_election ../scheduler/src/plugins/limit.c:336 + #1 plugin_job_election ../scheduler/src/plugin.c:601 + #2 sched_allocator ../scheduler/src/sched_election.c:236 +``` +The replay's snapshot stores resreqs in the debug-dump format that bug-1888 wrote before its fix. Since the bug-1887 fix, `parse_resreq` rejects that text, so the snapshot loader logs the error but keeps the job with no schedobj. The allocator thread's job election then reads `job->schedobj` as NULL in the limit plugin. + +**Expected:** a job whose resreq cannot be parsed from a snapshot is either dropped (with the error logged) or kept pending with a reason, and never reaches election with a NULL schedobj; the simulator must not crash on an old replay. + +**Impact:** crash (SIGSEGV) of `xce_scheduler --sim` on any replay whose snapshot has unparsable resreqs, including every replay captured before the bug-1888 fix. Suspected: the same path could crash a normal-mode scheduler loading such a snapshot (not tried). + +**Found by:** feature-level validation smoke run of spec `scheduler-allocator-runner` (2026-09-30); not caused by that feature (limit.c, snapshot_load.c, sched_election.c untouched by it). + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (src/build.alloc, dedicated scheduler-allocator-runner build dir); ASAN_OPTIONS=detect_leaks=0 +- Code: scheduler-framework @ 28cff179-dirty +- Deployment: none (simulation mode, `xce_scheduler --sim`, no controller or workers) + +## Reproduction +1. `git checkout 28cff179` (or later); `bash src/tools/xce-build.sh scheduler asan build.alloc --jobs 8` +2. `cd src && nix develop . --command bash -c 'ASAN_OPTIONS=detect_leaks=0 ./build.alloc/scheduler/xce_scheduler --sim=/home/claude/tmp/bug-1897/scheduler-replay-20260928-064842.replay -l /tmp/sim.log'` +3. Observe `parse_resreq failed` errors in the log, then the ASan SEGV in `limit_job_election`. +- Frequency: deterministic (3/3 runs, 2026-09-30) +- Evidence: `/home/claude/tmp/bug-1897/` (replay file copy, `run.out` with the ASan report, `sim.log`) + +## Root cause +TODO: required for fixed/partial + +## Fix +TODO: required for fixed/partial + +## Verification +- Test: TODO: name + file; RED before the fix, GREEN after +- Suites: TODO: suite results and build profile +- Live: TODO: soak/deploy run, or n/a and why + +## References +- Related: none