chore(scheduler-allocator-runner): implementation complete; log bug-1897
Feature validation GO (ASAN scheduler suite 96/96, smoke replay clean on the allocation path). Roadmap entry marked implemented with follow-ups. bug-1897 (found by the validation smoke run, not caused by this feature): xce_scheduler --sim segfaults in limit_job_election on a job whose snapshot resreq failed to parse (pre-bug-1888-fix replays). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
28cff179b2
commit
1f932bf22d
GPG key ID:
80533536B0EB651C
4 changed files with 92 additions and 3 deletions
|
|
@ -1,9 +1,9 @@
|
|||
{
|
||||
"feature_name": "scheduler-allocator-runner",
|
||||
"created_at": "2026-09-29T03:45:23Z",
|
||||
"updated_at": "2026-09-30T07:34:49Z",
|
||||
"updated_at": "2026-09-30T10:20:39Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"phase": "implementation-complete",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
|
|
|||
|
|
@ -59,7 +59,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
- [x] ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
|
||||
- [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
|
||||
- [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.)
|
||||
- [ ] scheduler-allocator-runner — NEW 2026-09-28 (`/kiro-discovery`, Path C). The framework owns `job_alloc_fn` through a new `xce_sched_allocator_iface_t` (accepts guard + plan iterator) and a single `sched_allocator_run()`; both `my_alloc()`s are deleted (seqjob/parjob), and `alloc_iter.h` `init` is retyped to `xce_sched_core_t*`. Behaviour must stay identical (bug-1882/1767/1768). Closes `issues/new_requirements.md` item 4. Brief: `.kiro/specs/scheduler-allocator-runner/brief.md`. Dependencies: scheduler-allocation, scheduler-core (both done).
|
||||
- [x] scheduler-allocator-runner — NEW 2026-09-28 (`/kiro-discovery`, Path C). The framework owns `job_alloc_fn` through a new `xce_sched_allocator_iface_t` (accepts guard + plan iterator) and a single `sched_allocator_run()`; both `my_alloc()`s are deleted (seqjob/parjob), and `alloc_iter.h` `init` is retyped to `xce_sched_core_t*`. Behaviour must stay identical (bug-1882/1767/1768). Closes `issues/new_requirements.md` item 4. Brief: `.kiro/specs/scheduler-allocator-runner/brief.md`. Dependencies: scheduler-allocation, scheduler-core (both done). **IMPLEMENTED 2026-09-30** (11/11 tasks, e2266582..28cff179; validate-impl GO; scheduler ASAN suite 96/96): runner in `src/scheduler/src/alloc_runner.c`; decided along the way: the runner clears `alloc_req.host` before every candidate so every short host gets pend reasons (both allocators), and fixes bug-1893 (shared reasons were never recorded). Follow-ups: publish pend reasons only when changed; sim-mode parjob allocation parity fixture; bug-1892; bug-1897.
|
||||
- [ ] scheduler-alter-pend-reasons — NEW 2026-09-28 (`/kiro-discovery`, Path E). An ALTER-resreq job keeps a pend reason for EVERY alternative instead of only the last (`scheduler.c:424` FIXME). Wire shape (user decision): additive `alternatives @3: List(JobPendReason)` on `JobPendReason`, per-alternative attribution; carried end-to-end through controller job record, WAL preason fact, recordings and `job_preason_eq`. Also fixes the confirmed `obj->reason` overwrite leak (`scheduler.c:388`). Brief: `.kiro/specs/scheduler-alter-pend-reasons/brief.md`. Dependencies: libxce-alloc-preason-equality (HARD: extends its `job_preason_eq`); sequence after scheduler-allocator-runner (its "no pend-reason content change" rule).
|
||||
- [ ] application-profile — NEW 2026-09-29 (`/kiro-discovery`, Job Sandbox Execution initiative, Path D; reshaped from `job-execution-mode` by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF `-app` precedent) in cluster config: execution mode + options (`direct`|`sandbox`, room for `container`/`vm`), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (`--app`), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; only profile-marked overridable attributes may be overridden; access by RBAC permission (new `app:use`/`app:view` with `app` domain type, folded into this spec); apiserver checks profile existence; controller resolves at submit and persists effective values on the job (wire additive, WAL/snapshot field-completeness). apiserver never checks node accounts; worker fails a `direct` job with a clear reason if the exec user is missing. Worker capability advertisement + scheduler placement. Brief: `.kiro/specs/application-profile/brief.md`. Dependencies: none.
|
||||
- [ ] job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (`direct` refactored + `systemd-sandbox`; non-systemd backends later). Sandbox = existing transient unit + `DynamicUser` with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), `StandardOutput=file:` output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. Viability-checked 2026-09-29. Brief: `.kiro/specs/job-sandbox-execution/brief.md`. Dependencies: application-profile; remote-task-root fix (Direct-Implementation).
|
||||
|
|
|
|||
|
|
@ -16658,6 +16658,33 @@
|
|||
],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1896.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1897",
|
||||
"status": "open",
|
||||
"status_updated": "2026-09-30",
|
||||
"status_note": "newly logged",
|
||||
"component": "scheduler",
|
||||
"severity": "crash",
|
||||
"timestamp": "2026-09-30T10:19:59Z",
|
||||
"last_seen": "2026-09-30T10:19:59Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xce_scheduler --sim segfaults in limit_job_election on a job whose snapshot resreq failed to parse\n\n**Observed:** replaying `data/scheduler-replay-20260928-064842.replay` in simulation mode crashes before any decision, 3/3 runs:\n```\n[ERROR] [xce_sched_core_load_snapshot] xce_sched_core_load_snapshot parse_resreq failed for jobid 4667606771186008064: Primary: ...\n==2723571==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x555555a7cb6e ... T3)\n #0 limit_job_election ../scheduler/src/plugins/limit.c:336\n #1 plugin_job_election ../scheduler/src/plugin.c:601\n #2 sched_allocator ../scheduler/src/sched_election.c:236\n```\nThe replay's snapshot stores resreqs in the debug-dump format that bug-1888 wrote before its fix. Since the bug-1887 fix, `parse_resreq` rejects that text, so the snapshot loader logs the error but keeps the job with no schedobj. The allocator thread's job election then reads `job->schedobj` as NULL in the limit plugin.\n\n**Expected:** a job whose resreq cannot be parsed from a snapshot is either dropped (with the error logged) or kept pending with a reason, and never reaches election with a NULL schedobj; the simulator must not crash on an old replay.\n\n**Impact:** crash (SIGSEGV) of `xce_scheduler --sim` on any replay whose snapshot has unparsable resreqs, including every replay captured before the bug-1888 fix. Suspected: the same path could crash a normal-mode scheduler loading such a snapshot (not tried).\n\n**Found by:** feature-level validation smoke run of spec `scheduler-allocator-runner` (2026-09-30); not caused by that feature (limit.c, snapshot_load.c, sched_election.c untouched by it).",
|
||||
"file": "src/scheduler/src/plugins/limit.c, src/scheduler/src/snapshot_load.c",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"sim",
|
||||
"snapshot",
|
||||
"limit"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1887",
|
||||
"bug-1888"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1897.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
62
.wolf/bugs/bug-1897.md
Normal file
62
.wolf/bugs/bug-1897.md
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
---
|
||||
id: "bug-1897"
|
||||
status: "open"
|
||||
status_updated: "2026-09-30"
|
||||
status_note: "newly logged"
|
||||
component: "scheduler"
|
||||
severity: "crash"
|
||||
timestamp: "2026-09-30T10:19:59Z"
|
||||
last_seen: "2026-09-30T10:19:59Z"
|
||||
occurrences: 1
|
||||
files: ["src/scheduler/src/plugins/limit.c", "src/scheduler/src/snapshot_load.c"]
|
||||
tags: ["scheduler", "sim", "snapshot", "limit"]
|
||||
related: ["bug-1887", "bug-1888"]
|
||||
---
|
||||
|
||||
# bug-1897: xce_scheduler --sim segfaults in limit_job_election on a job whose snapshot resreq failed to parse
|
||||
|
||||
## Description
|
||||
**Observed:** replaying `data/scheduler-replay-20260928-064842.replay` in simulation mode crashes before any decision, 3/3 runs:
|
||||
```
|
||||
[ERROR] [xce_sched_core_load_snapshot] xce_sched_core_load_snapshot parse_resreq failed for jobid 4667606771186008064: Primary: ...
|
||||
==2723571==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x555555a7cb6e ... T3)
|
||||
#0 limit_job_election ../scheduler/src/plugins/limit.c:336
|
||||
#1 plugin_job_election ../scheduler/src/plugin.c:601
|
||||
#2 sched_allocator ../scheduler/src/sched_election.c:236
|
||||
```
|
||||
The replay's snapshot stores resreqs in the debug-dump format that bug-1888 wrote before its fix. Since the bug-1887 fix, `parse_resreq` rejects that text, so the snapshot loader logs the error but keeps the job with no schedobj. The allocator thread's job election then reads `job->schedobj` as NULL in the limit plugin.
|
||||
|
||||
**Expected:** a job whose resreq cannot be parsed from a snapshot is either dropped (with the error logged) or kept pending with a reason, and never reaches election with a NULL schedobj; the simulator must not crash on an old replay.
|
||||
|
||||
**Impact:** crash (SIGSEGV) of `xce_scheduler --sim` on any replay whose snapshot has unparsable resreqs, including every replay captured before the bug-1888 fix. Suspected: the same path could crash a normal-mode scheduler loading such a snapshot (not tried).
|
||||
|
||||
**Found by:** feature-level validation smoke run of spec `scheduler-allocator-runner` (2026-09-30); not caused by that feature (limit.c, snapshot_load.c, sched_election.c untouched by it).
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (src/build.alloc, dedicated scheduler-allocator-runner build dir); ASAN_OPTIONS=detect_leaks=0
|
||||
- Code: scheduler-framework @ 28cff179-dirty
|
||||
- Deployment: none (simulation mode, `xce_scheduler --sim`, no controller or workers)
|
||||
|
||||
## Reproduction
|
||||
1. `git checkout 28cff179` (or later); `bash src/tools/xce-build.sh scheduler asan build.alloc --jobs 8`
|
||||
2. `cd src && nix develop . --command bash -c 'ASAN_OPTIONS=detect_leaks=0 ./build.alloc/scheduler/xce_scheduler --sim=/home/claude/tmp/bug-1897/scheduler-replay-20260928-064842.replay -l /tmp/sim.log'`
|
||||
3. Observe `parse_resreq failed` errors in the log, then the ASan SEGV in `limit_job_election`.
|
||||
- Frequency: deterministic (3/3 runs, 2026-09-30)
|
||||
- Evidence: `/home/claude/tmp/bug-1897/` (replay file copy, `run.out` with the ASan report, `sim.log`)
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
Loading…
Add table
Add a link
Reference in a new issue