lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/issues/new_requirements.md
Rongsong Shen bf0ba68b22
chore(scheduler-allocator-runner): validation passed; new_requirements item 4 resolved (task 5.2) 
Clean ASAN scheduler suite 96/96 with no expectation changes in existing
tests beyond wiring; scheduler invariant audit of the allocation path
clean (driver state-equivalence, core I/O isolation, picker/allocator
rendezvous, snapshot parity). issues/new_requirements.md item 4 marked
resolved.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 10:11:10 +00:00

135 lines
14 KiB
Markdown
Raw Blame History

1. Enhance to scheduler simulator
- enhance `info job` command for specific job to:
- for pending job, show the pending reason
- for running job, show the resource usage information(include reserved and real used)
- can we add more info commands for exploring internal state of scheduler in simulation mode?
- add option to trace the full decision made by scheduler
- which job has been selected to selected and why
- trace the full process to resreq selection
- trace the full of planner to resreq selection
- trace plan part to determine the host selection
- trace the allocation details(such as resource allocation and limits applied)
- trace the reason of why host can/cannot meet resreq
- we can trace the run of all jobs or just single job
2. Job pending reason
- for any job which is in PEND state, there should be a pend reason attached
- if the job has not been scheduled, we can set the reason to "The job is waiting to scheduler"
- if the job is not ready to be scheduled, show the reason of why job is not ready to be scheduled, it can be:
- because of queue?
- time constraint?
- job dependency?
- others?
- If the job can not be run in some host, give out the reason, such as:
- host status?
- select considition?
- resource can not be meet
- optimize the pendding reason set between scheduler and controller, old push back when pending reason changed
3. Enhance harness tool xce_test to get detail information of job:
```sh
xce/client>jobsub -q xce -u rshen -R "rusage[mem=2048]" -a 40 -c "sleep 100 && hostname"
jobsubmit ok: JobId: 0-14-1789369200-0
xce/client>joblist
JobId User Queue State SubmitTime StartTime FinishTime MinTasks MaxTasks Command
0-15-1789108916-0[0] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[1] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:36,000 4 4 sleep 100 && hostname
0-15-1789108916-0[2] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[3] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[4] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[5] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:40,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[6] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:40,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[7] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:41,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[8] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:41,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[9] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:41,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108940-0[0] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:42:21,000 2026-09-11 14:44:02,000 1 1 sleep 100 && hostname
0-15-1789108940-0[1] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:42:22,000 2026-09-11 14:44:02,000 1 1 sleep 100 && hostname
0-15-1789108940-0[2] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:23,000 1 1 sleep 100 && hostname
0-15-1789108940-0[3] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:23,000 1 1 sleep 100 && hostname
0-15-1789108940-0[4] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:24,000 1 1 sleep 100 && hostname
0-15-1789108940-0[5] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:23,000 1 1 sleep 100 && hostname
0-15-1789108940-0[6] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:44:04,000 2026-09-11 14:45:44,000 1 1 sleep 100 && hostname
0-15-1789108940-0[7] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:44:04,000 2026-09-11 14:45:45,000 1 1 sleep 100 && hostname
0-15-1789108940-0[8] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:45:22,000 2026-09-11 14:47:02,000 1 1 sleep 100 && hostname
0-15-1789108940-0[9] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:45:22,000 2026-09-11 14:47:02,000 1 1 sleep 100 && hostname
0-14-1789369200-0[0] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[1] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[2] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[3] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[4] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[5] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[6] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[7] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[8] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[9] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[10] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[11] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[12] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[13] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[14] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[15] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[16] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[17] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[18] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:03,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[19] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:03,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[20] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:03,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[21] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[22] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[23] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[24] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[25] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[26] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[27] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[28] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[29] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[30] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[31] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[32] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[33] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[34] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[35] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[36] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[37] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[38] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[39] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
xce/client>jobinfo -j "0-14-1789369200-0[23]"
JobID: 0-14-1789369200-0[23]
state: PEND
state_raw: 0x1
Queue: xce
User: rshen
Slots: 1,1
Command: sleep 100 && hostname
Pend Reason:
```
we prefer to make jobinfo to also show resreq and pending reason information and maybe other informations
4. re-arch plugin interface to eliminate duplicated code(such as my_alloc()).
According the design of scheduler-allocation project we can make a general allocation function in plugin framework to make sequential and parallel allocation plugins do not need to export my_alloc().
Resolved (2026-09-30): see `.kiro/specs/scheduler-allocator-runner/`. The framework routine `sched_allocator_run()` (src/scheduler/src/alloc_runner.c) is now the single allocation entry point (`job_alloc_fn`) for both allocator plugins; seqjob and parjob supply only their policy via `allocator_iface` (plan generator, optional acceptance guard), and both private `my_alloc()` routines were removed.
5. Enhance controller to clean up jobs which meet following conditions:
- pre-defined parameter in cluster configuration for how long to keep a job before it can be clean uped
- Job is completed until not exceed a period (say job clean up period)
- No other jobs depends on this jobs (we just need a stub function to check depend jobs)
6. job begin time & endtime
- job will not be scheduled before begin time
- job should be finished before end time
7. job dependency
8. verify the cache reuse in scheduler session?
This should be based on trace
9. run job inside sandbox to decouple the submit user and execution user?
10. provision volume for input/output of job?
11. xce resource provider? resource consumer?
- worker provides:
- cpu, memory, swap?
- volume/data?
- license?
Reference in a new issue View git blame Copy permalink