lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/issues/new_requirements.md
Rongsong Shen 668e3f6893
chore: persist OpenWolf state, controller-peers-env-fallback spec, and scheduler backlog notes 
- .wolf/anatomy-index.json, anatomy.md, memory.md: OpenWolf hook-maintained
  project index and session log, refreshed by this session's activity.
- .kiro/specs/controller-peers-env-fallback/: spec scaffold (brief,
  requirements, design, research) from prior spec-init work.
- issues/new_requirements.md: scheduler simulator enhancement backlog
  (info-command improvements, pend-reason detail, decision tracing).

Excluded as non-persistent: issues/Issues.md (empty placeholder),
src/lib/testsuite/test_xcectx_reconnect_race.c (self-documented as an
ASAN stress reproduction only, not a committed regression test, and not
wired into meson.build), scheduler-replay-20260922-095207.replay
(4MB binary debug capture, no .replay files are tracked in this repo).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 03:19:52 +00:00

133 lines
14 KiB
Markdown
Raw Blame History

1. Enhance to scheduler simulator
- enhance `info job` command for specific job to:
- for pending job, show the pending reason
- for running job, show the resource usage information(include reserved and real used)
- can we add more info commands for exploring internal state of scheduler in simulation mode?
- add option to trace the full decision made by scheduler
- which job has been selected to selected and why
- trace the full process to resreq selection
- trace the full of planner to resreq selection
- trace plan part to determine the host selection
- trace the allocation details(such as resource allocation and limits applied)
- trace the reason of why host can/cannot meet resreq
- we can trace the run of all jobs or just single job
2. Job pending reason
- for any job which is in PEND state, there should be a pend reason attached
- if the job has not been scheduled, we can set the reason to "The job is waiting to scheduler"
- if the job is not ready to be scheduled, show the reason of why job is not ready to be scheduled, it can be:
- because of queue?
- time constraint?
- job dependency?
- others?
- If the job can not be run in some host, give out the reason, such as:
- host status?
- select considition?
- resource can not be meet
- optimize the pendding reason set between scheduler and controller, old push back when pending reason changed
3. Enhance harness tool xce_test to get detail information of job:
```sh
xce/client>jobsub -q xce -u rshen -R "rusage[mem=2048]" -a 40 -c "sleep 100 && hostname"
jobsubmit ok: JobId: 0-14-1789369200-0
xce/client>joblist
JobId User Queue State SubmitTime StartTime FinishTime MinTasks MaxTasks Command
0-15-1789108916-0[0] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[1] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:36,000 4 4 sleep 100 && hostname
0-15-1789108916-0[2] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[3] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[4] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[5] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:40,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[6] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:40,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[7] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:41,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[8] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:41,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[9] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:41,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108940-0[0] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:42:21,000 2026-09-11 14:44:02,000 1 1 sleep 100 && hostname
0-15-1789108940-0[1] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:42:22,000 2026-09-11 14:44:02,000 1 1 sleep 100 && hostname
0-15-1789108940-0[2] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:23,000 1 1 sleep 100 && hostname
0-15-1789108940-0[3] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:23,000 1 1 sleep 100 && hostname
0-15-1789108940-0[4] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:24,000 1 1 sleep 100 && hostname
0-15-1789108940-0[5] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:23,000 1 1 sleep 100 && hostname
0-15-1789108940-0[6] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:44:04,000 2026-09-11 14:45:44,000 1 1 sleep 100 && hostname
0-15-1789108940-0[7] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:44:04,000 2026-09-11 14:45:45,000 1 1 sleep 100 && hostname
0-15-1789108940-0[8] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:45:22,000 2026-09-11 14:47:02,000 1 1 sleep 100 && hostname
0-15-1789108940-0[9] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:45:22,000 2026-09-11 14:47:02,000 1 1 sleep 100 && hostname
0-14-1789369200-0[0] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[1] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[2] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[3] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[4] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[5] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[6] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[7] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[8] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[9] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[10] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[11] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[12] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[13] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[14] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[15] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[16] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[17] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[18] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:03,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[19] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:03,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[20] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:03,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[21] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[22] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[23] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[24] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[25] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[26] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[27] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[28] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[29] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[30] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[31] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[32] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[33] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[34] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[35] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[36] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[37] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[38] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[39] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
xce/client>jobinfo -j "0-14-1789369200-0[23]"
JobID: 0-14-1789369200-0[23]
state: PEND
state_raw: 0x1
Queue: xce
User: rshen
Slots: 1,1
Command: sleep 100 && hostname
Pend Reason:
```
we prefer to make jobinfo to also show resreq and pending reason information and maybe other informations
4. re-arch plugin interface to eliminate duplicated code(such as my_alloc()).
According the design of scheduler-allocation project we can make a general allocation function in plugin framework to make sequential and parallel allocation plugins do not need to export my_alloc().
5. Enhance controller to clean up jobs which meet following conditions:
- pre-defined parameter in cluster configuration for how long to keep a job before it can be clean uped
- Job is completed until not exceed a period (say job clean up period)
- No other jobs depends on this jobs (we just need a stub function to check depend jobs)
6. job begin time & endtime
- job will not be scheduled before begin time
- job should be finished before end time
7. job dependency
8. verify the cache reuse in scheduler session?
This should be based on trace
9. run job inside sandbox to decouple the submit user and execution user?
10. provision volume for input/output of job?
11. xce resource provider? resource consumer?
- worker provides:
- cpu, memory, swap?
- volume/data?
- license?
Reference in a new issue View git blame Copy permalink