Rongsong Shen
668e3f6893
- .wolf/anatomy-index.json, anatomy.md, memory.md: OpenWolf hook-maintained project index and session log, refreshed by this session's activity. - .kiro/specs/controller-peers-env-fallback/: spec scaffold (brief, requirements, design, research) from prior spec-init work. - issues/new_requirements.md: scheduler simulator enhancement backlog (info-command improvements, pend-reason detail, decision tracing). Excluded as non-persistent: issues/Issues.md (empty placeholder), src/lib/testsuite/test_xcectx_reconnect_race.c (self-documented as an ASAN stress reproduction only, not a committed regression test, and not wired into meson.build), scheduler-replay-20260922-095207.replay (4MB binary debug capture, no .replay files are tracked in this repo). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
14 KiB
14 KiB
- Enhance to scheduler simulator
- enhance
info jobcommand for specific job to:- for pending job, show the pending reason
- for running job, show the resource usage information(include reserved and real used)
- can we add more info commands for exploring internal state of scheduler in simulation mode?
- add option to trace the full decision made by scheduler
- which job has been selected to selected and why
- trace the full process to resreq selection
- trace the full of planner to resreq selection
- trace plan part to determine the host selection
- trace the allocation details(such as resource allocation and limits applied)
- trace the reason of why host can/cannot meet resreq
- we can trace the run of all jobs or just single job
- Job pending reason
- for any job which is in PEND state, there should be a pend reason attached
- if the job has not been scheduled, we can set the reason to "The job is waiting to scheduler"
- if the job is not ready to be scheduled, show the reason of why job is not ready to be scheduled, it can be:
- because of queue?
- time constraint?
- job dependency?
- others?
- If the job can not be run in some host, give out the reason, such as:
- host status?
- select considition?
- resource can not be meet
- optimize the pendding reason set between scheduler and controller, old push back when pending reason changed
- Enhance harness tool xce_test to get detail information of job:
xce/client>jobsub -q xce -u rshen -R "rusage[mem=2048]" -a 40 -c "sleep 100 && hostname"
jobsubmit ok: JobId: 0-14-1789369200-0
xce/client>joblist
JobId User Queue State SubmitTime StartTime FinishTime MinTasks MaxTasks Command
0-15-1789108916-0[0] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[1] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:36,000 4 4 sleep 100 && hostname
0-15-1789108916-0[2] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[3] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[4] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:41:56,000 2026-09-11 14:43:37,000 4 4 sleep 100 && hostname
0-15-1789108916-0[5] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:40,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[6] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:40,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[7] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:41,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[8] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:41,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108916-0[9] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:41,000 2026-09-11 14:45:21,000 4 4 sleep 100 && hostname
0-15-1789108940-0[0] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:42:21,000 2026-09-11 14:44:02,000 1 1 sleep 100 && hostname
0-15-1789108940-0[1] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:42:22,000 2026-09-11 14:44:02,000 1 1 sleep 100 && hostname
0-15-1789108940-0[2] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:23,000 1 1 sleep 100 && hostname
0-15-1789108940-0[3] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:23,000 1 1 sleep 100 && hostname
0-15-1789108940-0[4] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:24,000 1 1 sleep 100 && hostname
0-15-1789108940-0[5] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:43:43,000 2026-09-11 14:45:23,000 1 1 sleep 100 && hostname
0-15-1789108940-0[6] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:44:04,000 2026-09-11 14:45:44,000 1 1 sleep 100 && hostname
0-15-1789108940-0[7] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:44:04,000 2026-09-11 14:45:45,000 1 1 sleep 100 && hostname
0-15-1789108940-0[8] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:45:22,000 2026-09-11 14:47:02,000 1 1 sleep 100 && hostname
0-15-1789108940-0[9] rshen xce DONE 2026-09-14 14:58:57,000 2026-09-11 14:45:22,000 2026-09-11 14:47:02,000 1 1 sleep 100 && hostname
0-14-1789369200-0[0] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[1] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[2] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[3] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:01,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[4] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[5] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[6] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[7] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[8] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[9] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[10] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[11] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[12] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[13] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[14] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[15] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[16] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[17] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:02,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[18] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:03,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[19] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:03,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[20] rshen xce RUN 2026-09-14 15:00:00,000 2026-09-14 15:00:03,000 - 1 1 sleep 100 && hostname
0-14-1789369200-0[21] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[22] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[23] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[24] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[25] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[26] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[27] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[28] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[29] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[30] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[31] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[32] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[33] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[34] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[35] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[36] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[37] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[38] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
0-14-1789369200-0[39] rshen xce PEND 2026-09-14 15:00:00,000 - - 1 1 sleep 100 && hostname
xce/client>jobinfo -j "0-14-1789369200-0[23]"
JobID: 0-14-1789369200-0[23]
state: PEND
state_raw: 0x1
Queue: xce
User: rshen
Slots: 1,1
Command: sleep 100 && hostname
Pend Reason:
we prefer to make jobinfo to also show resreq and pending reason information and maybe other informations
- re-arch plugin interface to eliminate duplicated code(such as my_alloc()).
According the design of scheduler-allocation project we can make a general allocation function in plugin framework to make sequential and parallel allocation plugins do not need to export my_alloc().
- Enhance controller to clean up jobs which meet following conditions:
- pre-defined parameter in cluster configuration for how long to keep a job before it can be clean uped
- Job is completed until not exceed a period (say job clean up period)
- No other jobs depends on this jobs (we just need a stub function to check depend jobs)
- job begin time & endtime
- job will not be scheduled before begin time
- job should be finished before end time
-
job dependency
-
verify the cache reuse in scheduler session? This should be based on trace
-
run job inside sandbox to decouple the submit user and execution user?
-
provision volume for input/output of job?
-
xce resource provider? resource consumer?
- worker provides:
- cpu, memory, swap?
- volume/data?
- license?