lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/issues/simulator-1.md
Rongsong Shen 25fb5c115a
fix(scheduler): sim resource over-commit, byte-unit display, sync job-state-at-capture 
- bug-1799/1801: fix resInUse over-commit in sim replay -- sched_account_jobs's
  per-round re-derive pass now skips RECONCILING/UNAVAIL workers the same way
  core_reset_workers already does (bug-1751), closing the gap that let a
  worker's used slots exceed its own capacity across a resync replay
- bug-1804: revise scheduler-sim-eager-reconciliation spec (user-directed) so
  a captured job's job->state is set to RUN synchronously at dispatch capture,
  mirroring sched_commit_decision's normal-mode semantics (bug-1624), instead
  of deferring until its separately-recorded JOB_STATE_CHANGE event replays
- bug-1800: render bytes-valued CONSUMABLE resources (mem/swp/tmp) with the
  same human-scaled unit formatting already used for maxmem/maxswap/maxtmp
- bug-1802/1803: add then revert a break-on-ANY-event breakpoint kind after
  live feedback that it over-fires inside busy sessions; replace it with a
  prescan-based "next non-dummy session" hint in info summary instead

Full scheduler suite: 77/77 tests pass (asan profile).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-12 14:06:41 +00:00

242 lines
18 KiB
Markdown
Raw Blame History

Following is the scheduler runs in simulation mode:
rshen@pc13v ~/p/x/w/s/src (scheduler-framework)> ./build.dbg/scheduler/xce_scheduler -i -s /tmp/scheduler-replay-20260911-064117.replay
[2026-09-11 14:49:54,696] [801857] [ WARN] [xce_sched_core_load_snapshot] xce_sched_core_load_snapshot failed to resolve job picker plugin (rc=-2); picker rebuild will be skipped for this load
[sim] replay file contains 420 sessions
[sim] loaded: 0 jobs, snapshot @ ts=1789108877564564549 (2026-09-11T06:41:17Z)
[sim] (jobs above reflect this snapshot only -- step/continue to replay later events)
[sim] ready. type 'help' for commands.
(sim) info summary
[sim] summary: 0 jobs (0 pending, 0 running), 1 workers (1 online), 4 queues, 33 resources
(sim) step
(sim session=1 t=2026-09-11T06:41:17Z) step
(sim session=2 t=2026-09-11T06:41:18Z) step
(sim session=3 t=2026-09-11T06:41:19Z) step
(sim session=4 t=2026-09-11T06:41:20Z) step
(sim session=5 t=2026-09-11T06:41:21Z) step
(sim session=6 t=2026-09-11T06:41:22Z) step
(sim session=7 t=2026-09-11T06:41:23Z) step
[sim] t=1789108884567768744 (2026-09-11T06:41:24Z) session=8
[sim] event: RESVALS_DELTA n=11
- how can we step to the next session which is not dummy(something occurs)? or how can I known what's the next non dummy session? for above I just want to goto session 8 directly
for session 40:
(sim session=39 t=2026-09-11T06:41:55Z) step
[sim] t=1789108916579881546 (2026-09-11T06:41:56Z) session=40
[sim] event: JOB_PUSH njobs=10 jobids=[0-15-1789108916-0[0-9]]
[sim] DISPATCH jobid=0-15-1789108916-0[0]
[sim] DISPATCH jobid=0-15-1789108916-0[1]
[sim] DISPATCH jobid=0-15-1789108916-0[2]
[sim] DISPATCH jobid=0-15-1789108916-0[3]
[sim] DISPATCH jobid=0-15-1789108916-0[4]
[sim] PEND jobid=0-15-1789108916-0[5]
[sim] PEND jobid=0-15-1789108916-0[6]
[sim] PEND jobid=0-15-1789108916-0[7]
[sim] PEND jobid=0-15-1789108916-0[8]
[sim] PEND jobid=0-15-1789108916-0[9]
(sim session=40 t=2026-09-11T06:41:56Z) info worker pc13v
[sim] worker pc13v: status=online
[sim] -- Consumable/Capacity --
[sim] resource slots: value=22.00 used=0.00
[sim] resource mem: value=20684476416.00 used=0.00
[sim] resource swp: value=13132.17 used=0.00
[sim] resource tmp: value=576636968960.00 used=0.00
[sim] -- Load Metrics --
[sim] resource r1m: value=2.30
[sim] resource r15m: value=4.67
[sim] resource r15s: value=2.30
[sim] resource ut: value=1.16
[sim] resource pg: value=263.47
[sim] resource ls: value=1
[sim] resource it: value=3135.20
[sim] resource io: value=0
[sim] resource cpuf: value=1
[sim] resource maxcpus: value=22
[sim] resource maxmem: value=33182072832.000000 (30.90 GB)
[sim] resource maxswap: value=17179865088.000000 (16.00 GB)
[sim] resource maxtmp: value=1005921779712.000000 (936.84 GB)
[sim] resource ndisks: value=1
[sim] -- Static Hardware Attributes --
[sim] resource type: value=x86_64-linux-gnu
[sim] resource model: value=x86_64-linux-gnu
[sim] assigned jobs: 0-15-1789108916-0[0-4]*4
- The used resources value of slots is still 0.00? as we have dispatched 5 jobs?
Can we improve resource usage value display?
(sim session=48 t=2026-09-11T06:42:07Z) info worker pc13v
[sim] worker pc13v: status=online
[sim] -- Consumable/Capacity --
[sim] resource slots: value=22.00 used=20.00
[sim] resource mem: value=20684476416.00 used=87960930222080.00 <<<can we use GB/MB/...>>>
[sim] resource swp: value=13132.17 used=0.00 <<same as above>>
[sim] resource tmp: value=576636968960.00 used=0.00 <<same as above>>
[sim] -- Load Metrics --
[sim] resource r1m: value=2.30
[sim] resource r15m: value=4.67
[sim] resource r15s: value=2.30
[sim] resource ut: value=1.16
[sim] resource pg: value=263.47
[sim] resource ls: value=1
[sim] resource it: value=3135.20
[sim] resource io: value=0
[sim] resource cpuf: value=1
[sim] resource maxcpus: value=22
[sim] resource maxmem: value=33182072832.000000 (30.90 GB)
[sim] resource maxswap: value=17179865088.000000 (16.00 GB)
[sim] resource maxtmp: value=1005921779712.000000 (936.84 GB)
[sim] resource ndisks: value=1
[sim] -- Static Hardware Attributes --
[sim] resource type: value=x86_64-linux-gnu
[sim] resource model: value=x86_64-linux-gnu
[sim] assigned jobs: 0-15-1789108916-0[0-4]*4
Something wrong in session 98, the count value of slots used is inconsistent with assigned jobs in pc13v?
[sim] t=1789109023245433276 (2026-09-11T06:43:43Z) session=98
[sim] event: JOB_UPDATE jobid=0-15-1789108916-0[5]
[sim] event: JOB_UPDATE jobid=0-15-1789108916-0[6]
[sim] event: WORKERS_PUSH nworkers=1 names=[pc13v]
[sim] event: JOB_PUSH njobs=20 jobids=[0-15-1789108916-0[0-9],0-15-1789108940-0[0-9]]
[sim] event: WORKERS_PUSH nworkers=1 names=[pc13v]
[sim] event: JOB_PUSH njobs=20 jobids=[0-15-1789108916-0[0-9],0-15-1789108940-0[0-9]]
[sim] event: JOB_UPDATE jobid=0-15-1789108916-0[9]
[sim] event: JOB_UPDATE jobid=0-15-1789108940-0[0]
[sim] event: JOB_RUSAGE jobid=0-15-1789108940-0[0]
[sim] event: JOB_UPDATE jobid=0-15-1789108940-0[1]
[sim] event: JOB_RUSAGE jobid=0-15-1789108940-0[1]
[sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[5] state=RUN
[sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[6] state=RUN
[sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[7] state=RUN
[sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[8] state=RUN
[sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[9] state=RUN
[2026-09-11 06:43:43,245] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[0] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,245] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[1] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[2] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[3] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[4] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[5] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[6] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[7] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[7] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept)
[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[8] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[9] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[0] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[1] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[2] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[3] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[4] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[5] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[6] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[7] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[8] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[9] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[0] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[1] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[2] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[3] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[4] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[5] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[6] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[7] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[7] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept)
[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[8] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[8] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept)
[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[9] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[0] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[1] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[2] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[3] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[4] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[5] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[6] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[7] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[8] (rc=-17), merging state into existing tracked job instead of discarding
[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[9] (rc=-17), merging state into existing tracked job instead of discarding
[sim] DISPATCH jobid=0-15-1789108940-0[2]
[sim] DISPATCH jobid=0-15-1789108940-0[3]
[sim] DISPATCH jobid=0-15-1789108940-0[4]
[sim] DISPATCH jobid=0-15-1789108940-0[5]
[sim] PEND jobid=0-15-1789108940-0[6]
[sim] PEND jobid=0-15-1789108940-0[7]
[sim] PEND jobid=0-15-1789108940-0[8]
[sim] PEND jobid=0-15-1789108940-0[9]
(sim session=98 t=2026-09-11T06:43:43Z) info worker pc13v
[sim] worker pc13v: status=online
[sim] -- Consumable/Capacity --
[sim] resource slots: value=22.00 used=22.00
[sim] resource mem: value=20555202560.00 used=35184372088832.00
[sim] resource swp: value=13132.59 used=0.00
[sim] resource tmp: value=576442761216.00 used=0.00
[sim] -- Load Metrics --
[sim] resource r1m: value=0.84
[sim] resource r15m: value=4.24
[sim] resource r15s: value=0.84
[sim] resource ut: value=3.52
[sim] resource pg: value=1752
[sim] resource ls: value=1
[sim] resource it: value=3136.95
[sim] resource io: value=0
[sim] resource cpuf: value=1
[sim] resource maxcpus: value=22
[sim] resource maxmem: value=33182072832.000000 (30.90 GB)
[sim] resource maxswap: value=17179865088.000000 (16.00 GB)
[sim] resource maxtmp: value=1005921779712.000000 (936.84 GB)
[sim] resource ndisks: value=1
[sim] -- Static Hardware Attributes --
[sim] resource type: value=x86_64-linux-gnu
[sim] resource model: value=x86_64-linux-gnu
[sim] assigned job: jobid=0-15-1789108916-0[5]*4
[sim] assigned job: jobid=0-15-1789108916-0[6]*4
[sim] assigned job: jobid=0-15-1789108916-0[7]*4
[sim] assigned job: jobid=0-15-1789108916-0[8]*4
[sim] assigned job: jobid=0-15-1789108916-0[9]*4
[sim] assigned job: jobid=0-15-1789108940-0[0]
[sim] assigned job: jobid=0-15-1789108940-0[1]
[sim] assigned job: jobid=0-15-1789108940-0[2]
[sim] assigned job: jobid=0-15-1789108940-0[3]
[sim] assigned job: jobid=0-15-1789108940-0[4]
[sim] assigned job: jobid=0-15-1789108940-0[5]
(sim session=98 t=2026-09-11T06:43:54Z)
- it seems the scheduler has broken max slots constraint? Investigate why? check next session, the used slots get to 26
(sim session=99 t=2026-09-11T06:43:54Z) info worker pc13v
[sim] worker pc13v: status=online
[sim] -- Consumable/Capacity --
[sim] resource slots: value=22.00 used=26.00
[sim] resource mem: value=20555202560.00 used=35184372088832.00
[sim] resource swp: value=13132.59 used=0.00
[sim] resource tmp: value=576442761216.00 used=0.00
[sim] -- Load Metrics --
[sim] resource r1m: value=0.84
[sim] resource r15m: value=4.24
[sim] resource r15s: value=0.84
[sim] resource ut: value=3.52
[sim] resource pg: value=1752
[sim] resource ls: value=1
[sim] resource it: value=3136.95
[sim] resource io: value=0
[sim] resource cpuf: value=1
[sim] resource maxcpus: value=22
[sim] resource maxmem: value=33182072832.000000 (30.90 GB)
[sim] resource maxswap: value=17179865088.000000 (16.00 GB)
[sim] resource maxtmp: value=1005921779712.000000 (936.84 GB)
[sim] resource ndisks: value=1
[sim] -- Static Hardware Attributes --
[sim] resource type: value=x86_64-linux-gnu
[sim] resource model: value=x86_64-linux-gnu
[sim] assigned job: jobid=0-15-1789108916-0[5]*4
[sim] assigned job: jobid=0-15-1789108916-0[6]*4
[sim] assigned job: jobid=0-15-1789108916-0[7]*4
[sim] assigned job: jobid=0-15-1789108916-0[8]*4
[sim] assigned job: jobid=0-15-1789108916-0[9]*4
[sim] assigned job: jobid=0-15-1789108940-0[0]
[sim] assigned job: jobid=0-15-1789108940-0[1]
[sim] assigned job: jobid=0-15-1789108940-0[2]
[sim] assigned job: jobid=0-15-1789108940-0[3]
[sim] assigned job: jobid=0-15-1789108940-0[4]
[sim] assigned job: jobid=0-15-1789108940-0[5]
Reference in a new issue View git blame Copy permalink