Rongsong Shen
25fb5c115a
- bug-1799/1801: fix resInUse over-commit in sim replay -- sched_account_jobs's per-round re-derive pass now skips RECONCILING/UNAVAIL workers the same way core_reset_workers already does (bug-1751), closing the gap that let a worker's used slots exceed its own capacity across a resync replay - bug-1804: revise scheduler-sim-eager-reconciliation spec (user-directed) so a captured job's job->state is set to RUN synchronously at dispatch capture, mirroring sched_commit_decision's normal-mode semantics (bug-1624), instead of deferring until its separately-recorded JOB_STATE_CHANGE event replays - bug-1800: render bytes-valued CONSUMABLE resources (mem/swp/tmp) with the same human-scaled unit formatting already used for maxmem/maxswap/maxtmp - bug-1802/1803: add then revert a break-on-ANY-event breakpoint kind after live feedback that it over-fires inside busy sessions; replace it with a prescan-based "next non-dummy session" hint in info summary instead Full scheduler suite: 77/77 tests pass (asan profile). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
18 KiB
Following is the scheduler runs in simulation mode:
rshen@pc13v ~/p/x/w/s/src (scheduler-framework)> ./build.dbg/scheduler/xce_scheduler -i -s /tmp/scheduler-replay-20260911-064117.replay [2026-09-11 14:49:54,696] [801857] [ WARN] [xce_sched_core_load_snapshot] xce_sched_core_load_snapshot failed to resolve job picker plugin (rc=-2); picker rebuild will be skipped for this load [sim] replay file contains 420 sessions [sim] loaded: 0 jobs, snapshot @ ts=1789108877564564549 (2026-09-11T06:41:17Z) [sim] (jobs above reflect this snapshot only -- step/continue to replay later events) [sim] ready. type 'help' for commands. (sim) info summary [sim] summary: 0 jobs (0 pending, 0 running), 1 workers (1 online), 4 queues, 33 resources (sim) step (sim session=1 t=2026-09-11T06:41:17Z) step (sim session=2 t=2026-09-11T06:41:18Z) step (sim session=3 t=2026-09-11T06:41:19Z) step (sim session=4 t=2026-09-11T06:41:20Z) step (sim session=5 t=2026-09-11T06:41:21Z) step (sim session=6 t=2026-09-11T06:41:22Z) step (sim session=7 t=2026-09-11T06:41:23Z) step [sim] t=1789108884567768744 (2026-09-11T06:41:24Z) session=8 [sim] event: RESVALS_DELTA n=11
- how can we step to the next session which is not dummy(something occurs)? or how can I known what's the next non dummy session? for above I just want to goto session 8 directly
for session 40:
(sim session=39 t=2026-09-11T06:41:55Z) step [sim] t=1789108916579881546 (2026-09-11T06:41:56Z) session=40 [sim] event: JOB_PUSH njobs=10 jobids=[0-15-1789108916-0[0-9]] [sim] DISPATCH jobid=0-15-1789108916-0[0] [sim] DISPATCH jobid=0-15-1789108916-0[1] [sim] DISPATCH jobid=0-15-1789108916-0[2] [sim] DISPATCH jobid=0-15-1789108916-0[3] [sim] DISPATCH jobid=0-15-1789108916-0[4] [sim] PEND jobid=0-15-1789108916-0[5] [sim] PEND jobid=0-15-1789108916-0[6] [sim] PEND jobid=0-15-1789108916-0[7] [sim] PEND jobid=0-15-1789108916-0[8] [sim] PEND jobid=0-15-1789108916-0[9] (sim session=40 t=2026-09-11T06:41:56Z) info worker pc13v [sim] worker pc13v: status=online [sim] -- Consumable/Capacity -- [sim] resource slots: value=22.00 used=0.00 [sim] resource mem: value=20684476416.00 used=0.00 [sim] resource swp: value=13132.17 used=0.00 [sim] resource tmp: value=576636968960.00 used=0.00 [sim] -- Load Metrics -- [sim] resource r1m: value=2.30 [sim] resource r15m: value=4.67 [sim] resource r15s: value=2.30 [sim] resource ut: value=1.16 [sim] resource pg: value=263.47 [sim] resource ls: value=1 [sim] resource it: value=3135.20 [sim] resource io: value=0 [sim] resource cpuf: value=1 [sim] resource maxcpus: value=22 [sim] resource maxmem: value=33182072832.000000 (30.90 GB) [sim] resource maxswap: value=17179865088.000000 (16.00 GB) [sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) [sim] resource ndisks: value=1 [sim] -- Static Hardware Attributes -- [sim] resource type: value=x86_64-linux-gnu [sim] resource model: value=x86_64-linux-gnu [sim] assigned jobs: 0-15-1789108916-0[0-4]*4
- The used resources value of slots is still 0.00? as we have dispatched 5 jobs?
Can we improve resource usage value display?
(sim session=48 t=2026-09-11T06:42:07Z) info worker pc13v [sim] worker pc13v: status=online [sim] -- Consumable/Capacity -- [sim] resource slots: value=22.00 used=20.00 [sim] resource mem: value=20684476416.00 used=87960930222080.00 <<<can we use GB/MB/...>>> [sim] resource swp: value=13132.17 used=0.00 <> [sim] resource tmp: value=576636968960.00 used=0.00 <> [sim] -- Load Metrics -- [sim] resource r1m: value=2.30 [sim] resource r15m: value=4.67 [sim] resource r15s: value=2.30 [sim] resource ut: value=1.16 [sim] resource pg: value=263.47 [sim] resource ls: value=1 [sim] resource it: value=3135.20 [sim] resource io: value=0 [sim] resource cpuf: value=1 [sim] resource maxcpus: value=22 [sim] resource maxmem: value=33182072832.000000 (30.90 GB) [sim] resource maxswap: value=17179865088.000000 (16.00 GB) [sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) [sim] resource ndisks: value=1 [sim] -- Static Hardware Attributes -- [sim] resource type: value=x86_64-linux-gnu [sim] resource model: value=x86_64-linux-gnu [sim] assigned jobs: 0-15-1789108916-0[0-4]*4
Something wrong in session 98, the count value of slots used is inconsistent with assigned jobs in pc13v?
[sim] t=1789109023245433276 (2026-09-11T06:43:43Z) session=98 [sim] event: JOB_UPDATE jobid=0-15-1789108916-0[5] [sim] event: JOB_UPDATE jobid=0-15-1789108916-0[6] [sim] event: WORKERS_PUSH nworkers=1 names=[pc13v] [sim] event: JOB_PUSH njobs=20 jobids=[0-15-1789108916-0[0-9],0-15-1789108940-0[0-9]] [sim] event: WORKERS_PUSH nworkers=1 names=[pc13v] [sim] event: JOB_PUSH njobs=20 jobids=[0-15-1789108916-0[0-9],0-15-1789108940-0[0-9]] [sim] event: JOB_UPDATE jobid=0-15-1789108916-0[9] [sim] event: JOB_UPDATE jobid=0-15-1789108940-0[0] [sim] event: JOB_RUSAGE jobid=0-15-1789108940-0[0] [sim] event: JOB_UPDATE jobid=0-15-1789108940-0[1] [sim] event: JOB_RUSAGE jobid=0-15-1789108940-0[1] [sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[5] state=RUN [sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[6] state=RUN [sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[7] state=RUN [sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[8] state=RUN [sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[9] state=RUN [2026-09-11 06:43:43,245] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[0] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,245] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[1] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[2] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[3] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[4] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[5] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[6] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[7] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[7] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept) [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[8] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[9] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[0] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[1] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[2] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[3] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[4] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[5] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[6] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[7] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[8] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[9] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[0] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[1] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[2] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[3] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[4] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[5] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[6] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[7] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[7] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept) [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[8] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[8] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept) [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[9] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[0] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[1] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[2] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[3] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[4] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[5] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[6] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[7] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[8] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[9] (rc=-17), merging state into existing tracked job instead of discarding [sim] DISPATCH jobid=0-15-1789108940-0[2] [sim] DISPATCH jobid=0-15-1789108940-0[3] [sim] DISPATCH jobid=0-15-1789108940-0[4] [sim] DISPATCH jobid=0-15-1789108940-0[5] [sim] PEND jobid=0-15-1789108940-0[6] [sim] PEND jobid=0-15-1789108940-0[7] [sim] PEND jobid=0-15-1789108940-0[8] [sim] PEND jobid=0-15-1789108940-0[9] (sim session=98 t=2026-09-11T06:43:43Z) info worker pc13v [sim] worker pc13v: status=online [sim] -- Consumable/Capacity -- [sim] resource slots: value=22.00 used=22.00 [sim] resource mem: value=20555202560.00 used=35184372088832.00 [sim] resource swp: value=13132.59 used=0.00 [sim] resource tmp: value=576442761216.00 used=0.00 [sim] -- Load Metrics -- [sim] resource r1m: value=0.84 [sim] resource r15m: value=4.24 [sim] resource r15s: value=0.84 [sim] resource ut: value=3.52 [sim] resource pg: value=1752 [sim] resource ls: value=1 [sim] resource it: value=3136.95 [sim] resource io: value=0 [sim] resource cpuf: value=1 [sim] resource maxcpus: value=22 [sim] resource maxmem: value=33182072832.000000 (30.90 GB) [sim] resource maxswap: value=17179865088.000000 (16.00 GB) [sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) [sim] resource ndisks: value=1 [sim] -- Static Hardware Attributes -- [sim] resource type: value=x86_64-linux-gnu [sim] resource model: value=x86_64-linux-gnu [sim] assigned job: jobid=0-15-1789108916-0[5]*4 [sim] assigned job: jobid=0-15-1789108916-0[6]*4 [sim] assigned job: jobid=0-15-1789108916-0[7]*4 [sim] assigned job: jobid=0-15-1789108916-0[8]*4 [sim] assigned job: jobid=0-15-1789108916-0[9]*4 [sim] assigned job: jobid=0-15-1789108940-0[0] [sim] assigned job: jobid=0-15-1789108940-0[1] [sim] assigned job: jobid=0-15-1789108940-0[2] [sim] assigned job: jobid=0-15-1789108940-0[3] [sim] assigned job: jobid=0-15-1789108940-0[4] [sim] assigned job: jobid=0-15-1789108940-0[5] (sim session=98 t=2026-09-11T06:43:54Z)
- it seems the scheduler has broken max slots constraint? Investigate why? check next session, the used slots get to 26
(sim session=99 t=2026-09-11T06:43:54Z) info worker pc13v [sim] worker pc13v: status=online [sim] -- Consumable/Capacity -- [sim] resource slots: value=22.00 used=26.00 [sim] resource mem: value=20555202560.00 used=35184372088832.00 [sim] resource swp: value=13132.59 used=0.00 [sim] resource tmp: value=576442761216.00 used=0.00 [sim] -- Load Metrics -- [sim] resource r1m: value=0.84 [sim] resource r15m: value=4.24 [sim] resource r15s: value=0.84 [sim] resource ut: value=3.52 [sim] resource pg: value=1752 [sim] resource ls: value=1 [sim] resource it: value=3136.95 [sim] resource io: value=0 [sim] resource cpuf: value=1 [sim] resource maxcpus: value=22 [sim] resource maxmem: value=33182072832.000000 (30.90 GB) [sim] resource maxswap: value=17179865088.000000 (16.00 GB) [sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) [sim] resource ndisks: value=1 [sim] -- Static Hardware Attributes -- [sim] resource type: value=x86_64-linux-gnu [sim] resource model: value=x86_64-linux-gnu [sim] assigned job: jobid=0-15-1789108916-0[5]*4 [sim] assigned job: jobid=0-15-1789108916-0[6]*4 [sim] assigned job: jobid=0-15-1789108916-0[7]*4 [sim] assigned job: jobid=0-15-1789108916-0[8]*4 [sim] assigned job: jobid=0-15-1789108916-0[9]*4 [sim] assigned job: jobid=0-15-1789108940-0[0] [sim] assigned job: jobid=0-15-1789108940-0[1] [sim] assigned job: jobid=0-15-1789108940-0[2] [sim] assigned job: jobid=0-15-1789108940-0[3] [sim] assigned job: jobid=0-15-1789108940-0[4] [sim] assigned job: jobid=0-15-1789108940-0[5]