Following is the scheduler runs in simulation mode: rshen@pc13v ~/p/x/w/s/src (scheduler-framework)> ./build.dbg/scheduler/xce_scheduler -i -s /tmp/scheduler-replay-20260911-064117.replay [2026-09-11 14:49:54,696] [801857] [ WARN] [xce_sched_core_load_snapshot] xce_sched_core_load_snapshot failed to resolve job picker plugin (rc=-2); picker rebuild will be skipped for this load [sim] replay file contains 420 sessions [sim] loaded: 0 jobs, snapshot @ ts=1789108877564564549 (2026-09-11T06:41:17Z) [sim] (jobs above reflect this snapshot only -- step/continue to replay later events) [sim] ready. type 'help' for commands. (sim) info summary [sim] summary: 0 jobs (0 pending, 0 running), 1 workers (1 online), 4 queues, 33 resources (sim) step (sim session=1 t=2026-09-11T06:41:17Z) step (sim session=2 t=2026-09-11T06:41:18Z) step (sim session=3 t=2026-09-11T06:41:19Z) step (sim session=4 t=2026-09-11T06:41:20Z) step (sim session=5 t=2026-09-11T06:41:21Z) step (sim session=6 t=2026-09-11T06:41:22Z) step (sim session=7 t=2026-09-11T06:41:23Z) step [sim] t=1789108884567768744 (2026-09-11T06:41:24Z) session=8 [sim] event: RESVALS_DELTA n=11 - how can we step to the next session which is not dummy(something occurs)? or how can I known what's the next non dummy session? for above I just want to goto session 8 directly for session 40: (sim session=39 t=2026-09-11T06:41:55Z) step [sim] t=1789108916579881546 (2026-09-11T06:41:56Z) session=40 [sim] event: JOB_PUSH njobs=10 jobids=[0-15-1789108916-0[0-9]] [sim] DISPATCH jobid=0-15-1789108916-0[0] [sim] DISPATCH jobid=0-15-1789108916-0[1] [sim] DISPATCH jobid=0-15-1789108916-0[2] [sim] DISPATCH jobid=0-15-1789108916-0[3] [sim] DISPATCH jobid=0-15-1789108916-0[4] [sim] PEND jobid=0-15-1789108916-0[5] [sim] PEND jobid=0-15-1789108916-0[6] [sim] PEND jobid=0-15-1789108916-0[7] [sim] PEND jobid=0-15-1789108916-0[8] [sim] PEND jobid=0-15-1789108916-0[9] (sim session=40 t=2026-09-11T06:41:56Z) info worker pc13v [sim] worker pc13v: status=online [sim] -- Consumable/Capacity -- [sim] resource slots: value=22.00 used=0.00 [sim] resource mem: value=20684476416.00 used=0.00 [sim] resource swp: value=13132.17 used=0.00 [sim] resource tmp: value=576636968960.00 used=0.00 [sim] -- Load Metrics -- [sim] resource r1m: value=2.30 [sim] resource r15m: value=4.67 [sim] resource r15s: value=2.30 [sim] resource ut: value=1.16 [sim] resource pg: value=263.47 [sim] resource ls: value=1 [sim] resource it: value=3135.20 [sim] resource io: value=0 [sim] resource cpuf: value=1 [sim] resource maxcpus: value=22 [sim] resource maxmem: value=33182072832.000000 (30.90 GB) [sim] resource maxswap: value=17179865088.000000 (16.00 GB) [sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) [sim] resource ndisks: value=1 [sim] -- Static Hardware Attributes -- [sim] resource type: value=x86_64-linux-gnu [sim] resource model: value=x86_64-linux-gnu [sim] assigned jobs: 0-15-1789108916-0[0-4]*4 - The used resources value of slots is still 0.00? as we have dispatched 5 jobs? Can we improve resource usage value display? (sim session=48 t=2026-09-11T06:42:07Z) info worker pc13v [sim] worker pc13v: status=online [sim] -- Consumable/Capacity -- [sim] resource slots: value=22.00 used=20.00 [sim] resource mem: value=20684476416.00 used=87960930222080.00 <<>> [sim] resource swp: value=13132.17 used=0.00 <> [sim] resource tmp: value=576636968960.00 used=0.00 <> [sim] -- Load Metrics -- [sim] resource r1m: value=2.30 [sim] resource r15m: value=4.67 [sim] resource r15s: value=2.30 [sim] resource ut: value=1.16 [sim] resource pg: value=263.47 [sim] resource ls: value=1 [sim] resource it: value=3135.20 [sim] resource io: value=0 [sim] resource cpuf: value=1 [sim] resource maxcpus: value=22 [sim] resource maxmem: value=33182072832.000000 (30.90 GB) [sim] resource maxswap: value=17179865088.000000 (16.00 GB) [sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) [sim] resource ndisks: value=1 [sim] -- Static Hardware Attributes -- [sim] resource type: value=x86_64-linux-gnu [sim] resource model: value=x86_64-linux-gnu [sim] assigned jobs: 0-15-1789108916-0[0-4]*4 Something wrong in session 98, the count value of slots used is inconsistent with assigned jobs in pc13v? [sim] t=1789109023245433276 (2026-09-11T06:43:43Z) session=98 [sim] event: JOB_UPDATE jobid=0-15-1789108916-0[5] [sim] event: JOB_UPDATE jobid=0-15-1789108916-0[6] [sim] event: WORKERS_PUSH nworkers=1 names=[pc13v] [sim] event: JOB_PUSH njobs=20 jobids=[0-15-1789108916-0[0-9],0-15-1789108940-0[0-9]] [sim] event: WORKERS_PUSH nworkers=1 names=[pc13v] [sim] event: JOB_PUSH njobs=20 jobids=[0-15-1789108916-0[0-9],0-15-1789108940-0[0-9]] [sim] event: JOB_UPDATE jobid=0-15-1789108916-0[9] [sim] event: JOB_UPDATE jobid=0-15-1789108940-0[0] [sim] event: JOB_RUSAGE jobid=0-15-1789108940-0[0] [sim] event: JOB_UPDATE jobid=0-15-1789108940-0[1] [sim] event: JOB_RUSAGE jobid=0-15-1789108940-0[1] [sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[5] state=RUN [sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[6] state=RUN [sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[7] state=RUN [sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[8] state=RUN [sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[9] state=RUN [2026-09-11 06:43:43,245] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[0] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,245] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[1] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[2] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[3] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[4] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[5] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[6] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[7] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[7] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept) [2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[8] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[9] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[0] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[1] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[2] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[3] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[4] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[5] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[6] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[7] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[8] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[9] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[0] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[1] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[2] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[3] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[4] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[5] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[6] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[7] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[7] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept) [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[8] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[8] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept) [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[9] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[0] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[1] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[2] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[3] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[4] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[5] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[6] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[7] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[8] (rc=-17), merging state into existing tracked job instead of discarding [2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[9] (rc=-17), merging state into existing tracked job instead of discarding [sim] DISPATCH jobid=0-15-1789108940-0[2] [sim] DISPATCH jobid=0-15-1789108940-0[3] [sim] DISPATCH jobid=0-15-1789108940-0[4] [sim] DISPATCH jobid=0-15-1789108940-0[5] [sim] PEND jobid=0-15-1789108940-0[6] [sim] PEND jobid=0-15-1789108940-0[7] [sim] PEND jobid=0-15-1789108940-0[8] [sim] PEND jobid=0-15-1789108940-0[9] (sim session=98 t=2026-09-11T06:43:43Z) info worker pc13v [sim] worker pc13v: status=online [sim] -- Consumable/Capacity -- [sim] resource slots: value=22.00 used=22.00 [sim] resource mem: value=20555202560.00 used=35184372088832.00 [sim] resource swp: value=13132.59 used=0.00 [sim] resource tmp: value=576442761216.00 used=0.00 [sim] -- Load Metrics -- [sim] resource r1m: value=0.84 [sim] resource r15m: value=4.24 [sim] resource r15s: value=0.84 [sim] resource ut: value=3.52 [sim] resource pg: value=1752 [sim] resource ls: value=1 [sim] resource it: value=3136.95 [sim] resource io: value=0 [sim] resource cpuf: value=1 [sim] resource maxcpus: value=22 [sim] resource maxmem: value=33182072832.000000 (30.90 GB) [sim] resource maxswap: value=17179865088.000000 (16.00 GB) [sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) [sim] resource ndisks: value=1 [sim] -- Static Hardware Attributes -- [sim] resource type: value=x86_64-linux-gnu [sim] resource model: value=x86_64-linux-gnu [sim] assigned job: jobid=0-15-1789108916-0[5]*4 [sim] assigned job: jobid=0-15-1789108916-0[6]*4 [sim] assigned job: jobid=0-15-1789108916-0[7]*4 [sim] assigned job: jobid=0-15-1789108916-0[8]*4 [sim] assigned job: jobid=0-15-1789108916-0[9]*4 [sim] assigned job: jobid=0-15-1789108940-0[0] [sim] assigned job: jobid=0-15-1789108940-0[1] [sim] assigned job: jobid=0-15-1789108940-0[2] [sim] assigned job: jobid=0-15-1789108940-0[3] [sim] assigned job: jobid=0-15-1789108940-0[4] [sim] assigned job: jobid=0-15-1789108940-0[5] (sim session=98 t=2026-09-11T06:43:54Z) - it seems the scheduler has broken max slots constraint? Investigate why? check next session, the used slots get to 26 (sim session=99 t=2026-09-11T06:43:54Z) info worker pc13v [sim] worker pc13v: status=online [sim] -- Consumable/Capacity -- [sim] resource slots: value=22.00 used=26.00 [sim] resource mem: value=20555202560.00 used=35184372088832.00 [sim] resource swp: value=13132.59 used=0.00 [sim] resource tmp: value=576442761216.00 used=0.00 [sim] -- Load Metrics -- [sim] resource r1m: value=0.84 [sim] resource r15m: value=4.24 [sim] resource r15s: value=0.84 [sim] resource ut: value=3.52 [sim] resource pg: value=1752 [sim] resource ls: value=1 [sim] resource it: value=3136.95 [sim] resource io: value=0 [sim] resource cpuf: value=1 [sim] resource maxcpus: value=22 [sim] resource maxmem: value=33182072832.000000 (30.90 GB) [sim] resource maxswap: value=17179865088.000000 (16.00 GB) [sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) [sim] resource ndisks: value=1 [sim] -- Static Hardware Attributes -- [sim] resource type: value=x86_64-linux-gnu [sim] resource model: value=x86_64-linux-gnu [sim] assigned job: jobid=0-15-1789108916-0[5]*4 [sim] assigned job: jobid=0-15-1789108916-0[6]*4 [sim] assigned job: jobid=0-15-1789108916-0[7]*4 [sim] assigned job: jobid=0-15-1789108916-0[8]*4 [sim] assigned job: jobid=0-15-1789108916-0[9]*4 [sim] assigned job: jobid=0-15-1789108940-0[0] [sim] assigned job: jobid=0-15-1789108940-0[1] [sim] assigned job: jobid=0-15-1789108940-0[2] [sim] assigned job: jobid=0-15-1789108940-0[3] [sim] assigned job: jobid=0-15-1789108940-0[4] [sim] assigned job: jobid=0-15-1789108940-0[5]