[sim] replay file contains 9 sessions
[sim] loaded: 0 jobs, snapshot @ ts=1000 (1970-01-01T00:00:00Z)
[sim] (jobs above reflect this snapshot only -- step/continue to replay later events)
[sim] ready. type 'help' for commands.
[sim] available commands:

Execution Control:
  step [N]                              Advance N sessions (default 1), pause after.
  continue                              Run until next breakpoint hit or EOF.
  quit                                  Exit the scheduler immediately.
  help                                  Print this help listing.

Breakpoints:
  break <EVENT_TYPE> [key=value ...]    Set a breakpoint. Valid event types: CONFIG_PUSH, JOB_PUSH, WORKERS_PUSH, RESOURCES_PUSH, RESVALS_FULL, RESVALS_DELTA, JOB_STATE_CHANGE, JOB_UPDATE, QUEUE_UPDATE, JOB_RUSAGE, BEGIN_ROUND, DECISION.
  info break                            List breakpoints.
  delete <id>|all                       Delete breakpoint(s).

State Inspection:
  info session                          Simulation position.
  info job <jobid>                      Dump one job.
  info jobs                             List all job IDs.
  info queue <name>|all                 Dump queue(s).
  info worker <name>|all                Dump worker(s).
  info resource <name>|all              Dump resource(s).
  info summary                          Quick overview.
  info config                           Show sim parameters.
  info limit <name>|all                 Show limit setting(s) and usage.
  info trace <jobid>                    Re-print a job's latest decision trace.

Snapshot & Parameters:
  snapshot [FILE]                       Write a standalone snapshot.
  set output <FILE>|off                 Start/stop writing decisions.
  set verify on|off                     Toggle decision verification.
  set units raw|human|both              Show resource values raw, human-scaled, or both (default: both).

Trace:
  trace on all [OPTIONS]                Trace every job's decisions.
  trace on job <jobid> ... [OPTIONS]    Trace the listed jobs (id, id[k] or array id).
    level summary|full                  Host detail: counts per position or every host (default: summary).
    lines N                             Line bound per trace, 50-100000 (default: 1000).
    file PATH                           Also append each trace to PATH (last option).
  trace off                             Stop tracing from the next session.
  trace status                          Show trace settings and stored-trace memory.
[sim] info: usage: info <break|session|config|job|jobs|queue|worker|resource|summary|limit|trace> [arg]
[sim] config: output=(active, path unknown) verify=on events_processed=0 current_ts=1000 (1970-01-01T00:00:00Z) sessions=0
[sim] t=1001 (1970-01-01T00:00:00Z) session=1
[sim] event: WORKERS_PUSH nworkers=2 names=[w01,w02]
[sim] event: QUEUE_UPDATE queue=normal
[sim] event: RESVALS_FULL n=1
[sim] event: JOB_PUSH njobs=7 jobids=[0-0-0-101,0-0-0-102,0-0-0-103,0-0-0-104,0-0-0-105,0-0-0-106,0-0-0-107]
[sim] DISPATCH jobid=0-0-0-101
[sim] DISPATCH jobid=0-0-0-102
[sim] DISPATCH jobid=0-0-0-103
[sim] PEND jobid=0-0-0-104
[sim] PEND jobid=0-0-0-105
[sim] PEND jobid=0-0-0-106
[sim] PEND jobid=0-0-0-107
[sim] session: ts=2000 (1970-01-01T00:00:00Z) event_index=4 sessions=1
[sim]   last session: round=2000 dispatch=3 pend=4 preempt=0
[sim] summary: 7 jobs (4 pending, 3 running), 2 workers (2 online), 1 queues, 5 resources
[sim]   next non-dummy session: 5 -- run 'step 4' to reach it
[sim] jobs:
  0-0-0-101                  RUN  normal
  0-0-0-102                  RUN  normal
  0-0-0-103                  RUN  normal
  0-0-0-104                  PEND  normal
  0-0-0-105                  PEND  normal
  0-0-0-106                  PEND  normal
  0-0-0-107                  PEND  normal
[sim] info jobs: 7 job(s)
[sim] job 104 (0-0-0-104): state=PEND queue=normal priority=0 queue_time=1104 resreq=Primary:
  Tasks: min = 1, max = 1
  Rusages:
     res license value 1.000000 per 3 duration -1.000000 decay 0
 allocs=0
[sim] worker w01: status=online
[sim]   -- Consumable/Capacity --
[sim]   resource slots: value=4.00 used=3.00
[sim]   resource mem: value=16384.000000 (16.00 KB) used=2048.000000 (2.00 KB)
[sim]   resource limit=host_slots: value=3.00 used=3.00
[sim]   assigned job: jobid=0-0-0-101*2
[sim]   assigned job: jobid=0-0-0-102
[sim] worker w02: status=online
[sim]   -- Consumable/Capacity --
[sim]   resource slots: value=4.00 used=3.00
[sim]   resource mem: value=16384.000000 (16.00 KB) used=2048.000000 (2.00 KB)
[sim]   resource limit=host_slots: value=3.00 used=3.00
[sim]   assigned job: jobid=0-0-0-101*2
[sim]   assigned job: jobid=0-0-0-103
[sim] resource license: desc= valtype=numeric ordtype=incr flags=GLOBAL,SHARED,CONSUMABLE interval=0
[sim]   value=2.00 used=2.00
[sim] limit: cluster_slots
[sim] USER = all
[sim] QUEUE = all
[sim] PROJECT = all
[sim] HOST = all
[sim] 
[sim] slots = 6.000000
[sim]   usage limit=cluster_slots: accounted=6.00 capacity=6.00
[sim] limit: host_slots
[sim] USER = all
[sim] QUEUE = all
[sim] PROJECT = all
[sim] HOST = all
[sim] 
[sim] slots = 75.000000%
[sim]   note: this limit's value is a percentage of a resource that is not shared across the cluster, so usage for it is tracked separately per host even though the configuration above does not show an explicit host scope
[sim]   usage limit=host_slots host=w01: accounted=3.00 capacity=3.00
[sim]   usage limit=host_slots host=w02: accounted=3.00 capacity=3.00
[sim] units set to raw
[sim] worker w01: status=online
[sim]   -- Consumable/Capacity --
[sim]   resource slots: value=4.00 used=3.00
[sim]   resource mem: value=16384.000000 used=2048.000000
[sim]   resource limit=host_slots: value=3.00 used=3.00
[sim]   assigned job: jobid=0-0-0-101*2
[sim]   assigned job: jobid=0-0-0-102
[sim] units set to human
[sim] worker w01: status=online
[sim]   -- Consumable/Capacity --
[sim]   resource slots: value=4.00 used=3.00
[sim]   resource mem: value=16.00 KB used=2.00 KB
[sim]   resource limit=host_slots: value=3.00 used=3.00
[sim]   assigned job: jobid=0-0-0-101*2
[sim]   assigned job: jobid=0-0-0-102
[sim]   -- Consumable/Capacity --
[sim] resource license: desc= valtype=numeric ordtype=incr flags=GLOBAL,SHARED,CONSUMABLE interval=0
[sim]   value=2.00 used=2.00
[sim] resource limit=cluster_slots: desc=(none) valtype=numeric ordtype=decr flags=GLOBAL,SHARED,RELEASE,CONSUMABLE,LIMIT interval=0
[sim]   value=6.00 used=6.00
[sim] units set to both
[sim] worker w02: status=online
[sim]   -- Consumable/Capacity --
[sim]   resource slots: value=4.00 used=3.00
[sim]   resource mem: value=16384.000000 (16.00 KB) used=2048.000000 (2.00 KB)
[sim]   resource limit=host_slots: value=3.00 used=3.00
[sim]   assigned job: jobid=0-0-0-101*2
[sim]   assigned job: jobid=0-0-0-103
[sim] queue normal: priority=1 status=open RUNLIMIT=n/a MAX_RUNNING=n/a job_pick_plugin=(inherit)
[sim]   queued job: jobid=0-0-0-104 priority=0
[sim]   queued job: jobid=0-0-0-105 priority=0
[sim]   queued job: jobid=0-0-0-106 priority=0
[sim]   queued job: jobid=0-0-0-107 priority=0
[sim] verify set to off
[sim] config: output=(active, path unknown) verify=off events_processed=4 current_ts=2000 (1970-01-01T00:00:00Z) sessions=1
[sim] verify set to on
[sim] snapshot written: snap.replay (ts=2000, 1970-01-01T00:00:00Z)
[sim] output started: set_output.replay
[sim] breakpoint 1 set: DECISION jobid=104 type=DISPATCH
[sim] breakpoints:
[sim]   1: DECISION jobid=104 type=DISPATCH
[sim] round 2-4 (3 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 0 dispatch, 4 pend, 0 preempt -- jobids=0-0-0-104,0-0-0-105,0-0-0-106,0-0-0-107
[sim] hit breakpoint 1 @ decision: DISPATCH jobid=0-0-0-104
[sim] session: ts=6000 (1970-01-01T00:00:00Z) event_index=6 sessions=5
[sim]   last session: round=6000 dispatch=1 pend=0 preempt=0
[sim] job 104 (0-0-0-104): state=RUN queue=normal priority=0 queue_time=1104 resreq=Primary:
  Tasks: min = 1, max = 1
  Rusages:
     res license value 1.000000 per 3 duration -1.000000 decay 0
 allocs=1
[sim] breakpoint 1 deleted
[sim] breakpoint 2 set: BEGIN_ROUND
[sim] round 5-5 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 3 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-104,0-0-0-105,0-0-0-106,0-0-0-107
[sim] hit breakpoint 2 @ BEGIN_ROUND round=7000
[sim] session: ts=6000 (1970-01-01T00:00:00Z) event_index=6 sessions=6
[sim]   last session: round=7000 dispatch=0 pend=0 preempt=0
[sim] summary: 7 jobs (1 pending, 4 running), 2 workers (2 online), 1 queues, 5 resources
[sim]   next non-dummy session: 8 -- run 'step 2' to reach it
[sim] deleted 1 breakpoint(s)
[sim] round 6-7 (2 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 0 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-107
[sim] t=8500 (1970-01-01T00:00:00Z) session=8
[sim] event: JOB_PUSH njobs=1 jobids=[0-0-0-108]
[sim] round 8-8 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 1 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-107,0-0-0-108
[sim] round 9-9 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 0 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-107
[sim] replay complete. 9 session(s) processed.
[sim] output written to: set_output.replay
[sim] simulation finished -- state remains available for inspection (info job/queue/worker/resource/summary). type 'quit' to exit.
[sim] summary: 8 jobs (1 pending, 4 running), 2 workers (2 online), 1 queues, 5 resources
[sim]   next non-dummy session: none remaining
[sim] jobs:
  0-0-0-101                  DONE  normal
  0-0-0-102                  EXIT  normal
  0-0-0-103                  DONE  normal
  0-0-0-104                  RUN  normal
  0-0-0-105                  RUN  normal
  0-0-0-106                  RUN  normal
  0-0-0-107                  PEND  normal
  0-0-0-108                  RUN  normal
[sim] info jobs: 8 job(s)
[sim] config: output=set_output.replay verify=on events_processed=8 current_ts=10000 (1970-01-01T00:00:00Z) sessions=9
[sim] output stopped: set_output.replay (4 event(s) written)
