The prior verification pass (commit 1c9cfc1, same session) restarted
controller+scheduler+worker but left the apiserver running, so it only
ever saw the apiserver's stale, still-correct in-memory job cache from
before the restart -- never the controller's actual post-recovery
state. Repeating the test with the apiserver also restarted (matching
the original report's "restart all daemons" exactly) reproduces the
bug immediately: EXEC_HOST stays permanently blank across 15 polls/30s
while STATUS correctly shows RUN throughout.
Root cause fully traced: dispatch_a_job_() (job.c:1343-1372) sets
job.allocs in memory at dispatch-commit time but the durable
WAL_JOB_DISPATCH fact (persist_submit_job_fact, wal_jobstate_t
{jobid,state}) never carries the allocation. On a full restart,
restore_job() replays only the pre-dispatch WAL_JOB_SUBMIT payload
(allocs empty by construction) and apply_committed_entry()'s
WAL_JOB_DISPATCH case routes through apply_job_update_transition(),
which by design "only ever touches state" -- job.allocs is never
reconstructed. Controller log corroborates: recovery trace shows the
WAL_JOB_DISPATCH replay applying silently (no log on the APPLIED
path), followed by each job's WAL_JOB_START fact correctly logging a
now-redundant "duplicate UPDATE ... no-op".
Fix direction confirmed with the user: WAL_JOB_DISPATCH's payload
needs to carry the scheduler's resource allocation, not just
{jobid, state}, with a corresponding decode/apply path in
apply_committed_entry() -- not implemented here, this is a
reproduce-and-root-cause pass only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Live-reproduced and root-caused on branch scheduler-framework via a
single-node deploy:
- bug-1673 (gitea #40): apiserver hosts-list MAX_SLOT stuck at 0 for
workers not exact-name-matched in hosts.yaml -- display-only, C-side
scheduling correctly uses the wildcard 'default' entry's slot count.
Reporter's secondary "hard 8-job cap" claim did NOT reproduce.
- bug-1674 (gitea #41): `xcecli jobs list -l` fails outright (not just
missing timestamp fields as reported) -- two call sites validate
field lists against JobSubmitRequest instead of JobInfo.
- bug-1675 (gitea #42): connection-refused right after apiserver
startup surfaces a raw OS error string instead of a friendly message.
- bug-1676 (gitea #44): FROM_HOST always blank -- capnp schema has no
from_host field, and OnJobUpdated's blind store overwrite wipes the
apiserver-local value set at creation on the first controller push.
Gitea issue #43 (EXEC_HOST lost after daemon restart) was NOT
reproduced after two restart cycles with WAL-preserving state --
EXEC_HOST/state recovered correctly in every poll. Not logged as a bug;
a related design gap (WAL_JOB_DISPATCH only carries {jobid, state}, not
allocs) was flagged as an unconfirmed latent risk, not a live finding.
bug-1677: scripts/gitea-sync.py's sync_bugs() reads buglog.json as a
bare list but the real format is the {version, bugs: [...]} wrapper --
would crash on any real run. Found via static review while checking
the gitea_issue cross-reference field convention; not executed live.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
39 duplicate IDs (one tripled: bug-909) in the bug-875-bug-949 range,
where a restarted counter re-used IDs already held by June 21-22
entries. Kept each original (earliest timestamp) untouched and
renumbered the 40 later collisions to bug-1633..bug-1672 (new global
max, up from 1632), preserving array order. Verified no related_bugs
reference pointed at a renumbered entry -- the 10 existing references
to bug-924 all predate the Aug 17 collision and correctly resolve to
the original ENOSPC/overlayfs entry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Committed at 90be04bd. Dashboard is LAN-only and never exposed
externally, so no history scrub -- just stop tracking the token file
so it's not re-added on the next wolf bookkeeping commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Elevates the normal-mode/simulation-mode driver equivalence guarantee
from a one-line architectural description in CLAUDE.md to an enforced
steering constraint (scheduler-driver-equivalence.md), backed by
bug-1611/bug-1613 as concrete evidence of violation. Scopes the fix as
its own spec (scheduler-sim-decision-parity, requirements + design)
rather than reopening the closed scheduler-core spec or folding it
into scheduler-sim-repl-verbosity's explicitly narrower boundary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Elevates the normal-mode/simulation-mode driver equivalence guarantee
from a one-line architectural description in CLAUDE.md to an enforced
steering constraint (scheduler-driver-equivalence.md), backed by
bug-1611/bug-1613 as concrete evidence of violation. Scopes the fix as
its own spec (scheduler-sim-decision-parity, requirements + design)
rather than reopening the closed scheduler-core spec or folding it
into scheduler-sim-repl-verbosity's explicitly narrower boundary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- .claude/settings.json: worktree.baseRef = "head"
- .kiro/specs/scheduler-core/.solomd: session tracking state
- .wolf/{anatomy,cerebrum,memory,token-ledger}.json, hooks/_session.json:
accumulated OpenWolf bookkeeping (includes this session's fmt_jobid work)
- .gitignore: ignore Python __pycache__/*.pyc (scripts/gitea-sync.py) and
.agent-shell/ (session transcripts can capture raw env dumps/secrets --
one already had a live-looking ANTHROPIC_API_KEY in a captured `env`
output; excluded from this commit entirely, never checked in)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Worker uses plain getopt_long(), not argp, so it has no free --version
handling like controller/scheduler. Adds a long-only OPT_VERSION=257
option and a handling case that computes xce_compose_version() into a
static buffer and returns before any startup work (check_user()/
xcecd_init/xcecd_start). Both startup INFO log branches (--peers and
no-peers) extended to log the identical buffer.
Live-verified: --version and both startup-log branches are
byte-identical (captured via a real daemon run under sudo).
Also updates bug-1519 (pre-existing, unrelated ASAN "Joining already
joined thread" abort during worker shutdown): independently reproduced
under real root during this task's review, refining its prior "never
reproduced under real root" note -- not a userns/unshare artifact, still
unfixed, does not affect this task's code path.
Task 3.3/13 of build-time-versioning. This closes out the "Core: C
daemon version display" phase (tasks 3.1-3.3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
repl_print_event_summary() (interactive REPL event echo) now prefixes
each echoed event with "[sim] t=<ns> (<human>) session=<N>", using the
same xce_core_now()/repl_format_ts_human()/session_count sources "info
session" already reports. Uses session_count+1, not the raw count:
events stream in and are echoed individually before the ReplaySession/
begin_round message that increments session_count is even read (file
order is event, event, ..., ReplaySession), so an echoed event always
belongs to the session about to begin.
Add sim_prescan_session_count() (sim.c): scans the replay file from the
current position via shape-only classification (mirroring sim_loop.c's
own ReplayEvent/ReplaySession/SnapshotEntry dispatch), counts
ReplaySession messages, then rewinds via fseek back to the exact ftell()
position -- same rewind technique sim_locate_and_load_snapshot() already
uses. Wired into sched_sim_driver_start(): prints "[sim] replay file
contains N sessions" once at load, non-fatal on scan failure.
Add test_replay_load_reports_total_session_count; extend
test_step_echoes_events_before_session_decisions with t=/session=1
assertions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apply_committed_entry()'s shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/
CKPT/PREASON dispatch case only ever called the generic, state-only
apply_job_update_transition(j, state) -- start_time/finish_time/run_time
were never touched, so `xcecli joblist` always showed StartTime=0 for
RUN/DONE/EXIT jobs and FinishTime=0 for DONE jobs (EXIT jobs got a
finish_time only via the separate, richer WAL_JOB_FAIL dispatch path).
Fix is additive: stamp time(NULL) keyed on wal_type right after
apply_job_update_transition() succeeds, still under the same
lock(j, RTTI_LOCK_WRITE) -- WAL_JOB_START sets start_time, WAL_JOB_FINISH
sets finish_time and derives run_time. No change to
apply_job_update_transition()'s signature or the dispatch structure.
Add test_ace_job_start_sets_start_time and
test_ace_job_finish_sets_finish_time_and_run_time to
test_apply_committed_entry.c, following the existing tace_seed_job
fixture pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
