lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore(wolf): update buglog and anatomy/memory index

Browse source 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
This commit is contained in:
Rongsong Shen 2026-09-07 07:32:25 +00:00
parent 0d3f925c4d
commit 37ea6dfd9f
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
4 changed files with 857 additions and 65 deletions
Download patch file Download diff file Expand all files Collapse all files

83
.wolf/buglog.json
View file

@ -20703,22 +20703,24 @@
"timestamp": "2026-09-02T03:47:00Z",
"error_message": "Live simulation run (xce_scheduler --sim, user-observed, not reproduced by me in this session) shows a JOBS_PUSH batch of 50 jobs where the first 22 (indices [0]-[21]) receive [sim] DISPATCH decisions, immediately followed by [sim] PEND decisions for ALL 50 jobs (indices [0]-[49]) -- including the 22 that had JUST been dispatched. Design expectation (confirmed correct, see root_cause): a job that has received a DISPATCH decision must be treated as RUN immediately, without waiting for any downstream job-start acknowledgment, and must never be reconsidered as a fresh scheduling candidate afterward -- in BOTH the live scheduler and simulation/replay mode.",
"file": "src/scheduler/src/sim_loop.c, src/scheduler/src/plugins/builtin_fcfs.c, src/scheduler/src/scheduler.c",
"root_cause": "NOT YET ROOT-CAUSED -- logged from a live-observed symptom, investigated only via static code reading this session (no live repro attempted), per explicit instruction to log and defer rather than context-switch into a full debug pass. Confirmed via source read: sched_commit_decision() (scheduler.c:1435-1455) DOES set job->state = JOB_STAT_RUN synchronously at the moment a DISPATCH decision is committed, with no wait for any job-start ack -- this is shared code (sched_allocator()/sched_commit_decision()) used by BOTH the normal-mode driver and the sim driver alike (scheduler/CLAUDE.md: \"Both drive the identical core session/decision-callback protocol\"), so the intended \"dispatch implies RUN, immediately\" design IS mechanically implemented at this layer already -- this is not a missing feature. builtin_fcfs.c's on_dispatch() carries its own documented, pre-existing concurrency-hazard comment (\"this runs in the allocator thread and can race the picker thread's next_job() -- jbuf's capacity-1 handoff unblocks the picker for job N+1 before this call for job N returns\"), guarded by fcfs_plugin_data_t.lock for candidate-list mutations, but reading the code alone does not confirm whether that race (or something else -- e.g. two back-to-back sim sessions/rounds failing to preserve job->state or the picker's candidate set between them) is what produces ALL 50 jobs reappearing in one PEND batch, including the 22 already-dispatched ones. Related prior findings: bug-1567 (fixed 2026-08-17) documented that simulation mode ONCE unconditionally PENDed every candidate with no real dispatch at all, deliberately deferred to a then-not-yet-designed scheduler-allocation spec -- the pasted live log now shows genuine DISPATCH decisions in sim mode, meaning that gap has since been closed (scheduler-sim-decision-parity, referenced by bug-1623, 2026-08-22, looks like the likely spec, not yet confirmed which task/commit). bug-1623 also documents sim_loop.c's sim_reconcile_unaccounted_jobs() logic (per-round decoded-decision matching for RUN-state unaccounted jobs) as a separate, related mechanism in the same file worth checking during a real debug pass.",
"fix": "NOT FIXED -- deliberately deferred per explicit instruction (log and continue other work, do not context-switch into a full debug pass now). Needs a dedicated investigation (e.g. /kiro-debug) scoped to sim_loop.c's session/round-driving loop to determine: (a) whether this is one scheduling session emitting a genuine duplicate/second decision for already-dispatched jobs (a real bug in the shared allocator/picker pipeline or in sim_loop.c's own session driving), or two distinct back-to-back sessions where job->state / the picker's candidate set was not correctly carried over between them; (b) whether the identical scenario is reproducible against the NORMAL-mode driver (scheduler.c/driver.c), which would point at the shared sched_allocator/builtin_fcfs code rather than anything sim-specific, or whether it only reproduces in --sim, which would point at sim_loop.c's own session/round handling (possibly interacting with bug-1623's sim_reconcile_unaccounted_jobs()). A live, controlled repro (fixed job count, fixed resource limits, single JOBS_PUSH batch, --sim-verify or a new targeted unit test) should be built first rather than relying on the user's own live-run log.",
"root_cause": "ROOT-CAUSED FOR REAL (2026-09-07), and turns out to be a SPEC CONFLICT, not a logic bug -- a patch was attempted and REVERTED after it broke a pre-existing, passing regression test. Confirmed mechanism: sim mode's REAL DISPATCH decisions come entirely from sim_reconcile_unaccounted_jobs() (sim_loop.c), which reconstructs a job's dispatch PURELY from replay data (populating job->allocs/job->accounts, reporting a synthesized DISPATCH decision) but never calls picker->picker_iface->on_dispatch() -- unlike the real/live allocator path (sched_allocator(), scheduler.c line 507-510), which ALWAYS calls on_dispatch() to retire the job from the picker's own candidate list (fcfs_queue_t.pendJobs) before building the decision. Because reconciliation skips this, the job stays a live candidate in the picker's bookkeeping. sim_picker_thread_fn() (sim_loop.c) -- a SEPARATE, always-running thread that walks next_job() every round and unconditionally PENDs whatever it is still offered, since it has no real resource-matching allocation of its own -- keeps re-offering and re-PENDing that job every subsequent round until an explicit JOB_STATE_CHANGE-to-RUN event (also replayed from the original recording, can legitimately arrive several rounds later) removes it from pendJobs via builtin_fcfs_on_job_state_change(). CRITICAL DISCOVERY: this exact behavior is NOT accidental -- it is DOCUMENTED, INTENTIONAL, and covered by an existing passing test, test_sim_verify_no_false_mismatch_across_round_separated_dispatch (test_sim_verify.c, scheduler-sim-eager-reconciliation spec task 3.2), whose own header comment states verbatim: 'sim_reconcile_unaccounted_jobs() captures job->accounts but never calls picker_iface->on_dispatch() to retire the job from the picker's own internal pendJobs (design.md's Non-Goals/Revalidation Triggers explicitly leave live-picker DISPATCH emission to a future scheduler-allocation spec)'. That spec's own design.md (.kiro/specs/scheduler-sim-eager-reconciliation/design.md, Non-Goals section) explicitly lists 'Any change to simulation mode's live picker (still PEND-only for freshly-arriving jobs, per scheduler-allocation)' as OUT OF SCOPE. A fix attempt (calling on_dispatch() from the reconciliation FOUND branch, mirroring sched_allocator()'s exact ordering) was implemented with a new TDD test, confirmed the bug reproduces deterministically (RED), and confirmed the fix resolves it (GREEN) -- but broke test_sim_verify_no_false_mismatch_across_round_separated_dispatch, whose fixture explicitly bakes in 'PEND(jobid) recorded in the SAME round as its own DISPATCH, and again for every round until the RUN confirmation arrives' as the CORRECT, expected --sim-verify-clean shape. The fix was REVERTED (both the sim_loop.c change and its new test) rather than kept, since it silently violates an already-agreed, tested design contract from a sibling spec.",
"fix": "FIXED FOR REAL (2026-09-07), after a first fix attempt was correctly reverted (see prior root_cause note) upon discovering it broke test_sim_verify_no_false_mismatch_across_round_separated_dispatch. That test's OWN fixture turned out to be baking in the bug's exact symptom as 'expected' -- but its parent spec's own design.md ('Integration with --sim-verify') already documented the CORRECT assumption ('the picker never re-decides an already-resolved job'), which the implementation had silently drifted from. This was a genuine implementation/test drift from an already-agreed design intent, not a live spec conflict requiring a product decision -- confirmed by re-reading both scheduler-sim-decision-parity's and scheduler-sim-eager-reconciliation's own design.md files (both updated in place with 2026-09-07 correction notes explaining this). Fix: sim_reconcile_unaccounted_jobs() (sim_loop.c) now resolves the job picker and calls picker_iface->on_dispatch() immediately after reconstructing job->allocs/job->accounts in its FOUND branch, mirroring sched_allocator()'s own live-dispatch ordering (scheduler.c line 507-515) exactly. This retires the reconciled job from the picker's own pendJobs bookkeeping the moment its dispatch is reconstructed, so sim_picker_thread_fn()'s unconditional-PEND thread never re-offers/re-PENDs it on a later round. TDD: new test test_reconcile_removes_reconciled_job_from_picker_candidates (test_sim_reconcile_unaccounted.c) directly proves a job is no longer offered by next_job() after reconciliation, using sim_bootstrap_picker() to register a real, resolvable builtin_fcfs instance -- confirmed RED before the fix, GREEN after. test_sim_verify_no_false_mismatch_across_round_separated_dispatch (test_sim_verify.c) was updated to match the corrected behavior: round 1 now records DISPATCH-only (new helper append_replay_session_with_dispatch(), no accompanying PEND), and rounds 2-3 are now empty ReplaySessions (matching round 4's own existing pattern) instead of baking in a stale re-PEND echo as expected. Full scheduler suite: 76/77 pass (same pre-existing unrelated test_core_events.c failure, bug-1762, untouched by this fix).",
"tags": [
"builtin_fcfs",
"discovered-live",
"dispatch",
"dispatch-implies-run",
"fixed",
"job-state",
"needs-debug",
"pend",
"race-condition-suspected",
"scheduler",
"scheduler-allocation",
"sim-mode",
"sim_loop-c",
"builtin_fcfs",
"dispatch",
"pend",
"job-state",
"dispatch-implies-run",
"race-condition-suspected",
"needs-debug",
"discovered-live",
"user-reported",
"not-fixed"
"spec-conflict",
"user-reported"
],
"related_bugs": [
"bug-1567",
@ -21124,6 +21126,63 @@
],
"occurrences": 1,
"last_seen": "2026-09-06T21:10:00Z"
},
{
"id": "bug-1778",
"timestamp": "2026-09-07T06:50:00Z",
"error_message": "Live-observed (interactive sim session=29, replaying a real captured trace): three plain (non-array) jobs already at JOB_STAT_DONE (0-14-1788751409-0, 0-14-1788751435-0, 0-14-1788751504-0) still appear in `info worker pc13v`'s 'assigned jobs' listing (each annotated '*2' via bug-1776's slot-count fix), alongside the genuinely-running array job elements. User: 'The finished job(exit/done) should be de-assign from worker.'",
"file": "src/scheduler/src/repl_info.c",
"root_cause": "PARTIALLY CONFIRMED, NOT FULLY ROOT-CAUSED this session (user asked to record, not fix). Confirmed facts: (1) repl_info_worker_one()'s assigned-jobs loop (repl_info.c, ~line 686) gates solely on `job->allocs != NULL`, with no independent job->state check -- if allocs is non-NULL for a terminal job, it will always display, by design (mirrors the exact structural gap bug-1764 already documented for this same function). (2) The worker's ACTUAL resource accounting is NOT confused by this: the same session's 'resource slots: value=22.00 used=20.00' correctly reflects only the 5 genuinely-running 4-slot array elements (5*4=20), excluding the 3 DONE jobs entirely -- proving core.resmgr/resInUse (the real admission-control state) is fine; only the assigned-jobs DISPLAY loop (which walks job->allocs directly, a separate field) is stale for these 3 jobs. (3) This is NOT a regression of bug-1764's fix: verified sim_loop.c:713-714 still correctly excludes JOB_IS_DONE(job->state)||JOB_IS_EXIT(job->state) from sim_reconcile_unaccounted_jobs()'s resurrection path, so that specific documented mechanism is not what's reintroducing these allocs. (4) core_events.c's XCE_CORE_EVENT_JOB_STATE_CHANGE (line ~1155-1163) and XCE_CORE_EVENT_JOB_UPDATE (line ~1207-1215) handlers both correctly free_JobAlloc_ptr()+null job->allocs atomically alongside the state write whenever `!JOB_IS_RUN(new_state)` -- so if either of THESE handlers was what drove these 3 jobs to DONE, allocs should already be NULL; something else must be true. Leading unconfirmed hypothesis: snapshot_load.c:490-493 deep-copies wj->allocs from the .replay snapshot's wire payload UNCONDITIONALLY whenever wj->allocs != NULL, with no check against the snapshot's own wj->state -- if the live production recording's snapshot-capture moment (snapshot_wire.c's save side) captured these 3 jobs' allocs and state in a mutually inconsistent way (e.g. a save-time field-read ordering gap, or the snapshot genuinely predates their JOB_STATE_CHANGE/JOB_UPDATE terminal cleanup while a separately-recorded later event updated state to DONE without a matching allocs-clearing event reaching this replay stream), the replay would faithfully reproduce that inconsistency verbatim. NOT verified with a live trace/gdb this session -- flagged per this project's own persistence.md field-completeness concern (a state that 'recovers correctly' while an unrelated field goes stale is the exact class of bug that steering calls out), not asserted as confirmed.",
"fix": "INTERIM DISPLAY-LAYER MITIGATION APPLIED (2026-09-07, per user request). repl_info_worker_one()'s assigned-jobs loop (repl_info.c) now also excludes JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state), in addition to the existing job->allocs != NULL check -- mirrors the same 'terminal is permanent' guard job_state_regresses_terminal()/sim_reconcile_unaccounted_jobs() already apply elsewhere. TDD: new test test_info_worker_assigned_job_listing_excludes_terminal_job_despite_stale_allocs (test_repl_resource_table.c) directly assigns the SAME allocs struct to a DONE job and a RUN job on the same worker, forcing the DONE job's state after assigning allocs (reproducing the exact state/allocs inconsistency without needing to root-cause how it arises live) -- confirms the DONE job is excluded while the RUN job still shows. Full scheduler suite: 76/77 pass (same pre-existing unrelated test_core_events.c failure as before, bug-1762). NOTE: this masks the symptom at the display layer only -- the underlying question of WHY job->allocs stays non-NULL for a terminal job (suspected snapshot save/load inconsistency) remains unresolved; a real fix should also prevent/clear the stale allocs at its source.",
"tags": [
"job-lifecycle",
"stale-state",
"repl",
"snapshot",
"scheduler",
"related-to-bug-1776",
"assigned-jobs",
"persistence",
"needs-live-repro",
"related-to-bug-1764",
"interim-fix",
"sim",
"fixed"
],
"related_bugs": [
"bug-1764",
"bug-1776"
],
"occurrences": 1,
"last_seen": "2026-09-07T06:50:00Z"
},
{
"id": "bug-1779",
"timestamp": "2026-09-07T06:58:00Z",
"error_message": "Live-observed (interactive sim session=19, replaying a real captured trace, 'step 9'): within the SAME batch of scheduler decisions, jobs 0-12-1788762745-0[0..4] receive a DISPATCH decision, then immediately ALSO receive a PEND decision in the same step's output -- and `info jobs` afterward shows all 10 array elements (0-12-1788762745-0[0..9]) as PEND, including the 5 that were just dispatched. User: 'when a job has been dispatched (in scheduler), we should treat it as RUN, so it should be excluded from PEND ... we need to remove 0-12-1788762745-0[0-4] from PEND but show in RUN.'",
"file": "src/scheduler/src/sim_loop.c",
"root_cause": "ROOT-CAUSED FOR REAL (2026-09-07), and turns out to be a SPEC CONFLICT, not a logic bug -- a patch was attempted and REVERTED after it broke a pre-existing, passing regression test. Confirmed mechanism: sim mode's REAL DISPATCH decisions come entirely from sim_reconcile_unaccounted_jobs() (sim_loop.c), which reconstructs a job's dispatch PURELY from replay data (populating job->allocs/job->accounts, reporting a synthesized DISPATCH decision) but never calls picker->picker_iface->on_dispatch() -- unlike the real/live allocator path (sched_allocator(), scheduler.c line 507-510), which ALWAYS calls on_dispatch() to retire the job from the picker's own candidate list (fcfs_queue_t.pendJobs) before building the decision. Because reconciliation skips this, the job stays a live candidate in the picker's bookkeeping. sim_picker_thread_fn() (sim_loop.c) -- a SEPARATE, always-running thread that walks next_job() every round and unconditionally PENDs whatever it is still offered, since it has no real resource-matching allocation of its own -- keeps re-offering and re-PENDing that job every subsequent round until an explicit JOB_STATE_CHANGE-to-RUN event (also replayed from the original recording, can legitimately arrive several rounds later) removes it from pendJobs via builtin_fcfs_on_job_state_change(). CRITICAL DISCOVERY: this exact behavior is NOT accidental -- it is DOCUMENTED, INTENTIONAL, and covered by an existing passing test, test_sim_verify_no_false_mismatch_across_round_separated_dispatch (test_sim_verify.c, scheduler-sim-eager-reconciliation spec task 3.2), whose own header comment states verbatim: 'sim_reconcile_unaccounted_jobs() captures job->accounts but never calls picker_iface->on_dispatch() to retire the job from the picker's own internal pendJobs (design.md's Non-Goals/Revalidation Triggers explicitly leave live-picker DISPATCH emission to a future scheduler-allocation spec)'. That spec's own design.md (.kiro/specs/scheduler-sim-eager-reconciliation/design.md, Non-Goals section) explicitly lists 'Any change to simulation mode's live picker (still PEND-only for freshly-arriving jobs, per scheduler-allocation)' as OUT OF SCOPE. A fix attempt (calling on_dispatch() from the reconciliation FOUND branch, mirroring sched_allocator()'s exact ordering) was implemented with a new TDD test, confirmed the bug reproduces deterministically (RED), and confirmed the fix resolves it (GREEN) -- but broke test_sim_verify_no_false_mismatch_across_round_separated_dispatch, whose fixture explicitly bakes in 'PEND(jobid) recorded in the SAME round as its own DISPATCH, and again for every round until the RUN confirmation arrives' as the CORRECT, expected --sim-verify-clean shape. The fix was REVERTED (both the sim_loop.c change and its new test) rather than kept, since it silently violates an already-agreed, tested design contract from a sibling spec.",
"fix": "FIXED FOR REAL (2026-09-07), after a first fix attempt was correctly reverted (see prior root_cause note) upon discovering it broke test_sim_verify_no_false_mismatch_across_round_separated_dispatch. That test's OWN fixture turned out to be baking in the bug's exact symptom as 'expected' -- but its parent spec's own design.md ('Integration with --sim-verify') already documented the CORRECT assumption ('the picker never re-decides an already-resolved job'), which the implementation had silently drifted from. This was a genuine implementation/test drift from an already-agreed design intent, not a live spec conflict requiring a product decision -- confirmed by re-reading both scheduler-sim-decision-parity's and scheduler-sim-eager-reconciliation's own design.md files (both updated in place with 2026-09-07 correction notes explaining this). Fix: sim_reconcile_unaccounted_jobs() (sim_loop.c) now resolves the job picker and calls picker_iface->on_dispatch() immediately after reconstructing job->allocs/job->accounts in its FOUND branch, mirroring sched_allocator()'s own live-dispatch ordering (scheduler.c line 507-515) exactly. This retires the reconciled job from the picker's own pendJobs bookkeeping the moment its dispatch is reconstructed, so sim_picker_thread_fn()'s unconditional-PEND thread never re-offers/re-PENDs it on a later round. TDD: new test test_reconcile_removes_reconciled_job_from_picker_candidates (test_sim_reconcile_unaccounted.c) directly proves a job is no longer offered by next_job() after reconciliation, using sim_bootstrap_picker() to register a real, resolvable builtin_fcfs instance -- confirmed RED before the fix, GREEN after. test_sim_verify_no_false_mismatch_across_round_separated_dispatch (test_sim_verify.c) was updated to match the corrected behavior: round 1 now records DISPATCH-only (new helper append_replay_session_with_dispatch(), no accompanying PEND), and rounds 2-3 are now empty ReplaySessions (matching round 4's own existing pattern) instead of baking in a stale re-PEND echo as expected. Full scheduler suite: 76/77 pass (same pre-existing unrelated test_core_events.c failure, bug-1762, untouched by this fix).",
"tags": [
"array-job",
"builtin_fcfs",
"decision",
"dispatch",
"fixed",
"job-lifecycle",
"needs-live-repro",
"pend",
"picker",
"scheduler",
"scheduler-allocation",
"sim",
"spec-conflict"
],
"related_bugs": [
"bug-1778"
],
"occurrences": 1,
"last_seen": "2026-09-07T06:58:00Z"
}
]
}