chore(wolf): checkpoint buglog.json (1062 bugs)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg

.wolf/buglog.json

@@ -17963,7 +17963,7 @@
       "error_message": "Discovered analyzing a real 108-session replay (xce_scheduler -i -s scheduler-replay-20260820-070103.replay, continue to EOF): sim_print_decision() (sim_loop.c) never printed a single DISPATCH or PREEMPT line across the entire replay -- only PEND -- even though 22 of the 30 jobs pushed at round 32 clearly transitioned to running (confirmed by ongoing JOB_UPDATE/JOB_RUSAGE ticks for jobids [0]-[21] every few rounds through round 107). An operator watching only decision output has zero confirmation those 22 jobs ever dispatched; they just silently stop appearing in PEND.",
       "file": "src/scheduler/src/sim_loop.c",
       "root_cause": "Not yet root-caused with certainty. Plausible, benign explanation: in --sim/-s REPLAY mode, jobs already resolved (dispatched) in the ORIGINAL recorded run may be replayed purely as state deltas (JOB_UPDATE/JOB_RUSAGE) without the live picker/allocator re-deriving a fresh XCE_DECISION_DISPATCH -- only jobs still outstanding (PEND) get freshly re-decided every round, which is why PEND is the only decision op ever observed. Unconfirmed against sim_loop.c's actual replay-vs-live-decide branching; needs a source read of how sim_loop.c distinguishes replayed-decision playback from freshly-computed decisions before concluding this is 'working as designed' vs a real visibility gap.",
-      "fix": "Not yet fixed. If confirmed benign (dispatch legitimately has no fresh decision object in replay mode), the round-summary rollup being designed for bug-1609/1610 should not label its dispatch count misleadingly (a permanent 'X dispatch' column reading 0 for an entire replay looks like a bug even when it isn't) -- consider deriving a running-job delta from info-jobs-equivalent state instead of the decision tally for replay-mode display. If NOT benign (a real decision-visibility gap), this is a separate, more serious finding than the formatting issues in bug-1609/1610.\n\n--- STATUS UPDATE 2026-08-24 (xce-audit-invariants scheduler audit) ---\n\nRoot cause confirmed and the literal mechanism is now built: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709, landed via scheduler-sim-decision-parity tasks 2.1-3, 2026-08-22) reconstructs and reports a decision for a RUN-state job whose DISPATCH decision was only in the recorded replay stream. BUT its trigger gate (sim_loop.c:592) requires JOB_IS_RUN(job->state) before searching, and the search only scans the CURRENT round's decoded_session (decoded fresh at sim_loop.c:871, freed at line 999 -- no cross-round persistence). So reconciliation only fires when a job's DISPATCH decision and its RUN confirmation land in the exact same replay round. This bug's own original scenario (dispatch at round 32, RUN confirmation through round 107) never matches -- the round-32 DISPATCH decision is gone by the time the job reaches RUN state. The scheduler-sim-decision-parity spec's own task 4.3 implementation notes explicitly acknowledge this residual does NOT fix bug-1611's dominant real-world manifestation. A follow-up spec already exists for this exact residual: .kiro/specs/scheduler-sim-eager-reconciliation/ (created 2026-08-22, phase=requirements-generated, ready_for_implementation=false -- design/tasks not yet generated). Its diagnosis: decouple allocation CAPTURE from the JOB_IS_RUN gate (capture as soon as a DISPATCH decision is seen for a job with job->accounts == NULL, regardless of state), leaving sched_account_jobs()'s existing JOB_IS_RUN + job->accounts != NULL gate to keep deferring actual charging until state transitions to RUN. Fix location: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709), trigger gate at line 592, plus its ownership-transfer logic (609-693) needs a double-capture guard since capture would no longer imply state==RUN. Do NOT fix inline as a drive-by patch per scheduler-driver-equivalence.md's testing discipline -- promote scheduler-sim-eager-reconciliation to design/tasks instead.",
+      "fix": "Not yet fixed. If confirmed benign (dispatch legitimately has no fresh decision object in replay mode), the round-summary rollup being designed for bug-1609/1610 should not label its dispatch count misleadingly (a permanent 'X dispatch' column reading 0 for an entire replay looks like a bug even when it isn't) -- consider deriving a running-job delta from info-jobs-equivalent state instead of the decision tally for replay-mode display. If NOT benign (a real decision-visibility gap), this is a separate, more serious finding than the formatting issues in bug-1609/1610.\n\n--- STATUS UPDATE 2026-08-24 (xce-audit-invariants scheduler audit) ---\n\nRoot cause confirmed and the literal mechanism is now built: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709, landed via scheduler-sim-decision-parity tasks 2.1-3, 2026-08-22) reconstructs and reports a decision for a RUN-state job whose DISPATCH decision was only in the recorded replay stream. BUT its trigger gate (sim_loop.c:592) requires JOB_IS_RUN(job->state) before searching, and the search only scans the CURRENT round's decoded_session (decoded fresh at sim_loop.c:871, freed at line 999 -- no cross-round persistence). So reconciliation only fires when a job's DISPATCH decision and its RUN confirmation land in the exact same replay round. This bug's own original scenario (dispatch at round 32, RUN confirmation through round 107) never matches -- the round-32 DISPATCH decision is gone by the time the job reaches RUN state. The scheduler-sim-decision-parity spec's own task 4.3 implementation notes explicitly acknowledge this residual does NOT fix bug-1611's dominant real-world manifestation. A follow-up spec already exists for this exact residual: .kiro/specs/scheduler-sim-eager-reconciliation/ (created 2026-08-22, phase=requirements-generated, ready_for_implementation=false -- design/tasks not yet generated). Its diagnosis: decouple allocation CAPTURE from the JOB_IS_RUN gate (capture as soon as a DISPATCH decision is seen for a job with job->accounts == NULL, regardless of state), leaving sched_account_jobs()'s existing JOB_IS_RUN + job->accounts != NULL gate to keep deferring actual charging until state transitions to RUN. Fix location: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709), trigger gate at line 592, plus its ownership-transfer logic (609-693) needs a double-capture guard since capture would no longer imply state==RUN. Do NOT fix inline as a drive-by patch per scheduler-driver-equivalence.md's testing discipline -- promote scheduler-sim-eager-reconciliation to design/tasks instead.\n\n--- STATUS UPDATE 2026-08-27 (scheduler-sim-eager-reconciliation task 1.1) ---\n\nCore production fix landed: sim_reconcile_unaccounted_jobs() trigger gate (sim_loop.c:592) had !JOB_IS_RUN(job->state) removed, leaving job->accounts == NULL as the sole capture-eligibility gate -- exactly the diagnosis from the 2026-08-24 status update. Match/capture/report mechanics (609-693) were confirmed already safe for early (pre-RUN) capture without changes (see scheduler-sim-eager-reconciliation/research.md's uncharge-safety research log -- sched_uncharge_account() has zero production callers, accounting is reset-and-resum not incremental, so no double-capture guard beyond the existing job->accounts != NULL check was needed). Old test test_reconcile_skips_non_run_state_job (which asserted the bug as correct behavior) replaced with test_reconcile_captures_non_run_state_job_when_dispatch_recorded (inverted expectation). RED/GREEN verified: new test failed under the old gate (job->allocs stayed NULL, confirmed via a real build+run before the fix), then passed after the one-line fix; full sched_sim_reconcile_unaccounted_test suite green (10/10, all prior regression tests unchanged). This is task 1.1 of the spec only -- task 1.2 (capture-safety unit coverage), task 2.1 (resInUse deferral integration test), and task 3.x (contention/--sim-verify/full-regression validation) are still pending before the spec itself is implementation-complete.",
       "tags": [
         "repl",
         "sim",
@@ -17977,8 +17977,8 @@
         "bug-1609",
         "bug-1610"
       ],
-      "occurrences": 2,
-      "last_seen": "2026-08-24T21:30:00Z"
+      "occurrences": 3,
+      "last_seen": "2026-08-27T03:35:00Z"
     },
     {
       "id": "bug-1612",
@@ -19563,6 +19563,108 @@
       "related_bugs": [],
       "occurrences": 1,
       "last_seen": "2026-08-26T14:25:00Z"
+    },
+    {
+      "id": "bug-1717",
+      "timestamp": "2026-08-26T00:00:00Z",
+      "error_message": "ASAN heap-use-after-free abort in controller_unit_test: commit_resolve_pending (commit.c:807) racing fsync_worker (persist.c:867) against tap_teardown (test_apply_pump.c:243)",
+      "file": "src/controller/commit.c",
+      "root_cause": "Background fsync_worker thread appears to still be resolving a pending commit via commit_resolve_pending() while a test teardown (tap_teardown in test_apply_pump.c) is freeing/tearing down state concurrently -- a shutdown-ordering race between the async fsync/commit-resolution thread and per-test teardown. Observed once during xce-test-rc-file task 1.2 review (ad-hoc standalone controller_unit_test run); did NOT reproduce on a second standalone run or in the canonical xce-build.sh run (both completed cleanly with 1213 Tests 2 Failures 0 Ignored, the 2 failures being the unrelated tracked bug-1561). Not caused by and unrelated to rcfile.c/test_rcfile.c (xce-test-rc-file spec) -- flagged as a pre-existing flake surfaced incidentally during that task-local review.",
+      "fix": "Not yet root-caused or fixed -- logged for tracking since it did not reproduce reliably enough to investigate further in that review pass. Next occurrence should capture the full ASAN report (not just the top frames) and check whether test_apply_pump.c's teardown waits for fsync_worker to fully quiesce before freeing shared state.",
+      "tags": [
+        "flaky",
+        "asan",
+        "use-after-free",
+        "controller_unit_test",
+        "commit.c",
+        "persist.c",
+        "fsync_worker",
+        "test_apply_pump.c",
+        "teardown-race",
+        "unreproduced"
+      ],
+      "related_bugs": [],
+      "occurrences": 1,
+      "last_seen": "2026-08-26T00:00:00Z"
+    },
+    {
+      "id": "bug-1718",
+      "timestamp": "2026-08-26T16:23:00Z",
+      "error_message": "A python3 pty-driver embedded via a heredoc on a command that ALSO receives piped stdin silently discards the piped input",
+      "file": "src/controller/testsuite/test_rcfile_integration.sh",
+      "root_cause": "First draft of run_interactive_pty() did: printf '%s\\n' \"$script\" | python3 - \"$XCE_TEST\" ... <<PYEOF ... PYEOF. python3 - reads its ENTIRE program from stdin until EOF; a <<HEREDOC redirect on the same command line takes precedence over the pipe for that command's stdin (the pipe's implicit fd0 assignment is overridden by the explicit redirect), so the heredoc (the Python source) became the process's only stdin content. Once python3 finished reading its own program from that stream, stdin was already at EOF -- the script's own sys.stdin.read() (meant to receive the piped xce_test command script) returned empty every time, silently. No error was raised; it just produced an empty commands list, which would have made every interactive-scenario assertion trivially fail once real command sending began (caught before ever running the script standalone, via careful re-reading of the draft rather than via a failing run).",
+      "fix": "Write the Python driver ONCE to its own temp file via mktemp + cat >\"$PTY_DRIVER\" <<PYEOF ... PYEOF at script startup (cleaned up by an EXIT trap), then invoke it as printf ... | python3 \"$PTY_DRIVER\" args... -- this leaves the pipe as the sole, unambiguous stdin source for the running Python process, since python3 <file> does not consume stdin to obtain its own source.",
+      "tags": [
+        "bash",
+        "python",
+        "pty",
+        "heredoc",
+        "stdin",
+        "xce-test-rc-file",
+        "task-3.1"
+      ],
+      "related_bugs": [],
+      "occurrences": 1,
+      "last_seen": "2026-08-26T16:23:00Z"
+    },
+    {
+      "id": "bug-1719",
+      "timestamp": "2026-08-26T17:14:00Z",
+      "error_message": "resource-value-verification skill's deployment failed: 'controller binary missing' immediately after a scheduler-only xce-build.sh run that had just succeeded",
+      "file": "src/tools/xce-build.sh",
+      "root_cause": "During xce-test-rc-file task 3.3 (compatibility verification, no product-code change), ran `nix develop ./src --command bash src/tools/xce-build.sh scheduler` (no --incremental) to fill a gap where xce.build/scheduler/xce_scheduler had never been linked. xce-build.sh's documented default behavior (see its own top-of-file comment) does a full `rm -rf \"$BUILD_DIR\"` + fresh `meson setup` on every invocation unless --incremental is passed, then `meson compile` only rebuilds the ONE requested component -- so the plain `scheduler` invocation silently wiped xce.build/controller/xce_controller and xce.build/controller/xce_test (built moments earlier by task 3.1/3.2's controller build, containing the rc-file feature under test) even though I never touched controller source. A prior attempt in the same session had additionally passed an explicit build_dir arg (`src/xce.build`) which -- combined with xce-build.sh's own internal cd into src/ -- produced a bogus nested `src/src/xce.build` directory instead of reusing the real one (harmless, fully gitignored via meson's own auto-generated per-builddir `.gitignore: *`, but wasted a build cycle and caused brief confusion investigating why `git status`/`git check-ignore` showed nothing for it).",
+      "fix": "Rebuilt controller incrementally (`xce-build.sh controller asan xce.build --incremental`) to restore xce_controller/xce_test alongside the already-built scheduler, without wiping it back out. Confirmed both controller_unit_test (1213 Tests 0 Failures -- even bug-1561's usually-flaky 2 tests passed clean this run) and the scheduler suite (53/54, only the pre-existing tracked bug-1560 symlink-escape failure) came back healthy. General lesson for any task that needs a SECOND component built into an ALREADY-built xce.build dir: always pass --incremental, and never pass an explicit build_dir positional unless it already matches xce-build.sh's own default relative-path resolution (omit the 3rd arg and let it default to 'xce.build' resolved from src/tools/xce-build.sh's own cwd, don't hand it a repo-root-relative 'src/xce.build' path).",
+      "tags": [
+        "xce-build.sh",
+        "incremental",
+        "build-dir-clobber",
+        "scheduler",
+        "controller",
+        "xce-test-rc-file",
+        "task-3.3"
+      ],
+      "related_bugs": [],
+      "occurrences": 1,
+      "last_seen": "2026-08-26T17:14:00Z"
+    },
+    {
+      "id": "bug-1720",
+      "timestamp": "2026-08-26T17:17:00Z",
+      "error_message": "xce-test-resources run-resource-verify.sh: apiserver layer UNREACHABLE for all 9 injected resources; apiserver.log showed 'API Server fails to start: listen tcp :8080: bind: address already in use'",
+      "file": "environment (stale orphaned process, not a code defect)",
+      "root_cause": "A leftover /tmp/apiserver process (pid 1458981, launched via /tmp/xce-deploy-bug1701/scripts/start-apiserver.sh) from an unrelated, already-concluded prior session's bug-1701 investigation was still running and bound to :8080 -- 7.5 hours old (started 09:44, found at 17:18) with its own deploy directory (/tmp/xce-deploy-bug1701) already deleted, i.e. genuinely abandoned rather than an active concurrent session. This is a general sandbox hygiene gap: xce-test-resources' own trap-based cleanup only covers processes IT started: nothing in this sandbox periodically reaps orphaned daemons left behind by an unrelated script/session that itself never got to run its own cleanup (e.g. was killed externally, or its subagent's context ended before the trap fired).",
+      "fix": "Identified via `lsof -i :8080` / `ss -ltnp`, confirmed genuinely orphaned (dead deploy dir, long uptime, no relation to this session's own work), then `kill -9` on both the wrapper script and the apiserver binary PID. Re-ran run-resource-verify.sh immediately after and got a clean PASS (9 resources, 3 layers, zero mismatches/unreachable/rejected). General lesson: an apiserver-layer UNREACHABLE result in this skill should prompt checking `lsof -i :8080` / `pgrep -af apiserver` for a stale bind-conflict BEFORE suspecting a propagation regression -- the skill's own FAILURE/UNREACHABLE line doesn't distinguish 'never started due to port conflict' from 'started but never logged the trace line', both surface identically as UNREACHABLE.",
+      "tags": [
+        "xce-test-resources",
+        "apiserver",
+        "port-conflict",
+        "orphaned-process",
+        "sandbox-hygiene",
+        "xce-test-rc-file",
+        "task-3.3"
+      ],
+      "related_bugs": [],
+      "occurrences": 1,
+      "last_seen": "2026-08-26T17:17:00Z"
+    },
+    {
+      "id": "bug-1721",
+      "timestamp": "2026-08-26T00:00:00Z",
+      "error_message": "clean_failover_preservation.sh FAIL: cluster.bin missing under <repo>/ha/data (deploy prerequisite)",
+      "file": "src/controller/testsuite/clean_failover_preservation.sh",
+      "root_cause": "The script hardcodes its DATADIR precondition default to $REPO_ROOT/ha/data (a pre-generated cluster.bin + TLS cert template directory it expects to already exist), but ha/data/ was deliberately deleted from the repo by an unrelated commit (ee8473bd, admin-config-tool: drop ha/data/ entirely, match real client TLS). The script was never updated afterward, so its own documented precondition check now fails unconditionally in any fresh checkout/worktree that has not manually recreated that directory out-of-band. Discovered while adding --norc to the script (xce-test-rc-file spec follow-up) and independently verifying the edit did not break anything -- confirmed via git history that the failure reproduces identically on the pre-edit script too, so it is unrelated to that change.",
+      "fix": "Not fixed -- out of scope for the xce-test-rc-file spec. Needs either: (a) DATADIR default updated to point at a still-valid TLS/config template location consistent with admin-config-tool's TLS changes, or (b) the script updated to self-generate its own template via deploy-ha.sh/xce-deploy-ha before the precondition check, matching how run-ha-soak.sh and run-resource-verify.sh already self-deploy their own scratch clusters instead of depending on a fixed pre-existing directory.",
+      "tags": [
+        "test-harness",
+        "stale-fixture",
+        "clean_failover_preservation.sh",
+        "ha-term-safety",
+        "pre-existing",
+        "admin-config-tool-regression"
+      ],
+      "related_bugs": [],
+      "occurrences": 1,
+      "last_seen": "2026-08-26T00:00:00Z"
     }
   ]
 }