diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 30b7f285..8102b4ab 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -16201,7 +16201,7 @@ "error_message": "test_recordstart_symlink_escape_replies_error (src/scheduler/testsuite/test_admin_request.c:850) FAILED: Expected 'invalid record file name' Was 'permission denied accessing record file'. Surfaced only now because bug-1559's fix made xce-build.sh's scheduler case run the full `meson test --suite scheduler` for the first time -- this test was never reached by the old sched_test-only invocation.", "file": "src/scheduler/testsuite/test_admin_request.c, src/scheduler/src/recording.c", "root_cause": "Not yet investigated. Test plants a symlink inside record_dir whose target resolves outside record_dir, then issues a recordStart admin request expecting rejection with error string 'invalid record file name' (the canonical-path containment check). Instead the admin path is returning a 'permission denied accessing record file' error, implying some permission-check code path is being hit and short-circuiting before (or instead of) the containment check in this sandbox environment (uid 1000, non-root) -- ordering between the two checks in recording.c not yet confirmed by direct read. Unrelated to component-failover-adoption / bug-1548/1549/1558/1559 -- discovered purely as a side effect of bug-1559's coverage-gap fix making this test reachable.", - "fix": "NOT FIXED -- out of scope for the component-failover-adoption remediation that surfaced it. Needs its own investigation of recording.c's record-file validation check ordering. NEW NUANCE (2026-08-20, build-time-versioning task 6.1): because xce-build.sh has `set -euo pipefail` and its scheduler case's RUN_TESTS block runs the FULL `meson test --suite scheduler` unconditionally (asan profile only), this one persistently-failing test makes the WHOLE `xce-build.sh scheduler asan ...` invocation exit non-zero EVEN THOUGH `meson compile` (all 430/430 targets) already succeeded and the real xce_scheduler binary was produced correctly -- any downstream script/wrapper that treats a non-zero xce-build.sh exit as \"build failed\" will incorrectly abort/skip real post-build verification (e.g. checking --version output) on the asan profile specifically. src/tools/test-build-version-regression.sh (task 6.1's new regression check) works around this deliberately: it captures xce-build.sh's exit code via `cmd || rc=$?` (NOT a bare unguarded call, which would trigger its own set -e), prints a WARNING distinguishing \"compile succeeded, only the bundled test run failed\" from a genuine build failure, and continues to check the binary/--version output regardless -- the binary-existence check right after is what still catches a REAL compile failure. Root cause of the underlying test failure itself remains unfixed/uninvestigated.", + "fix": "NOT FIXED -- out of scope for the component-failover-adoption remediation that surfaced it. Needs its own investigation of recording.c's record-file validation check ordering. NEW NUANCE (2026-08-20, build-time-versioning task 6.1): because xce-build.sh has `set -euo pipefail` and its scheduler case's RUN_TESTS block runs the FULL `meson test --suite scheduler` unconditionally (asan profile only), this one persistently-failing test makes the WHOLE `xce-build.sh scheduler asan ...` invocation exit non-zero EVEN THOUGH `meson compile` (all 430/430 targets) already succeeded and the real xce_scheduler binary was produced correctly -- any downstream script/wrapper that treats a non-zero xce-build.sh exit as \"build failed\" will incorrectly abort/skip real post-build verification (e.g. checking --version output) on the asan profile specifically. src/tools/test-build-version-regression.sh (task 6.1's new regression check) works around this deliberately: it captures xce-build.sh's exit code via `cmd || rc=$?` (NOT a bare unguarded call, which would trigger its own set -e), prints a WARNING distinguishing \"compile succeeded, only the bundled test run failed\" from a genuine build failure, and continues to check the binary/--version output regardless -- the binary-existence check right after is what still catches a REAL compile failure. Root cause of the underlying test failure itself remains unfixed/uninvestigated.\n\nRe-observed a 5th time 2026-08-23 during the scheduler-dispatch-commit feature's final GO/NO-GO release-gate verification run (`xce-build.sh scheduler asan build-asan --incremental`, tree fully committed, zero uncommitted diff in src/scheduler/). meson compile succeeded (467/467 targets, zero compiler errors/warnings-as-errors), xce_scheduler and all testsuite binaries linked cleanly; the bundled `meson test --suite scheduler` run's only failure was this same test (Expected 'invalid record file name' Was 'permission denied accessing record file'), still unrelated to job.c/handler.c dispatch-commit work. Confirmed the feature's own targets separately, both green: `scheduler dispatch commit test` (5 Tests 0 Failures) and `scheduler handlers test` (7 Tests 0 Failures, including the new test_handle_notify_job_dispatch_fail_enqueues_pend case).", "tags": [ "scheduler", "recording", @@ -16215,8 +16215,8 @@ "related_bugs": [ "bug-1559" ], - "occurrences": 4, - "last_seen": "2026-08-20T00:00:00.000Z" + "occurrences": 5, + "last_seen": "2026-08-23T21:53:00Z" }, { "id": "bug-909", @@ -16304,7 +16304,7 @@ "error_message": "controller_unit_test: test_conf_trim_keeps_only_n_versions and test_conf_get_saved_tag_no_config (test_config_repl.c) FAIL during an unrelated ASAN controller build/test run (bug-451 fix session)", "file": "src/controller/testsuite/test_config_repl.c", "root_cause": "Not investigated -- discovered incidentally while verifying an unrelated fix (bug-451, persist_job_hook_fun's RUSAGE arm). Neither test_config_repl.c nor config_repl.c were touched this session (confirmed via git diff, zero changes). Coincides with a burst of 'persist: flusher fdatasync FAILED (errno=5 Input/output error)' ERROR lines in the same test run, matching this project's well-documented overlayfs sandbox fdatasync artifact class (same family as bug-924, bug-1470, bug-1553) rather than a genuine regression.", - "fix": "NOT FIXED -- out of scope for the bug-451 session that surfaced it. Needs confirmation this is the same overlayfs artifact (re-run on a real fs) before treating it as a real regression in test_config_repl.c's config-version-trim/tag-save logic. Re-observed 2026-08-17 during resource-value-verification task 1.1 (resval_typeresolve extraction) full controller ASAN build/test run: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3, test_conf_get_saved_tag_no_config Expected 0 Was ), same accompanying persist_flush/flusher fdatasync errno=5 burst. Zero diff in test_config_repl.c/config_repl.c this session either (task 1.1 boundary was testsuite/resval_typeresolve.{c,h}, testsuite/test_resval_typeresolve.c, meson.build only) -- further corroborates the overlayfs sandbox artifact theory over a genuine regression, still unconfirmed on a real fs. Re-observed again 2026-08-17 during resource-value-verification task 2.1 (resupdate fallback + -t flag wiring, GREEN phase): identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst, in the same full `/xce-build controller` run that also confirmed resval_typeresolve_unit_test's own Unity summary at 4 Tests 0 Failures (task 2.1's GREEN evidence). Task 2.1's diff was scoped to testsuite/resval_typeresolve.{c,h} and testsuite/test.c only (confirmed via git diff --stat) -- test_config_repl.c/config_repl.c untouched, per this bug's own do-not-chase guidance in tasks.md's Implementation Notes.\n\nRe-observed a 5th time 2026-08-18 during a full `nix develop ./src --command bash src/tools/xce-build.sh all asan build-asan --incremental` verification run on the scheduler-framework branch (post bug-1584/1585 scheduler fixes). Identical signature: same 2 tests, same Expected/Was values (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. This run recompiled ZERO C source files (ninja reported \"no work to do\" for all C targets -- only the xceadmin custom target/wheel regenerated), so the failure cannot be attributed to any change in this session -- strongest evidence yet that this is purely an environment (overlayfs fdatasync) artifact, not a code regression. Scheduler suite's own single failure in the same run (test_recordstart_symlink_escape_replies_error, bug-1560) is separate and expected. Re-observed a 6th time 2026-08-19 during a controller ASAN build/test run verifying the StartTime/FinishTime persist.c fix (raft-apply-decoupling job-fact timestamp stamping, unrelated files persist.c/test_apply_committed_entry.c) -- identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst. Session diff confirmed scoped to persist.c/test_apply_committed_entry.c only -- further corroborates environment-artifact theory.\n\nRe-observed a 7th time 2026-08-21 verifying a pure output-formatting edit to src/controller/testsuite/test.c (raw %lu jobid printf sites replaced with fmt_jobid()+%s, 10 call sites, zero logic change) via `xce-build.sh controller asan --incremental` on the scheduler-framework branch. Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to test.c only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. controller/xce_test (the binary test.c itself compiles into) built and linked cleanly with zero warnings/errors tied to the fmt_jobid change; only controller_unit_test (a separate binary, testsuite_test_config_repl.c.o) carries these 2 pre-existing failures. Further corroborates environment-artifact theory over regression.\n\nRe-observed an 8th time 2026-08-23 during scheduler-dispatch-commit task 3.3's full controller ASAN build/test verification run (job.c dispatch-failure retry restructuring + new test_dispatch_fail_retry.c). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to job.c and testsuite/ files only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. Further corroborates environment-artifact theory over regression.\n\nRe-observed a 9th time 2026-08-23 during scheduler-dispatch-commit task 4.1's full controller ASAN build/test verification run (job.c handle_jobdispatch() double-notify removal + new test_jobdispatch_no_double_notify.c). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to job.c (handle_jobdispatch() loop body only) and new/edited testsuite files -- test_config_repl.c/config_repl.c untouched. Further corroborates environment-artifact theory over regression.", + "fix": "NOT FIXED -- out of scope for the bug-451 session that surfaced it. Needs confirmation this is the same overlayfs artifact (re-run on a real fs) before treating it as a real regression in test_config_repl.c's config-version-trim/tag-save logic. Re-observed 2026-08-17 during resource-value-verification task 1.1 (resval_typeresolve extraction) full controller ASAN build/test run: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3, test_conf_get_saved_tag_no_config Expected 0 Was ), same accompanying persist_flush/flusher fdatasync errno=5 burst. Zero diff in test_config_repl.c/config_repl.c this session either (task 1.1 boundary was testsuite/resval_typeresolve.{c,h}, testsuite/test_resval_typeresolve.c, meson.build only) -- further corroborates the overlayfs sandbox artifact theory over a genuine regression, still unconfirmed on a real fs. Re-observed again 2026-08-17 during resource-value-verification task 2.1 (resupdate fallback + -t flag wiring, GREEN phase): identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst, in the same full `/xce-build controller` run that also confirmed resval_typeresolve_unit_test's own Unity summary at 4 Tests 0 Failures (task 2.1's GREEN evidence). Task 2.1's diff was scoped to testsuite/resval_typeresolve.{c,h} and testsuite/test.c only (confirmed via git diff --stat) -- test_config_repl.c/config_repl.c untouched, per this bug's own do-not-chase guidance in tasks.md's Implementation Notes.\n\nRe-observed a 5th time 2026-08-18 during a full `nix develop ./src --command bash src/tools/xce-build.sh all asan build-asan --incremental` verification run on the scheduler-framework branch (post bug-1584/1585 scheduler fixes). Identical signature: same 2 tests, same Expected/Was values (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. This run recompiled ZERO C source files (ninja reported \"no work to do\" for all C targets -- only the xceadmin custom target/wheel regenerated), so the failure cannot be attributed to any change in this session -- strongest evidence yet that this is purely an environment (overlayfs fdatasync) artifact, not a code regression. Scheduler suite's own single failure in the same run (test_recordstart_symlink_escape_replies_error, bug-1560) is separate and expected. Re-observed a 6th time 2026-08-19 during a controller ASAN build/test run verifying the StartTime/FinishTime persist.c fix (raft-apply-decoupling job-fact timestamp stamping, unrelated files persist.c/test_apply_committed_entry.c) -- identical 2 tests, identical Expected/Was values, identical fdatasync errno=5 burst. Session diff confirmed scoped to persist.c/test_apply_committed_entry.c only -- further corroborates environment-artifact theory.\n\nRe-observed a 7th time 2026-08-21 verifying a pure output-formatting edit to src/controller/testsuite/test.c (raw %lu jobid printf sites replaced with fmt_jobid()+%s, 10 call sites, zero logic change) via `xce-build.sh controller asan --incremental` on the scheduler-framework branch. Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to test.c only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. controller/xce_test (the binary test.c itself compiles into) built and linked cleanly with zero warnings/errors tied to the fmt_jobid change; only controller_unit_test (a separate binary, testsuite_test_config_repl.c.o) carries these 2 pre-existing failures. Further corroborates environment-artifact theory over regression.\n\nRe-observed an 8th time 2026-08-23 during scheduler-dispatch-commit task 3.3's full controller ASAN build/test verification run (job.c dispatch-failure retry restructuring + new test_dispatch_fail_retry.c). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to job.c and testsuite/ files only (git diff --stat) -- test_config_repl.c/config_repl.c untouched. Further corroborates environment-artifact theory over regression.\n\nRe-observed a 9th time 2026-08-23 during scheduler-dispatch-commit task 4.1's full controller ASAN build/test verification run (job.c handle_jobdispatch() double-notify removal + new test_jobdispatch_no_double_notify.c). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. Session diff confirmed scoped to job.c (handle_jobdispatch() loop body only) and new/edited testsuite files -- test_config_repl.c/config_repl.c untouched. Further corroborates environment-artifact theory over regression.\n\nRe-observed a 10th time 2026-08-23 during the scheduler-dispatch-commit feature's final GO/NO-GO release-gate verification run (`xce-build.sh controller asan build-asan --incremental`, tree fully committed, zero uncommitted diff in src/controller/job.c, handler.c, or testsuite/). Identical signature: same 2 tests (test_conf_trim_keeps_only_n_versions Expected 2 Was 3; test_conf_get_saved_tag_no_config Expected 0 Was large-garbage-uint64), same accompanying persist_flush/flusher fdatasync errno=5 burst. A dedicated scoped rerun of controller_unit_test with -t filters covering only the scheduler-dispatch-commit feature's own groups (test_dispatch_commit, test_dispatch_fail_terminal, test_dispatch_fail_retry, test_jobdispatch_no_double_notify, test_commit_barrier_reply, test_wal_job_dispatch_fact, test_job_dispatch, test_job_dispatch_fail_count) came back 62 Tests 0 Failures 0 Ignored, exit 0 -- confirms these 2 pre-existing failures are fully isolated to test_config_repl.c and do not touch this feature's own test surface.", "tags": [ "controller", "testsuite", @@ -16320,8 +16320,8 @@ "bug-1470", "bug-1553" ], - "occurrences": 9, - "last_seen": "2026-08-23T18:00:00Z" + "occurrences": 10, + "last_seen": "2026-08-23T21:53:00Z" }, { "id": "bug-912", @@ -18240,7 +18240,7 @@ "error_message": "None yet observed at runtime -- found via code trace while answering an architectural question about job RUN-state boundaries and over/under-commit safety.", "file": "src/scheduler/src/scheduler.c, src/scheduler/src/core_session.c, src/scheduler/src/resource.c", "root_cause": "sched_account_jobs() (scheduler.c:1258-1292) re-derives core->sharedInUse/worker->resInUse every round from scratch, but gates on JOB_IS_RUN(job->state) before checking job->accounts. A job that has been DISPATCHED (job->accounts set synchronously at commit-decision time via sched_commit_job_alloc(), scheduler.c:1446) but not yet confirmed RUN by the worker is still job->state==PEND, so it is skipped by this resum. Since xce_core_begin_session() (core_session.c:66-108) unconditionally zeroes sharedInUse/resInUse every round (normal-mode cadence: 1s, main.c:453-460), that job's charge -- correct within the round it was dispatched -- is wiped at the very next round and not restored until the RUN confirmation is processed. The admission check other jobs go through (sched_check_host_satisfy_/sched_check_shared_satisfy_, resource.c:252-370, called from seqjob_allocator.c:130) reads exactly this same resInUse/sharedInUse, with no separate reserved/committed pool (xce_sched_job_t has only state/allocs/accounts, sched/types.h:96-116) -- so once wiped, the resource looks free again and can be granted to a second job, a genuine over-commit/double-booking risk under any dispatch-to-worker-confirmation latency exceeding ~1 round (ordinary for real worker RPC + fork/exec).", - "fix": "NOT YET FIXED -- found during investigation, not yet remediated. Candidate direction: sched_account_jobs() (or an equivalent pass) needs to also re-sum jobs with accounts != NULL regardless of RUN state (mirroring the same decouple-from-state-gate pattern already being applied to simulation-mode capture in the scheduler-sim-eager-reconciliation spec), OR resInUse/sharedInUse charges need to survive the per-round reset for dispatched-but-unconfirmed jobs some other way. Needs its own spec -- this is normal-mode live-accounting correctness, explicitly out of scope for scheduler-sim-eager-reconciliation (which only touches simulation-mode capture/observability).", + "fix": "FIXED (2026-08-23, scheduler-dispatch-commit tasks 2.1 + 3.1). Scheduler side (task 2.1, commit 8607348c): sched_commit_decision()'s DISPATCH branch now sets job->state = JOB_STAT_RUN synchronously, in the same operation that already sets job->accounts/job->allocs -- no separate fix to sched_account_jobs()'s JOB_IS_RUN gate was needed; the job simply reaches RUN from the round it's dispatched, so the existing gate naturally includes it every subsequent round regardless of confirmation latency. Controller side (task 3.1, commit d9aedc4b): dispatch_a_job_() gates the job.allocs commit + JOB_STAT_RUN transition together on xce_startjob_req() succeeding, keeping both sides consistent. Regression-tested end-to-end: test_dispatch_commit_survives_several_rounds_before_confirmation and test_dispatch_commit_multi_round_gap_matches_same_round_confirmation (src/scheduler/testsuite/test_scheduler_dispatch_commit.c, task 5.1) prove resInUse survives a 5-round gap with no confirmation event, and match a same-round-confirmation baseline; RED-phase independently verified via a real revert/rebuild/rerun of the fix by both the task 5.1 implementer and its reviewer. Full HA soak (sequential + chaotic, 10 rounds each) confirms no new job loss or divergence, task 5.4.", "tags": [ "scheduler", "resource-accounting", @@ -18256,8 +18256,8 @@ "bug-1611", "bug-1613" ], - "occurrences": 1, - "last_seen": "2026-08-22T19:53:00Z" + "occurrences": 2, + "last_seen": "2026-08-23T21:45:00.000Z" }, { "id": "bug-1625", @@ -18265,7 +18265,7 @@ "error_message": "None yet observed at runtime -- found via code trace while designing scheduler-dispatch-commit. Referenced throughout that spec's requirements.md/research.md/design.md by this id; being logged now after noticing it was never actually appended to buglog.json.", "file": "src/scheduler/src/handler.c", "root_cause": "The controller already reports dispatch-RPC failure back to the scheduler via NotifyData_data_jobDispatchFail (notify_dispatch(), src/controller/job.c:1108-1125), but sched_handle_notify()'s switch (handler.c:499-544) has no case for it -- it falls to default: rc = -EINVAL. A dispatch failure is therefore silently dropped by the scheduler: job->accounts/job->allocs (set synchronously at dispatch-decision-commit time by sched_commit_decision()) are never released, leaving stale committed-resource bookkeeping behind indefinitely.", - "fix": "NOT YET FIXED -- scoped as Requirement 3 of the scheduler-dispatch-commit spec (.kiro/specs/scheduler-dispatch-commit/, design-generated as of 2026-08-23). Design: add case NotifyData_data_jobDispatchFail to sched_handle_notify(), applying JOB_STAT_PEND via the existing XCE_SCHED_OP_JSTATE_CHANGE enqueue path (same shape as adjacent schedExitJob/schedFailJob cases) -- the existing core_events.c free-on-non-RUN logic then releases job->accounts/job->allocs with no new cleanup code needed.", + "fix": "FIXED (2026-08-23, scheduler-dispatch-commit task 2.2, commit 281a026d). Added case NotifyData_data_jobDispatchFail to sched_handle_notify()'s switch (handler.c), applying JOB_STAT_PEND via the existing XCE_SCHED_OP_JSTATE_CHANGE enqueue path -- exactly as designed. No new cleanup code needed: core_events.c's existing free-on-non-RUN branch releases job->accounts/job->allocs automatically once the enqueued PEND change is applied. Regression-tested: test_handle_notify_job_dispatch_fail_enqueues_pend (src/scheduler/testsuite/test_handlers.c) proves the notify enqueues the correct op; independently reviewed and approved.", "tags": [ "scheduler", "controller", @@ -18278,8 +18278,8 @@ "related_bugs": [ "bug-1624" ], - "occurrences": 1, - "last_seen": "2026-08-23T05:20:00Z" + "occurrences": 2, + "last_seen": "2026-08-23T21:45:00.000Z" }, { "id": "bug-1626", @@ -18398,7 +18398,7 @@ "error_message": "New test test_rpc_failure_bound_resets_after_intervening_success (test_dispatch_fail_retry.c) fails: 'Req 4.1 (bug-1631, NOT YET FIXED): a retryable dispatch failure must return the job to PEND -- fails here only because this job previously had a REAL successful dispatch (step 2)'. Expected JOB_STAT_PEND, got JOB_STAT_RUN (and separately, job.allocs stays non-NULL instead of being released).", "file": "src/controller/job.c", "root_cause": "dispatch_a_job_()'s four retryable-failure resolution branches (XCE_DISPATCH_FAIL_SITE_WORKER_LOOKUP/_WORKER_NULL/_WORKER_NOT_READY/_RPC_FAILURE, job.c ~lines 1040-1435, all four added/restructured by scheduler-dispatch-commit task 3.3) never write j->job.state or j->job.allocs on the 'resolved_retryable' (within-bound) path -- only job.dispatch_fail_count is touched. This is CORRECT for a job's very FIRST dispatch attempt: the job was never anything but PEND and allocs was never committed, so 'no reversal needed' (design.md's own 'Controller Dispatch Commit' state-model text: 'on RPC failure, state is untouched -- still PEND, never entered RUN, so no reversal is needed, per the Research Log finding') is vacuously true. That reasoning silently assumes the job has NEVER had a prior successful dispatch. It breaks the moment dispatch_a_job_() is called again on a job that previously succeeded (went RUN, committed allocs) and now fails retryably again -- exactly the scenario Requirement 4.3's own text ('consecutive... dispatch failures with no intervening successful dispatch') and task 3.1's own observable-completion criterion ('a job that fails, succeeds, then fails again is evaluated against the bound as if it had only failed once') explicitly anticipate. In that scenario job.state incorrectly stays RUN and job.allocs incorrectly stays committed across the retry, violating Requirement 4.1 ('the Scheduler and Controller shall return the job to a pending state') and Requirement 4.4 ('the Scheduler and Controller shall each release the resource-allocation and commitment data'). The terminal (bound-exceeded) resolution path is separately confirmed affected the same way: mark_job_fail()/apply_job_fail_transition() (job.c ~2303-2325) unconditionally sets job.state = JOB_STAT_EXIT (so the STATE outcome is correct on the terminal path) but never touches job.allocs either, so Requirement 4.5 is also affected on that path. Every existing retryable-failure test (test_dispatch_fail_retry.c's other three tests, test_rpc_failure_bounded_escalation's own 4-consecutive-failure test) only ever drives a job that starts and stays PEND with allocs always NULL throughout, so none of them could have caught this -- only a real intervening success (constructed by the new fail->succeed->fail scenario, scheduler-dispatch-commit task 5.2) exposes it. This is a design-level gap, not just an implementation oversight: design.md's own Controller Dispatch-Failure Classification / Controller Dispatch Commit sections both assert-by-omission that job.state/job.allocs need no reversal on any of the six failure paths, without qualifying that assertion to 'only true for a job's first-ever dispatch attempt'.", - "fix": "NOT YET FIXED. Discovered live while implementing scheduler-dispatch-commit task 5.2 (test-only task; the fix requires modifying src/controller/job.c, out of that task's test-only boundary -- reported as BLOCKED). Candidate fix: in each of the four retryable branches' 'resolved_retryable = true' case (still under the existing lock(j, RTTI_LOCK_WRITE)), add `j->job.state = JOB_STAT_PEND;` and release any committed j->job.allocs (free_JobAlloc_ptr(&j->job.allocs); j->job.allocs = NULL;) before unlock(j) -- mirroring, in reverse, what Controller Dispatch Commit's own success path (job.c ~1343-1372) does when transitioning PEND->RUN. Needs a design-review pass first: design.md's 'Controller Dispatch-Failure Classification' and 'Controller Dispatch Commit' sections both need their state-model text corrected to no longer claim the failure paths need no reversal unconditionally. Regression test already exists and is committed: test_rpc_failure_bound_resets_after_intervening_success (src/controller/testsuite/test_dispatch_fail_retry.c) -- its Req 4.1/4.4/4.5 assertions are the fix's own acceptance test and are currently RED against unmodified job.c (its Req 4.3 counter-reset assertions are unaffected and correctly GREEN, confirmed via a real revert/rebuild/rerun of job.c's `j->dispatch_fail_count = 0;` reset line).", + "fix": "NOT YET FIXED. Discovered live while implementing scheduler-dispatch-commit task 5.2 (test-only task; the fix requires modifying src/controller/job.c, out of that task's test-only boundary -- reported as BLOCKED). Candidate fix: in each of the four retryable branches' 'resolved_retryable = true' case (still under the existing lock(j, RTTI_LOCK_WRITE)), add `j->job.state = JOB_STAT_PEND;` and release any committed j->job.allocs (free_JobAlloc_ptr(&j->job.allocs); j->job.allocs = NULL;) before unlock(j) -- mirroring, in reverse, what Controller Dispatch Commit's own success path (job.c ~1343-1372) does when transitioning PEND->RUN. Needs a design-review pass first: design.md's 'Controller Dispatch-Failure Classification' and 'Controller Dispatch Commit' sections both need their state-model text corrected to no longer claim the failure paths need no reversal unconditionally. UPDATE (2026-08-23, task 5.2 completion): a kiro-debug investigation (HIGH confidence, independently re-verified even more exhaustively by task 5.2's own reviewer via a full trace of every pendJobs-reinsertion and controller-side state-mutation call site) confirmed this scenario is genuinely UNREACHABLE via any real caller in the current, fully-implemented system -- the scheduler's picker permanently removes a job from its PEND candidate pool the instant it dispatches it (task 2.1), and nothing already-implemented ever re-adds a successfully-dispatched (RUN) job back to that pool. Only the not-yet-implemented scheduler-preemption spec (confirmed still requirements-generated, no approvals) could someday create that path. Per that finding, the COMMITTED version of test_rpc_failure_bound_resets_after_intervening_success (src/controller/testsuite/test_dispatch_fail_retry.c, commit 16d90319) deliberately does NOT assert the job.state/job.allocs reversal described above -- it keeps only the Req 4.3 counter-reset assertions (which are correct and GREEN). There is no live RED test for this bug today; it remains an intentionally-deferred, documented gap to revisit once scheduler-preemption (or an equivalent mechanism) actually creates a reachable RUN-then-retryable-redispatch path -- not before.", "tags": [ "controller", "dispatch-failure", @@ -18417,8 +18417,8 @@ "bug-1626", "bug-1627" ], - "occurrences": 1, - "last_seen": "2026-08-23T19:30:00Z" + "occurrences": 2, + "last_seen": "2026-08-23T21:45:00.000Z" }, { "id": "bug-1632",