Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
43 lines
8.2 KiB
Markdown
43 lines
8.2 KiB
Markdown
---
|
|
id: "bug-1631"
|
|
status: "fixed"
|
|
status_updated: "2026-09-27"
|
|
status_note: "0c1bac44"
|
|
component: "controller"
|
|
severity: "unknown"
|
|
timestamp: "2026-08-23T19:30:00Z"
|
|
last_seen: "2026-09-19T09:00:00Z"
|
|
occurrences: 2
|
|
files: ["src/controller/job.c"]
|
|
tags: ["controller", "dispatch-failure", "job-state", "job-allocs", "retry", "scheduler-dispatch-commit", "requirement-4.1", "requirement-4.4", "requirement-4.5", "design-gap"]
|
|
related: ["bug-1625", "bug-1626", "bug-1627"]
|
|
fix_commits: ["0c1bac44"]
|
|
legacy: true
|
|
---
|
|
|
|
# bug-1631: New test test_rpc_failure_bound_resets_after_intervening_success (test_dispatch_fail_retry.c)…
|
|
|
|
## Description
|
|
New test test_rpc_failure_bound_resets_after_intervening_success (test_dispatch_fail_retry.c) fails: 'Req 4.1 (bug-1631, NOT YET FIXED): a retryable dispatch failure must return the job to PEND -- fails here only because this job previously had a REAL successful dispatch (step 2)'. Expected JOB_STAT_PEND, got JOB_STAT_RUN (and separately, job.allocs stays non-NULL instead of being released).
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
dispatch_a_job_()'s four retryable-failure resolution branches (XCE_DISPATCH_FAIL_SITE_WORKER_LOOKUP/_WORKER_NULL/_WORKER_NOT_READY/_RPC_FAILURE, job.c ~lines 1040-1435, all four added/restructured by scheduler-dispatch-commit task 3.3) never write j->job.state or j->job.allocs on the 'resolved_retryable' (within-bound) path -- only job.dispatch_fail_count is touched. This is CORRECT for a job's very FIRST dispatch attempt: the job was never anything but PEND and allocs was never committed, so 'no reversal needed' (design.md's own 'Controller Dispatch Commit' state-model text: 'on RPC failure, state is untouched -- still PEND, never entered RUN, so no reversal is needed, per the Research Log finding') is vacuously true. That reasoning silently assumes the job has NEVER had a prior successful dispatch. It breaks the moment dispatch_a_job_() is called again on a job that previously succeeded (went RUN, committed allocs) and now fails retryably again -- exactly the scenario Requirement 4.3's own text ('consecutive... dispatch failures with no intervening successful dispatch') and task 3.1's own observable-completion criterion ('a job that fails, succeeds, then fails again is evaluated against the bound as if it had only failed once') explicitly anticipate. In that scenario job.state incorrectly stays RUN and job.allocs incorrectly stays committed across the retry, violating Requirement 4.1 ('the Scheduler and Controller shall return the job to a pending state') and Requirement 4.4 ('the Scheduler and Controller shall each release the resource-allocation and commitment data'). The terminal (bound-exceeded) resolution path is separately confirmed affected the same way: mark_job_fail()/apply_job_fail_transition() (job.c ~2303-2325) unconditionally sets job.state = JOB_STAT_EXIT (so the STATE outcome is correct on the terminal path) but never touches job.allocs either, so Requirement 4.5 is also affected on that path. Every existing retryable-failure test (test_dispatch_fail_retry.c's other three tests, test_rpc_failure_bounded_escalation's own 4-consecutive-failure test) only ever drives a job that starts and stays PEND with allocs always NULL throughout, so none of them could have caught this -- only a real intervening success (constructed by the new fail->succeed->fail scenario, scheduler-dispatch-commit task 5.2) exposes it. This is a design-level gap, not just an implementation oversight: design.md's own Controller Dispatch-Failure Classification / Controller Dispatch Commit sections both assert-by-omission that job.state/job.allocs need no reversal on any of the six failure paths, without qualifying that assertion to 'only true for a job's first-ever dispatch attempt'.
|
|
|
|
## Fix
|
|
NOT YET FIXED. Discovered live while implementing scheduler-dispatch-commit task 5.2 (test-only task; the fix requires modifying src/controller/job.c, out of that task's test-only boundary -- reported as BLOCKED). Candidate fix: in each of the four retryable branches' 'resolved_retryable = true' case (still under the existing lock(j, RTTI_LOCK_WRITE)), add `j->job.state = JOB_STAT_PEND;` and release any committed j->job.allocs (free_JobAlloc_ptr(&j->job.allocs); j->job.allocs = NULL;) before unlock(j) -- mirroring, in reverse, what Controller Dispatch Commit's own success path (job.c ~1343-1372) does when transitioning PEND->RUN. Needs a design-review pass first: design.md's 'Controller Dispatch-Failure Classification' and 'Controller Dispatch Commit' sections both need their state-model text corrected to no longer claim the failure paths need no reversal unconditionally. UPDATE (2026-08-23, task 5.2 completion): a kiro-debug investigation (HIGH confidence, independently re-verified even more exhaustively by task 5.2's own reviewer via a full trace of every pendJobs-reinsertion and controller-side state-mutation call site) confirmed this scenario is genuinely UNREACHABLE via any real caller in the current, fully-implemented system -- the scheduler's picker permanently removes a job from its PEND candidate pool the instant it dispatches it (task 2.1), and nothing already-implemented ever re-adds a successfully-dispatched (RUN) job back to that pool. Only the not-yet-implemented scheduler-preemption spec (confirmed still requirements-generated, no approvals) could someday create that path. Per that finding, the COMMITTED version of test_rpc_failure_bound_resets_after_intervening_success (src/controller/testsuite/test_dispatch_fail_retry.c, commit 16d90319) deliberately does NOT assert the job.state/job.allocs reversal described above -- it keeps only the Req 4.3 counter-reset assertions (which are correct and GREEN). There is no live RED test for this bug today; it remains an intentionally-deferred, documented gap to revisit once scheduler-preemption (or an equivalent mechanism) actually creates a reachable RUN-then-retryable-redispatch path -- not before.
|
|
|
|
ADDENDUM (2026-09-01, worker-rejoin-resync design re-validation): worker-rejoin-resync's design.md now explicitly names this bug as the reason Requirement 4.6 (reconciling an 'unrecognized' RUN job back to PEND, eligible for re-dispatch) must not be shipped without the candidate fix described above. design.md's `record_job_reconcile_pending_fact()` is specified to release `job->job.allocs` at the same point it sets the job PEND -- the exact fix this entry already proposed, now scoped to a concrete call site instead of a hypothetical future one. Still NOT YET FIXED IN CODE (worker-rejoin-resync is design-phase only as of this note) -- this addendum records that the 'unreachable, deliberately deferred' status ends the moment that spec is implemented, and the fix must land in the same change as Requirement 4.6, not as a follow-up.
|
|
|
|
RESOLVED IN CODE (2026-09-01, worker-rejoin-resync task 2.1): record_job_reconcile_pending_fact() (src/controller/job.c, after mark_job_update) now performs exactly the fix this entry proposed -- free_JobAlloc_ptr(&job->job.allocs) immediately after a successful (JOB_TRANSITION_APPLIED) apply_job_update_transition(job, JOB_STAT_PEND), under the same caller-held write lock, before the durable WAL_JOB_UPDATE fact is appended. This is called when a worker reports a queried job as 'unrecognized' during rejoin reconciliation (Requirement 4.6) -- exactly the RUN-job-returned-to-PEND-and-re-admitted-to-the-dispatch-pool precondition this entry's own root_cause said was needed to make the gap reachable. Regression-tested directly: test_reconcile_pending_applies_releases_allocs_and_notifies (test_worker_resync.c) seeds a real JOB_STAT_RUN job with a real non-NULL allocation and asserts allocs == NULL immediately after the call. Independently reviewed and approved (kiro-review). dispatch_a_job_()'s own four retryable-failure branches (the original bug-1631 site) are still untouched by this fix -- they remain correct for a job's first-ever dispatch attempt, which is still the only reachable case THERE; this fix closes the gap at the new call site this feature introduces, not by changing dispatch_a_job_ itself.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Related: bug-1625, bug-1626, bug-1627
|