lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(worker-rejoin-resync): fail-fast disconnected-worker dispatch (task 2.3)

Browse source 
Split dispatch_a_job_()'s combined "(w->ep == NULL) || (w->host.status
!= 0)" check into two branches, checked in order: w->ep == NULL now
resolves directly via record_job_reconcile_pending_fact() (task 2.1) --
no dispatch_fail_count increment, no mark_job_fail(), never reaching
xce_classify_dispatch_fail() at all -- before the code ever tests
w->host.status != 0, which keeps its existing retry-then-terminal-fail
behavior completely unchanged (byte-identical body, confirmed against
HEAD).

Completes Task 2 (all controller job-table reconciliation outcomes:
2.1 reconcile-to-pending, 2.2 reconcile-assign, 2.3 this split).

RED phase caught a genuine pre-existing test bug: test_worker_notready_
resolves_retryable was seeding a NULL-ep worker while claiming to test
the host.status!=0 retry path -- it never actually exercised that path.
Corrected to use a connected-but-busy worker, with self-asserting
TEST_ASSERT_NOT_NULL(w->ep)/TEST_ASSERT_TRUE(status!=0) guards so this
can't silently drift again.

2 new tests, including a full JOB_TRANSITION_APPLIED path proof (WAL
LSN advance, hook-fires-exactly-once per bug-1679, allocs released per
bug-1631) through this exact call site, both using correctly-timed
canary references on both the job and worker objects to catch any
over-release under ASAN. Full controller suite: 1227 tests, 0 failures,
zero regressions across every other dispatch_a_job_ failure site.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
Rongsong Shen 2026-09-01 12:13:26 +00:00
parent ecfb4c942f
commit ea0658ecf4
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
3 changed files with 390 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

4
.kiro/specs/worker-rejoin-resync/tasks.md
View file

@ -36,7 +36,7 @@
- _Boundary: xceadmin config-build_
- _Depends: 1.1_
- [ ] 2. Core: controller job-table reconciliation outcomes
- [x] 2. Core: controller job-table reconciliation outcomes
- [x] 2.1 Return an unrecognized job to pending state, releasing its resource allocation
- When a worker reports a queried job as unrecognized, transition that job back to pending using the existing state-transition guard so a terminal or already-more-final job is never overwritten
- Release any resource allocation the job was carrying, so a job returned to pending never carries a stale allocation from a prior dispatch into its next dispatch attempt
@ -53,7 +53,7 @@
- _Requirements: 4.1, 4.3, 4.4, 4.7, 4.8, 7.1, 7.2, 7.3_
- _Boundary: job.c additions_
- [ ] 2.3 Fail a dispatch to a disconnected worker immediately, without retry or job failure
- [x] 2.3 Fail a dispatch to a disconnected worker immediately, without retry or job failure
- When the controller attempts to dispatch a job to a worker it is not currently connected to, fail that specific dispatch attempt right away and return the job to pending, reusing the pending-and-release behavior from task 2.1, rather than retrying the attempt or exhausting a retry count toward permanent failure
- Leave the existing behavior for every other reason a worker might not be ready (locked, busy, closed, or mid-reconciliation) completely unchanged — confirm those cases still retry and eventually fail exactly as before
- Observable completion: dispatching to a disconnected worker returns the job to pending on the first attempt with the job's failure-retry count untouched and no permanent failure recorded, while dispatching to a worker that is connected but otherwise not ready still follows the pre-existing retry-then-fail behavior unchanged