feat(worker-rejoin-resync): reconcile unrecognized job to pending (task 2.1)
Add record_job_reconcile_pending_fact(cluster_t*, job_t*) to job.c: eagerly applies apply_job_update_transition(job, JOB_STAT_PEND) (real, live mutation, not a deferred peek -- matching dispatch_a_job_'s established pattern, not record_job_finish_fact's), releases job->job.allocs on an applied (non-no-op) transition, then durably appends via the generic persist_submit_job_fact(WAL_JOB_UPDATE). Caller holds job's lock throughout and is responsible for calling persist_notify_job_updated_already_persisted() after unlocking -- this function deliberately does not call it itself. Closes bug-1631 at this new call site: a RUN job with a real allocation reconciled to PEND (Requirement 4.6, worker reports it unrecognized) would otherwise carry a stale allocation into its next dispatch attempt, exactly the gap bug-1631 documented as dormant until something re-admits an already-dispatched job to the scheduler's PEND pool -- this feature is that path. dispatch_a_job_ itself is unchanged (still correct for a job's first-ever dispatch, the only case reachable there). 3 tests: allocs release + eager-mutation-before-unlock proof, no-op on already-pending, and the Requirement 4.4 terminal-state-preservation proof (rejects overwriting JOB_STAT_DONE). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
parent
eda2eb1bf0
commit
cfe66dde16
7 changed files with 433 additions and 2 deletions
|
|
@ -37,7 +37,7 @@
|
|||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 2. Core: controller job-table reconciliation outcomes
|
||||
- [ ] 2.1 Return an unrecognized job to pending state, releasing its resource allocation
|
||||
- [x] 2.1 Return an unrecognized job to pending state, releasing its resource allocation
|
||||
- When a worker reports a queried job as unrecognized, transition that job back to pending using the existing state-transition guard so a terminal or already-more-final job is never overwritten
|
||||
- Release any resource allocation the job was carrying, so a job returned to pending never carries a stale allocation from a prior dispatch into its next dispatch attempt
|
||||
- Durably record the transition and notify the scheduler of the new pending state through the existing notification path, without waiting for a later replay to discover the change
|
||||
|
|
|
|||
|
|
@ -18411,7 +18411,7 @@
|
|||
"error_message": "New test test_rpc_failure_bound_resets_after_intervening_success (test_dispatch_fail_retry.c) fails: 'Req 4.1 (bug-1631, NOT YET FIXED): a retryable dispatch failure must return the job to PEND -- fails here only because this job previously had a REAL successful dispatch (step 2)'. Expected JOB_STAT_PEND, got JOB_STAT_RUN (and separately, job.allocs stays non-NULL instead of being released).",
|
||||
"file": "src/controller/job.c",
|
||||
"root_cause": "dispatch_a_job_()'s four retryable-failure resolution branches (XCE_DISPATCH_FAIL_SITE_WORKER_LOOKUP/_WORKER_NULL/_WORKER_NOT_READY/_RPC_FAILURE, job.c ~lines 1040-1435, all four added/restructured by scheduler-dispatch-commit task 3.3) never write j->job.state or j->job.allocs on the 'resolved_retryable' (within-bound) path -- only job.dispatch_fail_count is touched. This is CORRECT for a job's very FIRST dispatch attempt: the job was never anything but PEND and allocs was never committed, so 'no reversal needed' (design.md's own 'Controller Dispatch Commit' state-model text: 'on RPC failure, state is untouched -- still PEND, never entered RUN, so no reversal is needed, per the Research Log finding') is vacuously true. That reasoning silently assumes the job has NEVER had a prior successful dispatch. It breaks the moment dispatch_a_job_() is called again on a job that previously succeeded (went RUN, committed allocs) and now fails retryably again -- exactly the scenario Requirement 4.3's own text ('consecutive... dispatch failures with no intervening successful dispatch') and task 3.1's own observable-completion criterion ('a job that fails, succeeds, then fails again is evaluated against the bound as if it had only failed once') explicitly anticipate. In that scenario job.state incorrectly stays RUN and job.allocs incorrectly stays committed across the retry, violating Requirement 4.1 ('the Scheduler and Controller shall return the job to a pending state') and Requirement 4.4 ('the Scheduler and Controller shall each release the resource-allocation and commitment data'). The terminal (bound-exceeded) resolution path is separately confirmed affected the same way: mark_job_fail()/apply_job_fail_transition() (job.c ~2303-2325) unconditionally sets job.state = JOB_STAT_EXIT (so the STATE outcome is correct on the terminal path) but never touches job.allocs either, so Requirement 4.5 is also affected on that path. Every existing retryable-failure test (test_dispatch_fail_retry.c's other three tests, test_rpc_failure_bounded_escalation's own 4-consecutive-failure test) only ever drives a job that starts and stays PEND with allocs always NULL throughout, so none of them could have caught this -- only a real intervening success (constructed by the new fail->succeed->fail scenario, scheduler-dispatch-commit task 5.2) exposes it. This is a design-level gap, not just an implementation oversight: design.md's own Controller Dispatch-Failure Classification / Controller Dispatch Commit sections both assert-by-omission that job.state/job.allocs need no reversal on any of the six failure paths, without qualifying that assertion to 'only true for a job's first-ever dispatch attempt'.",
|
||||
"fix": "NOT YET FIXED. Discovered live while implementing scheduler-dispatch-commit task 5.2 (test-only task; the fix requires modifying src/controller/job.c, out of that task's test-only boundary -- reported as BLOCKED). Candidate fix: in each of the four retryable branches' 'resolved_retryable = true' case (still under the existing lock(j, RTTI_LOCK_WRITE)), add `j->job.state = JOB_STAT_PEND;` and release any committed j->job.allocs (free_JobAlloc_ptr(&j->job.allocs); j->job.allocs = NULL;) before unlock(j) -- mirroring, in reverse, what Controller Dispatch Commit's own success path (job.c ~1343-1372) does when transitioning PEND->RUN. Needs a design-review pass first: design.md's 'Controller Dispatch-Failure Classification' and 'Controller Dispatch Commit' sections both need their state-model text corrected to no longer claim the failure paths need no reversal unconditionally. UPDATE (2026-08-23, task 5.2 completion): a kiro-debug investigation (HIGH confidence, independently re-verified even more exhaustively by task 5.2's own reviewer via a full trace of every pendJobs-reinsertion and controller-side state-mutation call site) confirmed this scenario is genuinely UNREACHABLE via any real caller in the current, fully-implemented system -- the scheduler's picker permanently removes a job from its PEND candidate pool the instant it dispatches it (task 2.1), and nothing already-implemented ever re-adds a successfully-dispatched (RUN) job back to that pool. Only the not-yet-implemented scheduler-preemption spec (confirmed still requirements-generated, no approvals) could someday create that path. Per that finding, the COMMITTED version of test_rpc_failure_bound_resets_after_intervening_success (src/controller/testsuite/test_dispatch_fail_retry.c, commit 16d90319) deliberately does NOT assert the job.state/job.allocs reversal described above -- it keeps only the Req 4.3 counter-reset assertions (which are correct and GREEN). There is no live RED test for this bug today; it remains an intentionally-deferred, documented gap to revisit once scheduler-preemption (or an equivalent mechanism) actually creates a reachable RUN-then-retryable-redispatch path -- not before.\n\nADDENDUM (2026-09-01, worker-rejoin-resync design re-validation): worker-rejoin-resync's design.md now explicitly names this bug as the reason Requirement 4.6 (reconciling an 'unrecognized' RUN job back to PEND, eligible for re-dispatch) must not be shipped without the candidate fix described above. design.md's `record_job_reconcile_pending_fact()` is specified to release `job->job.allocs` at the same point it sets the job PEND -- the exact fix this entry already proposed, now scoped to a concrete call site instead of a hypothetical future one. Still NOT YET FIXED IN CODE (worker-rejoin-resync is design-phase only as of this note) -- this addendum records that the 'unreachable, deliberately deferred' status ends the moment that spec is implemented, and the fix must land in the same change as Requirement 4.6, not as a follow-up.",
|
||||
"fix": "NOT YET FIXED. Discovered live while implementing scheduler-dispatch-commit task 5.2 (test-only task; the fix requires modifying src/controller/job.c, out of that task's test-only boundary -- reported as BLOCKED). Candidate fix: in each of the four retryable branches' 'resolved_retryable = true' case (still under the existing lock(j, RTTI_LOCK_WRITE)), add `j->job.state = JOB_STAT_PEND;` and release any committed j->job.allocs (free_JobAlloc_ptr(&j->job.allocs); j->job.allocs = NULL;) before unlock(j) -- mirroring, in reverse, what Controller Dispatch Commit's own success path (job.c ~1343-1372) does when transitioning PEND->RUN. Needs a design-review pass first: design.md's 'Controller Dispatch-Failure Classification' and 'Controller Dispatch Commit' sections both need their state-model text corrected to no longer claim the failure paths need no reversal unconditionally. UPDATE (2026-08-23, task 5.2 completion): a kiro-debug investigation (HIGH confidence, independently re-verified even more exhaustively by task 5.2's own reviewer via a full trace of every pendJobs-reinsertion and controller-side state-mutation call site) confirmed this scenario is genuinely UNREACHABLE via any real caller in the current, fully-implemented system -- the scheduler's picker permanently removes a job from its PEND candidate pool the instant it dispatches it (task 2.1), and nothing already-implemented ever re-adds a successfully-dispatched (RUN) job back to that pool. Only the not-yet-implemented scheduler-preemption spec (confirmed still requirements-generated, no approvals) could someday create that path. Per that finding, the COMMITTED version of test_rpc_failure_bound_resets_after_intervening_success (src/controller/testsuite/test_dispatch_fail_retry.c, commit 16d90319) deliberately does NOT assert the job.state/job.allocs reversal described above -- it keeps only the Req 4.3 counter-reset assertions (which are correct and GREEN). There is no live RED test for this bug today; it remains an intentionally-deferred, documented gap to revisit once scheduler-preemption (or an equivalent mechanism) actually creates a reachable RUN-then-retryable-redispatch path -- not before.\n\nADDENDUM (2026-09-01, worker-rejoin-resync design re-validation): worker-rejoin-resync's design.md now explicitly names this bug as the reason Requirement 4.6 (reconciling an 'unrecognized' RUN job back to PEND, eligible for re-dispatch) must not be shipped without the candidate fix described above. design.md's `record_job_reconcile_pending_fact()` is specified to release `job->job.allocs` at the same point it sets the job PEND -- the exact fix this entry already proposed, now scoped to a concrete call site instead of a hypothetical future one. Still NOT YET FIXED IN CODE (worker-rejoin-resync is design-phase only as of this note) -- this addendum records that the 'unreachable, deliberately deferred' status ends the moment that spec is implemented, and the fix must land in the same change as Requirement 4.6, not as a follow-up.\n\nRESOLVED IN CODE (2026-09-01, worker-rejoin-resync task 2.1): record_job_reconcile_pending_fact() (src/controller/job.c, after mark_job_update) now performs exactly the fix this entry proposed -- free_JobAlloc_ptr(&job->job.allocs) immediately after a successful (JOB_TRANSITION_APPLIED) apply_job_update_transition(job, JOB_STAT_PEND), under the same caller-held write lock, before the durable WAL_JOB_UPDATE fact is appended. This is called when a worker reports a queried job as 'unrecognized' during rejoin reconciliation (Requirement 4.6) -- exactly the RUN-job-returned-to-PEND-and-re-admitted-to-the-dispatch-pool precondition this entry's own root_cause said was needed to make the gap reachable. Regression-tested directly: test_reconcile_pending_applies_releases_allocs_and_notifies (test_worker_resync.c) seeds a real JOB_STAT_RUN job with a real non-NULL allocation and asserts allocs == NULL immediately after the call. Independently reviewed and approved (kiro-review). dispatch_a_job_()'s own four retryable-failure branches (the original bug-1631 site) are still untouched by this fix -- they remain correct for a job's first-ever dispatch attempt, which is still the only reachable case THERE; this fix closes the gap at the new call site this feature introduces, not by changing dispatch_a_job_ itself.",
|
||||
"tags": [
|
||||
"controller",
|
||||
"dispatch-failure",
|
||||
|
|
|
|||
|
|
@ -183,6 +183,34 @@ extern job_transition_result_t apply_job_update_transition(job_t*, int);
|
|||
* RTTI_LOCK_WRITE. */
|
||||
extern job_transition_result_t apply_job_postdone_transition(job_t*);
|
||||
|
||||
/* worker-rejoin-resync task 2.1 (Req 4.6, 5.4, 7.1-7.3): returns a job the
|
||||
* worker reports as unrecognized back to PEND, releasing any resource
|
||||
* allocation it was carrying (bug-1631) so a later redispatch never
|
||||
* inherits a stale allocation from the prior dispatch. Follows
|
||||
* dispatch_a_job_()'s own eager-mutation-plus-explicit-notify pattern, NOT
|
||||
* record_job_finish_fact's deferred one -- see design.md's "Controller:
|
||||
* job.c" Components section for the full rationale.
|
||||
*
|
||||
* Precondition: caller holds `job` under RTTI_LOCK_WRITE on entry and does
|
||||
* not release it until this call returns -- matching apply_job_update_
|
||||
* transition's own calling convention (e.g. mark_job_update's callers).
|
||||
* This function does not lock/unlock `job` itself.
|
||||
*
|
||||
* Returns the job_transition_result_t of the underlying apply_job_update_
|
||||
* transition(job, JOB_STAT_PEND) call, cast to int, so the caller can tell
|
||||
* JOB_TRANSITION_APPLIED apart from JOB_TRANSITION_REJECTED_BACKWARD/
|
||||
* JOB_TRANSITION_NOOP_DUPLICATE exactly as that guard's other callers
|
||||
* already do. Only on JOB_TRANSITION_APPLIED does this function release
|
||||
* job->job.allocs (if non-NULL) and durably append the fact -- a rejected
|
||||
* or no-op outcome touches neither (Req 4.3/4.4).
|
||||
*
|
||||
* Unlike record_job_finish_fact/record_job_fail_fact, the caller (not this
|
||||
* function) must call persist_notify_job_updated_already_persisted(c, job)
|
||||
* itself, AFTER releasing the write lock, exactly once, whenever this
|
||||
* returns JOB_TRANSITION_APPLIED -- omitting it reproduces bug-1679 at this
|
||||
* call site. */
|
||||
extern int record_job_reconcile_pending_fact(cluster_t*, job_t*);
|
||||
|
||||
extern int on_job_start(cluster_t*, job_pidinfo_t*);
|
||||
extern int on_job_finish(cluster_t*, xce_jobfinish_t*);
|
||||
extern int on_job_fail(cluster_t*, job_failinfo_t*);
|
||||
|
|
|
|||
|
|
@ -2866,6 +2866,71 @@ static int mark_job_update(cluster_t* c, job_t* job, int state)
|
|||
return 0;
|
||||
}
|
||||
|
||||
/* worker-rejoin-resync task 2.1 (Req 4.1-4.6, 5.4, 7.1-7.3): returns a job
|
||||
* the worker reports as unrecognized back to PEND, releasing any resource
|
||||
* allocation it was carrying so a subsequent redispatch attempt never
|
||||
* inherits a stale allocation from a prior dispatch (bug-1631, confirmed
|
||||
* live and deliberately left unfixed until this feature provided the first
|
||||
* real re-admission path -- see design.md's Error Handling section).
|
||||
*
|
||||
* Follows dispatch_a_job_()'s own established pattern (job.c, its `if (rc
|
||||
* >= 0) { ... }` success block): eager, synchronous mutation under the
|
||||
* write lock the caller already holds, immediately followed by a durable
|
||||
* WAL append, immediately followed (by the CALLER, after unlocking) by an
|
||||
* explicit persist_notify_job_updated_already_persisted() call -- NOT
|
||||
* record_job_finish_fact's/record_job_fail_fact's purely-deferred pattern
|
||||
* (mutation withheld until apply_committed_entry replays the committed
|
||||
* fact). The fact recorded here is something the worker already confirmed
|
||||
* (or, in this case, failed to confirm) really happened -- ground truth
|
||||
* already changed outside this controller -- which is exactly dispatch_a_
|
||||
* job_'s own justification for going eager rather than deferred (design.md,
|
||||
* "Controller: job.c" Components section).
|
||||
*
|
||||
* Precondition: caller holds `job` under RTTI_LOCK_WRITE on entry and keeps
|
||||
* holding it until this call returns -- this function does not lock/unlock
|
||||
* `job` itself, matching apply_job_update_transition's own calling
|
||||
* convention (its other caller, mark_job_update above, is called under the
|
||||
* exact same already-held write lock).
|
||||
*
|
||||
* apply_job_update_transition()'s existing job_state_is_backward guard is
|
||||
* the ONLY commit/idempotency guard exercised here -- reused unchanged,
|
||||
* never duplicated: a terminal job's transition is rejected
|
||||
* (JOB_TRANSITION_REJECTED_BACKWARD, Req 4.4) and an already-PEND job's
|
||||
* transition is a no-op (JOB_TRANSITION_NOOP_DUPLICATE, Req 4.3) -- neither
|
||||
* outcome touches job->job.allocs or appends a WAL fact. Only on
|
||||
* JOB_TRANSITION_APPLIED does this release job->job.allocs (if non-NULL)
|
||||
* and durably append the fact via the existing generic persist_submit_job_
|
||||
* fact() -- the plain {jobid, state} shape is sufficient here, same as
|
||||
* every other WAL_JOB_UPDATE caller (no extra fields, unlike DISPATCH/
|
||||
* START/FINISH's richer siblings).
|
||||
*
|
||||
* Returns the job_transition_result_t of the underlying transition, cast to
|
||||
* int, so the caller can tell APPLIED apart from REJECTED_BACKWARD/
|
||||
* NOOP_DUPLICATE. Unlike record_job_finish_fact/record_job_fail_fact, this
|
||||
* function does NOT call persist_notify_job_updated_already_persisted --
|
||||
* the caller must do so itself, after releasing the write lock, exactly
|
||||
* once, for every JOB_TRANSITION_APPLIED result (cluster.h's doc comment on
|
||||
* the prototype carries this same contract). Omitting it reproduces
|
||||
* bug-1679 at this call site. */
|
||||
int record_job_reconcile_pending_fact(cluster_t* c, job_t* job)
|
||||
{
|
||||
job_transition_result_t result;
|
||||
|
||||
result = apply_job_update_transition(job, JOB_STAT_PEND);
|
||||
if (result != JOB_TRANSITION_APPLIED)
|
||||
return (int)result;
|
||||
|
||||
if (job->job.allocs != NULL) {
|
||||
free_JobAlloc_ptr(&(job->job.allocs));
|
||||
job->job.allocs = NULL;
|
||||
}
|
||||
|
||||
(void)persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, job->job.jobid,
|
||||
JOB_STAT_PEND);
|
||||
|
||||
return (int)result;
|
||||
}
|
||||
|
||||
/* Guarded state-transition core for a postdone fact, extracted from
|
||||
* on_job_postdone's old inline mutation (raft-apply-decoupling task 6.1
|
||||
* round 2, Req 3.3/7.1-7.3). Unlike a generic UPDATE, postdone's formula
|
||||
|
|
|
|||
|
|
@ -191,6 +191,7 @@ controller_unit_test_src = [
|
|||
'testsuite/test_dispatch_fail_terminal.c',
|
||||
'testsuite/test_dispatch_fail_retry.c',
|
||||
'testsuite/test_jobdispatch_no_double_notify.c',
|
||||
'testsuite/test_worker_resync.c',
|
||||
'testsuite/test_upgrade_snapshot_restore.c',
|
||||
'testsuite/test_upgrade_snapshot_apply.c',
|
||||
'testsuite/test_bug857_regression.c',
|
||||
|
|
|
|||
|
|
@ -109,6 +109,7 @@ extern void test_dispatch_commit(void);
|
|||
extern void test_dispatch_fail_terminal(void);
|
||||
extern void test_dispatch_fail_retry(void);
|
||||
extern void test_jobdispatch_no_double_notify(void);
|
||||
extern void test_worker_resync(void);
|
||||
|
||||
void setUp(void) { return; }
|
||||
|
||||
|
|
@ -287,6 +288,7 @@ int main(int argc, char* argv[])
|
|||
TEST(test_dispatch_fail_terminal);
|
||||
TEST(test_dispatch_fail_retry);
|
||||
TEST(test_jobdispatch_no_double_notify);
|
||||
TEST(test_worker_resync);
|
||||
|
||||
return UNITY_END();
|
||||
}
|
||||
|
|
|
|||
335
src/controller/testsuite/test_worker_resync.c
Normal file
335
src/controller/testsuite/test_worker_resync.c
Normal file
|
|
@ -0,0 +1,335 @@
|
|||
/* test_worker_resync.c — worker-rejoin-resync task 2.1
|
||||
*
|
||||
* Proves record_job_reconcile_pending_fact() (job.c) — the Requirement 4.6
|
||||
* new fact-recording function that returns a worker-unrecognized job to
|
||||
* PEND, releasing any resource allocation it was carrying (bug-1631's own
|
||||
* deliberately-deferred trigger condition, closed here — see design.md's
|
||||
* Error Handling section) and durably appending a plain WAL_JOB_UPDATE fact
|
||||
* -- following dispatch_a_job_()'s eager-mutation-plus-explicit-notify
|
||||
* pattern (design.md "Controller: job.c — record_job_reconcile_pending_
|
||||
* fact, record_job_reconcile_assign_fact"), NOT record_job_finish_fact's
|
||||
* deferred one.
|
||||
*
|
||||
* This task implements ONLY record_job_reconcile_pending_fact() itself --
|
||||
* no live caller exists yet (dispatch_a_job_'s disconnected-worker split is
|
||||
* task 2.3; resync.c's report-apply glue is a later task), so every test
|
||||
* here directly exercises the exact sequence a real caller will perform:
|
||||
* lock -> call record_job_reconcile_pending_fact() -> check the returned
|
||||
* job_transition_result_t -> unlock -> (only when APPLIED) call
|
||||
* persist_notify_job_updated_already_persisted() itself -- proving the full
|
||||
* intended sequence end-to-end even though production wiring of that
|
||||
* sequence is a later task's job (design.md's Service Interface
|
||||
* Postconditions/Invariants: the caller, not this function, owns that
|
||||
* post-unlock notify call).
|
||||
*
|
||||
* Fixture mirrors test_dispatch_commit.c's dc_setup/dc_teardown exactly: a
|
||||
* bare cluster_t (jobmgr + hooksys) plus a real disk-backed persist_t
|
||||
* initialized with cluster == NULL, so wal_hook_append_classified()
|
||||
* (persist.c) takes the direct persist_wal_async() path for
|
||||
* persist_submit_job_fact()'s plain RETAIN-mode WAL_JOB_UPDATE append -- no
|
||||
* raft/commit-engine machinery needed. p->current_lsn before/after each
|
||||
* call proves (or disproves) a durable append happened, mirroring
|
||||
* test_dispatch_commit.c's test_dc_worker_not_found_leaves_job_pend_and_
|
||||
* alloc_uncommitted's own LSN-delta idiom.
|
||||
*/
|
||||
#include <rtti.h>
|
||||
|
||||
#include <capnp_c.h>
|
||||
#include <errno.h>
|
||||
#include <pthread.h>
|
||||
#include <stdarg.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#include <unity/unity.h>
|
||||
|
||||
#include "cluster.h"
|
||||
#include "controller.capnp.h"
|
||||
#include "hook.h"
|
||||
#include "jobj.h"
|
||||
#include "kvmgr.h"
|
||||
#include "persist.h"
|
||||
#include "wal_types.h"
|
||||
#include "xce/cluster_def.h"
|
||||
#include "xce/types.h"
|
||||
#include "xcealloc.h"
|
||||
#include "xstring.h"
|
||||
|
||||
/* jobj.c's job type table and the process-global xstring table are shared
|
||||
* across every test file in the controller_unit_test binary -- guard
|
||||
* idempotently, mirroring every sibling fixture in this suite
|
||||
* (test_wal_job_dispatch_fact.c's g_wjd_jobobj_ready). */
|
||||
static int g_twr_jobobj_ready;
|
||||
|
||||
static cluster_t g_twr;
|
||||
static char g_twr_tmpdir[256];
|
||||
|
||||
static void twr_setup(void)
|
||||
{
|
||||
kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
|
||||
.nslots = 32, .delta = 16 };
|
||||
persist_config_t cfg;
|
||||
|
||||
if (makeString((char*)"twr-probe") == NULL)
|
||||
xstring_table_init(64);
|
||||
|
||||
if (!g_twr_jobobj_ready) {
|
||||
TEST_ASSERT_EQUAL_INT(0, job_obj_init(32, 32));
|
||||
g_twr_jobobj_ready = 1;
|
||||
}
|
||||
|
||||
memset(&g_twr, 0, sizeof(g_twr));
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
xce_hook_init(&g_twr.hooksys, XCE_CONTROLLER_HOOK_MAX));
|
||||
|
||||
g_twr.jobmgr = makeInstance(&xkvmgr_type, &int_params);
|
||||
TEST_ASSERT_NOT_NULL(g_twr.jobmgr);
|
||||
|
||||
/* Real disk-backed persist, cluster == NULL: wal_hook_append_
|
||||
* classified() (persist.c) takes the direct persist_wal_async() path --
|
||||
* no raft/commit-engine machinery needed for a plain RETAIN-mode fact
|
||||
* append (mirrors test_dispatch_commit.c's dc_setup exactly, which
|
||||
* exercises dispatch_a_job_'s own real persist_submit_job_fact()-family
|
||||
* call the same way). */
|
||||
cfg = persist_config_defaults();
|
||||
cfg.wal_sync_ms = 5000;
|
||||
strcpy(g_twr_tmpdir, "/tmp/xce_twr21_XXXXXX");
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(g_twr_tmpdir));
|
||||
persist_set_workdir(&g_twr.persist, g_twr_tmpdir, "ctrl-test");
|
||||
TEST_ASSERT_EQUAL_INT(0, persist_init(&g_twr.persist, &cfg, NULL));
|
||||
TEST_ASSERT_EQUAL_INT(0, persist_start(&g_twr.persist));
|
||||
}
|
||||
|
||||
static void twr_teardown(void)
|
||||
{
|
||||
char cmd[512];
|
||||
|
||||
persist_stop(&g_twr.persist);
|
||||
persist_fini(&g_twr.persist);
|
||||
|
||||
if (g_twr.jobmgr)
|
||||
dereference(g_twr.jobmgr);
|
||||
xce_hook_fini(&g_twr.hooksys);
|
||||
|
||||
snprintf(cmd, sizeof(cmd), "rm -rf %s", g_twr_tmpdir);
|
||||
if (system(cmd) != 0)
|
||||
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", g_twr_tmpdir);
|
||||
}
|
||||
|
||||
/* Creates a job directly in the fixture's jobmgr, bypassing dispatch/replay
|
||||
* entirely -- for seeding fact-transition tests, mirroring
|
||||
* test_wal_job_dispatch_fact.c's wjd_seed_job / test_dispatch_commit.c's
|
||||
* dc_seed_ready. */
|
||||
static job_submit_data_t g_twr_seed_submit;
|
||||
|
||||
static job_t* twr_seed_job(uint64_t jobid, uint32_t state)
|
||||
{
|
||||
xce_job_t jdef;
|
||||
|
||||
memset(&g_twr_seed_submit, 0, sizeof(g_twr_seed_submit));
|
||||
g_twr_seed_submit.queue = (char*)"q";
|
||||
g_twr_seed_submit.user = (char*)"tester";
|
||||
|
||||
memset(&jdef, 0, sizeof(jdef));
|
||||
jdef.jobid = jobid;
|
||||
jdef.state = state;
|
||||
jdef.submit = &g_twr_seed_submit;
|
||||
|
||||
return makeJob(g_twr.jobmgr, &jdef);
|
||||
}
|
||||
|
||||
/* One host ("twr-worker"), one resource ("slots", value 1.0) -- the same
|
||||
* minimal shape test_wal_job_dispatch_fact.c's/test_dispatch_commit.c's own
|
||||
* dispatch-alloc tests use. Caller owns the result (release via
|
||||
* free_JobAlloc_ptr, or hand ownership to job->job.allocs). */
|
||||
static xce_job_alloc_t* twr_make_test_alloc(uint64_t jobid)
|
||||
{
|
||||
return make_job_alloc(jobid, 1,
|
||||
make_host_allocs(1,
|
||||
make_host_alloc((char*)"twr-worker", 1,
|
||||
make_allocs(1,
|
||||
make_alloc((char*)"slots", NULL, 0, 1.0f, 0)))));
|
||||
}
|
||||
|
||||
static uint64_t twr_current_lsn(void)
|
||||
{
|
||||
uint64_t lsn;
|
||||
|
||||
pthread_mutex_lock(&g_twr.persist.lock);
|
||||
lsn = g_twr.persist.current_lsn;
|
||||
pthread_mutex_unlock(&g_twr.persist.lock);
|
||||
return lsn;
|
||||
}
|
||||
|
||||
/* Spy hook: counts HOOK_AFTER_EVENT/HOOK_DETAIL_JOB_UPDATE fires -- the
|
||||
* exact hook sched_job_update_fun/apiserver_job_update_fun subscribe to in
|
||||
* production, mirroring test_dispatch_commit.c's dc_job_update_spy_hook. */
|
||||
static int g_twr_update_hook_fires;
|
||||
|
||||
static int twr_job_update_spy_hook(int type, int situation, int detail,
|
||||
void* data, va_list ap)
|
||||
{
|
||||
(void)type; (void)data; (void)ap;
|
||||
if (situation == HOOK_AFTER_EVENT && detail == HOOK_DETAIL_JOB_UPDATE)
|
||||
g_twr_update_hook_fires++;
|
||||
return 0;
|
||||
}
|
||||
|
||||
#define TWR_JOBID_APPLIED 81001ULL
|
||||
#define TWR_JOBID_NOOP 81002ULL
|
||||
#define TWR_JOBID_TERMINAL 81003ULL
|
||||
|
||||
/* THE core proof (Req 4.6, 7.1-7.3; bug-1631 regression): a job that was
|
||||
* dispatched (real allocs committed, state RUN) and reconciles to
|
||||
* unrecognized ends the call -- synchronously, BEFORE any commit round-trip,
|
||||
* matching dispatch_a_job_'s own eager pattern, NOT record_job_finish_
|
||||
* fact's deferred one -- with PEND state and no allocation, a durably
|
||||
* appended fact, and (once the caller unlocks and notifies, exactly as a
|
||||
* real caller must) the scheduler's notify path fired exactly once. */
|
||||
static void test_reconcile_pending_applies_releases_allocs_and_notifies(void)
|
||||
{
|
||||
job_t* j;
|
||||
xce_job_alloc_t* alloc;
|
||||
int rc;
|
||||
uint64_t lsn_before, lsn_after;
|
||||
|
||||
twr_setup();
|
||||
g_twr_update_hook_fires = 0;
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
xce_add_hook(&g_twr.hooksys, XCE_CONTROLLER_HOOK_JOB,
|
||||
twr_job_update_spy_hook, NULL));
|
||||
|
||||
j = twr_seed_job(TWR_JOBID_APPLIED, JOB_STAT_RUN);
|
||||
TEST_ASSERT_NOT_NULL(j);
|
||||
|
||||
alloc = twr_make_test_alloc(TWR_JOBID_APPLIED);
|
||||
TEST_ASSERT_NOT_NULL(alloc);
|
||||
j->job.allocs = alloc; /* simulate dispatch_a_job_'s own committed alloc,
|
||||
* ownership transferred to job->job.allocs. */
|
||||
|
||||
lsn_before = twr_current_lsn();
|
||||
|
||||
lock(j, RTTI_LOCK_WRITE);
|
||||
rc = record_job_reconcile_pending_fact(&g_twr, j);
|
||||
|
||||
/* Observable completion (design.md): PEND + no allocation, asserted
|
||||
* IMMEDIATELY after the call returns, still under the write lock,
|
||||
* before any commit round-trip -- the mutation is eager, not
|
||||
* deferred. */
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(JOB_TRANSITION_APPLIED, rc,
|
||||
"an unrecognized RUN job must apply the PEND transition");
|
||||
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_PEND, j->job.state,
|
||||
"job must be PEND synchronously, before any WAL commit/apply "
|
||||
"round trip");
|
||||
TEST_ASSERT_NULL_MESSAGE(j->job.allocs,
|
||||
"bug-1631: a job reconciled from RUN back to PEND must release its "
|
||||
"resource allocation immediately, so a later redispatch never "
|
||||
"inherits a stale allocation from the prior dispatch");
|
||||
unlock(j);
|
||||
|
||||
lsn_after = twr_current_lsn();
|
||||
TEST_ASSERT_TRUE_MESSAGE(lsn_after > lsn_before,
|
||||
"record_job_reconcile_pending_fact must durably append a "
|
||||
"WAL_JOB_UPDATE fact for an applied transition");
|
||||
|
||||
/* Caller's responsibility, AFTER unlocking (design.md Postconditions/
|
||||
* Invariants): notify the scheduler through the existing notify path,
|
||||
* exactly once, for this actual mutation (Req 7.1-7.3). */
|
||||
persist_notify_job_updated_already_persisted(&g_twr, j);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_twr_update_hook_fires,
|
||||
"the scheduler notify path must fire exactly once for this "
|
||||
"reconciled pending transition (Req 7.2) when the caller notifies "
|
||||
"after unlocking -- omitting it would reproduce bug-1679 at this "
|
||||
"new call site");
|
||||
|
||||
xce_disable_hook(&g_twr.hooksys, XCE_CONTROLLER_HOOK_JOB,
|
||||
twr_job_update_spy_hook);
|
||||
twr_teardown();
|
||||
printf("test_reconcile_pending_applies_releases_allocs_and_notifies: "
|
||||
"PASS\n");
|
||||
}
|
||||
|
||||
/* A job with no allocation at all (e.g. never dispatched, already PEND):
|
||||
* reconciling it again must be a no-op per apply_job_update_transition's
|
||||
* own duplicate-state guard (JOB_TRANSITION_NOOP_DUPLICATE) -- no mutation,
|
||||
* no allocs touch (there is none to release), no durable fact, no crash
|
||||
* (Req 4.3). */
|
||||
static void test_reconcile_pending_noop_on_already_pending_job(void)
|
||||
{
|
||||
job_t* j;
|
||||
int rc;
|
||||
uint64_t lsn_before, lsn_after;
|
||||
|
||||
twr_setup();
|
||||
|
||||
j = twr_seed_job(TWR_JOBID_NOOP, JOB_STAT_PEND);
|
||||
TEST_ASSERT_NOT_NULL(j);
|
||||
TEST_ASSERT_NULL(j->job.allocs);
|
||||
|
||||
lsn_before = twr_current_lsn();
|
||||
|
||||
lock(j, RTTI_LOCK_WRITE);
|
||||
rc = record_job_reconcile_pending_fact(&g_twr, j);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(JOB_TRANSITION_NOOP_DUPLICATE, rc,
|
||||
"reconciling an already-PEND job must be treated as a duplicate "
|
||||
"no-op, matching apply_job_update_transition's own guard");
|
||||
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_PEND, j->job.state);
|
||||
TEST_ASSERT_NULL_MESSAGE(j->job.allocs,
|
||||
"a job that never held an allocation must still have none after a "
|
||||
"no-op reconcile");
|
||||
unlock(j);
|
||||
|
||||
lsn_after = twr_current_lsn();
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn_before, lsn_after,
|
||||
"a no-op reconcile must append no duplicate WAL fact (Req 4.3)");
|
||||
|
||||
twr_teardown();
|
||||
printf("test_reconcile_pending_noop_on_already_pending_job: PASS\n");
|
||||
}
|
||||
|
||||
/* A job the controller already knows is terminal (DONE) must never be
|
||||
* clobbered back to PEND by a reconciliation report (Req 4.4) -- the
|
||||
* existing job_state_is_backward guard inside apply_job_update_transition,
|
||||
* reused unmodified, rejects it. */
|
||||
static void test_reconcile_pending_rejects_terminal_job(void)
|
||||
{
|
||||
job_t* j;
|
||||
int rc;
|
||||
uint64_t lsn_before, lsn_after;
|
||||
|
||||
twr_setup();
|
||||
|
||||
j = twr_seed_job(TWR_JOBID_TERMINAL, JOB_STAT_DONE);
|
||||
TEST_ASSERT_NOT_NULL(j);
|
||||
|
||||
lsn_before = twr_current_lsn();
|
||||
|
||||
lock(j, RTTI_LOCK_WRITE);
|
||||
rc = record_job_reconcile_pending_fact(&g_twr, j);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(JOB_TRANSITION_REJECTED_BACKWARD, rc,
|
||||
"a terminal job must reject the backward PEND transition, never "
|
||||
"clobbering more-final state (Req 4.4)");
|
||||
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_DONE, j->job.state,
|
||||
"the job's terminal state must be preserved exactly");
|
||||
unlock(j);
|
||||
|
||||
lsn_after = twr_current_lsn();
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn_before, lsn_after,
|
||||
"a rejected reconcile must append no WAL fact");
|
||||
|
||||
twr_teardown();
|
||||
printf("test_reconcile_pending_rejects_terminal_job: PASS\n");
|
||||
}
|
||||
|
||||
/* ── Test group entry point ───────────────────────────────────────────── */
|
||||
|
||||
void test_worker_resync(void)
|
||||
{
|
||||
UnitySetTestFile(__FILE__);
|
||||
RUN_TEST(test_reconcile_pending_applies_releases_allocs_and_notifies);
|
||||
RUN_TEST(test_reconcile_pending_noop_on_already_pending_job);
|
||||
RUN_TEST(test_reconcile_pending_rejects_terminal_job);
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue