forked from lei.chen/myxce
feat(raft-apply-decoupling): defer job construction past the commit barrier (task 3.1)
Reorder handle_jobsubmit() so job objects are constructed via create_job() only after the WAL commit succeeds, not before. Adds jobid_range_exists() for a fast-fail pre-check with no WAL work, and persist_submit_job_payloads_and_commit() (persist.c) operating on raw xce_job_t* payloads instead of pre-existing job_t*. Commit remains synchronous -- converting to the continuation-based commit API is task 3.2. Fixes bug-1408 (found during review): the post-commit array- construction loop suppressed notification for an entire array batch if any one sibling element failed to construct, stranding other durably-committed and successfully-constructed jobs invisible until the next restart's replay. Notification now fires unconditionally for every successfully-constructed element, independent of sibling failures; the client reply still separately reports the batch's construction status.
This commit is contained in:
parent
8febd663d5
commit
6f1fa97fab
6 changed files with 871 additions and 399 deletions
|
|
@ -69,7 +69,7 @@
|
|||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 3. Command apply-after-commit ordering
|
||||
- [ ] 3.1 (P) Stop constructing the in-memory job object before the commit barrier
|
||||
- [x] 3.1 (P) Stop constructing the in-memory job object before the commit barrier
|
||||
- Change job submission to carry forward only the data needed to build the WAL entry payload, without inserting anything into the in-memory job table until the (still-synchronous, at this point) commit call returns success
|
||||
- Remove the now-unreachable code path that previously ran the durability step from inside the notification hook for job submission
|
||||
- Observable: a job submission whose commit fails or times out leaves no job object in the in-memory job table; a successful submission still replies synchronously exactly as before this task
|
||||
|
|
@ -193,6 +193,7 @@
|
|||
- **1.2**: implemented `commit_resolve()` + wired it into all 6 commit.c-internal paths that can advance `committed_lsn`/`writable_lsn` (the COMMITTED/majority-ack outcome). Chose `VTASK_LIVE_BLOCK` for dispatch (never admission-gated, unlike `VTASK_REQ_BLOCK` which can fast-reject — unacceptable for "exactly-once, never unresolved"). **Reviewer caught a real task-graph gap, not a defect in 1.2 itself**: converting `COMMIT_MODE_WAIT` away from a blocking wait also removes the *only* mechanism that ever detected a commit timing out (the caller's own `pthread_cond_timedwait` deadline) — nothing yet arms an equivalent timeout for async-registered continuations, and nothing yet wires demotion/log-truncation detection to call `commit_resolve(FAILED, ...)`. Without this, a pending async command during quorum loss would hang forever instead of failing — worse than today's behavior. Split out as new tasks **1.3** (timeout arming) and **1.4** (demotion/truncation wiring), inserted before the renumbered **1.5** (was 1.3, concurrency test — now exercises the real detection paths from 1.3/1.4 instead of calling `commit_resolve()` directly). Task 3.2 now also depends on 1.3/1.4, since its own "failure/timeout outcome replies" observable is meaningless without them.
|
||||
- **1.3**: added a dedicated per-commit-engine sweep thread (not the shared `delaymgr` — would violate boundary + delaymgr's 200ms poll granularity vs. `DEFAULT_SYNC_TIMEOUT_MS`=500ms would add timing jitter). Race rule: majority-satisfied always wins over an already-expired deadline when both are true under the same lock hold. **Round 1 REJECTED**: `commit_sweep_once()` computed the next-deadline snapshot under `ce->lock`, unlocked to dispatch, then slept on the *stale* snapshot — a `commit_submit_async()` registration/broadcast landing during that unlocked dispatch window was lost (no thread parked on the condvar to receive it), so the sweep thread could sleep on an absent/late deadline indefinitely (worst case: unbounded delay in a low-traffic quorum-loss scenario, violating the "identical failure timing" no-regression bar). **Fixed** by holding `ce->lock` continuously across scan→wait decisions (releasing only for the unlocked dispatch step) and always re-scanning under the lock immediately before ever calling `cond_wait`/`cond_timedwait` — the standard recheck-then-wait idiom. Round 2 reviewer independently reproduced the round-1 bug shape to confirm the fix actually closes it, not just narrows it. **Precedent for 1.4**: any future code that needs to interrupt the sweep thread's wait (e.g., a demotion/truncation handler resolving a pending LSN out-of-band) must go through the same lock+broadcast discipline `commit_submit_async()` uses, or it risks the identical lost-wakeup class of bug.
|
||||
- **1.4**: added `commit_resolve_pending_above(ce, lsn_floor, rc)` (snapshot-then-`commit_resolve()`-per-entry, no parallel bookkeeping). Truncation: a single new `on_truncate`/`truncate_cb_ctx` callback slot on `persist_t` (same inversion-of-control shape as the existing `on_durable_advance` hook) fires from `persist_truncate_wal_above` after releasing `p->lock`, only when a truncation actually removed entries — this ONE wrapper covers all three real truncation call sites (`raft_cb_become_leader`'s post-election truncate, the library's `on_truncate_from` conflict-reconcile, `ha_demotion_convergence_fn`'s post-demotion truncate), verified by the reviewer reading all three. Demotion: `raft_cb_become_follower` calls `commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY)` (lsn_floor=0 → fails everything — a demoted node can never authoritatively confirm anything pending; verified against the CP posture). **Known residual, non-blocking**: a fourth internal truncation path (`persist.c` inside `persist_wal_write_replicated`'s tip-conflict-repair branch, STANDBY replicated-apply only) calls `truncate_wal_above_locked()` directly, bypassing the new wrapper — safe today only because demotion resolves ALL pending continuations before role flips to STANDBY (so nothing can still be registered by the time that path runs), but this is an *implicit* ordering invariant, not an asserted one. Flag for whoever next touches the STANDBY replicated-apply path: add a defensive comment/assertion there, or route it through the wrapper too, before relying on it further. Also: bug-1407 was believed closed at this point (70/70 across 3 runs) — **see 1.5's note below, this was a false reading, not a real fix**.
|
||||
- **3.1**: reordered `handle_jobsubmit()` (job.c) so job objects are only constructed via `create_job()` AFTER the (still-synchronous) WAL commit succeeds — extracted `jobid_range_exists()` (fast-fail pre-check, no WAL work), `build_job_submit_payloads()` (WAL-payload construction with no jobmgr insertion), and a new persist.c sibling `persist_submit_job_payloads_and_commit()` (operates on raw `xce_job_t*` instead of pre-existing `job_t*`, single production call site in job.c — legitimately within `handle_jobsubmit`'s call chain, not a boundary spillover; no task-1 continuation API dependency, genuinely still synchronous). Took 3 review rounds. **Round 1 REJECTED — real defect, not documentation**: the post-commit array-construction loop suppressed the `AFTER_EVENT`/`HOOK_DETAIL_JOB_NEW` notification for an ENTIRE array batch if even one sibling element failed to construct (e.g. a jobid collision slipping past `jobid_range_exists()`'s inherent TOCTOU window, or a plain allocation failure — no race needed) — stranding OTHER elements that were durably committed AND successfully constructed, invisible to the purely event-driven scheduler/apiserver until the next restart's replay. Logged as bug-1408. **Round 2 fixed the defect** (moved the `construct_rc < 0` early-return to AFTER the jlist-build + hook-fire block, so every successfully-constructed sibling is always notified regardless of a sibling's failure; client reply still separately reports the batch's `construct_rc`) but was REJECTED again on a narrower, documentation-only ground: bug-1408's RED-phase writeup claimed a clean assertion FAIL, but under the standard full-group `--run test_commit_barrier_reply` invocation the failing assertion actually longjmps past the test's own `cbrp_teardown()` call (Unity `TEST_ABORT` semantics) and HANGS the next test — the same longjmp-past-teardown pathology already known from task 1.3's era. **Round 3 (parent-handled directly, no code regression risk)**: corrected bug-1408's buglog.json description to accurately describe the hang mode, and hardened the new test itself — every assertion-relevant observable is now captured into a local, `cbrp_teardown()` is called unconditionally, and assertions run only after teardown, so a future regression in this exact test can no longer skip cleanup and deadlock sibling tests. No production code changed in round 3; job.c's fix from round 2 was independently verified correct by the round-2 reviewer (traced jlist-build/hook-fire at job.c:717-766 running unconditionally before the `construct_rc` check at line 775) and needed no further changes. **Known low-severity residual, not blocking, flagged for a future task**: `apiserver_push_joblist` (apiserver.c) lacks the `nfiltered > 0` guard `sched.c`'s equivalent has, so a total-batch construction failure (every array element fails, jlist ends up empty) now sends an empty/no-op jobs push to every connected apiserver — informationless, can't corrupt state, but a minor behavioral quirk introduced by firing the hook unconditionally.
|
||||
- **2.4**: pure test-only close-out of task-group 2 — new `test_apply_reapply_idempotency.c` (3 tests), no production changes, approved first round. Covers three distinct cases beyond 2.2's existing coverage: (1) triple-reapplication of an identical WAL_JOB_SUBMIT through `apply_committed_entry()` with hook-fire-count AND pointer-identity assertions across all 3 calls (not just final state); (2) the `JOB_TRANSITION_NOOP_DUPLICATE` guard branch (state == current state) — a genuinely untested path, distinct from 2.2's forward-once and backward-rejected cases; (3) the task's named scenario, driven entirely through the real dispatch seam (not seeded/shortcut): real SUBMIT → real FINISH → stale out-of-order START (lower LSN, arriving after), asserting the job stays in its FINISH/terminal state and no third hook fires for the rejected START. Test assertions key on hook-fire COUNT and job STATE, never a specific `HOOK_DETAIL_*` value — so these tests stay valid even if 2.2's documented generic-hook-routing deviation is later corrected. **Task-group 2 (shared apply seam) is now complete**: 2.1 (guard cores extracted) → 2.2 (dispatch function built, generic job-state-family routing is a confirmed permanent deviation from design.md, tracked) → 2.3 (replay wired to the seam, duplicate-notification-on-restart hazard confirmed inert today but must be re-verified before 4.1/6.1/6.2/6.3) → 2.4 (idempotency proven through the real dispatch path). Next: task-group 3 (command apply-after-commit ordering), which is the first consumer of both this seam and task-group 1's commit-engine continuation API.
|
||||
- **2.3**: rewired `replay_wal()` (persist.c) to call `apply_committed_entry()` (task 2.2's seam) instead of `apply_wal_entry()` for restart replay's per-entry apply. `apply_wal_entry`/`apply_committed_entry` are siblings, not wrapper/wrappee — `apply_wal_entry` is NOT retired, it remains ha.c's live standby catch-up/replicated-apply dispatch (untouched, different call sites, out of this task's boundary). **Confirmed hazard, investigated to resolution by the reviewer (not left open)**: `replay_wal(p, p->checkpoint_lsn, ...)` replays the FULL WAL range since the last snapshot on every restart (`last_applied_lsn` is deliberately not persisted — recomputed at replay time per design.md), not just a narrow crash-gap tail. Since `apply_committed_entry` fires `AFTER_EVENT` hooks (unlike the old `apply_wal_entry`, which never did), this means every restart re-fires notification hooks for entries already applied+notified in a prior process life — a duplicate-notification hazard in principle. **Independently traced and confirmed inert today** for both existing hook consumers: scheduler hooks aren't registered until `on_scheduler_join()` (sched.c), which happens post-boot via network JOIN handshake, strictly after `persist_recover()` runs; apiserver hooks ARE registered before `persist_recover()` (`apiserver_init()` in `cluster_init_phase2()`), but every apiserver hook body gates its actual network notify on `apiservers[i].ep != NULL`, and the apiserver-join table starts empty each process life — a no-op except via a narrow reconnect race. **MUST be re-verified before tasks 4.1/6.1/6.2/6.3** make this path unambiguously live and broad (e.g. if a future task registers apiserver connections earlier, or scheduler hooks get registered pre-replay) — at that point this either needs a replay-mode flag suppressing hook-firing, or an explicit accepted-idempotent-duplicate-notification decision recorded in design.md. A loud in-line comment cross-referencing this note now sits at the `apply_committed_entry()` call site in `replay_wal()` (persist.c). Also confirmed: job-state facts now replay through 2.1's guarded `apply_job_update_transition()` instead of the previously-unguarded `restore_job_state()` — intentional (Req 3/7 online-replay divergence fix), verified no existing test depended on the old unguarded semantics. `apply_committed_entry`'s missing oversized-payload guard (vs. `apply_wal_entry`'s `MAX_WAL_PAYLOAD_BYTES`/`-EMSGSIZE` check) confirmed inert — `persist_wal_write_replicated()` already rejects oversized payloads at write time, before anything reaches disk.
|
||||
- **2.2**: implemented `apply_committed_entry(persist_t*, uint32_t wal_type, uint64_t lsn, const void* payload, uint32_t len)` in persist.c/persist.h, matching design.md's signature. Routes `WAL_JOB_SUBMIT` through `restore_job()` (idempotent on duplicate jobid — no-op, Req 7.1), `WAL_QUEUE_CTRL`/`WAL_WORKER_CTRL` through their existing restore primitives, and fires each type's post-mutation hook exactly once, after mutation. **Confirmed permanent, non-cosmetic deviation from design.md's Apply Seam wording**: ALL nine job-state fact WAL types (`WAL_JOB_{START,FINISH,FAIL,STOP,RESUME,UPDATE,RUSAGE,CKPT,PREASON}`) route through the single guard-only `apply_job_update_transition()` core from 2.1 — NOT the type-specific `apply_job_{start,finish,fail}_transition` cores — and fire one generic `HOOK_DETAIL_JOB_UPDATE` hook, not the type-specific `HOOK_DETAIL_JOB_START`/`_EXIT`/`_FAIL`. Root cause, independently verified by two review rounds: the on-disk wire format for every one of these types (`wal_jobstate_t {jobid, state}`) genuinely only carries the resulting state, not the richer `job_pidinfo_t`/`xce_jobfinish_t`/`job_failinfo_t` payload the type-specific cores/hooks require — this is a pre-existing wire-format reality (`apply_wal_entry` already collapsed these the same way before this task), not a shortcut invented here. **design.md's Apply Seam section and Fact Ingestion → Event Contract section currently describe per-type hook dispatch that this wire format cannot support as written** — this is a real, standing design/reality conflict, not yet reconciled in design.md itself. No task in the current plan (2.3, 4.1, 4.2, 6.1, 6.2, 6.3) enriches the wire format to close this gap — confirmed by reading all of them; task 4.1 is exclusively about removing receipt-time mutation/hook-firing, not payload enrichment. **Explicit blocking-style call-out for whoever implements 4.1/6.2/6.3**: wiring this dispatch function into a live fact-apply path will permanently change every worker-reported job start/finish/fail notification to a generic UPDATE detail instead of the type-specific one, unless the wire format is enriched first or design.md is corrected to formally accept this substitution — verify this is still the accepted behavior before wiring it live, don't assume it'll be fixed by then. Took 3 review rounds to land: round 1 rejected two doc-comment defects (a citation blaming the wrong crash site, and a dangling pointer to task 4.1 as the fix location); round 2's fix corrected the crash site (real one is `sched.c`'s `sched_job_start_fun`/`_exit_fun`/`_fail_fun`, which unconditionally dereference `info->jobid` before any NULL check) but fabricated a nonexistent function name (`encode_JobPIDInfo_ptr`) as supporting evidence; round 3 replaced it with a narrower, fully source-verified claim (apiserver.c's equivalent functions store the raw pointer into a notify union without dereferencing it at that call site — real, checked line-by-line). **Lesson**: when a reviewer catches a citation error, the safest fix is the narrowest claim you can personally verify by reading the actual code, not a broader-sounding replacement — round 2's attempt to explain "why the alternative is fine" reached for an unverified downstream mechanism and got caught fabricating it.
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -50,36 +50,76 @@ static job_t* create_job(cluster_t* c, xce_job_t* j)
|
|||
return job;
|
||||
}
|
||||
|
||||
uint64_t job_submit(void* c, xce_job_submit_t* submit)
|
||||
/* True if any jobid in [base_jobid, base_jobid + array_size) already exists
|
||||
* in c->jobmgr (raft-apply-decoupling task 3.1). Checked BEFORE any WAL work
|
||||
* so a colliding submission is still rejected immediately with no side
|
||||
* effect, matching the fast-fail behavior create_job()'s own key_find
|
||||
* guard used to provide when job construction ran before the commit
|
||||
* barrier — now that construction is deferred to after commit, a collision
|
||||
* detected only post-commit could no longer be cleanly reported as "never
|
||||
* happened". */
|
||||
static bool jobid_range_exists(cluster_t* c, uint64_t base_jobid,
|
||||
uint64_t array_size)
|
||||
{
|
||||
job_t* job = NULL;
|
||||
uint64_t base_id = submit->jobid;
|
||||
xce_job_t j = { .submit = submit->submit };
|
||||
time_t now = time(NULL);
|
||||
int i;
|
||||
for (uint64_t i = 0; i < array_size; i++) {
|
||||
job_t* existing = NULL;
|
||||
int found;
|
||||
|
||||
j.sub_time = now;
|
||||
j.queue_time = now;
|
||||
j.start_time = 0;
|
||||
j.finish_time = 0;
|
||||
j.pidinfo = NULL;
|
||||
j.state = JOB_STAT_PEND;
|
||||
j.allocs = NULL;
|
||||
j.run_time = 0;
|
||||
j.run_usage = NULL;
|
||||
j.exit_info = NULL;
|
||||
j.ckptinfo = NULL;
|
||||
j.pending_ctrl = 0;
|
||||
lock(c->jobmgr, RTTI_LOCK_READ);
|
||||
found = key_find(c->jobmgr, (void*)(base_jobid + i),
|
||||
(void**)&existing);
|
||||
unlock(c->jobmgr);
|
||||
|
||||
for (i = 0; i < submit->array_size; i++) {
|
||||
j.jobid = base_id + i;
|
||||
job = create_job(c, &j);
|
||||
if (job == NULL) {
|
||||
return -errno;
|
||||
if (found > 0 && existing != NULL)
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/* Build one xce_job_t submission payload per array element, WITHOUT
|
||||
* constructing or registering any job object (raft-apply-decoupling task
|
||||
* 3.1 / Req 2.1, 2.3) — these are plain heap structs carrying only the data
|
||||
* needed to build each entry's WAL payload; `free_job_submit_payloads` frees
|
||||
* them. `submit->submit` is borrowed (owned by the caller's request), never
|
||||
* duplicated or freed here. Returns 0 on success, -ENOMEM on allocation
|
||||
* failure (payloads already stored are left for the caller to free). */
|
||||
static int build_job_submit_payloads(uint64_t base_jobid,
|
||||
xce_job_submit_t* submit, xlist_t* plist)
|
||||
{
|
||||
time_t now = time(NULL);
|
||||
|
||||
xlist_init(plist, submit->array_size);
|
||||
|
||||
for (uint64_t i = 0; i < submit->array_size; i++) {
|
||||
xce_job_t* jp = calloc(1, sizeof(*jp));
|
||||
if (jp == NULL)
|
||||
return -ENOMEM;
|
||||
if (xlist_store(plist, jp) < 0) {
|
||||
free(jp);
|
||||
return -ENOMEM;
|
||||
}
|
||||
|
||||
jp->submit = submit->submit;
|
||||
jp->jobid = base_jobid + i;
|
||||
jp->sub_time = now;
|
||||
jp->queue_time = now;
|
||||
jp->state = JOB_STAT_PEND;
|
||||
}
|
||||
|
||||
return base_id;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int xlist_free_job_payload_(xce_job_t* j, void* arg)
|
||||
{
|
||||
(void)arg;
|
||||
free(j);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void free_job_submit_payloads(xlist_t* plist)
|
||||
{
|
||||
xlist_iter(plist, (xlist_iter_fn_t)xlist_free_job_payload_, NULL);
|
||||
xlist_fini(plist);
|
||||
}
|
||||
|
||||
int load_jobs(void* c)
|
||||
|
|
@ -562,8 +602,10 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
|
|||
uint64_t jobid;
|
||||
xce_job_submit_t* submit;
|
||||
xce_reply_t reply;
|
||||
xlist_t jlist;
|
||||
job_t* j = NULL;
|
||||
xlist_t plist; /* xce_job_t* WAL payloads, pre-commit */
|
||||
xlist_t jlist; /* job_t* for the post-commit notify hook */
|
||||
uint64_t* lsns = NULL;
|
||||
job_t* j = NULL;
|
||||
int i;
|
||||
int rc;
|
||||
|
||||
|
|
@ -574,15 +616,7 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
|
|||
if (NULLSTR(submit->submit->user) || NULLSTR(submit->submit->queue)
|
||||
|| (submit->submit->start_info == NULL)
|
||||
|| NULLSTR(submit->submit->start_info->cmdline)) {
|
||||
jobid = 0;
|
||||
errno = EINVAL;
|
||||
} else {
|
||||
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
|
||||
HOOK_DETAIL_JOB_NEW, c, submit);
|
||||
|
||||
jobid = job_submit(c, submit);
|
||||
}
|
||||
if (jobid == 0) {
|
||||
reply.status = -errno;
|
||||
reply.data_which = RequestReply_data_nodata;
|
||||
|
||||
|
|
@ -590,9 +624,114 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
|
|||
return -errno;
|
||||
}
|
||||
|
||||
/* Build job list for the AFTER_EVENT hook BEFORE replying to client.
|
||||
* The AFTER_EVENT hook writes WAL and waits for majority replication —
|
||||
* the client must not be acknowledged until the job is durable. */
|
||||
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
|
||||
HOOK_DETAIL_JOB_NEW, c, submit);
|
||||
|
||||
jobid = submit->jobid;
|
||||
|
||||
if (jobid_range_exists(c, jobid, submit->array_size)) {
|
||||
errno = EEXIST;
|
||||
reply.status = -errno;
|
||||
reply.data_which = RequestReply_data_nodata;
|
||||
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
return -errno;
|
||||
}
|
||||
|
||||
/* Build only the WAL payload data for each array element — no job
|
||||
* object is constructed or inserted into c->jobmgr until the commit
|
||||
* below succeeds (raft-apply-decoupling task 3.1 / Req 2.1, 2.3): a
|
||||
* commit that fails or times out must leave the job table untouched,
|
||||
* and the only way to guarantee that structurally is to never
|
||||
* construct anything until commit has already returned success. */
|
||||
rc = build_job_submit_payloads(jobid, submit, &plist);
|
||||
if (rc == 0) {
|
||||
lsns = calloc((size_t)plist.count, sizeof(*lsns));
|
||||
if (lsns == NULL)
|
||||
rc = -ENOMEM;
|
||||
}
|
||||
if (rc < 0) {
|
||||
free_job_submit_payloads(&plist);
|
||||
|
||||
reply.status = rc;
|
||||
reply.data_which = RequestReply_data_nodata;
|
||||
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* Explicit raft step (raft-apply-decoupling item #1 / bug-1306; task
|
||||
* 3.1): WAL-append + majority-commit barrier for every payload, reading
|
||||
* the commit verdict DIRECTLY. This used to run inside the AFTER_EVENT
|
||||
* hook (persist_job_hook_fun), but xce_run_hooks returns only the LAST
|
||||
* hook's rc, so a later notify subscriber (sched/apiserver) returning 0
|
||||
* CLOBBERED the -ETIMEDOUT — the client was acked for an entry that
|
||||
* never reached majority, which a new-term leader then legally
|
||||
* discarded (committed-job-loss). The client must not be acknowledged,
|
||||
* and no job object may exist, until the submission is durable on a
|
||||
* majority. */
|
||||
int commit_rc = persist_submit_job_payloads_and_commit(&c->persist,
|
||||
&plist, lsns);
|
||||
if (commit_rc < 0) {
|
||||
WARN("handle_jobsubmit: job not durable (rc=%d) — rejecting submission",
|
||||
commit_rc);
|
||||
free_job_submit_payloads(&plist);
|
||||
free(lsns);
|
||||
|
||||
reply.status = commit_rc;
|
||||
reply.data_which = RequestReply_data_nodata;
|
||||
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
return commit_rc;
|
||||
}
|
||||
|
||||
/* Committed on a majority — NOW construct and register each job object
|
||||
* (moved here from before the commit barrier; raft-apply-decoupling
|
||||
* task 3.1 / Req 2.1, 2.2, 2.3). A construction failure at this point
|
||||
* cannot be rolled back (the entry is already durably committed); it is
|
||||
* a genuine apply-time failure, not a retryable one (design.md Error
|
||||
* Strategy) — logged loud, and the submission is NOT reported as a
|
||||
* success. */
|
||||
int construct_rc = 0;
|
||||
for (i = 0; i < plist.count; i++) {
|
||||
xce_job_t* jp = (xce_job_t*)xlist_get(&plist, i);
|
||||
job_t* job = create_job(c, jp);
|
||||
|
||||
if (job == NULL) {
|
||||
int saved_errno = errno;
|
||||
|
||||
ERROR("handle_jobsubmit: job %lu committed at lsn %lu but "
|
||||
"could not be constructed (errno=%d)",
|
||||
(unsigned long)jp->jobid, (unsigned long)lsns[i],
|
||||
saved_errno);
|
||||
if (construct_rc == 0)
|
||||
construct_rc = -saved_errno;
|
||||
continue;
|
||||
}
|
||||
job->submit_lsn = lsns[i];
|
||||
}
|
||||
|
||||
free_job_submit_payloads(&plist);
|
||||
free(lsns);
|
||||
|
||||
/* Build the job list for the AFTER_EVENT hook — the same lookup used
|
||||
* before this task, just run after construction (which now itself runs
|
||||
* after commit) instead of before either.
|
||||
*
|
||||
* IMPORTANT (raft-apply-decoupling task 3.1 remediation): this lookup
|
||||
* and the hook it feeds run UNCONDITIONALLY here, even when
|
||||
* construct_rc < 0 for one or more array elements. Every element whose
|
||||
* create_job() succeeded above is already durably committed AND already
|
||||
* inserted into c->jobmgr — key_find below naturally finds only those
|
||||
* (a failed element was never inserted, so it is silently absent from
|
||||
* jlist with no extra bookkeeping needed). Both the scheduler and the
|
||||
* apiserver are purely event-driven off this hook with no periodic
|
||||
* reconciliation, so returning early here (as a previous version of
|
||||
* this function did whenever ANY sibling failed to construct) would
|
||||
* strand the successfully constructed siblings — real, durable jobs —
|
||||
* invisible until the next controller restart replays the WAL. A
|
||||
* whole-batch construction failure must never suppress notification for
|
||||
* the elements that individually succeeded. */
|
||||
xlist_init(&jlist, submit->array_size);
|
||||
|
||||
for (i = 0; i < submit->array_size; i++) {
|
||||
|
|
@ -615,31 +754,13 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
|
|||
}
|
||||
}
|
||||
|
||||
/* Explicit raft step (raft-apply-decoupling item #1 / bug-1306): WAL-append
|
||||
* + majority-commit barrier, reading the commit verdict DIRECTLY. This used
|
||||
* to run inside the AFTER_EVENT hook (persist_job_hook_fun), but xce_run_hooks
|
||||
* returns only the LAST hook's rc, so a later notify subscriber (sched/
|
||||
* apiserver) returning 0 CLOBBERED the -ETIMEDOUT — the client was acked for
|
||||
* an entry that never reached majority, which a new-term leader then legally
|
||||
* discarded (committed-job-loss). The client must not be acknowledged until
|
||||
* the job is durable on a majority. */
|
||||
int commit_rc = persist_submit_wal_and_commit(&c->persist, &jlist);
|
||||
if (commit_rc < 0) {
|
||||
WARN("handle_jobsubmit: job not durable (rc=%d) — rejecting submission",
|
||||
commit_rc);
|
||||
xlist_iter(&jlist, (xlist_iter_fn_t)xlist_deref_job_item_, NULL);
|
||||
xlist_fini(&jlist);
|
||||
reply.status = commit_rc;
|
||||
reply.data_which = RequestReply_data_nodata;
|
||||
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
return commit_rc;
|
||||
}
|
||||
|
||||
/* Committed on a majority — fire the AFTER_EVENT hook as NOTIFICATION only
|
||||
* (scheduler push, apiserver SSE), observing already-committed state. Its
|
||||
* return no longer gates durability: a notify-hook failure cannot un-commit
|
||||
* an already-durable job, so it is logged but never fails the client reply. */
|
||||
/* Committed + constructed — fire the AFTER_EVENT hook as NOTIFICATION
|
||||
* only (scheduler push, apiserver SSE), observing already-committed,
|
||||
* already-applied state. Its return no longer gates durability: a
|
||||
* notify-hook failure cannot un-commit an already-durable job, so it is
|
||||
* logged but never fails the client reply. Fired for every
|
||||
* successfully constructed element regardless of construct_rc (see the
|
||||
* comment above jlist's construction). */
|
||||
int notify_rc = xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB,
|
||||
HOOK_AFTER_EVENT,
|
||||
HOOK_DETAIL_JOB_NEW, c, &jlist);
|
||||
|
|
@ -651,7 +772,20 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
|
|||
|
||||
xlist_fini(&jlist);
|
||||
|
||||
/* Reply success only AFTER WAL write + majority commit succeeded. */
|
||||
if (construct_rc < 0) {
|
||||
/* At least one array element durably committed but could not be
|
||||
* constructed (see the ERROR logged at the point of failure above).
|
||||
* The client learns the submission did not fully succeed; the
|
||||
* siblings that DID construct successfully were already notified
|
||||
* above and must never have that notification gated on this. */
|
||||
reply.status = construct_rc;
|
||||
reply.data_which = RequestReply_data_nodata;
|
||||
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
return construct_rc;
|
||||
}
|
||||
|
||||
/* Reply success only AFTER WAL write + majority commit + construction. */
|
||||
reply.status = 0;
|
||||
reply.data_which = RequestReply_data_jobid;
|
||||
reply.data.jobid = jobid;
|
||||
|
|
|
|||
|
|
@ -3610,6 +3610,44 @@ static int wal_hook_append_classified(persist_t* p, uint32_t wal_type,
|
|||
return rc; /* local append rc only — facts have no commit timeout */
|
||||
}
|
||||
|
||||
/* Shared step for a single WAL_JOB_SUBMIT entry (raft-apply-decoupling task
|
||||
* 3.1): encode `job`, append, and block on the majority-commit barrier.
|
||||
* Returns 0 when the entry committed on a majority; a negative rc otherwise
|
||||
* (-ETIMEDOUT: no majority within the deadline; -XCE_ERR_NOT_READY: admission
|
||||
* cap; -ENOMEM: encode failure; or a local-write error). Whenever the LOCAL
|
||||
* WAL write succeeded (rc == 0, -ETIMEDOUT, or -XCE_ERR_NOT_READY — the entry
|
||||
* was appended locally and the LSN advanced even though replication is
|
||||
* degraded), *out_lsn is set to the entry's assigned LSN so a caller can tag
|
||||
* it for egress-watermark purposes even on a rejected submission. */
|
||||
static int submit_job_wal_entry_and_commit(persist_t* p, const xce_job_t* job,
|
||||
uint64_t* out_lsn)
|
||||
{
|
||||
uint8_t* buf = NULL;
|
||||
size_t sz = 0;
|
||||
int rc;
|
||||
|
||||
if (capnp_encode_job(job, &buf, &sz) != 0)
|
||||
return -ENOMEM;
|
||||
|
||||
/* ha-convergence-recovery (3.3): snapshot the LSN this submit entry
|
||||
* will receive BEFORE the append. persist_wal_async assigns
|
||||
* hdr.lsn = current_lsn then increments, so the pre-append current_lsn
|
||||
* is exactly the written entry's LSN. */
|
||||
pthread_mutex_lock(&p->lock);
|
||||
uint64_t submit_lsn = p->current_lsn;
|
||||
pthread_mutex_unlock(&p->lock);
|
||||
|
||||
int sync_rc = 0;
|
||||
rc = persist_wal_sync_append(p, WAL_JOB_SUBMIT, buf, (uint32_t)sz,
|
||||
&sync_rc);
|
||||
free(buf);
|
||||
|
||||
if (rc == 0 || rc == -ETIMEDOUT || rc == -XCE_ERR_NOT_READY)
|
||||
*out_lsn = submit_lsn;
|
||||
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* Explicit controller-side raft step for a job submission (raft-apply-decoupling
|
||||
* item #1 / bug-1306): WAL-append each job in `jlist` and block on the
|
||||
* majority-commit barrier, returning the commit verdict DIRECTLY to the caller
|
||||
|
|
@ -3633,32 +3671,36 @@ int persist_submit_wal_and_commit(persist_t* p, xlist_t* jlist)
|
|||
|
||||
for (int i = 0; i < jlist->count && rc == 0; i++) {
|
||||
job_t* j = (job_t*)xlist_get(jlist, i);
|
||||
uint8_t* buf = NULL;
|
||||
size_t sz = 0;
|
||||
if (capnp_encode_job(&j->job, &buf, &sz) != 0) {
|
||||
rc = -ENOMEM;
|
||||
break;
|
||||
}
|
||||
/* ha-convergence-recovery (3.3): snapshot the LSN this submit entry
|
||||
* will receive BEFORE the append. persist_wal_async assigns
|
||||
* hdr.lsn = current_lsn then increments, so the pre-append current_lsn
|
||||
* is exactly the written entry's LSN. */
|
||||
pthread_mutex_lock(&p->lock);
|
||||
uint64_t submit_lsn = p->current_lsn;
|
||||
pthread_mutex_unlock(&p->lock);
|
||||
uint64_t lsn = 0;
|
||||
|
||||
int sync_rc = 0;
|
||||
rc = persist_wal_sync_append(p, WAL_JOB_SUBMIT, buf, (uint32_t)sz,
|
||||
&sync_rc);
|
||||
free(buf);
|
||||
|
||||
/* Tag the job with its submit entry's LSN whenever the LOCAL WAL write
|
||||
* succeeded. persist_wal_sync_append returns the persist_wal_async
|
||||
* error only on local-write failure; on a replication-degraded outcome
|
||||
* (0 / -ETIMEDOUT / -ENODEV / -XCE_ERR_NOT_READY) the entry was appended
|
||||
* locally and the LSN advanced, so the egress tag is set. */
|
||||
rc = submit_job_wal_entry_and_commit(p, &j->job, &lsn);
|
||||
if (rc == 0 || rc == -ETIMEDOUT || rc == -XCE_ERR_NOT_READY)
|
||||
j->submit_lsn = submit_lsn;
|
||||
j->submit_lsn = lsn;
|
||||
}
|
||||
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* Sibling of persist_submit_wal_and_commit for a job submission whose job
|
||||
* object(s) do not exist yet (raft-apply-decoupling task 3.1 / Req 2.1, 2.3):
|
||||
* `payloads` is an xlist_t of xce_job_t* that have never been registered in
|
||||
* any job table. On full success (return 0), out_lsns[i] holds payloads[i]'s
|
||||
* committed LSN — the caller constructs and tags the job object AFTER this
|
||||
* call returns, so a failed/timed-out commit never leaves a job object
|
||||
* behind (there is nothing to construct until this succeeds). Returns the
|
||||
* first negative rc on failure. */
|
||||
int persist_submit_job_payloads_and_commit(persist_t* p, xlist_t* payloads,
|
||||
uint64_t* out_lsns)
|
||||
{
|
||||
int rc = 0;
|
||||
|
||||
if (p == NULL || payloads == NULL || out_lsns == NULL)
|
||||
return -EINVAL;
|
||||
|
||||
for (int i = 0; i < payloads->count && rc == 0; i++) {
|
||||
xce_job_t* j = (xce_job_t*)xlist_get(payloads, i);
|
||||
|
||||
rc = submit_job_wal_entry_and_commit(p, j, &out_lsns[i]);
|
||||
}
|
||||
|
||||
return rc;
|
||||
|
|
@ -3685,17 +3727,13 @@ int persist_job_hook_fun(int type, int situation, int detail,
|
|||
mode = wal_classify(wal_type, detail);
|
||||
|
||||
if (mode == WAL_MODE_WAIT) {
|
||||
/* WAL_JOB_SUBMIT durability (WAL append + majority-commit barrier) was
|
||||
* MOVED OUT of this hook into the explicit raft step
|
||||
* persist_submit_wal_and_commit(), called directly by handle_jobsubmit
|
||||
* (raft-apply-decoupling item #1 / bug-1306). Rationale: xce_run_hooks
|
||||
* returns only the LAST-registered hook's rc, so a later JOB AFTER_EVENT
|
||||
* subscriber (sched/apiserver, returning 0) CLOBBERED this branch's
|
||||
* -ETIMEDOUT — handle_jobsubmit then replied `jobsubmit ok` for an
|
||||
* entry that never reached majority, and a new-term leader legally
|
||||
* discarded it (committed-job-loss). The commit verdict must be read
|
||||
* from the protocol step directly, never as a hook return. This branch
|
||||
* is now a no-op; the AFTER_EVENT hook chain is notification-only. */
|
||||
/* WAL_JOB_SUBMIT durability (WAL append + majority-commit barrier) is
|
||||
* an explicit step invoked directly by handle_jobsubmit —
|
||||
* persist_submit_job_payloads_and_commit(), called strictly before
|
||||
* any job object is constructed (raft-apply-decoupling task 3.1 /
|
||||
* bug-1306) — never from this hook. This hook fires only after that
|
||||
* commit has already succeeded, purely as a notification, so there
|
||||
* is nothing left for it to do here. */
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -834,6 +834,264 @@ static void test_worker_join_no_hostconfig_error_reply_immediate(void)
|
|||
printf("test_worker_join_no_hostconfig_error_reply_immediate: PASS\n");
|
||||
}
|
||||
|
||||
/* ── handle_jobsubmit (raft-apply-decoupling task 3.1: apply-after-commit) ──
|
||||
*
|
||||
* Req 2.1/2.3: a job submission whose commit fails or times out must leave
|
||||
* NO job object in c->jobmgr — construction is deferred until after the
|
||||
* commit barrier succeeds. Req 2.2: a successful submission still replies
|
||||
* synchronously with the jobid, and by the time that reply is observable the
|
||||
* job is already visible in c->jobmgr. */
|
||||
|
||||
static job_startinfo_t g_jobsub_startinfo;
|
||||
static job_submit_data_t g_jobsub_data;
|
||||
static xce_job_submit_t g_jobsub_msg;
|
||||
static xce_request_t g_jobsub_req;
|
||||
static xce_peerinfo_t g_jobsub_from;
|
||||
|
||||
static void cbrp_arrange_jobsubmit(uint64_t jobid, uint64_t array_size)
|
||||
{
|
||||
memset(&g_jobsub_startinfo, 0, sizeof(g_jobsub_startinfo));
|
||||
g_jobsub_startinfo.cmdline = (char *)"/bin/true";
|
||||
|
||||
memset(&g_jobsub_data, 0, sizeof(g_jobsub_data));
|
||||
g_jobsub_data.queue = (char *)"normal";
|
||||
g_jobsub_data.user = (char *)"tester";
|
||||
g_jobsub_data.start_info = &g_jobsub_startinfo;
|
||||
|
||||
memset(&g_jobsub_msg, 0, sizeof(g_jobsub_msg));
|
||||
g_jobsub_msg.jobid = jobid;
|
||||
g_jobsub_msg.array_size = array_size;
|
||||
g_jobsub_msg.submit = &g_jobsub_data;
|
||||
|
||||
g_jobsub_from.name = (char *)"requester";
|
||||
g_jobsub_from.key = (char *)"";
|
||||
|
||||
memset(&g_jobsub_req, 0, sizeof(g_jobsub_req));
|
||||
g_jobsub_req.opcode = OPCode_jobsub;
|
||||
g_jobsub_req.seq = 15;
|
||||
g_jobsub_req.from = &g_jobsub_from;
|
||||
g_jobsub_req.body_which = Request_body_jobsubmsg;
|
||||
g_jobsub_req.body.jobsubmsg = &g_jobsub_msg;
|
||||
}
|
||||
|
||||
/* Direct c->jobmgr lookup — the task 3.1 observable is about the in-memory
|
||||
* job TABLE, not the wire reply, so it is checked independently of
|
||||
* cbrp_scan_replies/cbrp_reply_obs_t (which do not carry a jobid). */
|
||||
static bool cbrp_job_exists(uint64_t jobid)
|
||||
{
|
||||
job_t *j = NULL;
|
||||
int rc;
|
||||
|
||||
lock(g_hc.jobmgr, RTTI_LOCK_READ);
|
||||
rc = key_find(g_hc.jobmgr, (void *)jobid, (void **)&j);
|
||||
unlock(g_hc.jobmgr);
|
||||
|
||||
return (rc > 0 && j != NULL);
|
||||
}
|
||||
|
||||
static void test_jobsubmit_withheld_acks_no_phantom_job(void)
|
||||
{
|
||||
cbrp_setup(300);
|
||||
cbrp_arrange_jobsubmit(9100, 1);
|
||||
cbrp_drive_withheld(handle_jobsubmit, &g_jobsub_req, 300);
|
||||
TEST_ASSERT_FALSE_MESSAGE(cbrp_job_exists(9100),
|
||||
"a job submission whose commit times out must leave no job object "
|
||||
"in the in-memory job table (raft-apply-decoupling task 3.1, "
|
||||
"Req 2.1/2.3)");
|
||||
cbrp_teardown();
|
||||
printf("test_jobsubmit_withheld_acks_no_phantom_job: PASS\n");
|
||||
}
|
||||
|
||||
static void test_jobsubmit_success_reply_and_job_visible_after_commit(void)
|
||||
{
|
||||
cbrp_setup(1500);
|
||||
cbrp_arrange_jobsubmit(9101, 1);
|
||||
cbrp_drive_acked(handle_jobsubmit, &g_jobsub_req);
|
||||
TEST_ASSERT_TRUE_MESSAGE(cbrp_job_exists(9101),
|
||||
"a successful job submission must construct the job object once "
|
||||
"its commit succeeds, and still reply synchronously with the "
|
||||
"jobid (raft-apply-decoupling task 3.1, Req 2.2)");
|
||||
cbrp_teardown();
|
||||
printf("test_jobsubmit_success_reply_and_job_visible_after_commit: PASS\n");
|
||||
}
|
||||
|
||||
/* Captures every jobid delivered via the AFTER_EVENT/HOOK_DETAIL_JOB_NEW
|
||||
* notification hook (job.c's post-commit notify list). Needed by the
|
||||
* array-partial-construction-failure remediation test below to observe
|
||||
* that a durably committed AND successfully constructed sibling is still
|
||||
* notified even when another array element in the same submission fails to
|
||||
* construct (raft-apply-decoupling task 3.1 remediation). xce_run_hooks
|
||||
* also fires this same hook TYPE for HOOK_BEFORE_EVENT with a different
|
||||
* vararg shape (xce_job_submit_t*, not xlist_t*) — filtered out below
|
||||
* before touching ap. */
|
||||
#define CBRP_MAX_JOBNEW_NOTIFY 8
|
||||
static int g_jobnew_notify_count;
|
||||
static uint64_t g_jobnew_notify_jobids[CBRP_MAX_JOBNEW_NOTIFY];
|
||||
|
||||
static void cbrp_jobnew_capture_reset(void)
|
||||
{
|
||||
g_jobnew_notify_count = 0;
|
||||
memset(g_jobnew_notify_jobids, 0, sizeof(g_jobnew_notify_jobids));
|
||||
}
|
||||
|
||||
static int cbrp_jobnew_capture_fn(int type, int situation, int detail,
|
||||
void *data, va_list ap)
|
||||
{
|
||||
(void)type; (void)data;
|
||||
|
||||
if (detail != HOOK_DETAIL_JOB_NEW || situation != HOOK_AFTER_EVENT)
|
||||
return 0;
|
||||
|
||||
{
|
||||
xlist_t *jlist;
|
||||
int i;
|
||||
|
||||
(void)va_arg(ap, cluster_t *);
|
||||
jlist = va_arg(ap, xlist_t *);
|
||||
|
||||
if (jlist == NULL)
|
||||
return 0;
|
||||
|
||||
for (i = 0; i < jlist->count
|
||||
&& g_jobnew_notify_count < CBRP_MAX_JOBNEW_NOTIFY; i++) {
|
||||
job_t *j = (job_t *)xlist_get(jlist, i);
|
||||
if (j != NULL)
|
||||
g_jobnew_notify_jobids[g_jobnew_notify_count++] = j->job.jobid;
|
||||
}
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static bool cbrp_jobnew_notified(uint64_t jobid)
|
||||
{
|
||||
int i;
|
||||
|
||||
for (i = 0; i < g_jobnew_notify_count; i++) {
|
||||
if (g_jobnew_notify_jobids[i] == jobid)
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/* Array-partial-construction-failure remediation (round-2 fix, task 3.1):
|
||||
* when ONE array element's post-commit create_job() fails (a jobid
|
||||
* collision that slipped past jobid_range_exists()'s pre-check, or any
|
||||
* other construct-time failure), the OTHER array elements that constructed
|
||||
* successfully — and are already durably committed — must still fire the
|
||||
* AFTER_EVENT/JOB_NEW notification hook. Both scheduler and apiserver are
|
||||
* purely event-driven off this hook with no periodic reconciliation, so a
|
||||
* withheld notification strands an otherwise-healthy, already-durable job
|
||||
* until the next controller restart replays the WAL.
|
||||
*
|
||||
* The colliding element's failure is reproduced deterministically (no real
|
||||
* race) by exploiting the actual TOCTOU window between
|
||||
* jobid_range_exists()'s pre-check (a READ-locked snapshot taken BEFORE any
|
||||
* WAL work) and makeJob_'s own key_find (a WRITE-locked check made fresh at
|
||||
* construction time, AFTER the commit barrier): a stand-in job for the
|
||||
* second array element's jobid is inserted directly into c->jobmgr from the
|
||||
* test thread strictly after the pre-check has already run (its own WAL
|
||||
* entry is visible, which cannot happen until after the pre-check passes)
|
||||
* and strictly before the construction loop can run (its commit is still
|
||||
* pending on a withheld ACK). */
|
||||
static void test_jobsubmit_array_partial_construct_failure_notifies_survivors(void)
|
||||
{
|
||||
cbrp_reply_obs_t obs = { 0 };
|
||||
uint64_t lsn_before, wm0, wm1;
|
||||
job_submit_data_t collide_submit;
|
||||
bool elem0_wal_appeared, elem1_wal_appeared;
|
||||
bool elem0_exists, elem0_notified;
|
||||
|
||||
/* Every TEST_ASSERT_* below is deferred until after cbrp_teardown().
|
||||
* A failing Unity assertion longjmps out of this function (TEST_ABORT),
|
||||
* which would otherwise skip cbrp_teardown() and leak live
|
||||
* persist/commit-engine background threads into the NEXT test's setup,
|
||||
* deadlocking the whole group (see bug-1408's round-2 correction: this
|
||||
* exact failure mode was originally reproduced as a hang, not a clean
|
||||
* FAIL, under the standard --run test_commit_barrier_reply invocation).
|
||||
* Capture every observable into a local first, teardown unconditionally,
|
||||
* then assert on the captured locals. */
|
||||
|
||||
cbrp_setup(2000);
|
||||
cbrp_jobnew_capture_reset();
|
||||
TEST_ASSERT_EQUAL_INT(0, xce_add_hook(&g_hc.hooksys, XCE_CONTROLLER_HOOK_JOB,
|
||||
cbrp_jobnew_capture_fn, NULL));
|
||||
cbrp_arrange_jobsubmit(9200, 2); /* 9200 survives, 9201 collides */
|
||||
|
||||
pthread_mutex_lock(&g_hc.persist.lock);
|
||||
lsn_before = g_hc.persist.current_lsn;
|
||||
pthread_mutex_unlock(&g_hc.persist.lock);
|
||||
|
||||
g_op_arg.fn = handle_jobsubmit;
|
||||
g_op_arg.req = &g_jobsub_req;
|
||||
g_op_arg.rc = -9999;
|
||||
TEST_ASSERT_EQUAL_INT(0, pthread_create(&g_op_tid, NULL,
|
||||
cbrp_op_thread_fn, &g_op_arg));
|
||||
|
||||
/* Element 0's (jobid 9200) WAL entry appears first; ACK it so the
|
||||
* sequential payload loop advances to element 1. The settling sleep
|
||||
* before each ACK mirrors cbrp_drive_acked's proven pattern — it gives
|
||||
* the handler thread time to actually register its commit wait for the
|
||||
* just-appended LSN before the ACK is delivered. */
|
||||
wm0 = cbrp_poll_append(&g_hc.persist, lsn_before, 2000);
|
||||
elem0_wal_appeared = (wm0 > 0);
|
||||
cbrp_sleep_ms(150);
|
||||
if (elem0_wal_appeared)
|
||||
repl_on_ack(&g_hc.persist.repl, &g_hep_a, wm0);
|
||||
|
||||
/* Element 1's (jobid 9201) WAL entry appears once element 0 commits.
|
||||
* Its own ACK is deliberately withheld until the collision stand-in
|
||||
* below is in place. cbrp_poll_append's lsn_before is a current_lsn
|
||||
* SNAPSHOT (see its first use above, seeded from persist.current_lsn),
|
||||
* not an entry's assigned LSN — element 0 already advanced current_lsn
|
||||
* to wm0 + 1, so that (not wm0 itself) is the correct baseline to wait
|
||||
* past for element 1's append. */
|
||||
wm1 = cbrp_poll_append(&g_hc.persist, wm0 + 1, 2000);
|
||||
elem1_wal_appeared = (wm1 > wm0);
|
||||
cbrp_sleep_ms(150);
|
||||
|
||||
/* Deterministic TOCTOU-outcome injection (see block comment above). */
|
||||
memset(&collide_submit, 0, sizeof(collide_submit));
|
||||
cbrp_make_job(9201, &collide_submit);
|
||||
|
||||
if (elem1_wal_appeared)
|
||||
repl_on_ack(&g_hc.persist.repl, &g_hep_a, wm1);
|
||||
|
||||
cbrp_poll_reply(&g_req_ep, 2000, &obs);
|
||||
pthread_join(g_op_tid, NULL);
|
||||
|
||||
elem0_exists = cbrp_job_exists(9200);
|
||||
elem0_notified = cbrp_jobnew_notified(9200);
|
||||
|
||||
cbrp_teardown();
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(elem0_wal_appeared,
|
||||
"array element 0's WAL entry must appear");
|
||||
TEST_ASSERT_TRUE_MESSAGE(elem1_wal_appeared,
|
||||
"array element 1's WAL entry must appear");
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(obs.n_replies > 0,
|
||||
"the requester must still receive a reply when one array element "
|
||||
"fails post-commit construction");
|
||||
TEST_ASSERT_TRUE_MESSAGE(obs.last_status < 0,
|
||||
"a partially-failed array submission must report the construction "
|
||||
"failure to the client");
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(elem0_exists,
|
||||
"the non-colliding array element must still be constructed and "
|
||||
"present in c->jobmgr despite its sibling failing to construct");
|
||||
TEST_ASSERT_TRUE_MESSAGE(elem0_notified,
|
||||
"the successfully constructed sibling must fire the AFTER_EVENT/"
|
||||
"JOB_NEW notification hook even though array element 9201 failed "
|
||||
"to construct post-commit (raft-apply-decoupling task 3.1 "
|
||||
"remediation) — otherwise it is durably committed and constructed "
|
||||
"but invisible to the purely event-driven scheduler/apiserver "
|
||||
"until the next controller restart replays the WAL");
|
||||
|
||||
printf("test_jobsubmit_array_partial_construct_failure_notifies_survivors:"
|
||||
" PASS\n");
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════
|
||||
* Receipt-ACK verdict pins (ha-commit-barrier-rollout task 3.3, Req 2.3)
|
||||
*
|
||||
|
|
@ -1873,6 +2131,10 @@ void test_commit_barrier_reply(void)
|
|||
RUN_TEST(test_worker_join_withheld_acks_replies_error);
|
||||
RUN_TEST(test_worker_join_success_reply_only_after_commit);
|
||||
RUN_TEST(test_worker_join_no_hostconfig_error_reply_immediate);
|
||||
/* handle_jobsubmit (raft-apply-decoupling task 3.1, Req 2.1/2.2/2.3) */
|
||||
RUN_TEST(test_jobsubmit_withheld_acks_no_phantom_job);
|
||||
RUN_TEST(test_jobsubmit_success_reply_and_job_visible_after_commit);
|
||||
RUN_TEST(test_jobsubmit_array_partial_construct_failure_notifies_survivors);
|
||||
/* Receipt-ACK verdict pins (task 3.3, Req 2.3) — green on landing */
|
||||
RUN_TEST(test_jobdispatch_receipt_ack_no_wal_append);
|
||||
RUN_TEST(test_set_preason_receipt_reply_fact_retained);
|
||||
|
|
|
|||
|
|
@ -719,6 +719,19 @@ void persist_log_entry_metadata(const wal_entry_hdr_t* hdr);
|
|||
* a negative rc MUST reject the submission (never reply success). */
|
||||
int persist_submit_wal_and_commit(persist_t* p, xlist_t* jlist);
|
||||
|
||||
/* Sibling of persist_submit_wal_and_commit for callers that must not
|
||||
* construct any job object before the commit barrier succeeds
|
||||
* (raft-apply-decoupling task 3.1 / Req 2.1, 2.3): `payloads` is an xlist_t
|
||||
* of xce_job_t* — raw submission data that has never been registered in any
|
||||
* job table. WAL-appends + majority-commits each entry in order; on full
|
||||
* success (return 0), out_lsns[i] holds payloads[i]'s committed LSN, for the
|
||||
* caller to construct and tag the corresponding job object AFTER this call
|
||||
* returns — there is nothing to roll back on failure because nothing was
|
||||
* ever constructed. Returns the first negative rc on failure; out_lsns is
|
||||
* only valid up to (not including) the failed index. */
|
||||
int persist_submit_job_payloads_and_commit(persist_t* p, xlist_t* payloads,
|
||||
uint64_t* out_lsns);
|
||||
|
||||
int persist_job_hook_fun(int type, int situation, int detail,
|
||||
void* cls, va_list ap);
|
||||
int persist_queue_hook_fun(int type, int situation, int detail,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue