lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-apply-decoupling): split fact receipt to durable-append-only (task 4.1)

Browse source 
Convert on_job_start/on_job_finish/on_job_fail/on_job_postdone to
validate and durably append only -- BEFORE_EVENT validation hook
unchanged, AFTER_EVENT (post-mutation) hook no longer fires at
receipt, live state mutation removed. Adds persist_submit_job_fact()
(persist.c) since the durable append previously lived inside the
very AFTER_EVENT hook this task silences; wire format is unchanged
and identical to what apply_committed_entry() already routes.

mark_job_start/mark_job_finish (single caller each) renamed in place
to record_job_start_fact/record_job_finish_fact. mark_job_fail/
mark_job_update (shared with job_ctrl's client-command path, out of
boundary) left untouched; new durable-append-only siblings
record_job_fail_fact/record_job_update_fact added instead.

Accepts a narrow, permanent regression: on_job_postdone's finish_time
anomaly backstop has no wire-format equivalent and is dropped
(affects only a rare no-prior-exit-status path).

Two critical hazards discovered by review, documented in tasks.md,
MUST be resolved before task 6 wires apply_committed_entry() into a
live path: (1) persist_job_hook_fun still fires on HOOK_DETAIL_JOB_UPDATE
and will double-append every live-applied fact; (2) record_job_fail_fact/
record_job_update_fact precompute a final state that can be silently
clobbered by a concurrent intervening fact once apply is deferred by
the commit round-trip.
This commit is contained in:
Rongsong Shen 2026-07-04 07:00:09 +00:00
parent f357215a6d
commit 23cbfb48e1
5 changed files with 492 additions and 116 deletions
Download patch file Download diff file Expand all files Collapse all files

5
.kiro/specs/raft-apply-decoupling/tasks.md
View file

@ -91,7 +91,7 @@
- _Depends: 3.2_
- [ ] 4. Fact apply-after-commit split
- [ ] 4.1 (P) Change worker-fact receipt handling to validate and durably append only
- [x] 4.1 (P) Change worker-fact receipt handling to validate and durably append only
- Remove the state mutation and the post-mutation notification hook firing from the receipt-time handlers for job start, finish, fail, and update reports
- Keep the existing pre-mutation validation hook firing at receipt time, unchanged
- Observable: after a worker reports a job event, the job's in-memory state is unchanged immediately after the receipt handler returns
@ -193,6 +193,9 @@
- **1.2**: implemented `commit_resolve()` + wired it into all 6 commit.c-internal paths that can advance `committed_lsn`/`writable_lsn` (the COMMITTED/majority-ack outcome). Chose `VTASK_LIVE_BLOCK` for dispatch (never admission-gated, unlike `VTASK_REQ_BLOCK` which can fast-reject — unacceptable for "exactly-once, never unresolved"). **Reviewer caught a real task-graph gap, not a defect in 1.2 itself**: converting `COMMIT_MODE_WAIT` away from a blocking wait also removes the *only* mechanism that ever detected a commit timing out (the caller's own `pthread_cond_timedwait` deadline) — nothing yet arms an equivalent timeout for async-registered continuations, and nothing yet wires demotion/log-truncation detection to call `commit_resolve(FAILED, ...)`. Without this, a pending async command during quorum loss would hang forever instead of failing — worse than today's behavior. Split out as new tasks **1.3** (timeout arming) and **1.4** (demotion/truncation wiring), inserted before the renumbered **1.5** (was 1.3, concurrency test — now exercises the real detection paths from 1.3/1.4 instead of calling `commit_resolve()` directly). Task 3.2 now also depends on 1.3/1.4, since its own "failure/timeout outcome replies" observable is meaningless without them.
- **1.3**: added a dedicated per-commit-engine sweep thread (not the shared `delaymgr` — would violate boundary + delaymgr's 200ms poll granularity vs. `DEFAULT_SYNC_TIMEOUT_MS`=500ms would add timing jitter). Race rule: majority-satisfied always wins over an already-expired deadline when both are true under the same lock hold. **Round 1 REJECTED**: `commit_sweep_once()` computed the next-deadline snapshot under `ce->lock`, unlocked to dispatch, then slept on the *stale* snapshot — a `commit_submit_async()` registration/broadcast landing during that unlocked dispatch window was lost (no thread parked on the condvar to receive it), so the sweep thread could sleep on an absent/late deadline indefinitely (worst case: unbounded delay in a low-traffic quorum-loss scenario, violating the "identical failure timing" no-regression bar). **Fixed** by holding `ce->lock` continuously across scan→wait decisions (releasing only for the unlocked dispatch step) and always re-scanning under the lock immediately before ever calling `cond_wait`/`cond_timedwait` — the standard recheck-then-wait idiom. Round 2 reviewer independently reproduced the round-1 bug shape to confirm the fix actually closes it, not just narrows it. **Precedent for 1.4**: any future code that needs to interrupt the sweep thread's wait (e.g., a demotion/truncation handler resolving a pending LSN out-of-band) must go through the same lock+broadcast discipline `commit_submit_async()` uses, or it risks the identical lost-wakeup class of bug.
- **1.4**: added `commit_resolve_pending_above(ce, lsn_floor, rc)` (snapshot-then-`commit_resolve()`-per-entry, no parallel bookkeeping). Truncation: a single new `on_truncate`/`truncate_cb_ctx` callback slot on `persist_t` (same inversion-of-control shape as the existing `on_durable_advance` hook) fires from `persist_truncate_wal_above` after releasing `p->lock`, only when a truncation actually removed entries — this ONE wrapper covers all three real truncation call sites (`raft_cb_become_leader`'s post-election truncate, the library's `on_truncate_from` conflict-reconcile, `ha_demotion_convergence_fn`'s post-demotion truncate), verified by the reviewer reading all three. Demotion: `raft_cb_become_follower` calls `commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY)` (lsn_floor=0 → fails everything — a demoted node can never authoritatively confirm anything pending; verified against the CP posture). **Known residual, non-blocking**: a fourth internal truncation path (`persist.c` inside `persist_wal_write_replicated`'s tip-conflict-repair branch, STANDBY replicated-apply only) calls `truncate_wal_above_locked()` directly, bypassing the new wrapper — safe today only because demotion resolves ALL pending continuations before role flips to STANDBY (so nothing can still be registered by the time that path runs), but this is an *implicit* ordering invariant, not an asserted one. Flag for whoever next touches the STANDBY replicated-apply path: add a defensive comment/assertion there, or route it through the wrapper too, before relying on it further. Also: bug-1407 was believed closed at this point (70/70 across 3 runs) — **see 1.5's note below, this was a false reading, not a real fix**.
- **4.1**: converted the four worker-fact receipt handlers (`on_job_start`, `on_job_finish`, `on_job_fail`, `on_job_postdone` — NOT literally named `on_job_update`, task text's naming was approximate) to validate-and-durably-append-only: BEFORE_EVENT validation hook unchanged, AFTER_EVENT (post-mutation) hook no longer fires at receipt, live state mutation removed. **Key discovery**: the durable WAL append was NOT a separate call — it lived INSIDE the very `persist_job_hook_fun` AFTER_EVENT subscriber this task needed to silence, so a new `persist_submit_job_fact()` (persist.c) was added that calls the same unmodified `wal_hook_append_classified(WAL_MODE_RETAIN, wal_jobstate_t{jobid,state}, ...)` directly — verified wire-format-identical to before, `apply_committed_entry()`'s existing switch (task 2.2) routes it unchanged. `mark_job_start`/`mark_job_finish` (single caller each, verified via diff against 41d3620) renamed in place to `record_job_start_fact`/`record_job_finish_fact`; `mark_job_fail`/`mark_job_update` (shared with `job_ctrl`'s OUT-OF-BOUNDARY client-command path, 2 and 3 additional call sites) left untouched, with new durable-append-only siblings `record_job_fail_fact`/`record_job_update_fact` added instead — `job_ctrl`/`do_jobcontrol_op` confirmed byte-identical to pre-task by the reviewer. Approved first round. **Real, permanent regression accepted, not fixed (Req 5.1 required the removal)**: `on_job_postdone`'s `finish_time = time(NULL)` anomaly backstop is gone with no wire-format-level equivalent — concretely, a job hitting this rare no-prior-exit-status path now has `finish_time` stuck at 0 forever, which `job_should_retain()` (persist.c) evaluates as immediately GC-eligible instead of retained for the configured window. Narrow, rare-path, but real. **TWO CRITICAL HAZARDS DISCOVERED BY REVIEW, MUST BE RESOLVED BEFORE TASK 6 WIRES apply_committed_entry() INTO A LIVE PATH — currently inert only because replay runs before persist_start() registers hooks (cluster.c: persist_recover() at line ~1451 vs persist_start() at ~1694)**:
1. **Latent double-append**: `persist_job_hook_fun` is STILL registered on `(XCTL_HOOK_JOB, HOOK_AFTER_EVENT)` with no detail filter for `HOOK_DETAIL_JOB_UPDATE`, and does NOT early-return for that detail (it only early-returns for the WAIT-mode submit path). The moment task 6 wires `apply_committed_entry()` into live commit-advance, EVERY fact `apply_committed_entry()` applies will fire `HOOK_DETAIL_JOB_UPDATE`, which `persist_job_hook_fun` will catch and append AGAIN — a duplicate, mis-typed WAL entry per live-applied fact. **Task 6.1/6.2 must retire or detail-filter `persist_job_hook_fun`'s fact-handling branch before going live**, since no live receipt path fires AFTER_EVENT for facts anymore (this task removed the only other source).
2. **Merge-staleness on deferred apply**: `record_job_fail_fact`/`record_job_update_fact` precompute a FINAL target `state` at receipt time from a snapshot of current job state and bake it verbatim into the WAL entry (their own doc comments claim this "can never drift from what apply time will eventually compute" — that claim was TRUE only when receipt and apply were the same event; it becomes FALSE the moment they're separated by a commit round-trip, which is this entire feature's purpose). `apply_committed_entry()` routes all job-state facts through the generic `apply_job_update_transition`, which does a blind `job->job.state = state` overwrite — not the type-specific `|=` merge `apply_job_fail_transition` uses. Once two facts for the same job are in flight simultaneously (now newly possible, since apply is deferred), the second one to apply can silently clobber bits the first one set. Root cause is task 2.2's wire-format/generic-routing reality, not this task's diff — but no task in the current plan (4.2, 5.x, 6.x) tests or fixes this. **Task 6 planning must resolve this before wiring live apply** — likely requires either wire-format enrichment (baking the merge operation itself into the WAL entry, not just the precomputed result) or an apply-time re-derivation strategy.
- **3.3**: pure test-only close-out of task-group 3 — 4 new tests added to `test_commit_barrier_reply.c` (25→29), approved first round, zero production changes. Closes two gaps 3.2 flagged: (1) an explicit-FAILED test via `persist_truncate_wal_above`/`commit_on_truncate`/`commit_resolve_pending_above(-ENOENT)`, distinct from the existing timeout (`-ETIMEDOUT`) path; (2) the `array_size > 1` mixed-outcome gap — one element ACKed to real majority while two time out, proving the committed sibling's `persist_register_pending_apply` hand-off isn't blocked by failing siblings, zero phantom jobs for the failures, and exactly one batch-level failure reply (via `jobsubmit_reply_failure_once`'s ref-counted first-failure-wins gate). Also added two repeated-failure loop tests (5 independent cycles each, fresh jobid/LSN per iteration) proving the no-phantom-job guarantee isn't a one-shot fluke. **Confirmed correct, non-obvious harness detail**: this file's fixture wires job-submit continuations through a standalone `commit_engine_t g_hce`, NOT `cluster_t`'s embedded `c->commit` field that demotion (`raft_cb_become_follower`) resolves against — so truncation, not demotion, is the only deterministic way to force an explicit FAILED (non-timeout) outcome in this specific test file. Demotion-forces-FAILED already has its own dedicated coverage in `test_demotion_convergence.c` (tasks 1.4/1.5) via a different fixture — not a gap, a file-boundary distinction. **Non-blocking observation carried forward, not fixed here** (pre-existing since 3.2, correctly out of scope for a test-only task): the single batch-level failure reply carries no per-element granularity — a client can't tell "1 of 3 committed" from "all 3 failed" from the reply alone. **Task-group 3 (command apply-after-commit ordering) is now complete**: 3.1 (job construction deferred past commit, bug-1408 caught+fixed) → 3.2 (converted to real async continuation-based commit, majority-ack safety independently verified intact, several documented design deviations) → 3.3 (failure-path guarantee proven under repetition and array mixed-outcomes). **Standing operational blocker carried from 3.2, still true**: no live path applies a successfully-committed job submission until task 6 exists — this branch must not be soak-tested or deployed against live traffic yet.
- **3.2**: converted `handle_jobsubmit()` from 3.1's still-synchronous commit to a real async registration against task 1's continuation API. New `persist_submit_job_payloads_async()` (persist.c) appends the WAL entries via the same `ha_primary_log_append`/`persist_wal_async` primitive the pre-existing RETAIN-fact path (`wal_hook_append_classified`) already uses, then calls `commit_submit_async()` directly on `p->repl.ce` — bypassing `repl_wait_majority_ack`/`repl_submit_majority_commit` in repl.c, which is what design.md's text actually specifies as the integration site. **Independently verified safe** (reviewer traced the full ACK→`commit_resolve()` path in commit.c): majority-ack detection is driven entirely by `(ce, lsn)` events inside the commit engine, agnostic to which wrapper registered the continuation — no false-commit risk. But it's a real, permanent design deviation, not just an implementation detail: this creates a THIRD parallel "append+track a commit" implementation (alongside the old sync one, now alive only in 3 test files, and the RETAIN-fact one), narrowing the reusable primitive design.md wanted — a future command handler (e.g. a config-change command) gets no shared plumbing from this work and would need its own bespoke path. On FAILED/timeout, replies failure immediately via `jobsubmit_reply_failure_once()`, no job ever constructed. On COMMITTED, calls new `persist_register_pending_apply(p, lsn, on_applied, ctx)` — a hand-off linked list on `persist_t` for task 6.1 to consume — and does NOT apply or reply, per the task's explicit instruction. **Deleted (not preserved-dead) task 3.1's bug-1408 create_job-loop/notify block**: verified correct — `apply_committed_entry()`'s existing `WAL_JOB_SUBMIT` case (task 2.2) already fires `HOOK_DETAIL_JOB_NEW` per-entry independently, structurally immune to bug-1408's batch-suppression class; resurrecting the old loop in task 6 would double-construct/double-notify. Original code fully recoverable from `git show 6f1fa97:src/controller/job.c` if ever needed. **CRITICAL OPERATIONAL CALLOUT — this branch must NOT be soak-tested or deployed against live traffic until task 6 lands**: there is currently NO live code path that applies a successfully-committed job submission (only restart replay, task 2.3, does) — every successful job submission produces a durably-committed WAL entry but ZERO reply to the client, forever, until task 6.1/6.2 exist to consume the pending-apply hand-off and complete the reply. This is exactly what the task text sanctions ("no independently observable behavior until task 6"), not a bug, but it means this commit range is only safe in isolated unit-test form right now. **Hand-off mechanism for task 6.1's briefing**: `persist_register_pending_apply()`/`persist_has_pending_apply()` exist; there is NO "take and remove" accessor yet — task 6.1 must build the consumer side itself. The per-batch `jobsubmit_async_ctx_t` (ref-counted across array elements, frees only once every element has either failed-and-replied or been applied via `jobsubmit_apply_complete`) will not free for any batch containing a committed element until task 6 exists — an accepted, bounded-by-test-scope leak today, but would be unbounded growth under any live/soak run before task 6 lands. Outcome encoding uses a plain `int rc` (not `commit_outcome_t`+rc) to avoid a persist.h→commit.h circular dependency — verified unambiguous today (every real FAILED call site in commit.c passes non-zero rc) but not compiler-enforced; fragile by convention, not by construction. **New regression found by review, not yet fixed, non-blocking**: `array_size == 0` now hangs the client forever and leaks one `jobsubmit_async_ctx_t` per request (3.1's version replied synchronously even for this input; 3.2's async loop never runs so the ctx never releases) — confirmed NOT reachable via any current legitimate client (Go apiserver always sets `arraySize >= 1`; `xce_test`'s `cmd_jobsub` clamps `arsize <= 0` to 1), so flagged as a defense-in-depth hardening gap for a future task rather than fixed here. Also no `array_size > 1` mixed-outcome test exists yet (per-element independent resolution verified correct by code inspection, not by test) — fast-follow recommended.
- **3.1**: reordered `handle_jobsubmit()` (job.c) so job objects are only constructed via `create_job()` AFTER the (still-synchronous) WAL commit succeeds — extracted `jobid_range_exists()` (fast-fail pre-check, no WAL work), `build_job_submit_payloads()` (WAL-payload construction with no jobmgr insertion), and a new persist.c sibling `persist_submit_job_payloads_and_commit()` (operates on raw `xce_job_t*` instead of pre-existing `job_t*`, single production call site in job.c — legitimately within `handle_jobsubmit`'s call chain, not a boundary spillover; no task-1 continuation API dependency, genuinely still synchronous). Took 3 review rounds. **Round 1 REJECTED — real defect, not documentation**: the post-commit array-construction loop suppressed the `AFTER_EVENT`/`HOOK_DETAIL_JOB_NEW` notification for an ENTIRE array batch if even one sibling element failed to construct (e.g. a jobid collision slipping past `jobid_range_exists()`'s inherent TOCTOU window, or a plain allocation failure — no race needed) — stranding OTHER elements that were durably committed AND successfully constructed, invisible to the purely event-driven scheduler/apiserver until the next restart's replay. Logged as bug-1408. **Round 2 fixed the defect** (moved the `construct_rc < 0` early-return to AFTER the jlist-build + hook-fire block, so every successfully-constructed sibling is always notified regardless of a sibling's failure; client reply still separately reports the batch's `construct_rc`) but was REJECTED again on a narrower, documentation-only ground: bug-1408's RED-phase writeup claimed a clean assertion FAIL, but under the standard full-group `--run test_commit_barrier_reply` invocation the failing assertion actually longjmps past the test's own `cbrp_teardown()` call (Unity `TEST_ABORT` semantics) and HANGS the next test — the same longjmp-past-teardown pathology already known from task 1.3's era. **Round 3 (parent-handled directly, no code regression risk)**: corrected bug-1408's buglog.json description to accurately describe the hang mode, and hardened the new test itself — every assertion-relevant observable is now captured into a local, `cbrp_teardown()` is called unconditionally, and assertions run only after teardown, so a future regression in this exact test can no longer skip cleanup and deadlock sibling tests. No production code changed in round 3; job.c's fix from round 2 was independently verified correct by the round-2 reviewer (traced jlist-build/hook-fire at job.c:717-766 running unconditionally before the `construct_rc` check at line 775) and needed no further changes. **Known low-severity residual, not blocking, flagged for a future task**: `apiserver_push_joblist` (apiserver.c) lacks the `nfiltered > 0` guard `sched.c`'s equivalent has, so a total-batch construction failure (every array element fails, jlist ends up empty) now sends an empty/no-op jobs push to every connected apiserver — informationless, can't corrupt state, but a minor behavioral quirk introduced by firing the hook unconditionally.

188
src/controller/job.c
View file

@ -14,8 +14,10 @@
#include "xce/types.h"
#include "xstring.h"
static int mark_job_start(cluster_t*, job_t*, job_pidinfo_t*);
static int mark_job_finish(cluster_t*, job_t*, xce_jobfinish_t*);
static int record_job_start_fact(cluster_t*, job_t*, job_pidinfo_t*);
static int record_job_finish_fact(cluster_t*, job_t*, xce_jobfinish_t*);
static int record_job_fail_fact(cluster_t*, job_t*, job_failinfo_t*);
static int record_job_update_fact(cluster_t*, job_t*, int);
static int mark_job_fail(cluster_t*, job_t*, job_failinfo_t*);
static int mark_job_update(cluster_t*, job_t*, int);
@ -875,12 +877,14 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
}
/* Guarded state-transition core for a worker-reported START, extracted from
* mark_job_start (raft-apply-decoupling task 2.1, Req 3.3/7.1/7.2). Contains
* ONLY the monotonic-rank guard and the state mutation — it fires NO hook
* itself, so hook-firing can be relocated by a later task (the shared apply
* seam) without touching this guard logic again. Safe to invoke directly,
* bypassing the worker-report receipt path (on_job_start) entirely, as long
* as the caller already holds `job` under RTTI_LOCK_WRITE. */
* mark_job_start (raft-apply-decoupling task 2.1, Req 3.3/7.1/7.2; mark_job_start
* itself was renamed record_job_start_fact by task 4.1, which stopped calling
* this core from the receipt path — see that function's own doc comment).
* Contains ONLY the monotonic-rank guard and the state mutation — it fires NO
* hook itself, so hook-firing can be relocated by a later task (the shared
* apply seam) without touching this guard logic again. Safe to invoke
* directly, bypassing the worker-report receipt path (on_job_start) entirely,
* as long as the caller already holds `job` under RTTI_LOCK_WRITE. */
job_transition_result_t apply_job_start_transition(job_t* job,
job_pidinfo_t* info)
{
@ -917,14 +921,27 @@ job_transition_result_t apply_job_start_transition(job_t* job,
return JOB_TRANSITION_APPLIED;
}
static int mark_job_start(cluster_t* c, job_t* job, job_pidinfo_t* info)
/* raft-apply-decoupling task 4.1 (Req 5.1): renamed from mark_job_start —
* this is now durable-append-only. The pre-mutation BEFORE_EVENT validation
* hook still fires at the identical point, unchanged; the state mutation
* (apply_job_start_transition) and the AFTER_EVENT notification hook both
* relocate to run from apply_committed_entry() (task 2.2) once this fact's
* WAL entry commits to a majority — driven by the future apply pump
* (task 6). Until then, a worker-reported START has no live effect on job
* state; only the durable append happens here. */
static int record_job_start_fact(cluster_t* c, job_t* job, job_pidinfo_t* info)
{
/* Peek the identical guard apply_job_start_transition() re-checks
* internally, so this extraction changes NO observable behavior: today
* no hook fires on a rejected transition, and BEFORE_EVENT strictly
* precedes the mutation. The peek is a pure read of job->job.state, and
* every caller holds `job` under RTTI_LOCK_WRITE, so it cannot race with
* the core's own (redundant, defensive) re-check of the same guard. */
* precedes the (now-deferred) mutation. The peek is a pure read of
* job->job.state, and every caller holds `job` under RTTI_LOCK_WRITE, so
* it cannot race with the core's own (redundant, defensive) re-check of
* the same guard. Calling apply_job_start_transition here on the real
* `job` is safe and mutates nothing — its own internal guard re-check
* hits the identical condition and returns before any assignment; the
* call exists only to produce its WARN log line (logs + no-op,
* unchanged from before this task). */
if (job_state_is_backward(job->job.state, JOB_STAT_RUN)) {
apply_job_start_transition(job, info); /* logs + no-op, unchanged */
return 0;
@ -933,10 +950,13 @@ static int mark_job_start(cluster_t* c, job_t* job, job_pidinfo_t* info)
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_START, c, info, job);
apply_job_start_transition(job, info);
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_AFTER_EVENT,
HOOK_DETAIL_JOB_START, c, info, job);
/* Durable append only — no mutation, no AFTER_EVENT firing here (Req
* 5.1, 5.3). JOB_STAT_RUN is the resultant state a START always
* produces once past the guard above; apply_committed_entry() will
* re-validate the same guard against whatever state is current AT
* APPLY TIME before setting it. */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_START, job->job.jobid,
JOB_STAT_RUN);
return 0;
}
@ -961,7 +981,7 @@ int on_job_start(cluster_t* c, job_pidinfo_t* info)
}
lock(job, RTTI_LOCK_WRITE);
mark_job_start(c, job, info);
record_job_start_fact(c, job, info);
unlock(job);
dereference(job);
@ -970,11 +990,13 @@ int on_job_start(cluster_t* c, job_pidinfo_t* info)
}
/* Guarded state-transition core for a worker-reported FINISH, extracted from
* mark_job_finish (raft-apply-decoupling task 2.1, Req 3.3/7.1/7.2). Contains
* ONLY the monotonic-rank guard and the state mutation — fires NO hook, so
* hook-firing can be relocated later without touching this guard logic
* again. Safe to invoke directly, bypassing the worker-report receipt path
* (on_job_finish) entirely, as long as the caller already holds `job` under
* mark_job_finish (raft-apply-decoupling task 2.1, Req 3.3/7.1/7.2; mark_job_finish
* itself was renamed record_job_finish_fact by task 4.1, which stopped calling
* this core from the receipt path — see that function's own doc comment).
* Contains ONLY the monotonic-rank guard and the state mutation — fires NO
* hook, so hook-firing can be relocated later without touching this guard
* logic again. Safe to invoke directly, bypassing the worker-report receipt
* path (on_job_finish) entirely, as long as the caller already holds `job` under
* RTTI_LOCK_WRITE. */
job_transition_result_t apply_job_finish_transition(job_t* job,
xce_jobfinish_t* finish)
@ -1014,14 +1036,18 @@ job_transition_result_t apply_job_finish_transition(job_t* job,
return JOB_TRANSITION_APPLIED;
}
static int mark_job_finish(cluster_t* c, job_t* job, xce_jobfinish_t* finish)
/* raft-apply-decoupling task 4.1 (Req 5.1): renamed from mark_job_finish —
* durable-append-only, mirroring record_job_start_fact above. */
static int record_job_finish_fact(cluster_t* c, job_t* job,
xce_jobfinish_t* finish)
{
/* Peek the identical guard apply_job_finish_transition() re-checks
* internally, so this extraction changes NO observable behavior: today
* no hook fires on a duplicate-terminal FINISH, and BEFORE_EVENT strictly
* precedes the mutation. The peek is a pure read of job->job.state, and
* every caller holds `job` under RTTI_LOCK_WRITE, so it cannot race with
* the core's own (redundant, defensive) re-check of the same guard. */
* precedes the (now-deferred) mutation. The peek is a pure read of
* job->job.state, and every caller holds `job` under RTTI_LOCK_WRITE, so
* it cannot race with the core's own (redundant, defensive) re-check of
* the same guard. */
if (job_state_is_terminal(job->job.state)) {
apply_job_finish_transition(job, finish); /* logs + no-op, unchanged */
return 0;
@ -1030,10 +1056,12 @@ static int mark_job_finish(cluster_t* c, job_t* job, xce_jobfinish_t* finish)
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_EXIT, c, finish, job);
apply_job_finish_transition(job, finish);
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_AFTER_EVENT,
HOOK_DETAIL_JOB_EXIT, c, finish, job);
/* Durable append only — no mutation, no AFTER_EVENT firing here (Req
* 5.1, 5.3). The resultant state is deterministic from finish->exit_code
* alone, independent of current job state. */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_FINISH, job->job.jobid,
(finish->exit_code == 0) ? JOB_STAT_DONE
: JOB_STAT_EXIT);
return 0;
}
@ -1058,7 +1086,7 @@ int on_job_finish(cluster_t* c, xce_jobfinish_t* finish)
}
lock(job, RTTI_LOCK_WRITE);
mark_job_finish(c, job, finish);
record_job_finish_fact(c, job, finish);
unlock(job);
dereference(job);
@ -1100,6 +1128,12 @@ job_transition_result_t apply_job_fail_transition(job_t* job,
return JOB_TRANSITION_APPLIED;
}
/* Used only by job_ctrl's SIGKILL/SIGTERM not-started-job branch (a client
* command path, do_jobcontrol_op — out of raft-apply-decoupling task 4.1's
* boundary) — mutate-then-notify, unchanged. record_job_fail_fact below is
* the separate, durable-append-only sibling used by the worker-fact receipt
* path (on_job_fail); the two must stay independent since job_ctrl's
* synchronous command semantics are not this task's concern. */
static int mark_job_fail(cluster_t* c, job_t* job, job_failinfo_t* failinfo)
{
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
@ -1113,6 +1147,43 @@ static int mark_job_fail(cluster_t* c, job_t* job, job_failinfo_t* failinfo)
return 0;
}
/* raft-apply-decoupling task 4.1 (Req 5.1): the worker-fact receipt sibling
* of mark_job_fail above — durable-append-only, used only by on_job_fail.
* mark_job_fail itself is untouched (job_ctrl still needs its mutate-then-
* notify semantics; that command path is out of this task's boundary).
*
* mark_job_fail has no monotonic-rank guard (a FAIL always applies), so
* unlike record_job_start_fact/record_job_finish_fact there is no
* peek/rejected branch here — BEFORE_EVENT fires unconditionally, exactly as
* mark_job_fail's always does today.
*
* The resultant state depends on the CURRENT job state (PERR merges onto an
* existing DONE/EXIT terminal state rather than resetting it — job.c's
* pre-existing apply_job_fail_transition switch). That computation is
* reproduced here on a throwaway shallow copy of `job` — apply_job_fail_
* transition only ever touches `job->job.state`/`job->job.finish_time`
* (scalar fields, never pidinfo/exit_info), so a shallow copy is safe to
* discard afterward with no cleanup — rather than duplicating the switch,
* so this can never drift from what apply time will eventually compute. */
static int record_job_fail_fact(cluster_t* c, job_t* job,
job_failinfo_t* failinfo)
{
job_t scratch;
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_FAIL, c, failinfo, job);
scratch = *job;
apply_job_fail_transition(&scratch, failinfo);
/* Durable append only — no mutation of the real `job`, no AFTER_EVENT
* firing here (Req 5.1, 5.3). */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_FAIL, job->job.jobid,
(int)scratch.job.state);
return 0;
}
/* Guarded state-transition core for an UPDATE, extracted from
* mark_job_update (raft-apply-decoupling task 2.1, Req 3.3/7.1/7.2). Contains
* ONLY the monotonic-rank guards and the state mutation — fires NO hook, so
@ -1148,6 +1219,12 @@ job_transition_result_t apply_job_update_transition(job_t* job, int state)
return JOB_TRANSITION_APPLIED;
}
/* Used only by job_ctrl's SIGCONT/SIGSTOP suspend-resume branches (a client
* command path, do_jobcontrol_op — out of raft-apply-decoupling task 4.1's
* boundary) — mutate-then-notify, unchanged. record_job_update_fact below is
* the separate, durable-append-only sibling used by the worker-fact receipt
* path (on_job_postdone); the two must stay independent since job_ctrl's
* synchronous command semantics are not this task's concern. */
static int mark_job_update(cluster_t* c, job_t* job, int state)
{
int cur = job->job.state;
@ -1175,6 +1252,37 @@ static int mark_job_update(cluster_t* c, job_t* job, int state)
return 0;
}
/* raft-apply-decoupling task 4.1 (Req 5.1): the worker-fact receipt sibling
* of mark_job_update above — durable-append-only, used only by
* on_job_postdone. mark_job_update itself is untouched (job_ctrl still needs
* its mutate-then-notify semantics; that command path is out of this task's
* boundary). `state` here is the caller's already-computed target state
* (on_job_postdone computes it from the job's current state before calling
* this, exactly as it always has — that computation is untouched by this
* task; only what happens with the computed value changes). */
static int record_job_update_fact(cluster_t* c, job_t* job, int state)
{
int cur = job->job.state;
/* Peek the identical guards apply_job_update_transition() re-checks
* internally — same no-observable-behavior-change rationale as
* mark_job_update's peek above. */
if (job_state_is_backward(cur, state) || (state == cur)) {
apply_job_update_transition(job, state); /* logs + no-op, unchanged */
return 0;
}
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_UPDATE, c, job);
/* Durable append only — no mutation, no AFTER_EVENT firing here (Req
* 5.1, 5.3). */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, job->job.jobid,
state);
return 0;
}
int on_job_fail(cluster_t* c, job_failinfo_t* failinfo)
{
job_t* job = NULL;
@ -1195,7 +1303,7 @@ int on_job_fail(cluster_t* c, job_failinfo_t* failinfo)
}
lock(job, RTTI_LOCK_WRITE);
mark_job_fail(c, job, failinfo);
record_job_fail_fact(c, job, failinfo);
unlock(job);
dereference(job);
@ -1298,12 +1406,22 @@ int on_job_postdone(cluster_t* c, endpoint_t* ep, uint64_t jobid)
"has not been set",
job->job.jobid);
state |= JOB_STAT_EXIT;
/* ensure that job finish time has been set */
job->job.finish_time = time(NULL);
/* raft-apply-decoupling task 4.1 (Req 5.1): the backstop
* `job->job.finish_time = time(NULL)` write that used to live here
* is removed — no state mutation may happen at receipt time.
* finish_time is not part of the wal_jobstate_t wire format (only
* {jobid, state} travels — a pre-existing limitation, see
* apply_committed_entry's doc comment in persist.h), so this
* defensive backstop has no equivalent at apply time either; a
* post-done arriving with no prior exit status recorded no longer
* gets a defensive finish_time set anywhere. Low-severity residual,
* flagged in this task's status report — not fixed here (would
* require wire-format enrichment, explicitly out of this task's
* scope per persist.h's Apply Seam doc comment). */
}
state |= JOB_STAT_PDONE;
mark_job_update(c, job, state);
record_job_update_fact(c, job, state);
unlock(job);
dereference(job);

21
src/controller/persist.c
View file

@ -3931,6 +3931,27 @@ int persist_has_pending_apply(persist_t* p, uint64_t lsn)
return found;
}
/* raft-apply-decoupling task 4.1 (Req 5.1): the receipt-time durable-append
* primitive for a worker-reported job-state fact. Unlike persist_job_hook_fun
* below (an AFTER_EVENT hook that fires only once a mutation has already
* happened), this is a direct, hook-fan-out-free call: job.c's fact receipt
* handlers (record_job_start_fact/record_job_finish_fact/record_job_fail_fact/
* record_job_update_fact) call this INSTEAD of mutating and firing
* AFTER_EVENT, so no notification (scheduler dispatch, slot release,
* apiserver push) can ever fire for an uncommitted fact (Req 5.3). `state`
* is always WAL_MODE_RETAIN for every job-state fact type (verified against
* wal_classify's table) — hardcoded here rather than routed through
* wal_classify since none of these types are detail-disambiguated the way
* WAL_WORKER_CTRL is. */
int persist_submit_job_fact(persist_t* p, uint32_t wal_type, uint64_t jobid,
int state)
{
wal_jobstate_t js = { .jobid = jobid, .state = (uint32_t)state };
return wal_hook_append_classified(p, wal_type, WAL_MODE_RETAIN, &js,
sizeof(js));
}
/* P0 payload: just the detail value (4 bytes).
* Full Cap'n Proto serialization is wired in Task 15. */

376
src/controller/testsuite/test_commit_barrier_reply.c
View file

@ -2078,61 +2078,61 @@ static void test_fact_retention_across_quorum_loss(void)
}
/* ════════════════════════════════════════════════════════════════════════
* Idempotency / ordering guards for mark_job_* (task 6.1, Req 5.1/5.2/5.3)
* Idempotency / ordering guards for the fact-transition receipt handlers
* (guard extraction: task 2.1, Req 5.1/5.2/5.3 — landed and green)
* ════════════════════════════════════════════════════════════════════════
*
* RED-phase tests for the transition guards that task 6.2 adds at the top
* of mark_job_start / mark_job_finish / mark_job_fail / mark_job_update
* (job.c:583/638/696/726 pre-task anchors). Design §Idempotency Guards:
* - duplicate (already-reflected transition) → INFO log, NO-OP: state
* unchanged, BEFORE/AFTER_EVENT hooks NOT fired → NO WAL entry appended
* (current_lsn unmoved), caller-visible rc treated as success-no-op (5.1)
* Pins the transition guards task 2.1 extracted into apply_job_start_
* transition / apply_job_finish_transition / apply_job_fail_transition /
* apply_job_update_transition, and peeked at the top of the receipt-side
* wrappers (record_job_start_fact / record_job_finish_fact — renamed from
* mark_job_start / mark_job_finish by task 4.1 — and mark_job_fail /
* mark_job_update, still used by job_ctrl's separate command path). Design
* §Idempotency Guards:
* - duplicate (already-reflected transition) → INFO log, NO-OP: no WAL
* entry appended (current_lsn unmoved), caller-visible rc treated as
* success-no-op (5.1)
* - backward (terminal current state, non-terminal requested: START on a
* finished job, UPDATE clearing terminal bits) → WARN log, REJECT,
* terminal state preserved (5.2)
* - replay: re-applying the same transition converges to single-
* application state — no second WAL entry, identical final state (5.3)
* - mark_job_fail's existing fail-after-finish PERR-merge (job.c:702-716)
* is forward-compatible and must be PRESERVED by the guard (pin)
* finished job, UPDATE clearing terminal bits) → WARN log, REJECT, no
* WAL entry appended (5.2)
* - replay: re-applying the same transition appends no second WAL entry
* (5.3)
* - apply_job_fail_transition's existing fail-after-finish PERR-merge is
* forward-compatible and must be PRESERVED by the guard (pin)
*
* Drivers: the static mark_job_* are reached through their non-static
* callers on_job_start / on_job_finish / on_job_fail (job.c) — exactly the
* Drivers: on_job_start / on_job_finish / on_job_fail (job.c) — exactly the
* fact path used by test_fact_retention_across_quorum_loss, minus the
* opcode decode. These wrappers each return 0 unconditionally today, so the
* opcode decode. These wrappers each return 0 unconditionally, so the
* "success-no-op rc" observable (every caller treats a guarded-out
* transition as success) is asserted directly on the wrapper rc.
*
* Observability of state: the per-job WAL append count and the LAST
* recorded post-transition state come from cbrp_count_jobstate_entries
* (direct on-disk scan, defined above); the job's LIVE in-memory state is
* read via cbrp_job_live_state (key_find on g_hc.jobmgr). current_lsn is
* the "did a hook fire / WAL entry append" oracle (a guarded-out no-op runs
* no hooks → no persist_wal_async → current_lsn frozen).
* Observability: the per-job WAL append count and the LAST recorded
* post-transition state come from cbrp_count_jobstate_entries (direct
* on-disk scan, defined above) — current_lsn is the "did a WAL entry
* append" oracle (a guarded-out no-op never calls persist_wal_async →
* current_lsn frozen). cbrp_job_live_state remains meaningful in these
* tests only because cbrp_drive_to_done (task 4.1) now reaches its
* terminal precondition by applying the guard cores directly, as a
* test-only stand-in for the not-yet-built live apply pump — since task 4.1,
* the receipt handlers themselves never mutate live state on ANY path
* (guarded-out or accepted), so a passing guard test no longer implies
* "the guard blocked a live mutation," only "the guard blocked a WAL
* append." test_guard_forward_update_on_terminal_preserved and
* test_guard_fail_after_finish_preserves_perr_merge were updated by task 4.1
* to assert the forward-merge/PERR-merge outcome against the WAL-recorded
* fact instead of live state, plus an explicit live-state-unchanged pin.
*
* RED expectation against the UNGUARDED mark_job_* (no guard yet):
* - duplicate FINISH re-fires the AFTER_EVENT hook → a 2nd WAL_JOB_FINISH
* append, current_lsn advances → the "current_lsn unmoved / count==1"
* assertions FAIL.
* - START on a DONE job unconditionally sets state = JOB_STAT_RUN and
* appends WAL_JOB_START → the "state preserved DONE / no new append"
* assertions FAIL.
* - replay FINISH appends a duplicate entry → the "single append"
* assertion FAILS.
* The fail-after-finish PERR-merge pin and the success-no-op rc pins lock
* behavior that must hold both before and after the guard (preservation
* pins, like the task-3.1 validation pins); they ride this group so 6.2
* cannot silently regress the merge or the wrapper return contract.
*
* NOTE on the UPDATE-clears-terminal arm (5.2): mark_job_update is static
* with no public caller that produces a terminal→non-terminal state
* (on_job_postdone's update is forward — it OR's PDONE, preserving terminal;
* job_ctrl SIGCONT only clears suspend bits). mark_job_start IS the
* canonical "set state to the non-terminal JOB_STAT_RUN" overwrite, so the
* START-on-terminal case below is the faithful, publicly-reachable witness
* of the shared backward rule the design lists for START and UPDATE
* together. test_guard_forward_update_on_terminal_preserved additionally
* pins that a FORWARD update on a terminal job (on_job_postdone) is NOT
* over-rejected — the boundary the guard must respect.
* NOTE on the UPDATE-clears-terminal arm (5.2): mark_job_update/
* record_job_update_fact have no public caller that produces a
* terminal→non-terminal state (on_job_postdone's update is forward — it
* OR's PDONE, preserving terminal; job_ctrl SIGCONT only clears suspend
* bits). record_job_start_fact IS the canonical "set state to the
* non-terminal JOB_STAT_RUN" overwrite, so the START-on-terminal case below
* is the faithful, publicly-reachable witness of the shared backward rule
* the design lists for START and UPDATE together.
* test_guard_forward_update_on_terminal_preserved additionally pins that a
* FORWARD update on a terminal job (on_job_postdone) is NOT over-rejected —
* the boundary the guard must respect.
*/
/* Read a job's live in-memory state from g_hc.jobmgr (post-transition).
@ -2160,13 +2160,27 @@ static int cbrp_job_live_state(uint64_t jobid)
/* Drive a job to terminal DONE via start(jobid) → finish(jobid, exit 0).
* Returns current_lsn after the two forward facts (the watermark the
* duplicate/backward cases must NOT advance). */
* duplicate/backward cases must NOT advance).
*
* raft-apply-decoupling task 4.1: on_job_start/on_job_finish are now
* durable-append-only — they no longer mutate the job's LIVE state (that
* moves to apply_committed_entry(), driven by the not-yet-built live apply
* pump, task 6). This helper's callers still need the job's LIVE state to
* reach a genuine terminal DONE so they can exercise backward-reject /
* duplicate-noop / forward-merge guard behavior against a real terminal
* job, so — after driving the two facts through the real receipt path to
* verify the durable append — this also applies the identical two
* transitions directly through the task-2.1 guard cores
* (apply_job_start_transition/apply_job_finish_transition), a test-only
* stand-in for what the future apply pump will do live once an entry
* commits. This is not a production code path. */
static uint64_t cbrp_drive_to_done(uint64_t jobid)
{
static unsigned int gd_pids[1] = { 9999 };
job_pidinfo_t start;
xce_jobfinish_t fin;
uint64_t lsn;
job_t* job = NULL;
memset(&start, 0, sizeof(start));
start.jobid = jobid;
@ -2179,6 +2193,21 @@ static uint64_t cbrp_drive_to_done(uint64_t jobid)
fin.exit_code = 0; /* → JOB_STAT_DONE */
TEST_ASSERT_EQUAL_INT(0, on_job_finish(&g_hc, &fin));
lock(g_hc.jobmgr, RTTI_LOCK_READ);
key_find(g_hc.jobmgr, (void *)jobid, (void **)&job);
if (job != NULL) {
reference(job);
}
unlock(g_hc.jobmgr);
TEST_ASSERT_NOT_NULL_MESSAGE(job,
"job must exist in jobmgr after cbrp_make_job + on_job_start");
lock(job, RTTI_LOCK_WRITE);
apply_job_start_transition(job, &start);
apply_job_finish_transition(job, &fin);
unlock(job);
dereference(job);
TEST_ASSERT_EQUAL_INT_MESSAGE(JOB_STAT_DONE, cbrp_job_live_state(jobid),
"precondition: job must be DONE after start→finish(0)");
@ -2238,9 +2267,10 @@ static void test_guard_duplicate_finish_on_done_is_noop(void)
/* 5.2 — START on a finished (DONE) job is rejected: the terminal state is
* preserved (never overwritten with JOB_STAT_RUN), NO WAL_JOB_START entry is
* appended, and on_job_start returns a success-no-op code that the caller
* (do_notify) treats as success. mark_job_start is the canonical
* terminal→non-terminal overwrite, so this also witnesses the UPDATE-clears-
* terminal arm of the shared backward rule (5.2). */
* (do_notify) treats as success. record_job_start_fact (renamed from
* mark_job_start by task 4.1) is the canonical terminal→non-terminal
* overwrite, so this also witnesses the UPDATE-clears-terminal arm of the
* shared backward rule (5.2). */
static void test_guard_start_on_terminal_rejected_state_preserved(void)
{
static unsigned int rs_pids[1] = { 7001 };
@ -2341,15 +2371,25 @@ static void test_guard_replay_finish_equals_single_application(void)
}
/* 5.2 boundary pin — a FORWARD update on a terminal job must NOT be
* over-rejected by the guard. on_job_postdone is the reachable mark_job_update
* caller: on a DONE job it OR's JOB_STAT_PDONE (terminal preserved, strictly
* forward). This must still apply (state gains PDONE, an entry appends) after
* the guard lands — the guard rejects backward, never a forward merge.
* Preservation pin: green before and after task 6.2. */
* over-rejected by the guard. on_job_postdone is the reachable
* record_job_update_fact caller: on a DONE job it OR's JOB_STAT_PDONE
* (terminal preserved, strictly forward). This must still durably append
* (an entry appends, carrying the merged state) — the guard rejects
* backward, never a forward merge. Preservation pin: green before and after
* task 6.2.
*
* raft-apply-decoupling task 4.1: on_job_postdone is now durable-append-only
* — the oracle for "did the forward merge compute correctly" moves from live
* state to the WAL-recorded fact (cbrp_count_jobstate_entries), mirroring
* test_fact_retention_across_quorum_loss. The live-state-unchanged
* assertion is this task's own new pin (Req 5.1): live state must still show
* only DONE (from cbrp_drive_to_done's direct apply), never PDONE, since
* nothing applies this fact live yet. */
static void test_guard_forward_update_on_terminal_preserved(void)
{
uint64_t lsn_after_done, lsn_after_pdone;
int st;
uint32_t st_disk = 0;
int st_live;
cbrp_setup(3000);
cbrp_make_job(73, &g_job_submit);
@ -2363,31 +2403,54 @@ static void test_guard_forward_update_on_terminal_preserved(void)
pthread_mutex_lock(&g_hc.persist.lock);
lsn_after_pdone = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
st = cbrp_job_live_state(73);
/* Assert: terminal preserved, PDONE merged forward, the update appended */
TEST_ASSERT_TRUE_MESSAGE((st & JOB_STAT_DONE) != 0,
"a forward update on a terminal job must preserve the terminal bit");
TEST_ASSERT_TRUE_MESSAGE((st & JOB_STAT_PDONE) != 0,
"the forward update must merge JOB_STAT_PDONE (not be rejected)");
/* Assert: the update appended, carrying the correctly forward-merged
* state (terminal DONE preserved, PDONE merged in — not rejected). */
TEST_ASSERT_TRUE_MESSAGE(lsn_after_pdone > lsn_after_done,
"a forward update on a terminal job must still append its WAL entry "
"— the guard rejects backward transitions, never forward merges");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_UPDATE, 73,
&st_disk),
"exactly one UPDATE fact must be durably recorded (Req 5.1)");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_DONE) != 0,
"the recorded UPDATE fact must preserve the terminal bit");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_PDONE) != 0,
"the recorded UPDATE fact must merge JOB_STAT_PDONE (not be "
"rejected)");
/* Assert (Req 5.1, task 4.1's own observable): live state is UNCHANGED
* by the receipt handler — still just DONE, no PDONE — since nothing
* applies this fact live until the apply pump (task 6) exists. */
st_live = cbrp_job_live_state(73);
TEST_ASSERT_EQUAL_INT_MESSAGE(JOB_STAT_DONE, st_live,
"on_job_postdone must leave live job state unchanged (Req 5.1) — "
"the PDONE merge above is recorded in the WAL only, not applied "
"live, until the apply pump exists");
cbrp_teardown();
printf("test_guard_forward_update_on_terminal_preserved: PASS\n");
}
/* 5.2 pin — mark_job_fail's existing fail-after-finish PERR-merge
* (job.c:702-716) must be PRESERVED by the guard: a POST_EXEC-stage fail on a
* DONE job keeps DONE and OR's JOB_STAT_PERR (it does NOT reset the job to a
* bare JOB_STAT_EXIT). Forward-compatible terminal→terminal merge — the guard
* must not reject it. Preservation pin: green before and after task 6.2. */
/* 5.2 pin — apply_job_fail_transition's existing fail-after-finish PERR-merge
* must be PRESERVED: a POST_EXEC-stage fail on a DONE job keeps DONE and
* OR's JOB_STAT_PERR (it does NOT reset the job to a bare JOB_STAT_EXIT).
* Forward-compatible terminal→terminal merge — the guard must not reject it.
* Preservation pin: green before and after task 6.2.
*
* raft-apply-decoupling task 4.1: on_job_fail (via the new
* record_job_fail_fact) is now durable-append-only — it computes the
* PERR-merged target state on a throwaway copy of the job and durably
* appends it, without touching the real job's live state. The oracle for
* "did the merge compute correctly" moves from live state to the
* WAL-recorded fact (cbrp_count_jobstate_entries); the live-state-unchanged
* assertion is this task's own new pin (Req 5.1). */
static void test_guard_fail_after_finish_preserves_perr_merge(void)
{
job_failinfo_t failinfo;
uint64_t lsn_after_done, lsn_after_fail;
int st;
uint32_t st_disk = 0;
int st_live;
cbrp_setup(3000);
cbrp_make_job(74, &g_job_submit);
@ -2406,23 +2469,175 @@ static void test_guard_fail_after_finish_preserves_perr_merge(void)
pthread_mutex_lock(&g_hc.persist.lock);
lsn_after_fail = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
st = cbrp_job_live_state(74);
/* Assert: DONE preserved, PERR merged in — not reset to bare EXIT */
TEST_ASSERT_TRUE_MESSAGE((st & JOB_STAT_DONE) != 0,
"fail-after-finish must keep the DONE terminal bit (PERR-merge "
"behavior, job.c:702-716)");
TEST_ASSERT_TRUE_MESSAGE((st & JOB_STAT_PERR) != 0,
"fail-after-finish must OR JOB_STAT_PERR onto the existing terminal "
"state (PERR-merge behavior, job.c:702-716)");
/* Assert: the FAIL fact appended, carrying the correctly PERR-merged
* state — DONE preserved, PERR merged in, not reset to bare EXIT. */
TEST_ASSERT_TRUE_MESSAGE(lsn_after_fail > lsn_after_done,
"the PERR-merge fail must still append its WAL entry (forward "
"terminal→terminal merge — not a guarded-out no-op)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FAIL, 74,
&st_disk),
"exactly one FAIL fact must be durably recorded (Req 5.1)");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_DONE) != 0,
"the recorded FAIL fact must keep the DONE terminal bit "
"(PERR-merge behavior)");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_PERR) != 0,
"the recorded FAIL fact must OR JOB_STAT_PERR onto the existing "
"terminal state (PERR-merge behavior)");
/* Assert (Req 5.1, task 4.1's own observable): live state is UNCHANGED
* by the receipt handler — still just DONE, no PERR — since nothing
* applies this fact live until the apply pump (task 6) exists. */
st_live = cbrp_job_live_state(74);
TEST_ASSERT_EQUAL_INT_MESSAGE(JOB_STAT_DONE, st_live,
"on_job_fail must leave live job state unchanged (Req 5.1) — the "
"PERR merge above is recorded in the WAL only, not applied live, "
"until the apply pump exists");
cbrp_teardown();
printf("test_guard_fail_after_finish_preserves_perr_merge: PASS\n");
}
/* ════════════════════════════════════════════════════════════════════════
* raft-apply-decoupling task 4.1 (Req 5.1) — receipt handlers durably
* append only, no live mutation, no AFTER_EVENT firing
* ════════════════════════════════════════════════════════════════════════
*
* Spy hook counting BEFORE_EVENT vs AFTER_EVENT firings for
* XCE_CONTROLLER_HOOK_JOB — registered alongside persist_start's production
* hooks (persist_job_hook_fun etc., a fan-out, so both subscribers coexist).
* Used to pin: the pre-mutation validation hook (BEFORE_EVENT) still fires
* exactly once per accepted fact, unchanged; AFTER_EVENT never fires at
* receipt time any more (it now only fires from apply_committed_entry(),
* which nothing drives yet — task 6). */
static int g_cbrp_before_fires;
static int g_cbrp_after_fires;
static int cbrp_job_hook_spy(int type, int situation, int detail, void *data,
va_list ap)
{
(void)type; (void)detail; (void)data; (void)ap;
if (situation == HOOK_BEFORE_EVENT)
g_cbrp_before_fires++;
else if (situation == HOOK_AFTER_EVENT)
g_cbrp_after_fires++;
return 0;
}
/* The task's own RED/GREEN observable: after each of on_job_start/
* on_job_finish/on_job_fail/on_job_postdone returns, the job's LIVE
* in-memory state is unchanged — even though every one of the four facts
* durably appends with the correct target state, and the pre-mutation
* BEFORE_EVENT hook still fires exactly once per accepted fact. Against the
* pre-task-4.1 code (mutate-then-notify at receipt) this test's live-state-
* unchanged assertions fail on the very first fact (START moves live state
* PEND -> RUN immediately) — confirmed by running this test against a
* revert of job.c/persist.c/persist.h before applying this task's fix. */
static void test_receipt_facts_durably_append_without_live_mutation(void)
{
static unsigned int rf_pids[1] = { 6001 };
job_pidinfo_t start;
xce_jobfinish_t fin;
job_failinfo_t failinfo;
uint32_t st_disk = 0;
int live_before, live_after;
cbrp_setup(3000);
cbrp_make_job(90, &g_job_submit);
memset(&g_job_submit, 0, sizeof(g_job_submit));
g_cbrp_before_fires = 0;
g_cbrp_after_fires = 0;
TEST_ASSERT_EQUAL_INT(0,
xce_add_hook(&g_hc.hooksys, XCE_CONTROLLER_HOOK_JOB, cbrp_job_hook_spy,
NULL));
/* START */
live_before = cbrp_job_live_state(90);
memset(&start, 0, sizeof(start));
start.jobid = 90;
start.npids = 1;
start.pids = &rf_pids[0];
TEST_ASSERT_EQUAL_INT(0, on_job_start(&g_hc, &start));
live_after = cbrp_job_live_state(90);
TEST_ASSERT_EQUAL_INT_MESSAGE(live_before, live_after,
"on_job_start must leave live job state unchanged (Req 5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_START, 90,
&st_disk),
"on_job_start must durably append exactly one START fact (Req 5.1)");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, st_disk);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_cbrp_before_fires,
"the pre-mutation validation hook (BEFORE_EVENT) must still fire "
"unchanged at receipt time");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
"AFTER_EVENT must NOT fire at receipt time any more (Req 5.1, 5.3)");
/* FINISH */
live_before = cbrp_job_live_state(90);
memset(&fin, 0, sizeof(fin));
fin.jobid = 90;
fin.exit_code = 0;
TEST_ASSERT_EQUAL_INT(0, on_job_finish(&g_hc, &fin));
live_after = cbrp_job_live_state(90);
TEST_ASSERT_EQUAL_INT_MESSAGE(live_before, live_after,
"on_job_finish must leave live job state unchanged (Req 5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FINISH, 90,
&st_disk),
"on_job_finish must durably append exactly one FINISH fact (Req "
"5.1)");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_DONE, st_disk);
TEST_ASSERT_EQUAL_INT_MESSAGE(2, g_cbrp_before_fires,
"BEFORE_EVENT must fire again for the FINISH fact");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
"AFTER_EVENT must still never fire at receipt time");
/* FAIL */
live_before = cbrp_job_live_state(90);
memset(&failinfo, 0, sizeof(failinfo));
failinfo.jobid = 90;
failinfo.stage = JOB_EXEC_STAGE_POST_EXEC;
failinfo.code = 1;
failinfo.detail = (char *)"synthetic fail for no-mutation pin";
TEST_ASSERT_EQUAL_INT(0, on_job_fail(&g_hc, &failinfo));
live_after = cbrp_job_live_state(90);
TEST_ASSERT_EQUAL_INT_MESSAGE(live_before, live_after,
"on_job_fail must leave live job state unchanged (Req 5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FAIL, 90,
&st_disk),
"on_job_fail must durably append exactly one FAIL fact (Req 5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(3, g_cbrp_before_fires,
"BEFORE_EVENT must fire again for the FAIL fact — mark_job_fail has "
"no guard, so it always fires");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
"AFTER_EVENT must still never fire at receipt time");
/* UPDATE (via on_job_postdone) */
live_before = cbrp_job_live_state(90);
TEST_ASSERT_EQUAL_INT(0, on_job_postdone(&g_hc, &g_req_ep, 90));
live_after = cbrp_job_live_state(90);
TEST_ASSERT_EQUAL_INT_MESSAGE(live_before, live_after,
"on_job_postdone must leave live job state unchanged (Req 5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_UPDATE, 90,
&st_disk),
"on_job_postdone must durably append exactly one UPDATE fact (Req "
"5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(4, g_cbrp_before_fires,
"BEFORE_EVENT must fire again for the UPDATE fact");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
"AFTER_EVENT must NEVER fire at receipt time for any of the four "
"fact types (Req 5.1, 5.3) — it now only fires from "
"apply_committed_entry(), which nothing drives live until task 6");
cbrp_teardown();
printf("test_receipt_facts_durably_append_without_live_mutation: "
"PASS\n");
}
/* ── Test group entry point ──────────────────────────────────────────── */
void test_commit_barrier_reply(void)
@ -2470,14 +2685,15 @@ void test_commit_barrier_reply(void)
/* Fact retention across quorum loss (task 4.3, Req 3.1, 3.3, 3.4) —
* green on landing against tasks 1.2 + 4.2 output */
RUN_TEST(test_fact_retention_across_quorum_loss);
/* mark_job_* idempotency / ordering guards (task 6.1, Req 5.1/5.2/5.3)
* — the no-op / backward-reject / replay cases FAIL (red) against the
* unguarded mark_job_*; the forward-merge and PERR-merge cases are
* preservation pins (green before and after task 6.2) */
/* Fact-transition idempotency / ordering guards (extracted by task 2.1,
* already landed and green — the stale "RED-phase for task 6.2" framing
* this comment block used to carry predates that landing) */
RUN_TEST(test_guard_duplicate_finish_on_done_is_noop);
RUN_TEST(test_guard_start_on_terminal_rejected_state_preserved);
RUN_TEST(test_guard_replay_finish_equals_single_application);
RUN_TEST(test_guard_forward_update_on_terminal_preserved);
RUN_TEST(test_guard_fail_after_finish_preserves_perr_merge);
/* Fact receipt handlers durably append only (task 4.1, Req 5.1) */
RUN_TEST(test_receipt_facts_durably_append_without_live_mutation);
UNITY_END();
}

18
src/include/persist.h
View file

@ -832,6 +832,24 @@ int persist_job_hook_fun(int type, int situation, int detail,
void* cls, va_list ap);
int persist_queue_hook_fun(int type, int situation, int detail,
void* cls, va_list ap);
/* Durable-append-only entry point for a worker-reported job-state fact
* (raft-apply-decoupling task 4.1, Req 5.1): encodes {jobid, state} into the
* shared wal_jobstate_t wire format and appends it via the same
* wal_hook_append_classified() path persist_job_hook_fun already uses for
* these WAL types (always WAL_MODE_RETAIN for WAL_JOB_{START,FINISH,FAIL,
* UPDATE} — no commit timeout, never blocks) — but bypasses the hook fan-out
* entirely, so calling this fires NO AFTER_EVENT notification. The
* mutation this fact eventually drives, and the AFTER_EVENT notification
* itself, both happen later from apply_committed_entry() once this entry
* commits to a majority (task 2.2's existing dispatch). `state` is the
* caller-computed resultant state (e.g. JOB_STAT_RUN for a START, or the
* PERR-merged value for a FAIL) — this function performs no guard checks or
* merge logic of its own, it only encodes and appends.
* Returns the local-append rc (0 on success); facts have no commit timeout
* to report, mirroring wal_hook_append_classified's own contract. */
int persist_submit_job_fact(persist_t* p, uint32_t wal_type, uint64_t jobid,
int state);
int persist_worker_hook_fun(int type, int situation, int detail,
void* cls, va_list ap);