lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-apply-decoupling): split fact receipt to durable-append-only (task 4.1)

Browse source 
Convert on_job_start/on_job_finish/on_job_fail/on_job_postdone to
validate and durably append only -- BEFORE_EVENT validation hook
unchanged, AFTER_EVENT (post-mutation) hook no longer fires at
receipt, live state mutation removed. Adds persist_submit_job_fact()
(persist.c) since the durable append previously lived inside the
very AFTER_EVENT hook this task silences; wire format is unchanged
and identical to what apply_committed_entry() already routes.

mark_job_start/mark_job_finish (single caller each) renamed in place
to record_job_start_fact/record_job_finish_fact. mark_job_fail/
mark_job_update (shared with job_ctrl's client-command path, out of
boundary) left untouched; new durable-append-only siblings
record_job_fail_fact/record_job_update_fact added instead.

Accepts a narrow, permanent regression: on_job_postdone's finish_time
anomaly backstop has no wire-format equivalent and is dropped
(affects only a rare no-prior-exit-status path).

Two critical hazards discovered by review, documented in tasks.md,
MUST be resolved before task 6 wires apply_committed_entry() into a
live path: (1) persist_job_hook_fun still fires on HOOK_DETAIL_JOB_UPDATE
and will double-append every live-applied fact; (2) record_job_fail_fact/
record_job_update_fact precompute a final state that can be silently
clobbered by a concurrent intervening fact once apply is deferred by
the commit round-trip.
This commit is contained in:
Rongsong Shen 2026-07-04 07:00:09 +00:00
parent f357215a6d
commit 23cbfb48e1
5 changed files with 492 additions and 116 deletions
Download patch file Download diff file Expand all files Collapse all files

188
src/controller/job.c
View file

@ -14,8 +14,10 @@
#include "xce/types.h"
#include "xstring.h"
static int mark_job_start(cluster_t*, job_t*, job_pidinfo_t*);
static int mark_job_finish(cluster_t*, job_t*, xce_jobfinish_t*);
static int record_job_start_fact(cluster_t*, job_t*, job_pidinfo_t*);
static int record_job_finish_fact(cluster_t*, job_t*, xce_jobfinish_t*);
static int record_job_fail_fact(cluster_t*, job_t*, job_failinfo_t*);
static int record_job_update_fact(cluster_t*, job_t*, int);
static int mark_job_fail(cluster_t*, job_t*, job_failinfo_t*);
static int mark_job_update(cluster_t*, job_t*, int);
@ -875,12 +877,14 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
}
/* Guarded state-transition core for a worker-reported START, extracted from
* mark_job_start (raft-apply-decoupling task 2.1, Req 3.3/7.1/7.2). Contains
* ONLY the monotonic-rank guard and the state mutation — it fires NO hook
* itself, so hook-firing can be relocated by a later task (the shared apply
* seam) without touching this guard logic again. Safe to invoke directly,
* bypassing the worker-report receipt path (on_job_start) entirely, as long
* as the caller already holds `job` under RTTI_LOCK_WRITE. */
* mark_job_start (raft-apply-decoupling task 2.1, Req 3.3/7.1/7.2; mark_job_start
* itself was renamed record_job_start_fact by task 4.1, which stopped calling
* this core from the receipt path — see that function's own doc comment).
* Contains ONLY the monotonic-rank guard and the state mutation — it fires NO
* hook itself, so hook-firing can be relocated by a later task (the shared
* apply seam) without touching this guard logic again. Safe to invoke
* directly, bypassing the worker-report receipt path (on_job_start) entirely,
* as long as the caller already holds `job` under RTTI_LOCK_WRITE. */
job_transition_result_t apply_job_start_transition(job_t* job,
job_pidinfo_t* info)
{
@ -917,14 +921,27 @@ job_transition_result_t apply_job_start_transition(job_t* job,
return JOB_TRANSITION_APPLIED;
}
static int mark_job_start(cluster_t* c, job_t* job, job_pidinfo_t* info)
/* raft-apply-decoupling task 4.1 (Req 5.1): renamed from mark_job_start —
* this is now durable-append-only. The pre-mutation BEFORE_EVENT validation
* hook still fires at the identical point, unchanged; the state mutation
* (apply_job_start_transition) and the AFTER_EVENT notification hook both
* relocate to run from apply_committed_entry() (task 2.2) once this fact's
* WAL entry commits to a majority — driven by the future apply pump
* (task 6). Until then, a worker-reported START has no live effect on job
* state; only the durable append happens here. */
static int record_job_start_fact(cluster_t* c, job_t* job, job_pidinfo_t* info)
{
/* Peek the identical guard apply_job_start_transition() re-checks
* internally, so this extraction changes NO observable behavior: today
* no hook fires on a rejected transition, and BEFORE_EVENT strictly
* precedes the mutation. The peek is a pure read of job->job.state, and
* every caller holds `job` under RTTI_LOCK_WRITE, so it cannot race with
* the core's own (redundant, defensive) re-check of the same guard. */
* precedes the (now-deferred) mutation. The peek is a pure read of
* job->job.state, and every caller holds `job` under RTTI_LOCK_WRITE, so
* it cannot race with the core's own (redundant, defensive) re-check of
* the same guard. Calling apply_job_start_transition here on the real
* `job` is safe and mutates nothing — its own internal guard re-check
* hits the identical condition and returns before any assignment; the
* call exists only to produce its WARN log line (logs + no-op,
* unchanged from before this task). */
if (job_state_is_backward(job->job.state, JOB_STAT_RUN)) {
apply_job_start_transition(job, info); /* logs + no-op, unchanged */
return 0;
@ -933,10 +950,13 @@ static int mark_job_start(cluster_t* c, job_t* job, job_pidinfo_t* info)
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_START, c, info, job);
apply_job_start_transition(job, info);
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_AFTER_EVENT,
HOOK_DETAIL_JOB_START, c, info, job);
/* Durable append only — no mutation, no AFTER_EVENT firing here (Req
* 5.1, 5.3). JOB_STAT_RUN is the resultant state a START always
* produces once past the guard above; apply_committed_entry() will
* re-validate the same guard against whatever state is current AT
* APPLY TIME before setting it. */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_START, job->job.jobid,
JOB_STAT_RUN);
return 0;
}
@ -961,7 +981,7 @@ int on_job_start(cluster_t* c, job_pidinfo_t* info)
}
lock(job, RTTI_LOCK_WRITE);
mark_job_start(c, job, info);
record_job_start_fact(c, job, info);
unlock(job);
dereference(job);
@ -970,11 +990,13 @@ int on_job_start(cluster_t* c, job_pidinfo_t* info)
}
/* Guarded state-transition core for a worker-reported FINISH, extracted from
* mark_job_finish (raft-apply-decoupling task 2.1, Req 3.3/7.1/7.2). Contains
* ONLY the monotonic-rank guard and the state mutation — fires NO hook, so
* hook-firing can be relocated later without touching this guard logic
* again. Safe to invoke directly, bypassing the worker-report receipt path
* (on_job_finish) entirely, as long as the caller already holds `job` under
* mark_job_finish (raft-apply-decoupling task 2.1, Req 3.3/7.1/7.2; mark_job_finish
* itself was renamed record_job_finish_fact by task 4.1, which stopped calling
* this core from the receipt path — see that function's own doc comment).
* Contains ONLY the monotonic-rank guard and the state mutation — fires NO
* hook, so hook-firing can be relocated later without touching this guard
* logic again. Safe to invoke directly, bypassing the worker-report receipt
* path (on_job_finish) entirely, as long as the caller already holds `job` under
* RTTI_LOCK_WRITE. */
job_transition_result_t apply_job_finish_transition(job_t* job,
xce_jobfinish_t* finish)
@ -1014,14 +1036,18 @@ job_transition_result_t apply_job_finish_transition(job_t* job,
return JOB_TRANSITION_APPLIED;
}
static int mark_job_finish(cluster_t* c, job_t* job, xce_jobfinish_t* finish)
/* raft-apply-decoupling task 4.1 (Req 5.1): renamed from mark_job_finish —
* durable-append-only, mirroring record_job_start_fact above. */
static int record_job_finish_fact(cluster_t* c, job_t* job,
xce_jobfinish_t* finish)
{
/* Peek the identical guard apply_job_finish_transition() re-checks
* internally, so this extraction changes NO observable behavior: today
* no hook fires on a duplicate-terminal FINISH, and BEFORE_EVENT strictly
* precedes the mutation. The peek is a pure read of job->job.state, and
* every caller holds `job` under RTTI_LOCK_WRITE, so it cannot race with
* the core's own (redundant, defensive) re-check of the same guard. */
* precedes the (now-deferred) mutation. The peek is a pure read of
* job->job.state, and every caller holds `job` under RTTI_LOCK_WRITE, so
* it cannot race with the core's own (redundant, defensive) re-check of
* the same guard. */
if (job_state_is_terminal(job->job.state)) {
apply_job_finish_transition(job, finish); /* logs + no-op, unchanged */
return 0;
@ -1030,10 +1056,12 @@ static int mark_job_finish(cluster_t* c, job_t* job, xce_jobfinish_t* finish)
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_EXIT, c, finish, job);
apply_job_finish_transition(job, finish);
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_AFTER_EVENT,
HOOK_DETAIL_JOB_EXIT, c, finish, job);
/* Durable append only — no mutation, no AFTER_EVENT firing here (Req
* 5.1, 5.3). The resultant state is deterministic from finish->exit_code
* alone, independent of current job state. */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_FINISH, job->job.jobid,
(finish->exit_code == 0) ? JOB_STAT_DONE
: JOB_STAT_EXIT);
return 0;
}
@ -1058,7 +1086,7 @@ int on_job_finish(cluster_t* c, xce_jobfinish_t* finish)
}
lock(job, RTTI_LOCK_WRITE);
mark_job_finish(c, job, finish);
record_job_finish_fact(c, job, finish);
unlock(job);
dereference(job);
@ -1100,6 +1128,12 @@ job_transition_result_t apply_job_fail_transition(job_t* job,
return JOB_TRANSITION_APPLIED;
}
/* Used only by job_ctrl's SIGKILL/SIGTERM not-started-job branch (a client
* command path, do_jobcontrol_op — out of raft-apply-decoupling task 4.1's
* boundary) — mutate-then-notify, unchanged. record_job_fail_fact below is
* the separate, durable-append-only sibling used by the worker-fact receipt
* path (on_job_fail); the two must stay independent since job_ctrl's
* synchronous command semantics are not this task's concern. */
static int mark_job_fail(cluster_t* c, job_t* job, job_failinfo_t* failinfo)
{
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
@ -1113,6 +1147,43 @@ static int mark_job_fail(cluster_t* c, job_t* job, job_failinfo_t* failinfo)
return 0;
}
/* raft-apply-decoupling task 4.1 (Req 5.1): the worker-fact receipt sibling
* of mark_job_fail above — durable-append-only, used only by on_job_fail.
* mark_job_fail itself is untouched (job_ctrl still needs its mutate-then-
* notify semantics; that command path is out of this task's boundary).
*
* mark_job_fail has no monotonic-rank guard (a FAIL always applies), so
* unlike record_job_start_fact/record_job_finish_fact there is no
* peek/rejected branch here — BEFORE_EVENT fires unconditionally, exactly as
* mark_job_fail's always does today.
*
* The resultant state depends on the CURRENT job state (PERR merges onto an
* existing DONE/EXIT terminal state rather than resetting it — job.c's
* pre-existing apply_job_fail_transition switch). That computation is
* reproduced here on a throwaway shallow copy of `job` — apply_job_fail_
* transition only ever touches `job->job.state`/`job->job.finish_time`
* (scalar fields, never pidinfo/exit_info), so a shallow copy is safe to
* discard afterward with no cleanup — rather than duplicating the switch,
* so this can never drift from what apply time will eventually compute. */
static int record_job_fail_fact(cluster_t* c, job_t* job,
job_failinfo_t* failinfo)
{
job_t scratch;
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_FAIL, c, failinfo, job);
scratch = *job;
apply_job_fail_transition(&scratch, failinfo);
/* Durable append only — no mutation of the real `job`, no AFTER_EVENT
* firing here (Req 5.1, 5.3). */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_FAIL, job->job.jobid,
(int)scratch.job.state);
return 0;
}
/* Guarded state-transition core for an UPDATE, extracted from
* mark_job_update (raft-apply-decoupling task 2.1, Req 3.3/7.1/7.2). Contains
* ONLY the monotonic-rank guards and the state mutation — fires NO hook, so
@ -1148,6 +1219,12 @@ job_transition_result_t apply_job_update_transition(job_t* job, int state)
return JOB_TRANSITION_APPLIED;
}
/* Used only by job_ctrl's SIGCONT/SIGSTOP suspend-resume branches (a client
* command path, do_jobcontrol_op — out of raft-apply-decoupling task 4.1's
* boundary) — mutate-then-notify, unchanged. record_job_update_fact below is
* the separate, durable-append-only sibling used by the worker-fact receipt
* path (on_job_postdone); the two must stay independent since job_ctrl's
* synchronous command semantics are not this task's concern. */
static int mark_job_update(cluster_t* c, job_t* job, int state)
{
int cur = job->job.state;
@ -1175,6 +1252,37 @@ static int mark_job_update(cluster_t* c, job_t* job, int state)
return 0;
}
/* raft-apply-decoupling task 4.1 (Req 5.1): the worker-fact receipt sibling
* of mark_job_update above — durable-append-only, used only by
* on_job_postdone. mark_job_update itself is untouched (job_ctrl still needs
* its mutate-then-notify semantics; that command path is out of this task's
* boundary). `state` here is the caller's already-computed target state
* (on_job_postdone computes it from the job's current state before calling
* this, exactly as it always has — that computation is untouched by this
* task; only what happens with the computed value changes). */
static int record_job_update_fact(cluster_t* c, job_t* job, int state)
{
int cur = job->job.state;
/* Peek the identical guards apply_job_update_transition() re-checks
* internally — same no-observable-behavior-change rationale as
* mark_job_update's peek above. */
if (job_state_is_backward(cur, state) || (state == cur)) {
apply_job_update_transition(job, state); /* logs + no-op, unchanged */
return 0;
}
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_BEFORE_EVENT,
HOOK_DETAIL_JOB_UPDATE, c, job);
/* Durable append only — no mutation, no AFTER_EVENT firing here (Req
* 5.1, 5.3). */
(void)persist_submit_job_fact(&c->persist, WAL_JOB_UPDATE, job->job.jobid,
state);
return 0;
}
int on_job_fail(cluster_t* c, job_failinfo_t* failinfo)
{
job_t* job = NULL;
@ -1195,7 +1303,7 @@ int on_job_fail(cluster_t* c, job_failinfo_t* failinfo)
}
lock(job, RTTI_LOCK_WRITE);
mark_job_fail(c, job, failinfo);
record_job_fail_fact(c, job, failinfo);
unlock(job);
dereference(job);
@ -1298,12 +1406,22 @@ int on_job_postdone(cluster_t* c, endpoint_t* ep, uint64_t jobid)
"has not been set",
job->job.jobid);
state |= JOB_STAT_EXIT;
/* ensure that job finish time has been set */
job->job.finish_time = time(NULL);
/* raft-apply-decoupling task 4.1 (Req 5.1): the backstop
* `job->job.finish_time = time(NULL)` write that used to live here
* is removed — no state mutation may happen at receipt time.
* finish_time is not part of the wal_jobstate_t wire format (only
* {jobid, state} travels — a pre-existing limitation, see
* apply_committed_entry's doc comment in persist.h), so this
* defensive backstop has no equivalent at apply time either; a
* post-done arriving with no prior exit status recorded no longer
* gets a defensive finish_time set anywhere. Low-severity residual,
* flagged in this task's status report — not fixed here (would
* require wire-format enrichment, explicitly out of this task's
* scope per persist.h's Apply Seam doc comment). */
}
state |= JOB_STAT_PDONE;
mark_job_update(c, job, state);
record_job_update_fact(c, job, state);
unlock(job);
dereference(job);

21
src/controller/persist.c
View file

@ -3931,6 +3931,27 @@ int persist_has_pending_apply(persist_t* p, uint64_t lsn)
return found;
}
/* raft-apply-decoupling task 4.1 (Req 5.1): the receipt-time durable-append
* primitive for a worker-reported job-state fact. Unlike persist_job_hook_fun
* below (an AFTER_EVENT hook that fires only once a mutation has already
* happened), this is a direct, hook-fan-out-free call: job.c's fact receipt
* handlers (record_job_start_fact/record_job_finish_fact/record_job_fail_fact/
* record_job_update_fact) call this INSTEAD of mutating and firing
* AFTER_EVENT, so no notification (scheduler dispatch, slot release,
* apiserver push) can ever fire for an uncommitted fact (Req 5.3). `state`
* is always WAL_MODE_RETAIN for every job-state fact type (verified against
* wal_classify's table) — hardcoded here rather than routed through
* wal_classify since none of these types are detail-disambiguated the way
* WAL_WORKER_CTRL is. */
int persist_submit_job_fact(persist_t* p, uint32_t wal_type, uint64_t jobid,
int state)
{
wal_jobstate_t js = { .jobid = jobid, .state = (uint32_t)state };
return wal_hook_append_classified(p, wal_type, WAL_MODE_RETAIN, &js,
sizeof(js));
}
/* P0 payload: just the detail value (4 bytes).
* Full Cap'n Proto serialization is wired in Task 15. */

376
src/controller/testsuite/test_commit_barrier_reply.c
View file

@ -2078,61 +2078,61 @@ static void test_fact_retention_across_quorum_loss(void)
}
/* ════════════════════════════════════════════════════════════════════════
* Idempotency / ordering guards for mark_job_* (task 6.1, Req 5.1/5.2/5.3)
* Idempotency / ordering guards for the fact-transition receipt handlers
* (guard extraction: task 2.1, Req 5.1/5.2/5.3 — landed and green)
* ════════════════════════════════════════════════════════════════════════
*
* RED-phase tests for the transition guards that task 6.2 adds at the top
* of mark_job_start / mark_job_finish / mark_job_fail / mark_job_update
* (job.c:583/638/696/726 pre-task anchors). Design §Idempotency Guards:
* - duplicate (already-reflected transition) → INFO log, NO-OP: state
* unchanged, BEFORE/AFTER_EVENT hooks NOT fired → NO WAL entry appended
* (current_lsn unmoved), caller-visible rc treated as success-no-op (5.1)
* Pins the transition guards task 2.1 extracted into apply_job_start_
* transition / apply_job_finish_transition / apply_job_fail_transition /
* apply_job_update_transition, and peeked at the top of the receipt-side
* wrappers (record_job_start_fact / record_job_finish_fact — renamed from
* mark_job_start / mark_job_finish by task 4.1 — and mark_job_fail /
* mark_job_update, still used by job_ctrl's separate command path). Design
* §Idempotency Guards:
* - duplicate (already-reflected transition) → INFO log, NO-OP: no WAL
* entry appended (current_lsn unmoved), caller-visible rc treated as
* success-no-op (5.1)
* - backward (terminal current state, non-terminal requested: START on a
* finished job, UPDATE clearing terminal bits) → WARN log, REJECT,
* terminal state preserved (5.2)
* - replay: re-applying the same transition converges to single-
* application state — no second WAL entry, identical final state (5.3)
* - mark_job_fail's existing fail-after-finish PERR-merge (job.c:702-716)
* is forward-compatible and must be PRESERVED by the guard (pin)
* finished job, UPDATE clearing terminal bits) → WARN log, REJECT, no
* WAL entry appended (5.2)
* - replay: re-applying the same transition appends no second WAL entry
* (5.3)
* - apply_job_fail_transition's existing fail-after-finish PERR-merge is
* forward-compatible and must be PRESERVED by the guard (pin)
*
* Drivers: the static mark_job_* are reached through their non-static
* callers on_job_start / on_job_finish / on_job_fail (job.c) — exactly the
* Drivers: on_job_start / on_job_finish / on_job_fail (job.c) — exactly the
* fact path used by test_fact_retention_across_quorum_loss, minus the
* opcode decode. These wrappers each return 0 unconditionally today, so the
* opcode decode. These wrappers each return 0 unconditionally, so the
* "success-no-op rc" observable (every caller treats a guarded-out
* transition as success) is asserted directly on the wrapper rc.
*
* Observability of state: the per-job WAL append count and the LAST
* recorded post-transition state come from cbrp_count_jobstate_entries
* (direct on-disk scan, defined above); the job's LIVE in-memory state is
* read via cbrp_job_live_state (key_find on g_hc.jobmgr). current_lsn is
* the "did a hook fire / WAL entry append" oracle (a guarded-out no-op runs
* no hooks → no persist_wal_async → current_lsn frozen).
* Observability: the per-job WAL append count and the LAST recorded
* post-transition state come from cbrp_count_jobstate_entries (direct
* on-disk scan, defined above) — current_lsn is the "did a WAL entry
* append" oracle (a guarded-out no-op never calls persist_wal_async →
* current_lsn frozen). cbrp_job_live_state remains meaningful in these
* tests only because cbrp_drive_to_done (task 4.1) now reaches its
* terminal precondition by applying the guard cores directly, as a
* test-only stand-in for the not-yet-built live apply pump — since task 4.1,
* the receipt handlers themselves never mutate live state on ANY path
* (guarded-out or accepted), so a passing guard test no longer implies
* "the guard blocked a live mutation," only "the guard blocked a WAL
* append." test_guard_forward_update_on_terminal_preserved and
* test_guard_fail_after_finish_preserves_perr_merge were updated by task 4.1
* to assert the forward-merge/PERR-merge outcome against the WAL-recorded
* fact instead of live state, plus an explicit live-state-unchanged pin.
*
* RED expectation against the UNGUARDED mark_job_* (no guard yet):
* - duplicate FINISH re-fires the AFTER_EVENT hook → a 2nd WAL_JOB_FINISH
* append, current_lsn advances → the "current_lsn unmoved / count==1"
* assertions FAIL.
* - START on a DONE job unconditionally sets state = JOB_STAT_RUN and
* appends WAL_JOB_START → the "state preserved DONE / no new append"
* assertions FAIL.
* - replay FINISH appends a duplicate entry → the "single append"
* assertion FAILS.
* The fail-after-finish PERR-merge pin and the success-no-op rc pins lock
* behavior that must hold both before and after the guard (preservation
* pins, like the task-3.1 validation pins); they ride this group so 6.2
* cannot silently regress the merge or the wrapper return contract.
*
* NOTE on the UPDATE-clears-terminal arm (5.2): mark_job_update is static
* with no public caller that produces a terminal→non-terminal state
* (on_job_postdone's update is forward — it OR's PDONE, preserving terminal;
* job_ctrl SIGCONT only clears suspend bits). mark_job_start IS the
* canonical "set state to the non-terminal JOB_STAT_RUN" overwrite, so the
* START-on-terminal case below is the faithful, publicly-reachable witness
* of the shared backward rule the design lists for START and UPDATE
* together. test_guard_forward_update_on_terminal_preserved additionally
* pins that a FORWARD update on a terminal job (on_job_postdone) is NOT
* over-rejected — the boundary the guard must respect.
* NOTE on the UPDATE-clears-terminal arm (5.2): mark_job_update/
* record_job_update_fact have no public caller that produces a
* terminal→non-terminal state (on_job_postdone's update is forward — it
* OR's PDONE, preserving terminal; job_ctrl SIGCONT only clears suspend
* bits). record_job_start_fact IS the canonical "set state to the
* non-terminal JOB_STAT_RUN" overwrite, so the START-on-terminal case below
* is the faithful, publicly-reachable witness of the shared backward rule
* the design lists for START and UPDATE together.
* test_guard_forward_update_on_terminal_preserved additionally pins that a
* FORWARD update on a terminal job (on_job_postdone) is NOT over-rejected —
* the boundary the guard must respect.
*/
/* Read a job's live in-memory state from g_hc.jobmgr (post-transition).
@ -2160,13 +2160,27 @@ static int cbrp_job_live_state(uint64_t jobid)
/* Drive a job to terminal DONE via start(jobid) → finish(jobid, exit 0).
* Returns current_lsn after the two forward facts (the watermark the
* duplicate/backward cases must NOT advance). */
* duplicate/backward cases must NOT advance).
*
* raft-apply-decoupling task 4.1: on_job_start/on_job_finish are now
* durable-append-only — they no longer mutate the job's LIVE state (that
* moves to apply_committed_entry(), driven by the not-yet-built live apply
* pump, task 6). This helper's callers still need the job's LIVE state to
* reach a genuine terminal DONE so they can exercise backward-reject /
* duplicate-noop / forward-merge guard behavior against a real terminal
* job, so — after driving the two facts through the real receipt path to
* verify the durable append — this also applies the identical two
* transitions directly through the task-2.1 guard cores
* (apply_job_start_transition/apply_job_finish_transition), a test-only
* stand-in for what the future apply pump will do live once an entry
* commits. This is not a production code path. */
static uint64_t cbrp_drive_to_done(uint64_t jobid)
{
static unsigned int gd_pids[1] = { 9999 };
job_pidinfo_t start;
xce_jobfinish_t fin;
uint64_t lsn;
job_t* job = NULL;
memset(&start, 0, sizeof(start));
start.jobid = jobid;
@ -2179,6 +2193,21 @@ static uint64_t cbrp_drive_to_done(uint64_t jobid)
fin.exit_code = 0; /* → JOB_STAT_DONE */
TEST_ASSERT_EQUAL_INT(0, on_job_finish(&g_hc, &fin));
lock(g_hc.jobmgr, RTTI_LOCK_READ);
key_find(g_hc.jobmgr, (void *)jobid, (void **)&job);
if (job != NULL) {
reference(job);
}
unlock(g_hc.jobmgr);
TEST_ASSERT_NOT_NULL_MESSAGE(job,
"job must exist in jobmgr after cbrp_make_job + on_job_start");
lock(job, RTTI_LOCK_WRITE);
apply_job_start_transition(job, &start);
apply_job_finish_transition(job, &fin);
unlock(job);
dereference(job);
TEST_ASSERT_EQUAL_INT_MESSAGE(JOB_STAT_DONE, cbrp_job_live_state(jobid),
"precondition: job must be DONE after start→finish(0)");
@ -2238,9 +2267,10 @@ static void test_guard_duplicate_finish_on_done_is_noop(void)
/* 5.2 — START on a finished (DONE) job is rejected: the terminal state is
* preserved (never overwritten with JOB_STAT_RUN), NO WAL_JOB_START entry is
* appended, and on_job_start returns a success-no-op code that the caller
* (do_notify) treats as success. mark_job_start is the canonical
* terminal→non-terminal overwrite, so this also witnesses the UPDATE-clears-
* terminal arm of the shared backward rule (5.2). */
* (do_notify) treats as success. record_job_start_fact (renamed from
* mark_job_start by task 4.1) is the canonical terminal→non-terminal
* overwrite, so this also witnesses the UPDATE-clears-terminal arm of the
* shared backward rule (5.2). */
static void test_guard_start_on_terminal_rejected_state_preserved(void)
{
static unsigned int rs_pids[1] = { 7001 };
@ -2341,15 +2371,25 @@ static void test_guard_replay_finish_equals_single_application(void)
}
/* 5.2 boundary pin — a FORWARD update on a terminal job must NOT be
* over-rejected by the guard. on_job_postdone is the reachable mark_job_update
* caller: on a DONE job it OR's JOB_STAT_PDONE (terminal preserved, strictly
* forward). This must still apply (state gains PDONE, an entry appends) after
* the guard lands — the guard rejects backward, never a forward merge.
* Preservation pin: green before and after task 6.2. */
* over-rejected by the guard. on_job_postdone is the reachable
* record_job_update_fact caller: on a DONE job it OR's JOB_STAT_PDONE
* (terminal preserved, strictly forward). This must still durably append
* (an entry appends, carrying the merged state) — the guard rejects
* backward, never a forward merge. Preservation pin: green before and after
* task 6.2.
*
* raft-apply-decoupling task 4.1: on_job_postdone is now durable-append-only
* — the oracle for "did the forward merge compute correctly" moves from live
* state to the WAL-recorded fact (cbrp_count_jobstate_entries), mirroring
* test_fact_retention_across_quorum_loss. The live-state-unchanged
* assertion is this task's own new pin (Req 5.1): live state must still show
* only DONE (from cbrp_drive_to_done's direct apply), never PDONE, since
* nothing applies this fact live yet. */
static void test_guard_forward_update_on_terminal_preserved(void)
{
uint64_t lsn_after_done, lsn_after_pdone;
int st;
uint32_t st_disk = 0;
int st_live;
cbrp_setup(3000);
cbrp_make_job(73, &g_job_submit);
@ -2363,31 +2403,54 @@ static void test_guard_forward_update_on_terminal_preserved(void)
pthread_mutex_lock(&g_hc.persist.lock);
lsn_after_pdone = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
st = cbrp_job_live_state(73);
/* Assert: terminal preserved, PDONE merged forward, the update appended */
TEST_ASSERT_TRUE_MESSAGE((st & JOB_STAT_DONE) != 0,
"a forward update on a terminal job must preserve the terminal bit");
TEST_ASSERT_TRUE_MESSAGE((st & JOB_STAT_PDONE) != 0,
"the forward update must merge JOB_STAT_PDONE (not be rejected)");
/* Assert: the update appended, carrying the correctly forward-merged
* state (terminal DONE preserved, PDONE merged in — not rejected). */
TEST_ASSERT_TRUE_MESSAGE(lsn_after_pdone > lsn_after_done,
"a forward update on a terminal job must still append its WAL entry "
"— the guard rejects backward transitions, never forward merges");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_UPDATE, 73,
&st_disk),
"exactly one UPDATE fact must be durably recorded (Req 5.1)");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_DONE) != 0,
"the recorded UPDATE fact must preserve the terminal bit");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_PDONE) != 0,
"the recorded UPDATE fact must merge JOB_STAT_PDONE (not be "
"rejected)");
/* Assert (Req 5.1, task 4.1's own observable): live state is UNCHANGED
* by the receipt handler — still just DONE, no PDONE — since nothing
* applies this fact live until the apply pump (task 6) exists. */
st_live = cbrp_job_live_state(73);
TEST_ASSERT_EQUAL_INT_MESSAGE(JOB_STAT_DONE, st_live,
"on_job_postdone must leave live job state unchanged (Req 5.1) — "
"the PDONE merge above is recorded in the WAL only, not applied "
"live, until the apply pump exists");
cbrp_teardown();
printf("test_guard_forward_update_on_terminal_preserved: PASS\n");
}
/* 5.2 pin — mark_job_fail's existing fail-after-finish PERR-merge
* (job.c:702-716) must be PRESERVED by the guard: a POST_EXEC-stage fail on a
* DONE job keeps DONE and OR's JOB_STAT_PERR (it does NOT reset the job to a
* bare JOB_STAT_EXIT). Forward-compatible terminal→terminal merge — the guard
* must not reject it. Preservation pin: green before and after task 6.2. */
/* 5.2 pin — apply_job_fail_transition's existing fail-after-finish PERR-merge
* must be PRESERVED: a POST_EXEC-stage fail on a DONE job keeps DONE and
* OR's JOB_STAT_PERR (it does NOT reset the job to a bare JOB_STAT_EXIT).
* Forward-compatible terminal→terminal merge — the guard must not reject it.
* Preservation pin: green before and after task 6.2.
*
* raft-apply-decoupling task 4.1: on_job_fail (via the new
* record_job_fail_fact) is now durable-append-only — it computes the
* PERR-merged target state on a throwaway copy of the job and durably
* appends it, without touching the real job's live state. The oracle for
* "did the merge compute correctly" moves from live state to the
* WAL-recorded fact (cbrp_count_jobstate_entries); the live-state-unchanged
* assertion is this task's own new pin (Req 5.1). */
static void test_guard_fail_after_finish_preserves_perr_merge(void)
{
job_failinfo_t failinfo;
uint64_t lsn_after_done, lsn_after_fail;
int st;
uint32_t st_disk = 0;
int st_live;
cbrp_setup(3000);
cbrp_make_job(74, &g_job_submit);
@ -2406,23 +2469,175 @@ static void test_guard_fail_after_finish_preserves_perr_merge(void)
pthread_mutex_lock(&g_hc.persist.lock);
lsn_after_fail = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
st = cbrp_job_live_state(74);
/* Assert: DONE preserved, PERR merged in — not reset to bare EXIT */
TEST_ASSERT_TRUE_MESSAGE((st & JOB_STAT_DONE) != 0,
"fail-after-finish must keep the DONE terminal bit (PERR-merge "
"behavior, job.c:702-716)");
TEST_ASSERT_TRUE_MESSAGE((st & JOB_STAT_PERR) != 0,
"fail-after-finish must OR JOB_STAT_PERR onto the existing terminal "
"state (PERR-merge behavior, job.c:702-716)");
/* Assert: the FAIL fact appended, carrying the correctly PERR-merged
* state — DONE preserved, PERR merged in, not reset to bare EXIT. */
TEST_ASSERT_TRUE_MESSAGE(lsn_after_fail > lsn_after_done,
"the PERR-merge fail must still append its WAL entry (forward "
"terminal→terminal merge — not a guarded-out no-op)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FAIL, 74,
&st_disk),
"exactly one FAIL fact must be durably recorded (Req 5.1)");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_DONE) != 0,
"the recorded FAIL fact must keep the DONE terminal bit "
"(PERR-merge behavior)");
TEST_ASSERT_TRUE_MESSAGE((st_disk & JOB_STAT_PERR) != 0,
"the recorded FAIL fact must OR JOB_STAT_PERR onto the existing "
"terminal state (PERR-merge behavior)");
/* Assert (Req 5.1, task 4.1's own observable): live state is UNCHANGED
* by the receipt handler — still just DONE, no PERR — since nothing
* applies this fact live until the apply pump (task 6) exists. */
st_live = cbrp_job_live_state(74);
TEST_ASSERT_EQUAL_INT_MESSAGE(JOB_STAT_DONE, st_live,
"on_job_fail must leave live job state unchanged (Req 5.1) — the "
"PERR merge above is recorded in the WAL only, not applied live, "
"until the apply pump exists");
cbrp_teardown();
printf("test_guard_fail_after_finish_preserves_perr_merge: PASS\n");
}
/* ════════════════════════════════════════════════════════════════════════
* raft-apply-decoupling task 4.1 (Req 5.1) — receipt handlers durably
* append only, no live mutation, no AFTER_EVENT firing
* ════════════════════════════════════════════════════════════════════════
*
* Spy hook counting BEFORE_EVENT vs AFTER_EVENT firings for
* XCE_CONTROLLER_HOOK_JOB — registered alongside persist_start's production
* hooks (persist_job_hook_fun etc., a fan-out, so both subscribers coexist).
* Used to pin: the pre-mutation validation hook (BEFORE_EVENT) still fires
* exactly once per accepted fact, unchanged; AFTER_EVENT never fires at
* receipt time any more (it now only fires from apply_committed_entry(),
* which nothing drives yet — task 6). */
static int g_cbrp_before_fires;
static int g_cbrp_after_fires;
static int cbrp_job_hook_spy(int type, int situation, int detail, void *data,
va_list ap)
{
(void)type; (void)detail; (void)data; (void)ap;
if (situation == HOOK_BEFORE_EVENT)
g_cbrp_before_fires++;
else if (situation == HOOK_AFTER_EVENT)
g_cbrp_after_fires++;
return 0;
}
/* The task's own RED/GREEN observable: after each of on_job_start/
* on_job_finish/on_job_fail/on_job_postdone returns, the job's LIVE
* in-memory state is unchanged — even though every one of the four facts
* durably appends with the correct target state, and the pre-mutation
* BEFORE_EVENT hook still fires exactly once per accepted fact. Against the
* pre-task-4.1 code (mutate-then-notify at receipt) this test's live-state-
* unchanged assertions fail on the very first fact (START moves live state
* PEND -> RUN immediately) — confirmed by running this test against a
* revert of job.c/persist.c/persist.h before applying this task's fix. */
static void test_receipt_facts_durably_append_without_live_mutation(void)
{
static unsigned int rf_pids[1] = { 6001 };
job_pidinfo_t start;
xce_jobfinish_t fin;
job_failinfo_t failinfo;
uint32_t st_disk = 0;
int live_before, live_after;
cbrp_setup(3000);
cbrp_make_job(90, &g_job_submit);
memset(&g_job_submit, 0, sizeof(g_job_submit));
g_cbrp_before_fires = 0;
g_cbrp_after_fires = 0;
TEST_ASSERT_EQUAL_INT(0,
xce_add_hook(&g_hc.hooksys, XCE_CONTROLLER_HOOK_JOB, cbrp_job_hook_spy,
NULL));
/* START */
live_before = cbrp_job_live_state(90);
memset(&start, 0, sizeof(start));
start.jobid = 90;
start.npids = 1;
start.pids = &rf_pids[0];
TEST_ASSERT_EQUAL_INT(0, on_job_start(&g_hc, &start));
live_after = cbrp_job_live_state(90);
TEST_ASSERT_EQUAL_INT_MESSAGE(live_before, live_after,
"on_job_start must leave live job state unchanged (Req 5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_START, 90,
&st_disk),
"on_job_start must durably append exactly one START fact (Req 5.1)");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, st_disk);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_cbrp_before_fires,
"the pre-mutation validation hook (BEFORE_EVENT) must still fire "
"unchanged at receipt time");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
"AFTER_EVENT must NOT fire at receipt time any more (Req 5.1, 5.3)");
/* FINISH */
live_before = cbrp_job_live_state(90);
memset(&fin, 0, sizeof(fin));
fin.jobid = 90;
fin.exit_code = 0;
TEST_ASSERT_EQUAL_INT(0, on_job_finish(&g_hc, &fin));
live_after = cbrp_job_live_state(90);
TEST_ASSERT_EQUAL_INT_MESSAGE(live_before, live_after,
"on_job_finish must leave live job state unchanged (Req 5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FINISH, 90,
&st_disk),
"on_job_finish must durably append exactly one FINISH fact (Req "
"5.1)");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_DONE, st_disk);
TEST_ASSERT_EQUAL_INT_MESSAGE(2, g_cbrp_before_fires,
"BEFORE_EVENT must fire again for the FINISH fact");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
"AFTER_EVENT must still never fire at receipt time");
/* FAIL */
live_before = cbrp_job_live_state(90);
memset(&failinfo, 0, sizeof(failinfo));
failinfo.jobid = 90;
failinfo.stage = JOB_EXEC_STAGE_POST_EXEC;
failinfo.code = 1;
failinfo.detail = (char *)"synthetic fail for no-mutation pin";
TEST_ASSERT_EQUAL_INT(0, on_job_fail(&g_hc, &failinfo));
live_after = cbrp_job_live_state(90);
TEST_ASSERT_EQUAL_INT_MESSAGE(live_before, live_after,
"on_job_fail must leave live job state unchanged (Req 5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FAIL, 90,
&st_disk),
"on_job_fail must durably append exactly one FAIL fact (Req 5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(3, g_cbrp_before_fires,
"BEFORE_EVENT must fire again for the FAIL fact — mark_job_fail has "
"no guard, so it always fires");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
"AFTER_EVENT must still never fire at receipt time");
/* UPDATE (via on_job_postdone) */
live_before = cbrp_job_live_state(90);
TEST_ASSERT_EQUAL_INT(0, on_job_postdone(&g_hc, &g_req_ep, 90));
live_after = cbrp_job_live_state(90);
TEST_ASSERT_EQUAL_INT_MESSAGE(live_before, live_after,
"on_job_postdone must leave live job state unchanged (Req 5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_UPDATE, 90,
&st_disk),
"on_job_postdone must durably append exactly one UPDATE fact (Req "
"5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(4, g_cbrp_before_fires,
"BEFORE_EVENT must fire again for the UPDATE fact");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_cbrp_after_fires,
"AFTER_EVENT must NEVER fire at receipt time for any of the four "
"fact types (Req 5.1, 5.3) — it now only fires from "
"apply_committed_entry(), which nothing drives live until task 6");
cbrp_teardown();
printf("test_receipt_facts_durably_append_without_live_mutation: "
"PASS\n");
}
/* ── Test group entry point ──────────────────────────────────────────── */
void test_commit_barrier_reply(void)
@ -2470,14 +2685,15 @@ void test_commit_barrier_reply(void)
/* Fact retention across quorum loss (task 4.3, Req 3.1, 3.3, 3.4) —
* green on landing against tasks 1.2 + 4.2 output */
RUN_TEST(test_fact_retention_across_quorum_loss);
/* mark_job_* idempotency / ordering guards (task 6.1, Req 5.1/5.2/5.3)
* — the no-op / backward-reject / replay cases FAIL (red) against the
* unguarded mark_job_*; the forward-merge and PERR-merge cases are
* preservation pins (green before and after task 6.2) */
/* Fact-transition idempotency / ordering guards (extracted by task 2.1,
* already landed and green — the stale "RED-phase for task 6.2" framing
* this comment block used to carry predates that landing) */
RUN_TEST(test_guard_duplicate_finish_on_done_is_noop);
RUN_TEST(test_guard_start_on_terminal_rejected_state_preserved);
RUN_TEST(test_guard_replay_finish_equals_single_application);
RUN_TEST(test_guard_forward_update_on_terminal_preserved);
RUN_TEST(test_guard_fail_after_finish_preserves_perr_merge);
/* Fact receipt handlers durably append only (task 4.1, Req 5.1) */
RUN_TEST(test_receipt_facts_durably_append_without_live_mutation);
UNITY_END();
}

18
src/include/persist.h
View file

@ -832,6 +832,24 @@ int persist_job_hook_fun(int type, int situation, int detail,
void* cls, va_list ap);
int persist_queue_hook_fun(int type, int situation, int detail,
void* cls, va_list ap);
/* Durable-append-only entry point for a worker-reported job-state fact
* (raft-apply-decoupling task 4.1, Req 5.1): encodes {jobid, state} into the
* shared wal_jobstate_t wire format and appends it via the same
* wal_hook_append_classified() path persist_job_hook_fun already uses for
* these WAL types (always WAL_MODE_RETAIN for WAL_JOB_{START,FINISH,FAIL,
* UPDATE} — no commit timeout, never blocks) — but bypasses the hook fan-out
* entirely, so calling this fires NO AFTER_EVENT notification. The
* mutation this fact eventually drives, and the AFTER_EVENT notification
* itself, both happen later from apply_committed_entry() once this entry
* commits to a majority (task 2.2's existing dispatch). `state` is the
* caller-computed resultant state (e.g. JOB_STAT_RUN for a START, or the
* PERR-merged value for a FAIL) — this function performs no guard checks or
* merge logic of its own, it only encodes and appends.
* Returns the local-append rc (0 on success); facts have no commit timeout
* to report, mirroring wal_hook_append_classified's own contract. */
int persist_submit_job_fact(persist_t* p, uint32_t wal_type, uint64_t jobid,
int state);
int persist_worker_hook_fun(int type, int situation, int detail,
void* cls, va_list ap);