lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): WAL_JOB_START loses pidinfo/start_time on restart replay (bug-1687)

Browse source 
Same class as bug-1678: apply_job_start_transition() sets the real
job->job.pidinfo (worker-reported PIDs/PGIDs/exdata) and start_time, but
the WAL fact for WAL_JOB_START only carried the shared wal_jobstate_t
{jobid, state} header -- both apply seams (apply_committed_entry() for
primary boot replay, apply_wal_entry() for standby catch-up) restored
state only. pidinfo stayed permanently NULL after a restart; start_time
was fabricated with time(NULL) at replay instead of being carried,
drifting from the true value.

Mirrors bug-1678's exact fix shape: capnp_encode_pidinfo()/decode_pidinfo()
(persist.c, using the pre-existing JobPIDInfo capnp struct -- no schema
change needed) serialize job_pidinfo_t into the WAL payload, trailing a
new uint64_t start_time field appended after wal_jobstate_t.
persist_submit_job_start_fact() replaces the old state-only
persist_submit_job_fact() call for this type. Both apply seams updated;
a payload of exactly sizeof(wal_jobstate_t) (any entry written before
this fix) decodes with pidinfo left NULL and start_time fabricated via
time(NULL), exactly matching pre-fix behavior -- full backward
compatibility, verified by every pre-existing WAL_JOB_START test using
the old bare-struct shape.

Also fixed a genuine, unrelated deadlock this change's larger payload
exposed in test_commit_barrier_reply.c: cbrp_count_jobstate_entries_ex()
matched WAL entries by exact payload size (==), so once WAL_JOB_START
legitimately grew it silently undercounted to 0 and a downstream test
spun forever polling an uncounted target. Fixed the match to >= (jobid/
state always live at a fixed leading offset) -- bisected via /proc/*/wchan
futex inspection (no ptrace in this sandbox), confirmed as the actual
cause, not the teardown-ordering red herring first suspected.

Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension, added after bug-1678). 6 new tests in
test_wal_job_start_fact.c; RED/GREEN independently verified (4 tests
genuinely failed pre-fix with exact expected-vs-fabricated value
mismatches, 2 backward-compat tests correctly passed both before and
after). Full controller suite: 1146 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 09:23:00 +00:00
parent bc3f76b42e
commit 955efecd19
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
9 changed files with 992 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

2
src/controller/testsuite/controller_test.c
View file

@ -84,6 +84,7 @@ extern void test_apply_seam(void);
extern void test_apply_committed_entry(void);
extern void test_replay_apply_seam(void);
extern void test_wal_job_dispatch_fact(void);
extern void test_wal_job_start_fact(void);
extern void test_job_dispatch_fail_count(void);
extern void test_apply_reapply_idempotency(void);
extern void test_apply_correctness_verification(void);
@ -254,6 +255,7 @@ int main(int argc, char* argv[])
TEST(test_apply_committed_entry);
TEST(test_replay_apply_seam);
TEST(test_wal_job_dispatch_fact);
TEST(test_wal_job_start_fact);
TEST(test_job_dispatch_fail_count);
TEST(test_apply_reapply_idempotency);
TEST(test_apply_correctness_verification);

25
src/controller/testsuite/test_commit_barrier_reply.c
View file

@ -475,6 +475,22 @@ static void cbrp_teardown(void)
* range in job.c's process-lifetime in-flight registry. Clear it here
* so it never outlives this fixture. */
inflight_submits_reset_for_test();
/* bug-1687: persist_stop() must run BEFORE commit_engine_fini() --
* exactly the ordering bug-1452 already fixed in test_persist_truncate.c
* (that fix's own note flagged this file as a same-pattern, not-yet-
* observed-to-crash follow-up). commit_engine_fini() tears down
* g_hce.lock/g_hce.cond and, as its final step, nulls g_hce.p while the
* fsync_worker background thread (started by persist_start(), stopped
* only by persist_stop()'s pthread_join) can still be mid-flight inside
* its unlocked p->on_durable_advance(p->durable_cb_ctx) callback
* (persist.c's fsync_worker) -- a genuine TOCTOU data race with no
* synchronization between the two threads. Calling persist_stop() here,
* first, guarantees the flusher thread cannot be in-flight at all by the
* time commit_engine_fini() runs. The later persist_stop() call below
* becomes a safe, idempotent no-op on this second invocation (mirrors
* bug-1452's fix exactly -- fsync_thread_running is already 0 and wal_fd
* is already -1, so both of persist_stop()'s internal branches skip). */
persist_stop(&g_hc.persist);
commit_engine_fini(&g_hce);
persist_stop(&g_hc.persist);
persist_fini(&g_hc.persist);
@ -2410,7 +2426,14 @@ static int cbrp_count_jobstate_entries_ex(persist_t *p, uint32_t wal_type,
(int)pread(fd, &hdr, sizeof(hdr), (off_t)off),
"WAL segment scan: short header read");
if (hdr.type == wal_type
&& hdr.payload_len == sizeof(wal_jobstate_t)) {
&& hdr.payload_len >= sizeof(wal_jobstate_t)) {
/* bug-1687: >= not == -- WAL_JOB_START (like bug-1678's
* WAL_JOB_DISPATCH before it) can now carry trailing bytes
* (start_time + Cap'n Proto JobPIDInfo) beyond the fixed
* wal_jobstate_t header. jobid/state always live at the same
* fixed leading offset regardless of trailing bytes, so reading
* just sizeof(wal_jobstate_t) here remains correct for any
* payload_len >= that size. */
wal_jobstate_t js;
TEST_ASSERT_EQUAL_INT_MESSAGE(

673
src/controller/testsuite/test_wal_job_start_fact.c Normal file
View file

@ -0,0 +1,673 @@
/* test_wal_job_start_fact.c — bug-1687 regression suite
*
* Proves WAL_JOB_START's payload -- previously bare wal_jobstate_t{jobid,
* state}, exactly the same silent-field-drop bug class as bug-1678's
* WAL_JOB_DISPATCH -- now carries the worker-reported job_pidinfo_t (real
* pids/pgids/exdata) AND a receipt-time start_time through BOTH of
* persist.c's apply seams:
*
* - apply_committed_entry() (live-commit apply / boot replay dispatch)
* - apply_wal_entry() (standby catch-up / replicated-apply dispatch)
*
* and that a real, production on_job_start() call (the actual worker-START
* receipt handler) survives a real controller restart (WAL write +
* persist_recover() over a real on-disk WAL) with job.pidinfo and
* job.start_time intact, not just job.state (Req: mirrors bug-1678's own
* acceptance criterion, "after recover the job should get the same as it's
* state before shutdown").
*
* Also proves backward compatibility: a WAL_JOB_START entry with the OLD
* payload shape (exactly sizeof(wal_jobstate_t), no trailing bytes --
* simulating an entry written before this fix) still decodes cleanly, with
* pidinfo left NULL and start_time fabricated at apply time exactly as
* before this fix (apply_committed_entry) / left untouched exactly as
* before this fix (apply_wal_entry) -- no crash, no misread.
*
* Fixtures mirror test_wal_job_dispatch_fact.c's own conventions exactly:
* - wjs_setup/wjs_teardown: bare cluster_t (jobmgr + hooksys) + in-memory
* persist_t.
* - wjs_make_fresh_persist/wjs_teardown_persist: real disk-backed
* persist_t + persist_recover(), for restart-replay tests.
*/
#include <rtti.h>
#include <capnp_c.h>
#include <errno.h>
#include <stdarg.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unity/unity.h>
#include "cluster.h"
#include "controller.capnp.h"
#include "hook.h"
#include "jobj.h"
#include "persist.h"
#include "wal_types.h"
#include "xce/cluster_def.h"
#include "xce/types.h"
#include "xcealloc.h"
#include "xstring.h"
/* jobj.c's jobsub table and the process-global xstring table are shared
* across every test file in the controller_unit_test binary — guard
* idempotently, mirroring every sibling fixture in this suite. */
static int g_wjs_jobobj_ready;
static cluster_t g_wjs_c;
static persist_t g_wjs_p;
/* ── Spy hook: counts AFTER_EVENT firings, ignores all payload args ─────── */
static int g_wjs_hook_fires;
static int wjs_spy_hook(int type, int situation, int detail, void* data,
va_list ap)
{
(void)type; (void)detail; (void)data; (void)ap;
if (situation == HOOK_AFTER_EVENT)
g_wjs_hook_fires++;
return 0;
}
static void wjs_setup(void)
{
kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
.nslots = 32,
.delta = 16 };
if (makeString((char*)"wjs-probe") == NULL)
xstring_table_init(64);
if (!g_wjs_jobobj_ready) {
TEST_ASSERT_EQUAL_INT(0, job_obj_init(32, 32));
g_wjs_jobobj_ready = 1;
}
memset(&g_wjs_c, 0, sizeof(g_wjs_c));
TEST_ASSERT_EQUAL_INT(0,
xce_hook_init(&g_wjs_c.hooksys, XCE_CONTROLLER_HOOK_MAX));
g_wjs_c.jobmgr = makeInstance(&xkvmgr_type, &int_params);
TEST_ASSERT_NOT_NULL(g_wjs_c.jobmgr);
memset(&g_wjs_p, 0, sizeof(g_wjs_p));
g_wjs_p.cluster = &g_wjs_c;
g_wjs_hook_fires = 0;
}
static void wjs_teardown(void)
{
xce_hook_fini(&g_wjs_c.hooksys);
if (g_wjs_c.jobmgr)
dereference(g_wjs_c.jobmgr);
}
/* Look up a job_t wrapper by jobid (NULL if absent). */
static job_t* wjs_find_job(uint64_t jobid)
{
job_t* j = NULL;
lock(g_wjs_c.jobmgr, RTTI_LOCK_READ);
int found = key_find(g_wjs_c.jobmgr, (void*)jobid, (void**)&j);
unlock(g_wjs_c.jobmgr);
return (found > 0) ? j : NULL;
}
/* Creates a job directly in the fixture's jobmgr, bypassing dispatch/replay
* entirely — for seeding fact-transition tests, mirroring
* test_wal_job_dispatch_fact.c's wjd_seed_job. */
static job_submit_data_t g_wjs_seed_submit;
static job_t* wjs_seed_job(uint64_t jobid, uint32_t state)
{
xce_job_t jdef;
memset(&g_wjs_seed_submit, 0, sizeof(g_wjs_seed_submit));
g_wjs_seed_submit.queue = (char*)"q";
g_wjs_seed_submit.user = (char*)"tester";
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = jobid;
jdef.state = state;
jdef.submit = &g_wjs_seed_submit;
return makeJob(g_wjs_c.jobmgr, &jdef);
}
/* Build a wal_jobstate_t header + fixed uint64_t start_time + trailing
* Cap'n Proto JobPIDInfo bytes -- persist_submit_job_start_fact()'s payload
* shape -- mirroring wjd_build_dispatch_payload() (test_wal_job_dispatch_
* fact.c) since job.c/persist.c's own static encode helpers aren't linkable
* from a test TU. Caller owns *out. */
static int wjs_build_start_payload(uint64_t jobid, uint32_t state,
uint64_t start_time,
job_pidinfo_t* pidinfo,
uint8_t** out, size_t* out_len)
{
struct capn c;
struct capn_segment* cs;
struct JobPIDInfo cji;
JobPIDInfo_ptr jip;
uint8_t* infobuf;
size_t infolen;
wal_jobstate_t js = { .jobid = jobid, .state = state, ._pad = 0 };
capn_init_malloc(&c);
cs = capn_root(&c).seg;
encode_JobPIDInfo(cs, &cji, pidinfo);
jip = new_JobPIDInfo(cs);
write_JobPIDInfo(&cji, jip);
capn_setp(capn_root(&c), 0, jip.p);
infolen = capn_size(&c);
infobuf = malloc(infolen);
if (!infobuf) { capn_free(&c); return -ENOMEM; }
capn_write_mem(&c, infobuf, infolen, 0);
capn_free(&c);
*out_len = sizeof(js) + sizeof(start_time) + infolen;
*out = malloc(*out_len);
if (!*out) { free(infobuf); return -ENOMEM; }
memcpy(*out, &js, sizeof(js));
memcpy(*out + sizeof(js), &start_time, sizeof(start_time));
memcpy(*out + sizeof(js) + sizeof(start_time), infobuf, infolen);
free(infobuf);
return 0;
}
/* Real pids/pgids/exdata -- mirrors what a worker actually reports on a
* genuine job start. */
static unsigned int g_wjs_test_pids[3] = { 41001, 41002, 41003 };
static unsigned int g_wjs_test_pgids[2] = { 41001, 41500 };
static job_pidinfo_t wjs_make_test_pidinfo(uint64_t jobid)
{
job_pidinfo_t info;
memset(&info, 0, sizeof(info));
info.jobid = jobid;
info.npids = 3;
info.pids = g_wjs_test_pids;
info.npgids = 2;
info.pgids = g_wjs_test_pgids;
info.exdata = (char*)"wjs-exdata-blob";
return info;
}
static void wjs_assert_pidinfo_matches(job_pidinfo_t* got, uint64_t jobid)
{
TEST_ASSERT_NOT_NULL_MESSAGE(got,
"a WAL_JOB_START payload with trailing JobPIDInfo bytes must set "
"job.pidinfo (bug-1687), not just state");
TEST_ASSERT_EQUAL_UINT64(jobid, got->jobid);
TEST_ASSERT_EQUAL_INT(3, got->npids);
TEST_ASSERT_EQUAL_UINT32(41001, got->pids[0]);
TEST_ASSERT_EQUAL_UINT32(41002, got->pids[1]);
TEST_ASSERT_EQUAL_UINT32(41003, got->pids[2]);
TEST_ASSERT_EQUAL_INT(2, got->npgids);
TEST_ASSERT_EQUAL_UINT32(41001, got->pgids[0]);
TEST_ASSERT_EQUAL_UINT32(41500, got->pgids[1]);
TEST_ASSERT_EQUAL_STRING("wjs-exdata-blob", got->exdata);
}
/* ── apply_committed_entry(): live-commit apply / boot replay dispatch ──── */
static void test_wjs_apply_committed_entry_sets_pidinfo_and_start_time(void)
{
wjs_setup();
job_t* j = wjs_seed_job(72001, JOB_STAT_PEND);
TEST_ASSERT_NOT_NULL(j);
job_pidinfo_t info = wjs_make_test_pidinfo(72001);
uint64_t wire_start_time = 1700000001ULL;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0,
wjs_build_start_payload(72001, JOB_STAT_RUN, wire_start_time, &info,
&payload, &plen));
xce_add_hook(&g_wjs_c.hooksys, XCE_CONTROLLER_HOOK_JOB, wjs_spy_hook,
NULL);
int rc = apply_committed_entry(&g_wjs_p, WAL_JOB_START, 200, payload,
(uint32_t)plen);
free(payload);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, j->job.state);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(wire_start_time,
(uint64_t)j->job.start_time,
"WAL_JOB_START must adopt the wire-carried start_time (bug-1687), "
"not fabricate time(NULL) when one is present");
wjs_assert_pidinfo_matches(j->job.pidinfo, 72001);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_wjs_hook_fires,
"a forward WAL_JOB_START transition must still fire exactly one "
"notification");
wjs_teardown();
printf("test_wjs_apply_committed_entry_sets_pidinfo_and_start_time: "
"PASS\n");
}
static void
test_wjs_apply_committed_entry_backward_compat_old_payload(void)
{
wjs_setup();
job_t* j = wjs_seed_job(72002, JOB_STAT_PEND);
TEST_ASSERT_NOT_NULL(j);
TEST_ASSERT_NULL_MESSAGE(j->job.pidinfo,
"fixture must seed a job with pidinfo still unset");
wal_jobstate_t js = { .jobid = 72002, .state = JOB_STAT_RUN, ._pad = 0 };
time_t before = time(NULL);
int rc = apply_committed_entry(&g_wjs_p, WAL_JOB_START, 201, &js,
sizeof(js));
time_t after = time(NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, j->job.state);
TEST_ASSERT_TRUE_MESSAGE(
(j->job.start_time >= before) && (j->job.start_time <= after),
"an old-format (exactly sizeof(wal_jobstate_t)) WAL_JOB_START "
"payload must still fabricate start_time at apply time, unchanged "
"from before this fix");
TEST_ASSERT_NULL_MESSAGE(j->job.pidinfo,
"an old-format WAL_JOB_START payload must leave pidinfo NULL -- no "
"crash, no misread of absent trailing bytes");
wjs_teardown();
printf("test_wjs_apply_committed_entry_backward_compat_old_payload: "
"PASS\n");
}
/* ── apply_wal_entry(): standby catch-up / replicated-apply dispatch ────── */
static void
test_wjs_apply_wal_entry_sets_pidinfo_and_start_time_no_hook(void)
{
wjs_setup();
job_t* j = wjs_seed_job(72003, JOB_STAT_PEND);
TEST_ASSERT_NOT_NULL(j);
job_pidinfo_t info = wjs_make_test_pidinfo(72003);
uint64_t wire_start_time = 1700000003ULL;
uint8_t* payload = NULL;
size_t plen = 0;
TEST_ASSERT_EQUAL_INT(0,
wjs_build_start_payload(72003, JOB_STAT_RUN, wire_start_time, &info,
&payload, &plen));
xce_add_hook(&g_wjs_c.hooksys, XCE_CONTROLLER_HOOK_JOB, wjs_spy_hook,
NULL);
int rc = apply_wal_entry(&g_wjs_p, WAL_JOB_START, 202, payload,
(uint32_t)plen);
free(payload);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, j->job.state);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(wire_start_time,
(uint64_t)j->job.start_time,
"apply_wal_entry's catch-up dispatch must also recover the "
"wire-carried start_time (bug-1687), required for a promoted "
"standby");
wjs_assert_pidinfo_matches(j->job.pidinfo, 72003);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_wjs_hook_fires,
"apply_wal_entry's catch-up dispatch must never fire a notification "
"hook");
wjs_teardown();
printf("test_wjs_apply_wal_entry_sets_pidinfo_and_start_time_no_hook: "
"PASS\n");
}
static void test_wjs_apply_wal_entry_backward_compat_old_payload(void)
{
wjs_setup();
job_t* j = wjs_seed_job(72004, JOB_STAT_PEND);
TEST_ASSERT_NOT_NULL(j);
j->job.start_time = 12345; /* sentinel -- must survive untouched */
wal_jobstate_t js = { .jobid = 72004, .state = JOB_STAT_RUN, ._pad = 0 };
int rc = apply_wal_entry(&g_wjs_p, WAL_JOB_START, 203, &js, sizeof(js));
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, j->job.state);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(12345, (uint64_t)j->job.start_time,
"an old-format WAL_JOB_START payload must leave start_time "
"untouched on this seam, unchanged from before this fix");
TEST_ASSERT_NULL_MESSAGE(j->job.pidinfo,
"an old-format WAL_JOB_START payload must leave pidinfo NULL");
wjs_teardown();
printf("test_wjs_apply_wal_entry_backward_compat_old_payload: PASS\n");
}
/* ── Restart-replay Observables ──────────────────────────────────────── */
static persist_t* wjs_make_fresh_persist(const char* tmpdir)
{
persist_t* p = calloc(1, sizeof(persist_t));
if (!p) return NULL;
persist_config_t cfg = persist_config_defaults();
cfg.wal_sync_ms = 5000; /* slow fsync — no races in tests */
persist_set_workdir(p, tmpdir, "ctrl-test");
if (persist_init(p, &cfg, NULL) != 0) {
free(p);
return NULL;
}
if (persist_start(p) != 0) {
persist_fini(p);
free(p);
return NULL;
}
return p;
}
static void wjs_teardown_persist(persist_t* p)
{
persist_stop(p);
persist_fini(p);
free(p);
}
/* Encode helper mirroring test_wal_job_dispatch_fact.c's wjd_encode_job
* (capnp_encode_job is file-static in persist.c). */
static int wjs_encode_job(const xce_job_t* j, uint8_t** out, size_t* out_len)
{
struct capn c;
struct capn_segment* cs;
struct Job cj;
Job_ptr jp;
capn_init_malloc(&c);
cs = capn_root(&c).seg;
encode_Job(cs, &cj, (xce_job_t*)j);
jp = new_Job(cs);
write_Job(&cj, jp);
capn_setp(capn_root(&c), 0, jp.p);
*out_len = capn_size(&c);
*out = malloc(*out_len);
if (!*out) { capn_free(&c); return -ENOMEM; }
capn_write_mem(&c, *out, *out_len, 0);
capn_free(&c);
return 0;
}
/* Direct-payload restart replay: appends WAL_JOB_SUBMIT + a new-format
* WAL_JOB_START (real pidinfo + a fixed start_time) straight onto disk via
* persist_wal_async, then replays via a fresh persist_t's persist_recover()
* (a real restart). Isolates the WAL fact shape from the live caller,
* mirroring test_wal_job_dispatch_fact.c's own restart-replay test. */
static void test_wjs_restart_replay_recovers_pidinfo_and_start_time(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_wjs_replay_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
uint64_t wire_start_time = 1700009999ULL;
/* Phase 1: write the log to real disk. */
{
persist_t* p = wjs_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p);
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"q";
submit.user = (char*)"tester";
xce_job_t jdef;
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = 72005;
jdef.state = JOB_STAT_PEND;
jdef.submit = &submit;
uint8_t* job_payload = NULL;
size_t job_plen = 0;
TEST_ASSERT_EQUAL_INT(0,
wjs_encode_job(&jdef, &job_payload, &job_plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_SUBMIT, job_payload,
(uint32_t)job_plen));
free(job_payload);
job_pidinfo_t info = wjs_make_test_pidinfo(72005);
uint8_t* start_payload = NULL;
size_t start_plen = 0;
TEST_ASSERT_EQUAL_INT(0,
wjs_build_start_payload(72005, JOB_STAT_RUN, wire_start_time,
&info, &start_payload, &start_plen));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(p, WAL_JOB_START, start_payload,
(uint32_t)start_plen));
free(start_payload);
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
wjs_teardown_persist(p);
}
/* Phase 2: fresh persist_t + fresh cluster_t, recover from disk —
* simulates a controller restart. */
{
persist_t* p2 = wjs_make_fresh_persist(tmp);
TEST_ASSERT_NOT_NULL(p2);
wjs_setup();
p2->cluster = &g_wjs_c;
int rc = persist_recover(p2);
TEST_ASSERT_EQUAL_INT(0, rc);
job_t* j = wjs_find_job(72005);
TEST_ASSERT_NOT_NULL_MESSAGE(j,
"restart replay must construct the job via WAL_JOB_SUBMIT");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, j->job.state);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(wire_start_time,
(uint64_t)j->job.start_time,
"restart replay must recover the exact wire-carried start_time "
"(bug-1687), not a freshly fabricated time(NULL)");
wjs_assert_pidinfo_matches(j->job.pidinfo, 72005);
wjs_teardown();
wjs_teardown_persist(p2);
}
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp);
printf("test_wjs_restart_replay_recovers_pidinfo_and_start_time: "
"PASS\n");
}
/* bug-1687's primary regression test: drives the REAL production
* on_job_start() path (the actual worker-START receipt handler --
* record_job_start_fact()'s only caller) with a real job_pidinfo_t (real
* pids/pgids/exdata), through a real WAL-write + persist_recover() restart
* cycle -- the exact technique test_dc_success_appends_wal_job_dispatch_
* durably (test_dispatch_commit.c) already established for bug-1678.
* Asserts the job's pidinfo AND start_time survive the restart intact, not
* just state -- the audit's exact confirmed root cause. */
static void test_wjs_real_on_job_start_survives_restart(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_wjs_onstart_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t jobid = 72006;
time_t before_call, after_call;
/* Phase 1: real cluster + real disk persist. Seed WAL_JOB_SUBMIT
* directly (so replay can reconstruct the job) AND a live in-memory job
* object (so the real on_job_start() call site has something to find
* and lock) -- mirrors test_dc_success_appends_wal_job_dispatch_durably's
* dual-seed pattern exactly. Then drive an actual on_job_start() call
* and let IT append WAL_JOB_START -- this is the production code path
* under test, not a hand-built WAL entry. */
{
persist_config_t cfg;
if (makeString((char*)"wjs-onstart-probe") == NULL)
xstring_table_init(64);
if (!g_wjs_jobobj_ready) {
TEST_ASSERT_EQUAL_INT(0, job_obj_init(32, 32));
g_wjs_jobobj_ready = 1;
}
cluster_t c;
kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
.nslots = 32, .delta = 16 };
memset(&c, 0, sizeof(c));
TEST_ASSERT_EQUAL_INT(0,
xce_hook_init(&c.hooksys, XCE_CONTROLLER_HOOK_MAX));
c.jobmgr = makeInstance(&xkvmgr_type, &int_params);
TEST_ASSERT_NOT_NULL(c.jobmgr);
cfg = persist_config_defaults();
cfg.wal_sync_ms = 5000;
persist_set_workdir(&c.persist, tmp, "ctrl-test");
TEST_ASSERT_EQUAL_INT(0, persist_init(&c.persist, &cfg, NULL));
TEST_ASSERT_EQUAL_INT(0, persist_start(&c.persist));
/* Durable WAL_JOB_SUBMIT seed. */
{
job_submit_data_t wal_submit;
xce_job_t wal_jdef;
uint8_t* buf = NULL;
size_t sz = 0;
memset(&wal_submit, 0, sizeof(wal_submit));
wal_submit.queue = (char*)"oq";
wal_submit.user = (char*)"tester";
memset(&wal_jdef, 0, sizeof(wal_jdef));
wal_jdef.jobid = jobid;
wal_jdef.state = JOB_STAT_PEND;
wal_jdef.submit = &wal_submit;
TEST_ASSERT_EQUAL_INT(0,
wjs_encode_job(&wal_jdef, &buf, &sz));
TEST_ASSERT_EQUAL_INT(0,
persist_wal_async(&c.persist, WAL_JOB_SUBMIT, buf,
(uint32_t)sz));
free(buf);
}
/* Live in-memory job object for on_job_start() to find/lock. */
{
job_submit_data_t submit;
xce_job_t jdef;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"oq";
submit.user = (char*)"tester";
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = jobid;
jdef.state = JOB_STAT_PEND;
jdef.submit = &submit;
TEST_ASSERT_NOT_NULL(makeJob(c.jobmgr, &jdef));
}
/* The real worker-reported pids/pgids/exdata. */
job_pidinfo_t info = wjs_make_test_pidinfo(jobid);
before_call = time(NULL);
int rc = on_job_start(&c, &info);
after_call = time(NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(0, persist_flush(&c.persist));
persist_stop(&c.persist);
persist_fini(&c.persist);
dereference(c.jobmgr);
xce_hook_fini(&c.hooksys);
}
/* Phase 2: fresh persist_t + fresh cluster_t (jobmgr/hooksys only),
* recover from disk — simulates a controller restart. */
{
persist_t* p2 = wjs_make_fresh_persist(tmp);
cluster_t c2;
job_t* j2 = NULL;
int rc;
kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
.nslots = 32, .delta = 16 };
TEST_ASSERT_NOT_NULL(p2);
memset(&c2, 0, sizeof(c2));
TEST_ASSERT_EQUAL_INT(0,
xce_hook_init(&c2.hooksys, XCE_CONTROLLER_HOOK_MAX));
c2.jobmgr = makeInstance(&xkvmgr_type, &int_params);
TEST_ASSERT_NOT_NULL(c2.jobmgr);
p2->cluster = &c2;
rc = persist_recover(p2);
TEST_ASSERT_EQUAL_INT(0, rc);
lock(c2.jobmgr, RTTI_LOCK_READ);
key_find(c2.jobmgr, (void*)jobid, (void**)&j2);
unlock(c2.jobmgr);
TEST_ASSERT_NOT_NULL_MESSAGE(j2,
"restart replay must reconstruct the job via WAL_JOB_SUBMIT");
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_RUN, j2->job.state,
"restart replay must recover the job as RUNNING from the "
"WAL_JOB_START fact appended by the real on_job_start() call");
/* bug-1687: restart replay must recover not just STATE but the
* worker-reported pidinfo AND the true start_time -- the audit's
* exact confirmed root cause (mirrors bug-1678's own acceptance
* criterion for WAL_JOB_DISPATCH/job.allocs). */
wjs_assert_pidinfo_matches(j2->job.pidinfo, jobid);
TEST_ASSERT_TRUE_MESSAGE(
(j2->job.start_time >= before_call)
&& (j2->job.start_time <= after_call),
"restart replay must recover the true, receipt-time start_time "
"captured by the real on_job_start() call, not a fresh "
"apply-time time(NULL) fabrication");
xce_hook_fini(&c2.hooksys);
if (c2.jobmgr) dereference(c2.jobmgr);
wjs_teardown_persist(p2);
}
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp);
printf("test_wjs_real_on_job_start_survives_restart: PASS\n");
}
/* ── Test group entry point ───────────────────────────────────────────── */
void test_wal_job_start_fact(void)
{
UnitySetTestFile(__FILE__);
RUN_TEST(test_wjs_apply_committed_entry_sets_pidinfo_and_start_time);
RUN_TEST(test_wjs_apply_committed_entry_backward_compat_old_payload);
RUN_TEST(test_wjs_apply_wal_entry_sets_pidinfo_and_start_time_no_hook);
RUN_TEST(test_wjs_apply_wal_entry_backward_compat_old_payload);
RUN_TEST(test_wjs_restart_replay_recovers_pidinfo_and_start_time);
RUN_TEST(test_wjs_real_on_job_start_survives_restart);
}