lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU

Browse source 
Bundles the fixes validated this session across ha.c/persist.c/worker.c:

- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
  fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
  closed with a recovery-complete gate that closes at process-lifecycle
  start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
  instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
  apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
  check mirroring the existing job-count check (count_jobmgr_entries and
  count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
  close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
  applies mark_worker_unavail()'s mutations via a shared
  apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
  fixed for promoted vote endpoints via a shared votes_idx_for_peer()
  index-scale converter.

Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-16 10:20:02 +00:00
parent 3afe3d5549
commit 45400c3a28
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
18 changed files with 1499 additions and 195 deletions
Download patch file Download diff file Expand all files Collapse all files

534
src/controller/testsuite/test.c
View file

@ -36,9 +36,10 @@
typedef int (*cmd_func_t)(int, char*[]);
typedef struct {
char* name;
cmd_func_t func;
char* doc;
char* name;
cmd_func_t func;
char* doc;
const char* usage;
} command_t;
typedef struct {
@ -189,20 +190,370 @@ static setting_t settings[] = {
{ (char*)NULL, (char*)NULL, (char*)NULL }
};
/* Detailed per-command `help <cmd>` text (design: keep these OUT of the
* command_t table initializers below so the tables stay readable). Each
* constant's shape: first line = command name + flag/positional syntax,
* then one indented line per flag/positional explaining what it does and
* whether it's required, then (where useful) an `example:` line. Derived
* from actually reading each cmd_* handler's getopt() string and switch
* cases (or its positional argv[] handling) -- not guessed from the name.
*
* Commands that are identical (same func, same semantics) across more than
* one of the 4 command tables share ONE constant here, referenced from every
* table, so the text can't drift out of sync between contexts. */
static const char USAGE_HELP[] =
"help [<cmd>]\n"
" <cmd> optional command name; prints its detailed usage instead of\n"
" the one-line summary\n"
" With no argument, lists every command in the current context with its\n"
" one-line summary (unaffected by this detailed-usage text).\n"
"example: help resval\n";
static const char USAGE_QUIT[] =
"quit\n"
" (no arguments)\n"
" Leaves the current context (if any) and exits xce_test.\n";
static const char USAGE_ECHO[] =
"echo [<message...>]\n"
" <message...> zero or more words, printed space-joined followed by a\n"
" newline (always succeeds)\n"
" Useful to print the $jobid / $jobcount / $? auto-variables in scripts.\n"
"example: echo job is $jobid\n";
static const char USAGE_UP[] =
"..\n"
" (no arguments)\n"
" Leaves the current context and returns to its parent, closing the\n"
" underlying client connection for this context.\n";
static const char USAGE_WALDUMP[] =
"waldump <wal_dir|data_dir|seg_file> [more...]\n"
" <path> one or more paths: a seg-*.wal file, a wal/ dir, or a\n"
" node/data dir\n"
" Offline, read-only WAL inspector: lists every entry, decodes\n"
" JOB_SUBMIT jobids, and flags LSN gaps and a CRC-broken tail.\n"
" Returns nonzero if any path fails to load or any gap/CRC break is\n"
" found.\n"
"example: waldump /var/xce/ctrl-1/wal\n";
static const char USAGE_WALCOMPARE[] =
"walcompare [--report] [--committed <N>] <memberA> <memberB> [more...]\n"
" --report also emit a full per-member (lsn,term,pcrc,status)\n"
" machine-checkable table, not just the verdict\n"
" --committed <N> use LSN N as the shared committed boundary instead\n"
" of deriving it as the minimum CRC-intact tip across\n"
" members\n"
" <memberX> 2-16 member paths (seg-*.wal file, wal/ dir, or\n"
" node/data dir)\n"
" Returns 0 if all members' committed WAL prefixes are identical,\n"
" nonzero on divergence or load error.\n"
"example: walcompare --report /var/xce/ctrl-1/wal /var/xce/ctrl-2/wal\n";
static const char USAGE_SNAPVERIFY[] =
"snapverify <snap_file> [<snap_file> ...]\n"
" <snap_file> one or more snap-*.capnp or snap-*.bin checkpoint files\n"
" Offline structural validator: walks the same layout\n"
" persist_load_snapshot() parses, without restoring anything. Reports\n"
" the byte offset of the first structural defect and how many frames\n"
" parsed cleanly before it.\n"
" Returns nonzero if any file fails to verify.\n"
"example: snapverify /var/xce/ctrl-1/snap/snap-000123.capnp\n";
static const char USAGE_SNAPDIFF[] =
"snapdiff <source_file> <received_file>\n"
" <source_file> the primary's own locally generated checkpoint\n"
" (snap-*.capnp)\n"
" <received_file> the copy a standby assembled from the wire\n"
" (snap-*.bin)\n"
" Byte-level comparator; reports the first differing offset and\n"
" whether it lands on an OPCode_snapshotTransfer chunk boundary.\n"
" Returns 0 if identical, nonzero otherwise.\n"
"example: snapdiff snap-000123.capnp snap-000123.bin\n";
static const char USAGE_CONFIGCOMPARE[] =
"configcompare <member_dir> <member_dir> [<member_dir> ...]\n"
" <member_dir> a member's workdir/name conf-root directory (NOT a raw\n"
" config file path) -- the dir under which conf/current\n"
" resolves\n"
" Recomputes each member's active-config sha256 fresh from its own\n"
" on-disk bytes and reports whether all members match.\n"
" Returns 0 if all members are present and identical, nonzero if any\n"
" member is unreadable or divergent.\n"
"example: configcompare /var/xce/ctrl-1 /var/xce/ctrl-2 /var/xce/ctrl-3\n";
static const char USAGE_WORKER[] =
"worker\n"
" (no arguments)\n"
" Enters the worker context: connects to the controller cluster as a\n"
" worker client and starts the local delay-manager housekeeping loop.\n";
static const char USAGE_SCHEDULER[] =
"scheduler\n"
" (no arguments)\n"
" Enters the scheduler context: connects to the controller cluster as\n"
" a scheduler client.\n";
static const char USAGE_CLIENT[] =
"client\n"
" (no arguments)\n"
" Enters the client context: connects to the controller cluster as a\n"
" regular API client (job submission, resource/worker queries, etc.).\n";
static const char USAGE_SET[] =
"set\n"
"set <key> <value>\n"
"set -e | +e | continue-on-error\n"
" (no args) list all settings in the current context\n"
" (current value + default)\n"
" <key> <value> set setting <key> (e.g. masters, secret,\n"
" myname) to <value>\n"
" -e abort the read loop on the first failing\n"
" command (errexit)\n"
" +e | continue-on-error keep the read loop running after a\n"
" failing command\n"
"example: set masters ctrl1=localhost:9991,ctrl2=localhost:9992\n";
static const char USAGE_JOIN_WORKER[] =
"join [-f <cpufactor>] [-c <maxcpus>] [-m <maxmem>] [-s <maxswap>]\n"
" [-t <maxtmp>] [-d <ndisks>]\n"
" -f <cpufactor> relative CPU speed factor (default 1.0)\n"
" -c <maxcpus> max CPU count (default 4)\n"
" -m <maxmem> max memory in bytes (default 64)\n"
" -s <maxswap> max swap in bytes (default 1024)\n"
" -t <maxtmp> max tmp space in bytes (default 2048)\n"
" -d <ndisks> number of local disks (default 1)\n"
" Joins the cluster as a worker (hosttype LINUX64, hostmodel Intel64).\n"
"example: join -c 8 -m 16384\n";
static const char USAGE_JOBSTART[] =
"jstart -j <jobid> [-p <pid>]\n"
" -j <jobid> jobid to report a synthetic job-start fact for\n"
" (required)\n"
" -p <pid> pid to report (default: this xce_test process's own\n"
" pid)\n"
"example: jstart -j 12345 -p 4242\n";
static const char USAGE_JOBFINISH[] =
"jfinish -j <jobid> [-c <exitcode>]\n"
" -j <jobid> jobid to report a synthetic job-finish fact for\n"
" (required)\n"
" -c <exitcode> exit code to report (default 0)\n"
"example: jfinish -j 12345 -c 0\n";
static const char USAGE_JOBFAIL[] =
"jfail -j <jobid> [-c <code>] [-s <stage>]\n"
" -j <jobid> jobid to report a synthetic job-fail fact for\n"
" (required)\n"
" -c <code> failure code to report (default 0)\n"
" -s <stage> exec-stage code to report (default\n"
" JOB_EXEC_STAGE_POST_EXEC)\n"
"example: jfail -j 12345 -c 1\n";
static const char USAGE_RESUPDATE[] =
"resupdate -n <resname> -v <value> [-n <resname> -v <value> ...]\n"
" -n <resname> resource name to update (e.g. slots, mem, cpuf,\n"
" maxswap, maxtmp, maxcpus, ndisks, type, model,\n"
" dynres1, dynres2, r1m, r15s); may repeat for\n"
" multiple resources\n"
" -v <value> value for the immediately preceding -n (numeric for\n"
" most resources, string for type/model)\n"
" Order matters: pair each -n with its -v right after it.\n"
"example: resupdate -n slots -v 4 -n mem -v 8192\n";
static const char USAGE_HELLO[] =
"hello [-c <count>] [-m <message>]\n"
" -c <count> number of times to send the hello (default 1)\n"
" -m <message> greeting text to send (default \"Greeting from xce\n"
" client\")\n"
"example: hello -c 3 -m \"ping\"\n";
static const char USAGE_RESLIST[] =
"reslist\n"
" (no arguments)\n"
" Prints all cluster-wide resources (name, flags, interval,\n"
" description).\n";
static const char USAGE_WORKERLIST[] =
"workerlist\n"
" (no arguments)\n"
" Prints all joined workers (host, status, host type/model, cpu\n"
" factor, max cpu/mem/swap/tmp, ndisks).\n";
static const char USAGE_QCTRL[] =
"qctrl -q <queue> [-a] [-i] [-o] [-c]\n"
" -q <queue> queue name to control (required)\n"
" -a set queue ACTIVE (accepts dispatch)\n"
" -i set queue INACTIVE (rejects dispatch)\n"
" -o set queue OPEN (accepts new submissions)\n"
" -c set queue CLOSE (rejects new submissions)\n"
"example: qctrl -q normal -i\n";
static const char USAGE_HOSTCTRL[] =
"hostctrl -h <hostname> (-o|-c)\n"
" -h <hostname> host to administratively control (required)\n"
" -o open the host (allow scheduling)\n"
" -c close the host (admin-close; shows CLOSED in\n"
" workerlist)\n"
"example: hostctrl -h node01 -c\n";
static const char USAGE_UPGRADECONFIG[] =
"upgradeconfig <cluster.bin>\n"
" <cluster.bin> path to a capnp-encoded ClusterConfig blob (same\n"
" format `xceadmin config build` produces and\n"
" deploy-ha.sh installs)\n"
" Computes its SHA-256 and submits it as a configuration upgrade via\n"
" OP_upgradeConfig.\n"
"example: upgradeconfig /var/xce/cluster.bin\n";
static const char USAGE_JOBLIST[] =
"joblist\n"
" (no arguments)\n"
" Prints all jobs in the cluster (JobId, user, queue, state, times,\n"
" ntasks, command) and captures the count into $jobcount.\n";
static const char USAGE_EXPECT_JOBS[] =
"expect-jobs <N>\n"
" <N> non-negative integer; asserts the current job count equals N\n"
" Fails (nonzero) on a malformed N, a failed joblist query, or a count\n"
" mismatch.\n"
"example: expect-jobs 3\n";
static const char USAGE_EXPECT_JOB[] =
"expect-job <jobid>\n"
" <jobid> a JobId string (as printed by joblist / captured into\n"
" $jobid); asserts a job with this id is present in the\n"
" current job list\n"
" Fails (nonzero) on a missing argument, a failed joblist query, or if\n"
" the job is not found.\n"
"example: expect-job $jobid\n";
static const char USAGE_WAIT_PRIMARY[] =
"wait-primary [<timeout_ms>]\n"
" <timeout_ms> optional max wait in milliseconds (default 30000);\n"
" must be a non-negative integer\n"
" Blocks, polling every 200 ms, until the AUTHORITATIVE controller\n"
" status probe reaches the serving primary (a standby/hint answer does\n"
" not count), or the timeout elapses.\n"
" Returns 0 once a primary answers, nonzero (ETIMEDOUT) on timeout.\n"
"example: wait-primary 10000\n";
static const char USAGE_CTRLSTATUS[] =
"ctrlstatus\n"
" (no arguments)\n"
" Prints the responding controller's identity/role, cluster name,\n"
" peer topology, config version/checksum, and HA health (term,\n"
" committedLsn, appliedLsn, fenced) from a single any-node status\n"
" read.\n";
static const char USAGE_SHUTDOWN[] =
"shutdown\n"
" (no arguments)\n"
" Shuts down the xce controller and, on success, leaves the client\n"
" context (same as running `..`).\n";
static const char USAGE_JOBRUN[] =
"jobrun -j <jobid> [-m <host>[:<ntasks>]] [-m ...]\n"
" -j <jobid> jobid to dispatch (required, must be > 0)\n"
" -m <host>[:<n>] host to run tasks on, optional :ntasks count\n"
" (default 1); may be repeated up to 5 times\n"
"example: jobrun -j 12345 -m node01:2 -m node02\n";
static const char USAGE_JOBSUB[] =
"jobsub -u <user> -q <queue> -c <cmdline> [-R <resreq>]\n"
" [-n <min>[,<max>]] [-x <exdata>] [-a <arraysize>]\n"
" -u <user> submitting user (required)\n"
" -q <queue> target queue (required)\n"
" -c <cmdline> command line to run (required)\n"
" -R <resreq> resource requirement expression (default\n"
" \"type == any\")\n"
" -n <min>[,<max>] task count, or a min,max range (default 1)\n"
" -x <exdata> raw exdata JSON string (overrides the built-in\n"
" PATH env)\n"
" -a <arraysize> job array size (default 1)\n"
" On success, captures the submitted JobId into the $jobid\n"
" auto-variable.\n"
"example: jobsub -u alice -q normal -c \"sleep 10\" -n 1,4\n";
static const char USAGE_RESVAL[] =
"resval -n <resname> [-m <host>]\n"
" -n <resname> resource name to query (required)\n"
" -m <host> host to query (required unless the resource is a\n"
" shared/cluster-wide resource, not per-host)\n"
"example: resval -n slots -m myhost\n";
static const char USAGE_JOBKILL[] =
"jobkill -j <jobid> [-s <signo>]\n"
" -j <jobid> jobid to signal (required)\n"
" -s <signo> signal number to send (default SIGTERM)\n"
"example: jobkill -j 12345 -s 9\n";
static const char USAGE_JOBCKPT[] =
"jobckpt -j <jobid> [-k] [-m <method>]\n"
" -j <jobid> jobid to checkpoint (required)\n"
" -k kill the job immediately after checkpointing\n"
" -m <method> checkpoint method name (default \"default\")\n"
"example: jobckpt -j 12345 -m default\n";
static const char USAGE_JOBINFO[] =
"jobinfo -j <jobid> [-j <jobid> ...]\n"
" -j <jobid> jobid to show info for; may be repeated to show\n"
" several jobs in one call\n"
"example: jobinfo -j 12345\n";
static const char USAGE_TASKRUN[] =
"taskrun [-j <jobid>] [-m <host>[:<n>]]... [-u <user>] [-c <cmdline>]\n"
" [-x <exdata>] [-d <dir>] [-t <launch_type>]\n"
" -j <jobid> jobid this task run belongs to\n"
" -m <host>[:<n>] host to run on, optional :ntasks count\n"
" (default 1); may be repeated up to 5 times\n"
" -u <user> exec user, applied to every host\n"
" -c <cmdline> command line, applied to every host\n"
" -x <exdata> exdata string, applied to every host\n"
" -d <dir> exec home/cwd directory, applied to every host\n"
" -t <launch_type> launch type (default \"mpi\" if the total task\n"
" count across all -m hosts is > 1, else a\n"
" single-task default)\n"
"example: taskrun -j 12345 -m node01:2 -c \"hostname\" -u alice\n";
static const char USAGE_JOIN_SCHED[] =
"join\n"
" (no arguments)\n"
" Joins the cluster as a scheduler.\n";
static const char USAGE_GETCONF[] =
"getconf\n"
" (no arguments)\n"
" Prints the full cluster configuration (name, resources, queues,\n"
" workers, user/host groups, configs) known to the scheduler.\n";
static const char USAGE_SETPEND[] =
"setpend -j <jobid> -p <reason> [-p <reason> ...]\n"
" -j <jobid> jobid to set the pend reason for (required)\n"
" -p <reason> pend reason spec, may repeat: `<code>` sets the main\n"
" reason; `<host>:<code>` adds a host-based reason;\n"
" `<res>:<host>:<code>` adds a resource-based reason\n"
"example: setpend -j 12345 -p 5\n";
static command_t top_commands[] = {
{ "help", cmd_help, "Display this text" },
{ "worker", cmd_worker, "In worker context" },
{ "scheduler", cmd_scheduler, "In scheduler context" },
{ "client", cmd_client, "In client context" },
{ "set", cmd_set, "Show/Set settings" },
{ "echo", cmd_echo, "Print a message" },
{ "waldump", cmd_waldump, "Inspect WAL segments" },
{ "walcompare", cmd_walcompare, "Diff member committed WAL prefixes" },
{ "snapverify", cmd_snapverify, "Verify checkpoint file structure" },
{ "snapdiff", cmd_snapdiff, "Byte-diff two checkpoint files" },
{ "configcompare", cmd_configcompare, "Compare active config hash across members" },
{ "quit", cmd_quit, "Quit xce test" },
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL }
{ "help", cmd_help, "Display this text", USAGE_HELP },
{ "worker", cmd_worker, "In worker context", USAGE_WORKER },
{ "scheduler", cmd_scheduler, "In scheduler context", USAGE_SCHEDULER },
{ "client", cmd_client, "In client context", USAGE_CLIENT },
{ "set", cmd_set, "Show/Set settings", USAGE_SET },
{ "echo", cmd_echo, "Print a message", USAGE_ECHO },
{ "waldump", cmd_waldump, "Inspect WAL segments", USAGE_WALDUMP },
{ "walcompare", cmd_walcompare, "Diff member committed WAL prefixes",
USAGE_WALCOMPARE },
{ "snapverify", cmd_snapverify, "Verify checkpoint file structure",
USAGE_SNAPVERIFY },
{ "snapdiff", cmd_snapdiff, "Byte-diff two checkpoint files",
USAGE_SNAPDIFF },
{ "configcompare", cmd_configcompare, "Compare active config hash across members",
USAGE_CONFIGCOMPARE },
{ "quit", cmd_quit, "Quit xce test", USAGE_QUIT },
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL, (const char*)NULL }
};
static command_ctx_t topctx = { .parent = NULL,
@ -211,16 +562,16 @@ static command_ctx_t topctx = { .parent = NULL,
.commands = top_commands };
static command_t worker_commands[] = {
{ "..", cmd_up, "Goto upper context" },
{ "help", cmd_help, "Display this text" },
{ "quit", cmd_quit, "Quit xce test" },
{ "join", cmd_join_worker, "Join the cluster" },
{ "jstart", cmd_jobstart, "tell controller jobstart" },
{ "jfinish", cmd_jobfinish, "tell controller jobfinish" },
{ "jfail", cmd_jobfail, "tell controller jobfail" },
{ "resupdate", cmd_resupdate, "update res vals" },
{ "echo", cmd_echo, "Print a message" },
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL }
{ "..", cmd_up, "Goto upper context", USAGE_UP },
{ "help", cmd_help, "Display this text", USAGE_HELP },
{ "quit", cmd_quit, "Quit xce test", USAGE_QUIT },
{ "join", cmd_join_worker, "Join the cluster", USAGE_JOIN_WORKER },
{ "jstart", cmd_jobstart, "tell controller jobstart", USAGE_JOBSTART },
{ "jfinish", cmd_jobfinish, "tell controller jobfinish", USAGE_JOBFINISH },
{ "jfail", cmd_jobfail, "tell controller jobfail", USAGE_JOBFAIL },
{ "resupdate", cmd_resupdate, "update res vals", USAGE_RESUPDATE },
{ "echo", cmd_echo, "Print a message", USAGE_ECHO },
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL, (const char*)NULL }
};
static command_ctx_t workerctx = { .parent = &topctx,
@ -229,37 +580,41 @@ static command_ctx_t workerctx = { .parent = &topctx,
.commands = worker_commands };
static command_t client_commands[] = {
{ "..", cmd_up, "Goto upper context" },
{ "hello", cmd_hello, "Say hello to xce controller" },
{ "reslist", cmd_reslist, "Display cluster resources" },
{ "workerlist", cmd_workerlist, "list workers" },
{ "qctrl", cmd_qctrl, "queue control" },
{ "hostctrl", cmd_hostctrl, "host control" },
{ "..", cmd_up, "Goto upper context", USAGE_UP },
{ "hello", cmd_hello, "Say hello to xce controller", USAGE_HELLO },
{ "reslist", cmd_reslist, "Display cluster resources", USAGE_RESLIST },
{ "workerlist", cmd_workerlist, "list workers", USAGE_WORKERLIST },
{ "qctrl", cmd_qctrl, "queue control", USAGE_QCTRL },
{ "hostctrl", cmd_hostctrl, "host control", USAGE_HOSTCTRL },
{ "upgradeconfig", cmd_upgradeconfig,
"submit a config upgrade (cluster.bin file)" },
{ "joblist", cmd_joblist, "list jobs" },
{ "expect-jobs", cmd_expect_jobs, "assert current job count" },
{ "expect-job", cmd_expect_job, "assert a JobId is present" },
"submit a config upgrade (cluster.bin file)", USAGE_UPGRADECONFIG },
{ "joblist", cmd_joblist, "list jobs", USAGE_JOBLIST },
{ "expect-jobs", cmd_expect_jobs, "assert current job count", USAGE_EXPECT_JOBS },
{ "expect-job", cmd_expect_job, "assert a JobId is present", USAGE_EXPECT_JOB },
{ "wait-primary", cmd_wait_primary,
"wait until a primary serves [timeout_ms]" },
{ "ctrlstatus", cmd_ctrlstatus, "show controller/cluster status" },
{ "waldump", cmd_waldump, "Inspect WAL segments" },
{ "walcompare", cmd_walcompare, "Diff member committed WAL prefixes" },
{ "snapverify", cmd_snapverify, "Verify checkpoint file structure" },
{ "snapdiff", cmd_snapdiff, "Byte-diff two checkpoint files" },
{ "configcompare", cmd_configcompare, "Compare active config hash across members" },
{ "shutdown", cmd_shutdown, "shutdown xce controller" },
{ "jobrun", cmd_jobrun, "run a job" },
{ "jobsub", cmd_jobsub, "submit a job" },
{ "jobkill", cmd_jobkill, "signal a job" },
{ "jobckpt", cmd_jobckpt, "checkpoint a job" },
{ "jobinfo", cmd_jobinfo, "job information" },
{ "resval", cmd_resval, "get value of res" },
{ "taskrun", cmd_runtasks, "run tasks" },
{ "help", cmd_help, "Display this text" },
{ "echo", cmd_echo, "Print a message" },
{ "quit", cmd_quit, "Quit xce test" },
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL }
"wait until a primary serves [timeout_ms]", USAGE_WAIT_PRIMARY },
{ "ctrlstatus", cmd_ctrlstatus, "show controller/cluster status", USAGE_CTRLSTATUS },
{ "waldump", cmd_waldump, "Inspect WAL segments", USAGE_WALDUMP },
{ "walcompare", cmd_walcompare, "Diff member committed WAL prefixes",
USAGE_WALCOMPARE },
{ "snapverify", cmd_snapverify, "Verify checkpoint file structure",
USAGE_SNAPVERIFY },
{ "snapdiff", cmd_snapdiff, "Byte-diff two checkpoint files",
USAGE_SNAPDIFF },
{ "configcompare", cmd_configcompare, "Compare active config hash across members",
USAGE_CONFIGCOMPARE },
{ "shutdown", cmd_shutdown, "shutdown xce controller", USAGE_SHUTDOWN },
{ "jobrun", cmd_jobrun, "run a job", USAGE_JOBRUN },
{ "jobsub", cmd_jobsub, "submit a job", USAGE_JOBSUB },
{ "jobkill", cmd_jobkill, "signal a job", USAGE_JOBKILL },
{ "jobckpt", cmd_jobckpt, "checkpoint a job", USAGE_JOBCKPT },
{ "jobinfo", cmd_jobinfo, "job information", USAGE_JOBINFO },
{ "resval", cmd_resval, "get value of res", USAGE_RESVAL },
{ "taskrun", cmd_runtasks, "run tasks", USAGE_TASKRUN },
{ "help", cmd_help, "Display this text", USAGE_HELP },
{ "echo", cmd_echo, "Print a message", USAGE_ECHO },
{ "quit", cmd_quit, "Quit xce test", USAGE_QUIT },
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL, (const char*)NULL }
};
static command_ctx_t clientctx = { .parent = &topctx,
@ -268,14 +623,14 @@ static command_ctx_t clientctx = { .parent = &topctx,
.commands = client_commands };
static command_t scheduler_commands[] = {
{ "..", cmd_up, "Goto upper context" },
{ "help", cmd_help, "Display this text" },
{ "quit", cmd_quit, "Quit xce test" },
{ "join", cmd_join_sched, "Join the cluster" },
{ "getconf", cmd_getconf, "Get conf of cluster" },
{ "setpend", cmd_setpend, "set job pend reason" },
{ "echo", cmd_echo, "Print a message" },
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL }
{ "..", cmd_up, "Goto upper context", USAGE_UP },
{ "help", cmd_help, "Display this text", USAGE_HELP },
{ "quit", cmd_quit, "Quit xce test", USAGE_QUIT },
{ "join", cmd_join_sched, "Join the cluster", USAGE_JOIN_SCHED },
{ "getconf", cmd_getconf, "Get conf of cluster", USAGE_GETCONF },
{ "setpend", cmd_setpend, "set job pend reason", USAGE_SETPEND },
{ "echo", cmd_echo, "Print a message", USAGE_ECHO },
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL, (const char*)NULL }
};
static command_ctx_t schedulerctx = { .parent = &topctx,
@ -678,8 +1033,18 @@ int cmd_help(int argc, char* argv[])
for (i = 0; active_ctx->commands[i].name != NULL; i++) {
if (strcmp(argv[1], active_ctx->commands[i].name) == 0) {
printf("%s\t\t%s.\n", active_ctx->commands[i].name,
active_ctx->commands[i].doc);
/* `help <cmd>` prints the detailed per-command usage (flags,
* positionals, example) -- distinct from the compact one-line
* `doc` the bare `help` listing below still uses unchanged.
* Every table entry is expected to carry a non-NULL `usage`
* now; the `doc` fallback stays only as a defensive guard
* against a future entry that forgets to set it. */
if (active_ctx->commands[i].usage != NULL) {
fputs(active_ctx->commands[i].usage, stdout);
} else {
printf("%s\t\t%s.\n", active_ctx->commands[i].name,
active_ctx->commands[i].doc);
}
return 0;
}
}
@ -972,7 +1337,10 @@ int cmd_hello(int argc, char* argv[])
msg = optarg;
break;
default:
fprintf(stderr, " hello [-c count] [-w wait_usec] [-m message]\n");
/* -w was never implemented (getopt string above is "c:m:" only)
* -- this hint previously advertised a nonexistent -w wait_usec
* flag; corrected to match the actual accepted options. */
fprintf(stderr, " hello [-c count] [-m message]\n");
return -1;
break;
}
@ -1071,20 +1439,26 @@ int cmd_workerlist(int argc, char* argv[])
"MaxSwap", "MaxTmp", "NDisks");
for (i = 0; i < rc; i++) {
/* MaxMem is stored/wired as raw bytes (xce_resource_t.unit="bytes"
* for the maxmem resource definition) -- humanize for display only,
* every other column stays as-is. */
/* MaxMem/MaxSwap/MaxTmp are stored/wired as raw bytes (see
* is_bytes_valued_resource) -- humanize for display only, every
* other column stays as-is. */
char maxmem_buf[64];
char maxswap_buf[64];
char maxtmp_buf[64];
format_bytes_human(workers[i]->attrs->maxmem, maxmem_buf,
sizeof(maxmem_buf));
format_bytes_human(workers[i]->attrs->maxswap, maxswap_buf,
sizeof(maxswap_buf));
format_bytes_human(workers[i]->attrs->maxtmp, maxtmp_buf,
sizeof(maxtmp_buf));
printf("%-20s%-8s%-25s%-25s%12f%12d%12s%12f%12f%12d\n",
printf("%-20s%-8s%-25s%-25s%12f%12d%12s%12s%12s%12d\n",
workers[i]->name, worker_status_str(workers[i]->status),
workers[i]->attrs->hosttype, workers[i]->attrs->hostmodel,
workers[i]->attrs->cpufactor, workers[i]->attrs->maxcpus,
maxmem_buf, workers[i]->attrs->maxswap,
workers[i]->attrs->maxtmp, workers[i]->attrs->ndisks);
maxmem_buf, maxswap_buf, maxtmp_buf,
workers[i]->attrs->ndisks);
}
free_Worker_list(rc, workers);
@ -2038,15 +2412,19 @@ cleanup:
/*
* Resource names whose canonical stored/wire value is raw bytes (not a
* generic schema-driven unit system -- deliberately kept as a small,
* explicit, client-side list rather than a wire field, since exactly one
* resource needs this today and a wire-level unit hint would have grown
* every Resource/ResourceVal message regardless of whether it's used).
* CLI display picks the best-fit human unit adaptively (format_bytes_human);
* the stored value is never converted.
* explicit, client-side list rather than a wire field, since only a
* handful of resources need this and a wire-level unit hint would have
* grown every Resource/ResourceVal message regardless of whether it's
* used). CLI display picks the best-fit human unit adaptively
* (format_bytes_human); the stored value is never converted. Covers every
* memory/disk-scale built-in host attribute and load index (maxmem, mem,
* tmp, maxswap, maxtmp) -- deliberately NOT maxcpus/ndisks (plain counts)
* or cpuf/r1m/r15s (unitless factors/averages).
*/
static int is_bytes_valued_resource(const char* name)
{
static const char* const BYTES_VALUED_RESOURCES[] = { "maxmem" };
static const char* const BYTES_VALUED_RESOURCES[]
= { "maxmem", "mem", "tmp", "maxswap", "maxtmp" };
const int nres = (int)(sizeof(BYTES_VALUED_RESOURCES)
/ sizeof(BYTES_VALUED_RESOURCES[0]));
int i;