Rongsong Shen
da1a663f05
Stack overflows on user input:
- bug-1702 jobrun -m: host segment strncpy'd into a 256-byte buffer
unbounded (ASAN stack-buffer-overflow); now cmdutil_split_host_tasks()
refuses an over-long host.
- bug-1703 jobsub -n: min strncpy'd into a 64-byte buffer; now
cmdutil_parse_ntasks() parses in place, 1 <= min <= max.
- resupdate's 16-entry vals[] had no bound either; a 17th -n overflowed.
Wrong or silent results:
- bug-1574 resupdate type/model: set_resval() had no string case, left the
entry's name uninitialized, and crashed the encoder. Moved to
resval_typeresolve.c as resval_fill(), which fills every type.
- bug-1705 jobsub -x was overwritten (and leaked) by the default exdata;
cmdutil_default_exdata() only fills an unset one. jobsub's
execuser/exechome/execcwd copies leaked on every call.
- bug-1707 a trailing -n without -v was dropped silently; resupdate also
returned 0 after its RPC failed and "succeeded" with nothing to send.
- bug-1708 jobkill -s SIGKILL became signal 0 and was sent.
- bug-1709 jobinfo exited 0 with no -j, a malformed -j, or no hits.
- bug-1710 hello -c 0 sent nothing and reported success.
- bug-1712 a duplicate -v was reported as "no res name found".
- bug-1777 workerlist showed join-time maxmem/maxswap/maxtmp, not the
effective values (hosts.yaml overrides) the reply already carries in
resvals_to_sched and the scheduler uses.
- bug-1793 join's maxswap/maxtmp defaults were 1 KB / 2 KB in bytes; now
1 GiB / 2 GiB, and -m/-s/-t no longer overflow atoi() above 2 GB.
Error reporting and leaks:
- bug-1796 RPC failures printed perror() of a stale errno ("Connection
refused" / "Invalid argument" / "Inappropriate ioctl for device" for
the same failure). cmdutil_rpc_error() prints the RPC's own code
(E_XCE_* included) at all 19 sites.
- bug-1797 qctrl leaked the queue name on the RPC-error path; repeated
-q/-h leaked the earlier copy.
New src/harness/cmdutil.c (+ cmdutil_unit_test, 14 cases); 9 new
test_test_cli.c cases, 8 RED on the old xce_test; new resval_fill,
worker_effective_attr and join-default tests (join ones RED on the old
join_worker_args.c). Harness suite 6/6 (asan), harness_unit_test 141/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
7004 lines
245 KiB
C
7004 lines
245 KiB
C
#include <argp.h>
|
|
#include <errno.h>
|
|
#include <getopt.h>
|
|
#include <inttypes.h>
|
|
#include <limits.h> /* PATH_MAX -- cmd_source's realpath() canonicalization
|
|
* buffer (xce-test-cli-ux task 2.4) */
|
|
#include <signal.h>
|
|
#include <stdio.h>
|
|
#include <stdlib.h>
|
|
#include <string.h>
|
|
#include <sys/wait.h>
|
|
#include <time.h>
|
|
#include <unistd.h>
|
|
|
|
#include <editline/readline.h> /* libedit (BSD) -- provides readline()/
|
|
* add_history(), same as GNU readline's
|
|
* split readline.h/history.h pair */
|
|
|
|
#include <cjson/cJSON.h>
|
|
#include <cjson/s2j.h>
|
|
#include <openssl/sha.h> /* SHA256() — cmd_upgradeconfig hashes the config blob */
|
|
|
|
#include "client.h"
|
|
#include "config.h" /* XCE_MAXNAME_LEN */
|
|
#include "controller.capnp.h"
|
|
#include "xce/ctrlstatus_fmt.h" /* ctrlstatus_format_time_ms (task 6.3) */
|
|
#include "delay.h"
|
|
#include "flagspec.h" /* shared flagspec_t descriptor-table flag parsing,
|
|
* rejection (with typo suggestion), and help
|
|
* rendering (xce-test-cli-ux tasks 1.1/1.2); every
|
|
* flag-accepting command below migrates onto this
|
|
* (task 3.1) */
|
|
#include "idgen.h"
|
|
#include "interruptcall.h" /* xcetest_run_interruptible, xcetest_set_active_connection
|
|
* (xce-test-cli-ux tasks 1.4/1.5); every blocking xcecli_*
|
|
* call site below is wrapped through this (task 3.2).
|
|
*
|
|
* Every per-call-site heap block below (typedef'd as
|
|
* <cmd>_call_t, one per wrapped call site) is allocated
|
|
* via calloc(1, sizeof(*call)), never plain malloc() --
|
|
* found live, not just by inspection: cmd_jobrun's
|
|
* xce_job_alloc_t has two fields (nres_shared,
|
|
* allocs_shared) that fill_job_alloc() never sets, a
|
|
* PRE-EXISTING gap that was harmless on the old
|
|
* stack-allocated xce_job_alloc_t (whatever incidental
|
|
* value the stack frame happened to hold there rarely
|
|
* mattered) but corrupts the wire encoding when the SAME
|
|
* uninitialized fields instead come from freshly
|
|
* malloc()'d heap memory (confirmed: an ASAN
|
|
* allocation-size-too-big abort in alloc_packet() from a
|
|
* garbage nres_shared/allocs_shared, immediately fixed by
|
|
* switching this block's allocation to calloc()). Rather
|
|
* than auditing every wrapped request struct for a
|
|
* similarly-never-fully-initialized field one at a time,
|
|
* every call site's heap block is zero-initialized
|
|
* uniformly -- the same defensive default an
|
|
* accidentally-zero stack frame was already silently
|
|
* relied on for. */
|
|
#include "preason.h"
|
|
#include "walcompare.h" /* cross-member committed-prefix divergence (cmd_walcompare) */
|
|
#include "waldump.h" /* offline WAL inspector (cmd_waldump) */
|
|
#include "snapdiff.h" /* byte-level checkpoint-file comparator (cmd_snapdiff) */
|
|
#include "snapverify.h" /* offline checkpoint-file structural validator (cmd_snapverify) */
|
|
#include "configcompare.h" /* cross-member active-config comparator (cmd_configcompare) */
|
|
#include "confirm.h" /* destructive-command confirmation gate (xce-test-cli-ux task 2.1) */
|
|
#include "cmdutil.h" /* argument parsing + RPC-error reporting helpers */
|
|
#include "joblist_display.h" /* worker_status_str/job_state_str/format_job_time
|
|
* (xce-test-cli-ux task 2.5 remediation) */
|
|
#include "join_worker_args.h" /* join_worker_parse_attrs (xce-test-cli-ux
|
|
* task 2.6 remediation, bug-1557) */
|
|
#include "listfmt_json.h" /* listfmt_json_begin/new_row/finish -- JSON list
|
|
* output for workerlist/joblist/reslist
|
|
* (xce-test-cli-ux task 2.2, wired in task 3.4) */
|
|
#include "listrow_json.h" /* reslist_json_row/workerlist_json_row/
|
|
* joblist_json_row -- per-command JSON field
|
|
* mapping (xce-test-cli-ux task 3.4) */
|
|
#include "rcfile.h" /* startup rc-file path resolution + loading (xce-test-rc-file task 2.2) */
|
|
#include "resval_typeresolve.h" /* resval_resolve_type (resource-value-verification task 2.1) */
|
|
#include "xce/cluster.h" /* xce_parse_peers, xce_cluster_config_t */
|
|
#include "xce/types.h"
|
|
#include "xcetest_interrupt.h" /* xcetest_install_sigint, xcetest_interrupted (xce-test-cli-ux task 1.3) */
|
|
|
|
typedef int (*cmd_func_t)(int, char*[]);
|
|
|
|
/* command_t (xce-test-cli-ux task 3.5, Help Renderer, Requirements 8.1-8.7,
|
|
* 11.2, 12.2): `usage`/`flagspec`/`flagspec_count`/`usage_tail` together let
|
|
* cmd_help() assemble a command's detailed `help <cmd>` text in the order
|
|
* design.md mandates: description -> flags -> example -> related-command/
|
|
* context notes.
|
|
*
|
|
* usage For a command with a flagspec table (flagspec != NULL):
|
|
* its HEAD text only -- the usage-syntax line plus a
|
|
* 1-2 sentence purpose description (and any positional-
|
|
* argument documentation flagspec_render_help() can't
|
|
* express). For a command with NO flagspec table
|
|
* (flagspec == NULL): the ENTIRE detailed usage text, same
|
|
* as before this task -- printed as-is, nothing generated.
|
|
* flagspec NULL if this command takes no flags; otherwise the
|
|
* command's flagspec_t table (Flag Parsing Layer, tasks
|
|
* 1.1/1.2), rendered via flagspec_render_help() and
|
|
* printed immediately after `usage`.
|
|
* flagspec_count Element count of `flagspec`; meaningless when
|
|
* flagspec == NULL.
|
|
* usage_tail Printed after the generated flag block: the example
|
|
* invocation(s) and any related-command/scripting-state
|
|
* notes (Requirement 8.3/8.4). NULL when flagspec == NULL
|
|
* -- `usage` already carries everything for those
|
|
* commands, matching their pre-task-3.5 shape exactly.
|
|
*
|
|
* C aggregate-initializer rules zero-initialize any trailing members a
|
|
* `{ ... }` initializer omits, so every non-flag command_t entry (help,
|
|
* quit, echo, source, up, worker, scheduler, client, set, waldump,
|
|
* snapverify, snapdiff, configcompare, join_sched, getconf, expect-jobs,
|
|
* expect-job, wait-primary, ctrlstatus) needs NO changes at all: their
|
|
* initializers stay exactly 4 fields, and flagspec/flagspec_count/
|
|
* usage_tail all come out NULL/0 automatically. */
|
|
typedef struct {
|
|
char* name;
|
|
cmd_func_t func;
|
|
char* doc;
|
|
const char* usage;
|
|
const flagspec_t* flagspec;
|
|
size_t flagspec_count;
|
|
const char* usage_tail;
|
|
} command_t;
|
|
|
|
typedef struct {
|
|
char* key;
|
|
char* value;
|
|
char* defval;
|
|
} setting_t;
|
|
|
|
typedef struct _command_ctx {
|
|
struct _command_ctx* parent;
|
|
char* prompt;
|
|
setting_t* settings;
|
|
command_t* commands;
|
|
xcecli_ctx_t ctx;
|
|
} command_ctx_t;
|
|
|
|
typedef struct {
|
|
uint64_t jobid;
|
|
int pid;
|
|
time_t ts;
|
|
} run_job_t;
|
|
|
|
typedef struct {
|
|
uint64_t jobid;
|
|
uint32_t taskid;
|
|
int clientidx;
|
|
int pid;
|
|
time_t ts;
|
|
} run_task_t;
|
|
|
|
typedef struct {
|
|
char* host;
|
|
int ntasks;
|
|
} host_task_t;
|
|
|
|
#define MAX_RUN_JOBS (20)
|
|
#define MAX_TASK_HOSTS (5)
|
|
#define MAX_RUN_TASKS (50)
|
|
#define RUSAGE_UPDATE_INTERVAL (60)
|
|
#define MAX_LINE_LEN (40960)
|
|
|
|
#define S2J_JSON_SET_var_string_ELEMENT(to_json, from_struct, _element) \
|
|
if (((from_struct) != NULL) && (((from_struct)->_element) != NULL)) { \
|
|
cJSON_AddStringToObject((to_json), #_element, \
|
|
(from_struct)->_element); \
|
|
}
|
|
|
|
static run_job_t run_jobs[MAX_RUN_JOBS];
|
|
static run_task_t run_tasks[MAX_RUN_TASKS];
|
|
|
|
static int myreq_handler(xce_request_t*, void*);
|
|
|
|
extern int cmd_help(int, char*[]);
|
|
extern int cmd_up(int, char*[]);
|
|
extern int cmd_worker(int, char*[]);
|
|
extern int cmd_scheduler(int, char*[]);
|
|
extern int cmd_client(int, char*[]);
|
|
extern int cmd_quit(int, char*[]);
|
|
extern int cmd_hello(int, char*[]);
|
|
extern int cmd_shutdown(int, char*[]);
|
|
extern int cmd_set(int, char*[]);
|
|
extern int cmd_reslist(int, char*[]);
|
|
extern int cmd_join_worker(int, char*[]);
|
|
extern int cmd_jobstart(int, char*[]);
|
|
extern int cmd_jobfail(int, char*[]);
|
|
extern int cmd_join_sched(int, char*[]);
|
|
extern int cmd_workerlist(int, char*[]);
|
|
extern int cmd_joblist(int, char*[]);
|
|
extern int cmd_jobrun(int, char*[]);
|
|
extern int cmd_jobsub(int, char*[]);
|
|
extern int cmd_getconf(int, char*[]);
|
|
extern int cmd_jobfinish(int, char*[]);
|
|
extern int cmd_resupdate(int, char*[]);
|
|
extern int cmd_resval(int, char*[]);
|
|
extern int cmd_jobkill(int, char*[]);
|
|
extern int cmd_jobckpt(int, char*[]);
|
|
extern int cmd_qctrl(int, char*[]);
|
|
extern int cmd_hostctrl(int, char*[]);
|
|
extern int cmd_upgradeconfig(int, char*[]);
|
|
extern int cmd_runtasks(int, char*[]);
|
|
extern int cmd_setpend(int, char*[]);
|
|
extern int cmd_jobinfo(int, char*[]);
|
|
extern int cmd_echo(int, char*[]);
|
|
extern int cmd_expect_jobs(int, char*[]);
|
|
extern int cmd_expect_job(int, char*[]);
|
|
extern int cmd_wait_primary(int, char*[]);
|
|
extern int cmd_ctrlstatus(int, char*[]);
|
|
extern int cmd_waldump(int, char*[]);
|
|
extern int cmd_walcompare(int, char*[]);
|
|
extern int cmd_snapverify(int, char*[]);
|
|
extern int cmd_snapdiff(int, char*[]);
|
|
extern int cmd_configcompare(int, char*[]);
|
|
extern int cmd_source(int, char*[]);
|
|
static int worker_house_keeping(void*);
|
|
|
|
/* Session verdict / loop state (xce-test-e2e-primitives task 1.1).
|
|
*
|
|
* g_session_failed set to 1 the first time any command in the session
|
|
* fails; drives the process exit code in main().
|
|
* g_continue_on_error when set (via `set continue-on-error`), a failing
|
|
* command does not stop the read loop; the session still
|
|
* exits non-zero at the end.
|
|
* g_last_status status of the most recently executed command
|
|
* (0 == success, non-zero == failure). Future tasks
|
|
* expose this as the `$?` auto-variable.
|
|
*/
|
|
static int g_session_failed = 0;
|
|
static int g_continue_on_error = 0;
|
|
static int g_last_status = 0;
|
|
/* Interactive (TTY) sessions are a REPL: a failed command prints its error and
|
|
* the loop CONTINUES — never abort the read loop on a typo. The abort-on-first-
|
|
* failure + non-zero-exit verdict is a SCRIPTED/piped-mode concept only. Set
|
|
* from isatty(stdin) in main. */
|
|
static int g_interactive = 0;
|
|
|
|
/* Auto-variable state for the fixed `$jobid` / `$jobcount` substitution
|
|
* (xce-test-e2e-primitives task 3.1, Req 3.1/3.3/3.5/3.6).
|
|
*
|
|
* g_last_jobid most recently acknowledged JobId as a string; empty ("")
|
|
* means unset. Populated later by cmd_jobsub (task 4.1).
|
|
* g_last_jobcount most recent job count; -1 means unset. Populated by
|
|
* cmd_joblist on a successful `joblist` (task 5.1, Req 3.4);
|
|
* a failed joblist leaves it untouched so no fake count is
|
|
* advertised.
|
|
*
|
|
* Until a command sets one, it stays unset — which is exactly what makes an
|
|
* unset `$jobid`/`$jobcount` reference fail the command (Req 3.6). */
|
|
static char g_last_jobid[XCE_MAXNAME_LEN] = { 0 };
|
|
static int g_last_jobcount = -1;
|
|
|
|
static idgen_ctx_t idctx;
|
|
delaymgr_t delaymgr;
|
|
|
|
static setting_t settings[] = {
|
|
/* HA controller list: comma-separated "name=host:port" or "host:port"
|
|
* entries (same syntax as the controller --peers flag), enabling the
|
|
* test client to fail over across controller candidates. */
|
|
{ (char*)"masters", (char*)NULL, (char*)"localhost:9990" },
|
|
{ (char*)"secret", (char*)NULL, (char*)"keepSecret" },
|
|
{ (char*)"myname", (char*)NULL, (char*)"localhost" },
|
|
{ (char*)NULL, (char*)NULL, (char*)NULL }
|
|
};
|
|
|
|
/* Detailed per-command `help <cmd>` text (design: keep these OUT of the
|
|
* command_t table initializers below so the tables stay readable). Each
|
|
* constant's shape: first line = command name + flag/positional syntax,
|
|
* then one indented line per flag/positional explaining what it does and
|
|
* whether it's required, then (where useful) an `example:` line. Derived
|
|
* from actually reading each cmd_* handler's getopt() string and switch
|
|
* cases (or its positional argv[] handling) -- not guessed from the name.
|
|
*
|
|
* Commands that are identical (same func, same semantics) across more than
|
|
* one of the 4 command tables share ONE constant here, referenced from every
|
|
* table, so the text can't drift out of sync between contexts. */
|
|
|
|
static const char USAGE_HELP[] =
|
|
"help [<cmd>]\n"
|
|
" <cmd> optional command name; prints its detailed usage instead of\n"
|
|
" the one-line summary\n"
|
|
" With no argument, lists every command in the current context with its\n"
|
|
" one-line summary (unaffected by this detailed-usage text).\n"
|
|
"example: help resval\n";
|
|
|
|
static const char USAGE_QUIT[] =
|
|
"quit\n"
|
|
" (no arguments)\n"
|
|
" Leaves the current context (if any) and exits xce_test.\n";
|
|
|
|
static const char USAGE_ECHO[] =
|
|
"echo [<message...>]\n"
|
|
" <message...> zero or more words, printed space-joined followed by a\n"
|
|
" newline (always succeeds)\n"
|
|
" Useful to print the $jobid / $jobcount / $? auto-variables in scripts.\n"
|
|
"example: echo job is $jobid\n";
|
|
|
|
static const char USAGE_UP[] =
|
|
"..\n"
|
|
" (no arguments)\n"
|
|
" Leaves the current context and returns to its parent, closing the\n"
|
|
" underlying client connection for this context.\n";
|
|
|
|
static const char USAGE_WALDUMP[] =
|
|
"waldump <wal_dir|data_dir|seg_file> [more...]\n"
|
|
" <path> one or more paths: a seg-*.wal file, a wal/ dir, or a\n"
|
|
" node/data dir\n"
|
|
" Offline, read-only WAL inspector: lists every entry, decodes\n"
|
|
" JOB_SUBMIT jobids, and flags LSN gaps and a CRC-broken tail.\n"
|
|
" Returns nonzero if any path fails to load or any gap/CRC break is\n"
|
|
" found.\n"
|
|
"example: waldump /var/xce/ctrl-1/wal\n";
|
|
|
|
static const char USAGE_WALCOMPARE_HEAD[] =
|
|
"walcompare [--report] [--committed <N>] <memberA> <memberB> [more...]\n"
|
|
" <memberX> 2-16 member paths (seg-*.wal file, wal/ dir, or\n"
|
|
" node/data dir)\n"
|
|
" Diffs each member's committed WAL prefix (by LSN, term, and CRC) to\n"
|
|
" detect a durability divergence across replicas; --report\n"
|
|
" additionally emits a full per-member (lsn,term,pcrc,status)\n"
|
|
" machine-checkable table, not just the verdict. Returns 0 if all\n"
|
|
" members' committed WAL prefixes are identical, nonzero on\n"
|
|
" divergence or load error.\n";
|
|
|
|
static const char USAGE_WALCOMPARE_TAIL[] =
|
|
"example: walcompare --report /var/xce/ctrl-1/wal /var/xce/ctrl-2/wal\n";
|
|
|
|
static const char USAGE_SNAPVERIFY[] =
|
|
"snapverify <snap_file> [<snap_file> ...]\n"
|
|
" <snap_file> one or more snap-*.capnp or snap-*.bin checkpoint files\n"
|
|
" Offline structural validator: walks the same layout\n"
|
|
" persist_load_snapshot() parses, without restoring anything. Reports\n"
|
|
" the byte offset of the first structural defect and how many frames\n"
|
|
" parsed cleanly before it.\n"
|
|
" Returns nonzero if any file fails to verify.\n"
|
|
"example: snapverify /var/xce/ctrl-1/snap/snap-000123.capnp\n";
|
|
|
|
static const char USAGE_SNAPDIFF[] =
|
|
"snapdiff <source_file> <received_file>\n"
|
|
" <source_file> the primary's own locally generated checkpoint\n"
|
|
" (snap-*.capnp)\n"
|
|
" <received_file> the copy a standby assembled from the wire\n"
|
|
" (snap-*.bin)\n"
|
|
" Byte-level comparator; reports the first differing offset and\n"
|
|
" whether it lands on an OPCode_snapshotTransfer chunk boundary.\n"
|
|
" Returns 0 if identical, nonzero otherwise.\n"
|
|
"example: snapdiff snap-000123.capnp snap-000123.bin\n";
|
|
|
|
static const char USAGE_CONFIGCOMPARE[] =
|
|
"configcompare <member_dir> <member_dir> [<member_dir> ...]\n"
|
|
" <member_dir> a member's workdir/name conf-root directory (NOT a raw\n"
|
|
" config file path) -- the dir under which conf/current\n"
|
|
" resolves\n"
|
|
" Recomputes each member's active-config sha256 fresh from its own\n"
|
|
" on-disk bytes and reports whether all members match.\n"
|
|
" Returns 0 if all members are present and identical, nonzero if any\n"
|
|
" member is unreadable or divergent.\n"
|
|
"example: configcompare /var/xce/ctrl-1 /var/xce/ctrl-2 /var/xce/ctrl-3\n";
|
|
|
|
static const char USAGE_SOURCE[] =
|
|
"source <file>\n"
|
|
" <file> path to a file containing xce_test commands, one per line\n"
|
|
" (required)\n"
|
|
" Reads and executes each line of <file> in order, exactly as if it\n"
|
|
" had been typed at the current prompt -- including a nested `source`\n"
|
|
" command, so one script file can compose others. A file that does\n"
|
|
" not exist or cannot be read is rejected with an explicit error and\n"
|
|
" no line of it is executed; a file that would source itself,\n"
|
|
" directly or transitively, is rejected as a cycle instead of\n"
|
|
" hanging. A failing line inside the file follows the session's\n"
|
|
" current -e/+e/continue-on-error setting (see `help set`), exactly\n"
|
|
" as it does for lines typed at the top-level prompt.\n"
|
|
"example: source /path/to/script.xt\n";
|
|
|
|
static const char USAGE_WORKER[] =
|
|
"worker\n"
|
|
" (no arguments)\n"
|
|
" Enters the worker context: connects to the controller cluster as a\n"
|
|
" worker client and starts the local delay-manager housekeeping loop.\n";
|
|
|
|
static const char USAGE_SCHEDULER[] =
|
|
"scheduler\n"
|
|
" (no arguments)\n"
|
|
" Enters the scheduler context: connects to the controller cluster as\n"
|
|
" a scheduler client.\n";
|
|
|
|
static const char USAGE_CLIENT[] =
|
|
"client\n"
|
|
" (no arguments)\n"
|
|
" Enters the client context: connects to the controller cluster as a\n"
|
|
" regular API client (job submission, resource/worker queries, etc.).\n";
|
|
|
|
static const char USAGE_SET[] =
|
|
"set\n"
|
|
"set <key> <value>\n"
|
|
"set -e | +e | continue-on-error\n"
|
|
"set -y | +y\n"
|
|
" (no args) list all settings in the current context\n"
|
|
" (current value + default)\n"
|
|
" <key> <value> set setting <key> (e.g. masters, secret,\n"
|
|
" myname) to <value>\n"
|
|
" -e abort the read loop on the first failing\n"
|
|
" command (errexit)\n"
|
|
" +e | continue-on-error keep the read loop running after a\n"
|
|
" failing command\n"
|
|
" -y skip the confirmation prompt before a\n"
|
|
" destructive command (jobkill, shutdown,\n"
|
|
" qctrl -i/-c, hostctrl -c, upgradeconfig)\n"
|
|
" regardless of TTY state, until +y\n"
|
|
" +y require confirmation before those\n"
|
|
" commands again (default)\n"
|
|
"example: set masters ctrl1=localhost:9991,ctrl2=localhost:9992\n";
|
|
|
|
static const char USAGE_JOIN_WORKER_HEAD[] =
|
|
"join [-f <cpufactor>] [-c <maxcpus>] [-m <maxmem>] [-s <maxswap>]\n"
|
|
" [-t <maxtmp>] [-d <ndisks>]\n"
|
|
" Joins the cluster as a worker (hosttype LINUX64, hostmodel Intel64).\n";
|
|
|
|
static const char USAGE_JOIN_WORKER_TAIL[]
|
|
= "example: join -c 8 -m 17179869184 (16 GB in bytes)\n";
|
|
|
|
static const char USAGE_JOBSTART_HEAD[] =
|
|
"jstart -j <jobid> [-p <pid>]\n"
|
|
" Reports a synthetic job-start fact to the controller, simulating\n"
|
|
" what a real worker would report after actually launching the job\n"
|
|
" -- this does NOT execute anything; it is a fake-worker test\n"
|
|
" primitive only.\n";
|
|
|
|
static const char USAGE_JOBSTART_TAIL[] = "example: jstart -j 12345 -p 4242\n";
|
|
|
|
static const char USAGE_JOBFINISH_HEAD[] =
|
|
"jfinish -j <jobid> [-c <exitcode>]\n"
|
|
" Reports a synthetic job-finish fact to the controller, simulating\n"
|
|
" what a real worker would report after the job actually exited --\n"
|
|
" this does NOT execute anything; it is a fake-worker test primitive\n"
|
|
" only.\n";
|
|
|
|
static const char USAGE_JOBFINISH_TAIL[] = "example: jfinish -j 12345 -c 0\n";
|
|
|
|
static const char USAGE_JOBFAIL_HEAD[] =
|
|
"jfail -j <jobid> [-c <code>] [-s <stage>]\n"
|
|
" Reports a synthetic job-fail fact to the controller, simulating\n"
|
|
" what a real worker would report after the job actually failed --\n"
|
|
" this does NOT execute anything; it is a fake-worker test primitive\n"
|
|
" only.\n";
|
|
|
|
static const char USAGE_JOBFAIL_TAIL[] = "example: jfail -j 12345 -c 1\n";
|
|
|
|
static const char USAGE_RESUPDATE_HEAD[] =
|
|
"resupdate -n <resname> -v <value> [-t <numeric|string>]\n"
|
|
" [-n <resname> -v <value> [-t <numeric|string>] ...]\n"
|
|
" Injects or overrides a resource's reported value on this session's\n"
|
|
" simulated worker, for testing scheduler/resource-accounting\n"
|
|
" behavior without a real xcecd. Any resource name the cluster has\n"
|
|
" registered is accepted (e.g. slots, mem, cpuf, maxswap, maxtmp,\n"
|
|
" maxcpus, ndisks, type, model, dynres1, dynres2, r1m, r15s, r15m,\n"
|
|
" pg, ls, it, swp, io, tmp); -n may repeat to update multiple\n"
|
|
" resources in one call. -t forces the wire type for the\n"
|
|
" immediately preceding -n/-v pair; it is only meaningful for a\n"
|
|
" resource name NOT already known to this tool -- a known name\n"
|
|
" (e.g. mem, type) always keeps its own fixed type, and -t is\n"
|
|
" ignored for it, while an unrecognized name defaults to numeric\n"
|
|
" when -t is omitted. Order matters: pair each -n with its -v (and\n"
|
|
" optional -t) right after it.\n";
|
|
|
|
static const char USAGE_RESUPDATE_TAIL[] =
|
|
"example: resupdate -n slots -v 4 -n mem -v 8192\n"
|
|
"example: resupdate -n r15m -v 1.5\n"
|
|
"example: resupdate -n mylabel -v foo -t string\n";
|
|
|
|
static const char USAGE_HELLO_HEAD[] =
|
|
"hello [-c <count>] [-m <message>]\n"
|
|
" Sends a greeting RPC to the controller, optionally repeated, as a\n"
|
|
" basic connectivity smoke test for the active context's connection.\n";
|
|
|
|
static const char USAGE_HELLO_TAIL[] = "example: hello -c 3 -m \"ping\"\n";
|
|
|
|
static const char USAGE_RESLIST_HEAD[] =
|
|
"reslist [--json]\n"
|
|
" Prints all cluster-wide resources (name, flags, interval,\n"
|
|
" description). Use --json to emit the same fields as one JSON\n"
|
|
" object per resource instead of the human-readable table.\n";
|
|
|
|
static const char USAGE_RESLIST_TAIL[] = "example: reslist --json\n";
|
|
|
|
static const char USAGE_WORKERLIST_HEAD[] =
|
|
"workerlist [--json]\n"
|
|
" Prints all joined workers (host, status, host type/model, cpu\n"
|
|
" factor, max cpu/mem/swap/tmp, ndisks). Use --json to emit the same\n"
|
|
" fields as one JSON object per worker instead of the human-readable\n"
|
|
" table.\n";
|
|
|
|
static const char USAGE_WORKERLIST_TAIL[] = "example: workerlist --json\n";
|
|
|
|
static const char USAGE_QCTRL_HEAD[] =
|
|
"qctrl -q <queue> [-a] [-i] [-o] [-c]\n"
|
|
" Controls a queue's dispatch/submission state: -a sets it ACTIVE\n"
|
|
" (accepts dispatch), -i sets it INACTIVE (rejects dispatch), -o\n"
|
|
" sets it OPEN (accepts new submissions), and -c sets it CLOSE\n"
|
|
" (rejects new submissions). Exactly one of -a/-i/-o/-c is required.\n";
|
|
|
|
static const char USAGE_QCTRL_TAIL[] =
|
|
"example: qctrl -q normal -i\n"
|
|
" This is a destructive action only for -i/--deactivate and\n"
|
|
" -c/--close, which prompt for confirmation by default (-a/--activate\n"
|
|
" and -o/--open never prompt); use -y/--yes to skip the prompt for\n"
|
|
" this invocation, or `set -y` to skip it for the rest of the\n"
|
|
" session (see `help set`).\n";
|
|
|
|
static const char USAGE_HOSTCTRL_HEAD[] =
|
|
"hostctrl -h <hostname> (-o|-c)\n"
|
|
" Administratively controls a host's scheduling eligibility: -o\n"
|
|
" opens it (allow scheduling), -c closes it (admin-close; shows\n"
|
|
" CLOSED in workerlist). Exactly one of -o/-c is required.\n";
|
|
|
|
static const char USAGE_HOSTCTRL_TAIL[] =
|
|
"example: hostctrl -h node01 -c\n"
|
|
" This is a destructive action only for -c/--close, which prompts\n"
|
|
" for confirmation by default (-o/--open never prompts); use\n"
|
|
" -y/--yes to skip the prompt for this invocation, or `set -y` to\n"
|
|
" skip it for the rest of the session (see `help set`).\n";
|
|
|
|
static const char USAGE_UPGRADECONFIG_HEAD[] =
|
|
"upgradeconfig [-y|--yes] <cluster.bin>\n"
|
|
" <cluster.bin> path to a capnp-encoded ClusterConfig blob (same\n"
|
|
" format `xceadmin config build` produces and\n"
|
|
" deploy-ha.sh installs)\n"
|
|
" Computes its SHA-256 and submits it as a configuration upgrade via\n"
|
|
" OP_upgradeConfig.\n";
|
|
|
|
static const char USAGE_UPGRADECONFIG_TAIL[] =
|
|
"example: upgradeconfig /var/xce/cluster.bin\n"
|
|
" This is a destructive action and prompts for confirmation by\n"
|
|
" default; use -y/--yes to skip the prompt for this invocation, or\n"
|
|
" `set -y` to skip it for the rest of the session (see `help set`).\n";
|
|
|
|
static const char USAGE_JOBLIST_HEAD[] =
|
|
"joblist [--json]\n"
|
|
" Prints all jobs in the cluster (JobId, user, queue, state, times,\n"
|
|
" mintasks/maxtasks, command) and captures the count into $jobcount.\n"
|
|
" Use --json to emit the same fields as one JSON object per job\n"
|
|
" instead of the human-readable table.\n";
|
|
|
|
static const char USAGE_JOBLIST_TAIL[] = "example: joblist --json\n";
|
|
|
|
static const char USAGE_EXPECT_JOBS[] =
|
|
"expect-jobs <N>\n"
|
|
" <N> non-negative integer; asserts the current job count equals N\n"
|
|
" Fails (nonzero) on a malformed N, a failed joblist query, or a count\n"
|
|
" mismatch.\n"
|
|
"example: expect-jobs 3\n";
|
|
|
|
static const char USAGE_EXPECT_JOB[] =
|
|
"expect-job <jobid>\n"
|
|
" <jobid> a JobId string (as printed by joblist / captured into\n"
|
|
" $jobid); asserts a job with this id is present in the\n"
|
|
" current job list\n"
|
|
" Fails (nonzero) on a missing argument, a failed joblist query, or if\n"
|
|
" the job is not found.\n"
|
|
"example: expect-job $jobid\n";
|
|
|
|
static const char USAGE_WAIT_PRIMARY[] =
|
|
"wait-primary [<timeout_ms>]\n"
|
|
" <timeout_ms> optional max wait in milliseconds (default 30000);\n"
|
|
" must be a non-negative integer\n"
|
|
" Blocks, polling every 200 ms, until the AUTHORITATIVE controller\n"
|
|
" status probe reaches the serving primary (a standby/hint answer does\n"
|
|
" not count), or the timeout elapses.\n"
|
|
" Returns 0 once a primary answers, nonzero (ETIMEDOUT) on timeout.\n"
|
|
"example: wait-primary 10000\n";
|
|
|
|
static const char USAGE_CTRLSTATUS[] =
|
|
"ctrlstatus\n"
|
|
" (no arguments)\n"
|
|
" Prints the responding controller's identity/role, cluster name,\n"
|
|
" peer topology, config version/checksum, and HA health (term,\n"
|
|
" committedLsn, appliedLsn, fenced) from a single any-node status\n"
|
|
" read.\n";
|
|
|
|
static const char USAGE_SHUTDOWN_HEAD[] =
|
|
"shutdown [-y|--yes]\n"
|
|
" Shuts down the xce controller and, on success, leaves the client\n"
|
|
" context (same as running `..`).\n";
|
|
|
|
static const char USAGE_SHUTDOWN_TAIL[] =
|
|
"example: shutdown\n"
|
|
" This is a destructive action and prompts for confirmation by\n"
|
|
" default; use -y/--yes to skip the prompt for this invocation, or\n"
|
|
" `set -y` to skip it for the rest of the session (see `help set`).\n";
|
|
|
|
static const char USAGE_JOBRUN_HEAD[] =
|
|
"jobrun -j <jobid> [-m <host>[:<ntasks>]] [-m ...]\n"
|
|
" Dispatches an already-submitted job (jobid must be > 0) to one or\n"
|
|
" more hosts, allocating the given task counts on each.\n";
|
|
|
|
static const char USAGE_JOBRUN_TAIL[] =
|
|
"example: jobrun -j 12345 -m node01:2 -m node02\n";
|
|
|
|
static const char USAGE_JOBSUB_HEAD[] =
|
|
"jobsub -u <user> -q <queue> -c <cmdline> [-R <resreq>]\n"
|
|
" [-n <min>[,<max>]] [-x <exdata>] [-a <arraysize>]\n"
|
|
" Submits a new job to the cluster under the given user and queue.\n"
|
|
" -n accepts either a single task count, or a comma-separated\n"
|
|
" <min>,<max> pair to request an elastic task count the scheduler\n"
|
|
" may dispatch anywhere in that range; -x's exdata JSON, when\n"
|
|
" given, overrides the command's built-in PATH env instead of the\n"
|
|
" default.\n";
|
|
|
|
static const char USAGE_JOBSUB_TAIL[] =
|
|
" On success, captures the submitted JobId into the $jobid\n"
|
|
" auto-variable.\n"
|
|
"example: jobsub -u alice -q normal -c \"sleep 10\" -n 1,4\n";
|
|
|
|
static const char USAGE_RESVAL_HEAD[] =
|
|
"resval -n <resname> [-m <host>]\n"
|
|
" Queries the current value of one resource, cluster-wide or on a\n"
|
|
" specific host.\n";
|
|
|
|
static const char USAGE_RESVAL_TAIL[] = "example: resval -n slots -m myhost\n";
|
|
|
|
static const char USAGE_JOBKILL_HEAD[] =
|
|
"jobkill -j <jobid> [-s <signo>]\n"
|
|
" Sends a signal to a running job, terminating it by default.\n";
|
|
|
|
static const char USAGE_JOBKILL_TAIL[] =
|
|
"example: jobkill -j 12345 -s 9\n"
|
|
" This is a destructive action and prompts for confirmation by\n"
|
|
" default; use -y/--yes to skip the prompt for this invocation, or\n"
|
|
" `set -y` to skip it for the rest of the session (see `help set`).\n";
|
|
|
|
static const char USAGE_JOBCKPT_HEAD[] =
|
|
"jobckpt -j <jobid> [-k] [-m <method>]\n"
|
|
" Requests a checkpoint of a running job; -k additionally kills the\n"
|
|
" job immediately after checkpointing.\n";
|
|
|
|
static const char USAGE_JOBCKPT_TAIL[] =
|
|
"example: jobckpt -j 12345 -m default\n";
|
|
|
|
static const char USAGE_JOBINFO_HEAD[] =
|
|
"jobinfo -j <jobid> [-j <jobid> ...]\n"
|
|
" Prints detailed information (state, pend reasons, submit/exec\n"
|
|
" metadata) for one or more jobs; -j may be repeated to show\n"
|
|
" several jobs in one call.\n";
|
|
|
|
static const char USAGE_JOBINFO_TAIL[] = "example: jobinfo -j 12345\n";
|
|
|
|
static const char USAGE_TASKRUN_HEAD[] =
|
|
"taskrun [-j <jobid>] [-m <host>[:<n>]]... [-u <user>] [-c <cmdline>]\n"
|
|
" [-x <exdata>] [-d <dir>] [-t <launch_type>]\n"
|
|
" Reports one or more task launches for an existing job to the\n"
|
|
" controller -- the same call a real worker makes after actually\n"
|
|
" starting a task -- letting a test simulate multi-host task\n"
|
|
" placement without a live worker; -u/-c/-x/-d apply uniformly to\n"
|
|
" every -m host.\n";
|
|
|
|
static const char USAGE_TASKRUN_TAIL[] =
|
|
"example: taskrun -j 12345 -m node01:2 -c \"hostname\" -u alice\n";
|
|
|
|
static const char USAGE_JOIN_SCHED[] =
|
|
"join\n"
|
|
" (no arguments)\n"
|
|
" Joins the cluster as a scheduler.\n";
|
|
|
|
static const char USAGE_GETCONF[] =
|
|
"getconf\n"
|
|
" (no arguments)\n"
|
|
" Prints the full cluster configuration (name, resources, queues,\n"
|
|
" workers, user/host groups, configs) known to the scheduler.\n";
|
|
|
|
static const char USAGE_SETPEND_HEAD[] =
|
|
"setpend -j <jobid> -p <reason> [-p <reason> ...]\n"
|
|
" Sets a job's pend reason(s), simulating what the scheduler reports\n"
|
|
" for a job that cannot yet be dispatched. -p may repeat: `<code>`\n"
|
|
" sets the main reason; `<host>:<code>` adds a host-based reason;\n"
|
|
" `<res>:<host>:<code>` adds a resource-based reason.\n";
|
|
|
|
static const char USAGE_SETPEND_TAIL[] = "example: setpend -j 12345 -p 5\n";
|
|
|
|
/* xce-test-cli-ux task 3.5 (Help Renderer): forward-declare every
|
|
* flag-accepting command's flagspec_t table -- each one is actually
|
|
* DEFINED later in this file, immediately next to its cmd_* handler, as
|
|
* before this task -- with its current flag count as an explicit array
|
|
* bound, and give that count a name via the same NFLAGS macro each
|
|
* handler already uses to size its own getopt_long() option table
|
|
* (`struct option long_opts[X_NFLAGS + 1]`). This lets top_commands[]/
|
|
* worker_commands[]/client_commands[]/scheduler_commands[] below wire
|
|
* each entry's `.flagspec`/`.flagspec_count` directly to its table and
|
|
* macro, without requiring the table's full definition to appear first --
|
|
* a file-scope initializer only needs a constant expression, which a
|
|
* literal NFLAGS macro is (an array's true element count from a later
|
|
* sizeof() is not known this early).
|
|
*
|
|
* Safety net, not a hand-counted magic number left free to drift: C
|
|
* requires two declarations of the same array to agree on size once
|
|
* either one specifies a constant size, so if a future edit changes a
|
|
* table's entry count below without updating its bound here, the build
|
|
* fails immediately ("conflicting types for ...") instead of silently
|
|
* rendering a wrong flag count in generated help text. Each table's own
|
|
* `#define X_NFLAGS (sizeof(x_flags) / sizeof(x_flags[0]))` -- used
|
|
* exactly this way for tasks 1.1/3.1 -- is superseded by (and removed at)
|
|
* that same later definition site; this is the one and only place each
|
|
* NFLAGS macro is now defined. */
|
|
#define WALCOMPARE_NFLAGS 2
|
|
static const flagspec_t walcompare_flags[WALCOMPARE_NFLAGS];
|
|
#define HELLO_NFLAGS 2
|
|
static const flagspec_t hello_flags[HELLO_NFLAGS];
|
|
#define SHUTDOWN_NFLAGS 1
|
|
static const flagspec_t shutdown_flags[SHUTDOWN_NFLAGS];
|
|
#define RESLIST_NFLAGS 1
|
|
static const flagspec_t reslist_flags[RESLIST_NFLAGS];
|
|
#define WORKERLIST_NFLAGS 1
|
|
static const flagspec_t workerlist_flags[WORKERLIST_NFLAGS];
|
|
#define JOBLIST_NFLAGS 1
|
|
static const flagspec_t joblist_flags[JOBLIST_NFLAGS];
|
|
#define JOBINFO_NFLAGS 1
|
|
static const flagspec_t jobinfo_flags[JOBINFO_NFLAGS];
|
|
#define JOBRUN_NFLAGS 2
|
|
static const flagspec_t jobrun_flags[JOBRUN_NFLAGS];
|
|
#define JOBSUB_NFLAGS 7
|
|
static const flagspec_t jobsub_flags[JOBSUB_NFLAGS];
|
|
#define RESVAL_NFLAGS 2
|
|
static const flagspec_t resval_flags[RESVAL_NFLAGS];
|
|
#define JOBKILL_NFLAGS 3
|
|
static const flagspec_t jobkill_flags[JOBKILL_NFLAGS];
|
|
#define JOBCKPT_NFLAGS 3
|
|
static const flagspec_t jobckpt_flags[JOBCKPT_NFLAGS];
|
|
#define QCTRL_NFLAGS 6
|
|
static const flagspec_t qctrl_flags[QCTRL_NFLAGS];
|
|
#define HOSTCTRL_NFLAGS 4
|
|
static const flagspec_t hostctrl_flags[HOSTCTRL_NFLAGS];
|
|
#define UPGRADECONFIG_NFLAGS 1
|
|
static const flagspec_t upgradeconfig_flags[UPGRADECONFIG_NFLAGS];
|
|
#define RUNTASKS_NFLAGS 7
|
|
static const flagspec_t runtasks_flags[RUNTASKS_NFLAGS];
|
|
#define SETPEND_NFLAGS 2
|
|
static const flagspec_t setpend_flags[SETPEND_NFLAGS];
|
|
#define JOBSTART_NFLAGS 2
|
|
static const flagspec_t jobstart_flags[JOBSTART_NFLAGS];
|
|
#define JOBFINISH_NFLAGS 2
|
|
static const flagspec_t jobfinish_flags[JOBFINISH_NFLAGS];
|
|
#define JOBFAIL_NFLAGS 3
|
|
static const flagspec_t jobfail_flags[JOBFAIL_NFLAGS];
|
|
#define RESUPDATE_NFLAGS 3
|
|
static const flagspec_t resupdate_flags[RESUPDATE_NFLAGS];
|
|
|
|
static command_t top_commands[] = {
|
|
{ "help", cmd_help, "Display this text", USAGE_HELP },
|
|
{ "worker", cmd_worker, "In worker context", USAGE_WORKER },
|
|
{ "scheduler", cmd_scheduler, "In scheduler context", USAGE_SCHEDULER },
|
|
{ "client", cmd_client, "In client context", USAGE_CLIENT },
|
|
{ "set", cmd_set, "Show/Set settings", USAGE_SET },
|
|
{ "echo", cmd_echo, "Print a message", USAGE_ECHO },
|
|
{ "waldump", cmd_waldump, "Inspect WAL segments", USAGE_WALDUMP },
|
|
{ "walcompare", cmd_walcompare, "Diff member committed WAL prefixes",
|
|
USAGE_WALCOMPARE_HEAD, walcompare_flags, WALCOMPARE_NFLAGS,
|
|
USAGE_WALCOMPARE_TAIL },
|
|
{ "snapverify", cmd_snapverify, "Verify checkpoint file structure",
|
|
USAGE_SNAPVERIFY },
|
|
{ "snapdiff", cmd_snapdiff, "Byte-diff two checkpoint files",
|
|
USAGE_SNAPDIFF },
|
|
{ "configcompare", cmd_configcompare, "Compare active config hash across members",
|
|
USAGE_CONFIGCOMPARE },
|
|
{ "source", cmd_source, "Run commands from a file", USAGE_SOURCE },
|
|
{ "quit", cmd_quit, "Quit xce test", USAGE_QUIT },
|
|
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL, (const char*)NULL }
|
|
};
|
|
|
|
static command_ctx_t topctx = { .parent = NULL,
|
|
.prompt = "xce",
|
|
.settings = settings,
|
|
.commands = top_commands };
|
|
|
|
static command_t worker_commands[] = {
|
|
{ "..", cmd_up, "Goto upper context", USAGE_UP },
|
|
{ "help", cmd_help, "Display this text", USAGE_HELP },
|
|
{ "quit", cmd_quit, "Quit xce test", USAGE_QUIT },
|
|
{ "join", cmd_join_worker, "Join the cluster",
|
|
USAGE_JOIN_WORKER_HEAD, join_flags, JOIN_FLAGS_COUNT,
|
|
USAGE_JOIN_WORKER_TAIL },
|
|
{ "jstart", cmd_jobstart, "tell controller jobstart",
|
|
USAGE_JOBSTART_HEAD, jobstart_flags, JOBSTART_NFLAGS,
|
|
USAGE_JOBSTART_TAIL },
|
|
{ "jfinish", cmd_jobfinish, "tell controller jobfinish",
|
|
USAGE_JOBFINISH_HEAD, jobfinish_flags, JOBFINISH_NFLAGS,
|
|
USAGE_JOBFINISH_TAIL },
|
|
{ "jfail", cmd_jobfail, "tell controller jobfail",
|
|
USAGE_JOBFAIL_HEAD, jobfail_flags, JOBFAIL_NFLAGS, USAGE_JOBFAIL_TAIL },
|
|
{ "resupdate", cmd_resupdate, "update res vals",
|
|
USAGE_RESUPDATE_HEAD, resupdate_flags, RESUPDATE_NFLAGS,
|
|
USAGE_RESUPDATE_TAIL },
|
|
{ "echo", cmd_echo, "Print a message", USAGE_ECHO },
|
|
{ "source", cmd_source, "Run commands from a file", USAGE_SOURCE },
|
|
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL, (const char*)NULL }
|
|
};
|
|
|
|
static command_ctx_t workerctx = { .parent = &topctx,
|
|
.prompt = "worker",
|
|
.settings = settings,
|
|
.commands = worker_commands };
|
|
|
|
static command_t client_commands[] = {
|
|
{ "..", cmd_up, "Goto upper context", USAGE_UP },
|
|
{ "hello", cmd_hello, "Say hello to xce controller",
|
|
USAGE_HELLO_HEAD, hello_flags, HELLO_NFLAGS, USAGE_HELLO_TAIL },
|
|
{ "reslist", cmd_reslist, "Display cluster resources",
|
|
USAGE_RESLIST_HEAD, reslist_flags, RESLIST_NFLAGS, USAGE_RESLIST_TAIL },
|
|
{ "workerlist", cmd_workerlist, "list workers",
|
|
USAGE_WORKERLIST_HEAD, workerlist_flags, WORKERLIST_NFLAGS,
|
|
USAGE_WORKERLIST_TAIL },
|
|
{ "qctrl", cmd_qctrl, "queue control",
|
|
USAGE_QCTRL_HEAD, qctrl_flags, QCTRL_NFLAGS, USAGE_QCTRL_TAIL },
|
|
{ "hostctrl", cmd_hostctrl, "host control",
|
|
USAGE_HOSTCTRL_HEAD, hostctrl_flags, HOSTCTRL_NFLAGS,
|
|
USAGE_HOSTCTRL_TAIL },
|
|
{ "upgradeconfig", cmd_upgradeconfig,
|
|
"submit a config upgrade (cluster.bin file)",
|
|
USAGE_UPGRADECONFIG_HEAD, upgradeconfig_flags, UPGRADECONFIG_NFLAGS,
|
|
USAGE_UPGRADECONFIG_TAIL },
|
|
{ "joblist", cmd_joblist, "list jobs",
|
|
USAGE_JOBLIST_HEAD, joblist_flags, JOBLIST_NFLAGS, USAGE_JOBLIST_TAIL },
|
|
{ "expect-jobs", cmd_expect_jobs, "assert current job count", USAGE_EXPECT_JOBS },
|
|
{ "expect-job", cmd_expect_job, "assert a JobId is present", USAGE_EXPECT_JOB },
|
|
{ "wait-primary", cmd_wait_primary,
|
|
"wait until a primary serves [timeout_ms]", USAGE_WAIT_PRIMARY },
|
|
{ "ctrlstatus", cmd_ctrlstatus, "show controller/cluster status", USAGE_CTRLSTATUS },
|
|
{ "waldump", cmd_waldump, "Inspect WAL segments", USAGE_WALDUMP },
|
|
{ "walcompare", cmd_walcompare, "Diff member committed WAL prefixes",
|
|
USAGE_WALCOMPARE_HEAD, walcompare_flags, WALCOMPARE_NFLAGS,
|
|
USAGE_WALCOMPARE_TAIL },
|
|
{ "snapverify", cmd_snapverify, "Verify checkpoint file structure",
|
|
USAGE_SNAPVERIFY },
|
|
{ "snapdiff", cmd_snapdiff, "Byte-diff two checkpoint files",
|
|
USAGE_SNAPDIFF },
|
|
{ "configcompare", cmd_configcompare, "Compare active config hash across members",
|
|
USAGE_CONFIGCOMPARE },
|
|
{ "shutdown", cmd_shutdown, "shutdown xce controller",
|
|
USAGE_SHUTDOWN_HEAD, shutdown_flags, SHUTDOWN_NFLAGS,
|
|
USAGE_SHUTDOWN_TAIL },
|
|
{ "jobrun", cmd_jobrun, "run a job",
|
|
USAGE_JOBRUN_HEAD, jobrun_flags, JOBRUN_NFLAGS, USAGE_JOBRUN_TAIL },
|
|
{ "jobsub", cmd_jobsub, "submit a job",
|
|
USAGE_JOBSUB_HEAD, jobsub_flags, JOBSUB_NFLAGS, USAGE_JOBSUB_TAIL },
|
|
{ "jobkill", cmd_jobkill, "signal a job",
|
|
USAGE_JOBKILL_HEAD, jobkill_flags, JOBKILL_NFLAGS, USAGE_JOBKILL_TAIL },
|
|
{ "jobckpt", cmd_jobckpt, "checkpoint a job",
|
|
USAGE_JOBCKPT_HEAD, jobckpt_flags, JOBCKPT_NFLAGS, USAGE_JOBCKPT_TAIL },
|
|
{ "jobinfo", cmd_jobinfo, "job information",
|
|
USAGE_JOBINFO_HEAD, jobinfo_flags, JOBINFO_NFLAGS, USAGE_JOBINFO_TAIL },
|
|
{ "resval", cmd_resval, "get value of res",
|
|
USAGE_RESVAL_HEAD, resval_flags, RESVAL_NFLAGS, USAGE_RESVAL_TAIL },
|
|
{ "taskrun", cmd_runtasks, "run tasks",
|
|
USAGE_TASKRUN_HEAD, runtasks_flags, RUNTASKS_NFLAGS,
|
|
USAGE_TASKRUN_TAIL },
|
|
{ "help", cmd_help, "Display this text", USAGE_HELP },
|
|
{ "echo", cmd_echo, "Print a message", USAGE_ECHO },
|
|
{ "source", cmd_source, "Run commands from a file", USAGE_SOURCE },
|
|
{ "quit", cmd_quit, "Quit xce test", USAGE_QUIT },
|
|
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL, (const char*)NULL }
|
|
};
|
|
|
|
static command_ctx_t clientctx = { .parent = &topctx,
|
|
.prompt = "client",
|
|
.settings = settings,
|
|
.commands = client_commands };
|
|
|
|
static command_t scheduler_commands[] = {
|
|
{ "..", cmd_up, "Goto upper context", USAGE_UP },
|
|
{ "help", cmd_help, "Display this text", USAGE_HELP },
|
|
{ "quit", cmd_quit, "Quit xce test", USAGE_QUIT },
|
|
{ "join", cmd_join_sched, "Join the cluster", USAGE_JOIN_SCHED },
|
|
{ "getconf", cmd_getconf, "Get conf of cluster", USAGE_GETCONF },
|
|
{ "setpend", cmd_setpend, "set job pend reason",
|
|
USAGE_SETPEND_HEAD, setpend_flags, SETPEND_NFLAGS, USAGE_SETPEND_TAIL },
|
|
{ "echo", cmd_echo, "Print a message", USAGE_ECHO },
|
|
{ "source", cmd_source, "Run commands from a file", USAGE_SOURCE },
|
|
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL, (const char*)NULL }
|
|
};
|
|
|
|
static command_ctx_t schedulerctx = { .parent = &topctx,
|
|
.prompt = "scheduler",
|
|
.settings = settings,
|
|
.commands = scheduler_commands };
|
|
|
|
static command_ctx_t* active_ctx = &topctx;
|
|
|
|
static char prompt[256];
|
|
static int break_out = 0;
|
|
|
|
/* xce-test-cli-ux task 1.6, Requirement 4.3: describes the in-session
|
|
* `help' command as the way to list/learn available commands, in addition
|
|
* to argp's own auto-generated --help/-?/-V flag documentation below. */
|
|
static char doc[] =
|
|
"xce_test -- interactive/scripted test client for the controller, "
|
|
"worker, and scheduler\v"
|
|
"Once a session is running, type `help' at the prompt to list the "
|
|
"commands available in the current context, or `help <command>' for "
|
|
"details on a specific one.";
|
|
static char args_doc[] = "";
|
|
|
|
/* build-time-versioning task 3.2: no longer a compile-time initializer --
|
|
* argp reads this by pointer when --version/-V is processed, and
|
|
* xce_compose_version() (utils.h) is a runtime function, not a constant
|
|
* expression, so it cannot initialize a file-scope global. main() assigns
|
|
* this to a static buffer, computed via xce_compose_version(), before
|
|
* argp_parse() runs. See utils.h's NOTE FOR TASK 3.2 above MK_VERSION. */
|
|
const char* argp_program_version = NULL;
|
|
const char* argp_program_bug_address = BUG_EMAIL;
|
|
|
|
/* Long-option key (above ASCII to avoid collisions with short opts) --
|
|
* mirrors src/controller/main.c's OPT_PEERS/OPT_CONF_KEEP_N idiom. */
|
|
#define OPT_NORC 256
|
|
|
|
struct arguments {
|
|
int verbose;
|
|
int norc;
|
|
};
|
|
|
|
static struct argp_option options[] = {
|
|
{ "verbose", 'v', 0, 0, "Produce verbose logs" },
|
|
{ "norc", OPT_NORC, 0, 0, "Skip loading the rc file for this invocation" },
|
|
/* xce-test-cli-ux task 1.6, Requirement 4.1: argp's OWN built-in help
|
|
* alias is `-?` (via --help), NOT `-h` -- glibc argp does not add `-h`
|
|
* automatically. `-h` is hidden from the rendered help/usage text
|
|
* (OPTION_HIDDEN) so it doesn't duplicate the auto-generated
|
|
* "-?, --help" entry there; it is still a fully functional alias,
|
|
* handled explicitly in parse_opt() below via argp_state_help(). */
|
|
{ 0, 'h', 0, OPTION_HIDDEN, 0 },
|
|
{ 0 }
|
|
};
|
|
|
|
static error_t parse_opt(int key, char* arg, struct argp_state* state)
|
|
{
|
|
struct arguments* arguments = state->input;
|
|
|
|
switch (key) {
|
|
case 'v':
|
|
arguments->verbose++;
|
|
break;
|
|
case OPT_NORC:
|
|
arguments->norc = 1;
|
|
break;
|
|
case 'h':
|
|
/* ARGP_HELP_STD_HELP includes ARGP_HELP_EXIT_OK, so this prints
|
|
* the same help text as --help/-? and exits(0) -- it does not
|
|
* return. */
|
|
argp_state_help(state, stdout, ARGP_HELP_STD_HELP);
|
|
break;
|
|
default:
|
|
return ARGP_ERR_UNKNOWN;
|
|
}
|
|
|
|
return 0;
|
|
}
|
|
|
|
static struct argp argp = { options, parse_opt, args_doc, doc };
|
|
|
|
void sigbreak(int signo) { break_out = 1; }
|
|
|
|
/* xce-test-cli-ux task 1.3, Requirement 9.9: read-only accessor over
|
|
* `break_out` for code outside this file (interruptcall.c, task 1.4) to
|
|
* poll -- see xcetest_interrupt.h's file-header comment for the full
|
|
* contract (pure read, no reset side effect, break_out's own reset stays
|
|
* test.c-private). */
|
|
int xcetest_interrupted(void) { return break_out; }
|
|
|
|
/* xce-test-cli-ux task 2.1, Requirement 9.4/9.6/13.7: read-only accessor
|
|
* over `g_interactive` (this file's top-level-session TTY flag, set once
|
|
* in main() from isatty(STDIN_FILENO) and never changed after that) for
|
|
* code outside this file (confirm.c) to consult -- mirrors
|
|
* xcetest_interrupted()'s accessor pattern immediately above. Pure read;
|
|
* declared in confirm.h rather than xcetest_interrupt.h since confirm.c
|
|
* is its only consumer and g_interactive is unrelated to that header's
|
|
* SIGINT/EINTR concern. Deliberately the TOP-LEVEL session's stdin, not
|
|
* "is this line from a sourced file" -- that distinction is Requirement
|
|
* 13.7, a later task; this accessor's contract must already be correct
|
|
* for it, per design.md's Confirmation Gate invariant. */
|
|
int xcetest_is_interactive(void) { return g_interactive; }
|
|
|
|
static char* get_setting(char* key)
|
|
{
|
|
int i;
|
|
|
|
for (i = 0; active_ctx->settings[i].key; i++) {
|
|
if (strcmp(key, active_ctx->settings[i].key) != 0) {
|
|
continue;
|
|
}
|
|
|
|
if (active_ctx->settings[i].value != NULL) {
|
|
return active_ctx->settings[i].value;
|
|
}
|
|
|
|
return active_ctx->settings[i].defval;
|
|
}
|
|
|
|
fprintf(stderr, "Key `%s' is not in settings\n", key);
|
|
return NULL;
|
|
}
|
|
|
|
static xce_client_config_t config;
|
|
|
|
/* Holds the parsed HA controller list. Static so the hostname[] buffers it
|
|
* owns outlive enter_ctx(): xcecli_ctx_init() stores the config pointer (it
|
|
* does not copy masters[]), so config.masters[i] must reference storage that
|
|
* persists for the lifetime of the client context. */
|
|
static xce_cluster_config_t master_cfg;
|
|
|
|
int enter_ctx(command_ctx_t* ctx)
|
|
{
|
|
char* masters = get_setting("masters");
|
|
int i;
|
|
|
|
/* Parse the controller HA peer list with the shared libxce parser — the
|
|
* same "name=host:port,..."/"host:port,..." syntax used by the controller
|
|
* --peers flag — so the test client fails over across candidates exactly
|
|
* like the worker/scheduler/apiserver do. */
|
|
memset(&master_cfg, 0, sizeof(master_cfg));
|
|
if ((masters == NULL) || (xce_parse_peers(masters, &master_cfg) != 0) ||
|
|
(master_cfg.candidates_count < 1)) {
|
|
fprintf(stderr, "enter_ctx: invalid 'masters' setting `%s'\n",
|
|
masters ? masters : "(null)");
|
|
return -EINVAL;
|
|
}
|
|
|
|
config.nmasters = master_cfg.candidates_count;
|
|
if (config.nmasters > N_MAX_MASTERS) {
|
|
fprintf(stderr,
|
|
"enter_ctx: %d masters configured, using first %d "
|
|
"(N_MAX_MASTERS)\n",
|
|
config.nmasters, N_MAX_MASTERS);
|
|
config.nmasters = N_MAX_MASTERS;
|
|
}
|
|
|
|
for (i = 0; i < config.nmasters; i++) {
|
|
config.masters[i] = master_cfg.candidates[i].hostname;
|
|
config.master_ports[i] = master_cfg.candidates[i].port;
|
|
}
|
|
|
|
config.from = get_setting("myname");
|
|
config.key = get_setting("secret");
|
|
|
|
xcecli_ctx_init(&(ctx->ctx), &config);
|
|
xcecli_set_handler(ctx->ctx, &myreq_handler, NULL);
|
|
xcecli_start(ctx->ctx);
|
|
|
|
return 0;
|
|
}
|
|
|
|
int leave_ctx(command_ctx_t* ctx)
|
|
{
|
|
if (ctx == &topctx) {
|
|
return 0;
|
|
}
|
|
|
|
if (ctx == &workerctx) {
|
|
delaymgr_fini(&delaymgr);
|
|
}
|
|
|
|
xcecli_wakeup(active_ctx->ctx, (int)-EINTR);
|
|
|
|
/* xce-test-cli-ux task 3.2 remediation round 1 (bug-1795): the
|
|
* xcecli_wakeup() call just above is a best-effort nudge only -- it
|
|
* does not itself guarantee that a worker thread abandoned by a prior
|
|
* Ctrl-C on THIS connection (interruptcall.c's per-connection
|
|
* registry, tasks 1.4/1.5) has actually finished touching `ctx->ctx`
|
|
* before xcecli_stop()/xcecli_ctx_fini() below free it out from under
|
|
* that worker -- a genuine use-after-free if the operator's very next
|
|
* action after abandoning a hung command is `up`/`quit` (an entirely
|
|
* ordinary thing to do). xcetest_wait_connection_idle() blocks (a
|
|
* narrow, deliberately UNINTERRUPTIBLE wait -- see interruptcall.h)
|
|
* until that same registry proves no worker is active on this
|
|
* connection, then returns immediately, before teardown proceeds. */
|
|
xcetest_wait_connection_idle(ctx->ctx);
|
|
|
|
xcecli_stop(ctx->ctx);
|
|
xcecli_ctx_fini(ctx->ctx);
|
|
|
|
return 0;
|
|
}
|
|
|
|
static int fill_prompt(command_ctx_t* ctx, char* buf)
|
|
{
|
|
if (ctx->parent != NULL) {
|
|
fill_prompt(ctx->parent, buf);
|
|
strcat(buf, "/");
|
|
}
|
|
|
|
strcat(buf, ctx->prompt);
|
|
|
|
return strlen(buf);
|
|
}
|
|
|
|
command_t* find_command(char* name)
|
|
{
|
|
int i;
|
|
|
|
for (i = 0; active_ctx->commands[i].name; i++) {
|
|
if (strcmp(active_ctx->commands[i].name, name) == 0) {
|
|
return &(active_ctx->commands[i]);
|
|
}
|
|
}
|
|
|
|
return NULL;
|
|
}
|
|
|
|
/* xce-test-cli-ux task 2.3, Requirement 7: command-name tab completion.
|
|
*
|
|
* libedit API verification (design.md's flagged P0 risk, research.md's
|
|
* "libedit completion API may not perfectly mirror GNU readline" risk):
|
|
* confirmed by reading the ACTUAL linked headers in this build environment
|
|
* (this project's `src/flake.nix`-pinned libedit, pkg-config name
|
|
* "libedit", providing `editline/readline.h` -- the same header already
|
|
* included at the top of this file) plus a standalone pty-driven probe
|
|
* program built and run against this exact library. Findings:
|
|
*
|
|
* - The GNU-readline-style completion API IS present verbatim:
|
|
* `rl_completion_func_t* rl_attempted_completion_function` (signature
|
|
* `char** (*)(const char*, int, int)`, exactly matching this task's
|
|
* `xcetest_complete` shape from design.md) and
|
|
* `char** rl_completion_matches(const char*, rl_compentry_func_t*)`
|
|
* (matches on a `char* (*)(const char*, int)` generator, exactly
|
|
* `xcetest_complete_generator`'s shape). No lower-level `<histedit.h>`
|
|
* fallback was needed.
|
|
* - `rl_attempted_completion_over` exists and suppresses libedit's
|
|
* default filename-completion fallback -- used below so an empty
|
|
* match list in the active context yields NO completion rather than
|
|
* local filenames.
|
|
* - Single/multiple-match UI behavior is entirely the library's: a sole
|
|
* match is completed in full (plus a trailing space) with ZERO extra
|
|
* code (7.2); multiple matches with no further common prefix ring the
|
|
* bell on the first Tab and print the candidate list (input line left
|
|
* untouched -- confirmed via a raw pty capture) on a SECOND,
|
|
* consecutive Tap of Tab -- standard GNU-readline/bash double-Tab
|
|
* convention (7.3), not something this code implements. This
|
|
* implementation's own responsibility is narrowly the context-scoped
|
|
* candidate lookup below.
|
|
* - `rl_variable_bind("show-all-if-ambiguous", "on")` was probed and
|
|
* returns failure (1) against this libedit build -- that variable is
|
|
* not implemented here, so it is not used; the stock double-Tab
|
|
* behavior above is what ships.
|
|
*/
|
|
|
|
/* xcetest_complete_generator - the readline "compentry" generator: called
|
|
* repeatedly with state==0 on the first call (resets the scan) and
|
|
* state!=0 on each subsequent call, returning one more `strdup()`ed match
|
|
* per call (readline/libedit frees each returned string) and NULL once
|
|
* exhausted. Iterates active_ctx->commands[] -- the CURRENTLY active
|
|
* context at the moment Tab is pressed, read fresh on every call rather
|
|
* than captured at registration time -- so completion always tracks
|
|
* `worker`/`scheduler`/`client`/`..` context switches for the life of the
|
|
* session (Requirement 7.1/7.4: only the active context's commands are
|
|
* ever offered). Plain prefix match against `text`, same convention as
|
|
* find_command()'s exact-match lookup just above. */
|
|
static char* xcetest_complete_generator(const char* text, int state)
|
|
{
|
|
static int idx;
|
|
static size_t len;
|
|
|
|
if (state == 0) {
|
|
idx = 0;
|
|
len = strlen(text);
|
|
}
|
|
|
|
while (active_ctx->commands[idx].name != NULL) {
|
|
const char* name = active_ctx->commands[idx].name;
|
|
|
|
idx++;
|
|
if (strncmp(name, text, len) == 0) {
|
|
return strdup(name);
|
|
}
|
|
}
|
|
|
|
return NULL;
|
|
}
|
|
|
|
/* xcetest_complete - the readline "attempted completion" entry point,
|
|
* registered as rl_attempted_completion_function only when g_interactive
|
|
* (main(), below). `start`/`end` are byte offsets of the word being
|
|
* completed within rl_line_buffer; `start == 0` means the FIRST word on
|
|
* the line, i.e. the command name itself. Per this task's explicit
|
|
* boundary (command-NAME completion only -- no flag-value completion of
|
|
* jobids/hostnames/resource names), every other word position (start != 0)
|
|
* returns NULL and leaves rl_attempted_completion_over at its default so
|
|
* libedit's normal fallback applies there, unchanged from today's
|
|
* zero-customization behavior. For the command-name position, this always
|
|
* sets rl_attempted_completion_over so a context with zero matching
|
|
* commands offers NOTHING (never falls back to filename completion). */
|
|
static char** xcetest_complete(const char* text, int start, int end)
|
|
{
|
|
(void)end;
|
|
|
|
if (start != 0) {
|
|
return NULL;
|
|
}
|
|
|
|
rl_attempted_completion_over = 1;
|
|
|
|
return rl_completion_matches(text, xcetest_complete_generator);
|
|
}
|
|
|
|
/* Execute a command line. */
|
|
int execute_line(char* line)
|
|
{
|
|
int rc;
|
|
command_t* command;
|
|
int argc;
|
|
char** argv;
|
|
|
|
argc = parse_argv(line, &argv);
|
|
if (argc < 0) {
|
|
SET_ERROR(-argc);
|
|
perror("execute_line");
|
|
return -1;
|
|
}
|
|
|
|
if ((argv == NULL) || (argv[0] == NULL)) {
|
|
return 0;
|
|
}
|
|
|
|
command = find_command(argv[0]);
|
|
|
|
if (!command) {
|
|
fprintf(stderr, "%s: No such command for xce test.\n", argv[0]);
|
|
free_argv(argv, argc);
|
|
return (-1);
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.2, Requirement 1.1/1.5: this is the SINGLE call
|
|
* site for xcetest_set_active_connection() -- interruptcall.h's own
|
|
* "SEAM FOR A FUTURE CALLER (task 3.2)" comment names this exact
|
|
* dispatch chokepoint ("wherever main()/the dispatcher sets up each
|
|
* command's active connection context"). active_ctx->ctx (xcecli_ctx_t
|
|
* is `void*`, see client.h) is read fresh here, on every command
|
|
* dispatch, so the per-connection lock registry always keys off
|
|
* whichever connection is actually active for the command about to run
|
|
* -- correct across `worker`/`scheduler`/`client`/`..` context switches
|
|
* without needing a separate call in cmd_worker/cmd_scheduler/cmd_client/
|
|
* cmd_up. In topctx (no live connection; enter_ctx() is never called for
|
|
* it), active_ctx->ctx is the zero-initialized NULL it starts as -- a
|
|
* harmless, consistent key, since no topctx command makes a blocking RPC
|
|
* call. */
|
|
xcetest_set_active_connection(active_ctx->ctx);
|
|
|
|
/* xce-test-cli-ux task 3.2, Requirement 1.1/1.3/1.4: reset break_out
|
|
* immediately before dispatching THIS command, so its own blocking-RPC
|
|
* call sites (wrapped via xcetest_run_interruptible() below) observe a
|
|
* freshly-clear interrupt flag rather than a stale one left set by an
|
|
* EARLIER command that already reported itself as interrupted/failed
|
|
* (xcetest_interrupted() is a pure, non-consuming read -- see
|
|
* xcetest_interrupt.h -- so nothing else ever clears it on this
|
|
* command's behalf). Without this, a command run immediately after an
|
|
* interrupted one would itself be instantly, incorrectly reported as
|
|
* interrupted before ever running -- breaking exactly the "same
|
|
* connection accepts a subsequent command normally" guarantee this
|
|
* task exists to provide. */
|
|
break_out = 0;
|
|
|
|
rc = (command->func)(argc, argv);
|
|
|
|
free_argv(argv, argc);
|
|
return rc;
|
|
}
|
|
|
|
/* Substitute the fixed auto-variables `$?`, `$jobid`, `$jobcount` on a raw line
|
|
* before tokenizing (xce-test-e2e-primitives task 3.1, Req 3.1/3.3/3.5/3.6).
|
|
*
|
|
* Only WHOLE-TOKEN occurrences are recognized: a `$name` must be bounded by the
|
|
* start of the line / whitespace on its left and whitespace / end-of-line on its
|
|
* right. A `$name` glued to other non-whitespace (e.g. `x$jobid`, `$jobidx`) is
|
|
* NOT a recognized auto-variable and is copied through verbatim. No other `$...`
|
|
* syntax is recognized: an unknown whole token such as `$other` is also copied
|
|
* verbatim (it is simply not one of the three names).
|
|
*
|
|
* Replacements:
|
|
* $? -> decimal of g_last_status (always available).
|
|
* $jobid -> g_last_jobid if set (non-empty); unset => command failure.
|
|
* $jobcount -> decimal of g_last_jobcount if set (>=0); unset => failure.
|
|
*
|
|
* Writes the expanded line into `out` (capacity `outsz`, always NUL-terminated
|
|
* on success). Returns 0 on success, -EINVAL on an unset auto-var reference,
|
|
* -ENOSPC if the expansion would overflow `out`. On any error `out` content is
|
|
* unspecified and the caller must fail the command rather than run it. */
|
|
static int substitute_autovars(const char* line, char* out, size_t outsz)
|
|
{
|
|
const char* p = line;
|
|
size_t o = 0;
|
|
|
|
if ((line == NULL) || (out == NULL) || (outsz == 0)) {
|
|
return -EINVAL;
|
|
}
|
|
|
|
while (*p != '\0') {
|
|
/* A `$` starts a candidate auto-var only when it begins a token, i.e.
|
|
* the previous char is the line start or whitespace. */
|
|
int at_token_start
|
|
= (p == line) || (*(p - 1) == ' ') || (*(p - 1) == '\t');
|
|
|
|
if ((*p == '$') && at_token_start) {
|
|
const char* name = p + 1;
|
|
const char* end = name;
|
|
const char* repl = NULL;
|
|
char numbuf[32];
|
|
|
|
/* Scan to the token end (whitespace or NUL). */
|
|
while ((*end != '\0') && (*end != ' ') && (*end != '\t')) {
|
|
end++;
|
|
}
|
|
|
|
size_t namelen = (size_t)(end - name);
|
|
|
|
if ((namelen == 1) && (name[0] == '?')) {
|
|
snprintf(numbuf, sizeof(numbuf), "%d", g_last_status);
|
|
repl = numbuf;
|
|
} else if ((namelen == 5) && (strncmp(name, "jobid", 5) == 0)) {
|
|
if (g_last_jobid[0] == '\0') {
|
|
fprintf(stderr,
|
|
"autovar: $jobid referenced before set\n");
|
|
return -EINVAL;
|
|
}
|
|
repl = g_last_jobid;
|
|
} else if ((namelen == 8) && (strncmp(name, "jobcount", 8) == 0)) {
|
|
if (g_last_jobcount < 0) {
|
|
fprintf(stderr,
|
|
"autovar: $jobcount referenced before set\n");
|
|
return -EINVAL;
|
|
}
|
|
snprintf(numbuf, sizeof(numbuf), "%d", g_last_jobcount);
|
|
repl = numbuf;
|
|
}
|
|
|
|
if (repl != NULL) {
|
|
size_t rlen = strlen(repl);
|
|
|
|
if ((o + rlen) >= outsz) {
|
|
return -ENOSPC;
|
|
}
|
|
memcpy(out + o, repl, rlen);
|
|
o += rlen;
|
|
p = end; /* consume the whole `$name` token */
|
|
continue;
|
|
}
|
|
|
|
/* Not a recognized auto-var: copy the whole token verbatim. */
|
|
while (p < end) {
|
|
if ((o + 1) >= outsz) {
|
|
return -ENOSPC;
|
|
}
|
|
out[o++] = *p++;
|
|
}
|
|
continue;
|
|
}
|
|
|
|
if ((o + 1) >= outsz) {
|
|
return -ENOSPC;
|
|
}
|
|
out[o++] = *p++;
|
|
}
|
|
|
|
out[o] = '\0';
|
|
return 0;
|
|
}
|
|
|
|
/* Main-loop wrapper around execute_line that implements the session verdict
|
|
* (xce-test-e2e-primitives task 1.1):
|
|
* - blank / whitespace-only lines and `#`-comment lines are ignored;
|
|
* - normal lines run via execute_line; g_last_status / g_session_failed are
|
|
* updated from the result;
|
|
* - in default (abort) mode a failure stops the loop; under
|
|
* g_continue_on_error the loop keeps running but the failure is recorded.
|
|
*
|
|
* Returns 0 to keep the read loop running, non-zero to request that the loop
|
|
* stop (abort-on-failure). The leading `!` shell-escape and `$`-substitution
|
|
* are intentionally handled in later tasks (2.1 / 3.1); this function is the
|
|
* single chokepoint where they will be added.
|
|
*/
|
|
int preprocess_and_run(char* line)
|
|
{
|
|
char* p = line;
|
|
int rc;
|
|
|
|
/* Skip leading whitespace to classify the line. */
|
|
while ((*p == ' ') || (*p == '\t')) {
|
|
p++;
|
|
}
|
|
|
|
/* Ignore blank / whitespace-only lines and `#` comments. */
|
|
if ((*p == '\0') || (*p == '#')) {
|
|
return 0;
|
|
}
|
|
|
|
/* Shell-escape: a line whose first non-whitespace char is `!` runs the
|
|
* remainder via system(). The shell command's exit status becomes
|
|
* g_last_status (the `$?` auto-variable); a non-zero status is a command
|
|
* failure subject to the same abort-vs-continue rules as execute_line. The
|
|
* remainder is passed raw to the shell (no parse_argv / auto-var
|
|
* substitution — that is task 3.1). (xce-test-e2e-primitives task 2.1) */
|
|
if (*p == '!') {
|
|
const char* rest = p + 1;
|
|
int sysrc = system(rest);
|
|
int status;
|
|
|
|
if (sysrc == -1) {
|
|
/* system() itself failed to run the shell — treat as failure. */
|
|
status = 127;
|
|
} else if (WIFEXITED(sysrc)) {
|
|
status = WEXITSTATUS(sysrc);
|
|
} else {
|
|
/* Signaled / abnormal termination — non-zero failure status. */
|
|
status = 1;
|
|
}
|
|
|
|
g_last_status = status;
|
|
|
|
if (status != 0) {
|
|
g_session_failed = 1;
|
|
if (!g_continue_on_error && !g_interactive) {
|
|
return 1; /* abort: stop the read loop (scripted mode only) */
|
|
}
|
|
} else {
|
|
add_history(line);
|
|
}
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* Substitute the fixed auto-variables (`$?`, `$jobid`, `$jobcount`) before
|
|
* tokenizing (xce-test-e2e-primitives task 3.1). An unset `$jobid` /
|
|
* `$jobcount` reference (Req 3.6) — or an overflow — fails the command:
|
|
* we do NOT run execute_line, and the result is treated exactly like a
|
|
* failing command (g_last_status non-zero, g_session_failed, abort vs
|
|
* continue). The leading `!` shell-escape above is intentionally NOT
|
|
* substituted: that line goes raw to system() and the shell owns `$?`. */
|
|
{
|
|
char expanded[MAX_LINE_LEN];
|
|
int subrc = substitute_autovars(p, expanded, sizeof(expanded));
|
|
|
|
if (subrc < 0) {
|
|
g_last_status = subrc;
|
|
g_session_failed = 1;
|
|
if (!g_continue_on_error && !g_interactive) {
|
|
return 1; /* abort: stop the read loop (scripted mode only) */
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
rc = execute_line(expanded);
|
|
}
|
|
|
|
g_last_status = (rc < 0) ? (rc) : 0;
|
|
|
|
if (rc < 0) {
|
|
SET_ERROR(-rc);
|
|
g_session_failed = 1;
|
|
if (!g_continue_on_error) {
|
|
return 1; /* errexit (-e): stop the read loop on first failure */
|
|
}
|
|
} else {
|
|
add_history(line);
|
|
}
|
|
|
|
return 0;
|
|
}
|
|
|
|
int cmd_help(int argc, char* argv[])
|
|
{
|
|
int i;
|
|
int rc = 0;
|
|
/* xce-test-cli-ux task 3.6, Requirement 3.1/3.2: the command-list loop
|
|
* below is shared by two callers -- bare `help` (success, Req 3.2 result
|
|
* data) and the not-found fallback just above it (Req 3.1 diagnostic,
|
|
* "here's what IS available"). Route BOTH the not-found headline and
|
|
* that shared list to `out`, selected by which case we're in, rather
|
|
* than duplicating the loop. */
|
|
FILE* out = stdout;
|
|
|
|
if (argc > 1) {
|
|
char* cmd = argv[1];
|
|
|
|
/* Requirement 8.7: scan ONLY active_ctx->commands -- a name valid in
|
|
* a different context must never be found here. Do not add a
|
|
* fallback lookup into another context's table. */
|
|
for (i = 0; active_ctx->commands[i].name != NULL; i++) {
|
|
if (strcmp(argv[1], active_ctx->commands[i].name) == 0) {
|
|
/* `help <cmd>` prints the detailed per-command usage (flags,
|
|
* positionals, example) -- distinct from the compact one-line
|
|
* `doc` the bare `help` listing below still uses unchanged.
|
|
* Every table entry is expected to carry a non-NULL `usage`
|
|
* now; the `doc` fallback stays only as a defensive guard
|
|
* against a future entry that forgets to set it.
|
|
*
|
|
* xce-test-cli-ux task 3.5 (Help Renderer, design.md's
|
|
* "description -> flags -> example -> related" assembly
|
|
* order): when this entry carries a flagspec table, `usage`
|
|
* holds only the HEAD (syntax + description) -- print the
|
|
* generated flag block right after it, then `usage_tail`
|
|
* (example + related-command/context notes). A command with
|
|
* no flagspec table is unchanged from before this task:
|
|
* `usage` alone already carries everything. */
|
|
if (active_ctx->commands[i].usage != NULL) {
|
|
fputs(active_ctx->commands[i].usage, stdout);
|
|
|
|
if (active_ctx->commands[i].flagspec != NULL) {
|
|
char* flags_block = flagspec_render_help(
|
|
active_ctx->commands[i].flagspec,
|
|
active_ctx->commands[i].flagspec_count);
|
|
|
|
if (flags_block != NULL) {
|
|
fputs(flags_block, stdout);
|
|
free(flags_block);
|
|
}
|
|
|
|
if (active_ctx->commands[i].usage_tail != NULL) {
|
|
fputs(active_ctx->commands[i].usage_tail, stdout);
|
|
}
|
|
}
|
|
} else {
|
|
printf("%s\t\t%s.\n", active_ctx->commands[i].name,
|
|
active_ctx->commands[i].doc);
|
|
}
|
|
return 0;
|
|
}
|
|
}
|
|
|
|
/* bug-1711 / Requirement 11.2: grammar fix only ("match", not
|
|
* "matchs") -- does NOT add the cross-context lookup Requirement
|
|
* 8.7 explicitly excludes. */
|
|
out = stderr; /* Req 3.1: not-found is a diagnostic, not result data */
|
|
fprintf(out, "No commands match `%s'. Possibilities are:\n", cmd);
|
|
rc = -1;
|
|
}
|
|
|
|
for (i = 0; active_ctx->commands[i].name != NULL; i++) {
|
|
fprintf(out, "\t%-12s\t%s\n", active_ctx->commands[i].name,
|
|
active_ctx->commands[i].doc);
|
|
}
|
|
|
|
return rc;
|
|
}
|
|
|
|
/* echo <message...> — print the remainder of the line (argv[1..] joined by
|
|
* single spaces) followed by a newline and succeed (xce-test-e2e-primitives
|
|
* task 1.2, Req 6.3). `echo` with no argument prints an empty line and still
|
|
* succeeds. Returns 0 always so it never marks the session failed. */
|
|
int cmd_echo(int argc, char* argv[])
|
|
{
|
|
int i;
|
|
|
|
for (i = 1; i < argc; i++) {
|
|
if (i > 1) {
|
|
putchar(' ');
|
|
}
|
|
fputs(argv[i], stdout);
|
|
}
|
|
putchar('\n');
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* waldump <path> [<path> ...] — offline WAL inspector for failover/durability
|
|
* triage. Each path may be a seg-*.wal file, a wal/ dir, or a node/data dir.
|
|
* Read-only: lists every entry, decodes JOB_SUBMIT jobids, and flags LSN gaps
|
|
* (a missing committed prefix is THE failover job-loss signal) plus the CRC
|
|
* recovered tip. Returns nonzero if any path failed or any gap was found, so
|
|
* it doubles as an assertion in scripted E2E runs. */
|
|
int cmd_waldump(int argc, char* argv[])
|
|
{
|
|
if (argc < 2) {
|
|
fprintf(stderr,
|
|
"usage: waldump <wal_dir|data_dir|seg_file> [more...]\n");
|
|
return -1;
|
|
}
|
|
|
|
int rc = 0;
|
|
for (int i = 1; i < argc; i++) {
|
|
waldump_stats_t st;
|
|
int r = waldump_path(argv[i], stdout, &st);
|
|
if (r != 0) {
|
|
fprintf(stderr, "waldump: %s: %s\n", argv[i], strerror(-r));
|
|
rc = -1;
|
|
continue;
|
|
}
|
|
if (st.n_gaps > 0 || st.crc_broken)
|
|
rc = -1; /* surface the defect as a nonzero command verdict */
|
|
}
|
|
return rc;
|
|
}
|
|
|
|
/* walcompare [--report] [--committed N] <memberA> <memberB> [more...] — diff
|
|
* member WAL committed prefixes by (lsn -> term, pcrc) and FAIL (nonzero) on any
|
|
* committed-prefix divergence (ha-term-safety R6.1/R6.2/R6.3), reusing the
|
|
* waldump decoder. Each member path may be a seg-*.wal file, a wal/ dir, or a
|
|
* node/data dir (same resolution as waldump). With no --committed, the shared
|
|
* committed boundary = the minimum CRC-intact tip across members. The diverging
|
|
* LSN and per-member (term, pcrc) identities are reported. With --report, also
|
|
* emit a machine-checkable per-member `(lsn, term, pcrc, status)` table and a
|
|
* committed-vs-uncommitted classification for each divergence (task 1.1). Exit
|
|
* 0 = identical, nonzero = divergence or load error — so it doubles as a
|
|
* scripted assertion. */
|
|
/* Both flags are long-only (no short letter ever existed for either --
|
|
* cmd_walcompare was the one command using a fully hand-rolled long-flag
|
|
* parser, per xce-test-cli-ux task 3.1's explicit scope). Their synthetic
|
|
* getopt_long() `val`s are FLAGSPEC_LONG_ONLY_BASE + <index in this table>
|
|
* (flagspec.h) -- the enum below names those indices so the switch below
|
|
* doesn't rely on bare magic numbers. */
|
|
enum { WALCOMPARE_OPT_REPORT = 0, WALCOMPARE_OPT_COMMITTED = 1 };
|
|
|
|
static const flagspec_t walcompare_flags[] = {
|
|
{ 0, "report", "report", no_argument, 0, NULL },
|
|
{ 0, "committed", "committed", required_argument, 0,
|
|
"derive the shared boundary from members' CRC-intact tips" },
|
|
};
|
|
|
|
int cmd_walcompare(int argc, char* argv[])
|
|
{
|
|
uint64_t committed = 0; /* 0 = derive the shared boundary */
|
|
int report = 0; /* --report => full per-member machine table */
|
|
int opt;
|
|
struct option long_opts[WALCOMPARE_NFLAGS + 1];
|
|
char optstring[8];
|
|
char posix_optstring[sizeof(optstring) + 1];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(walcompare_flags, WALCOMPARE_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: walcompare: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
/* Leading '+' (GNU getopt extension): stop at the first non-option
|
|
* argument instead of permuting argv -- matches the pre-migration
|
|
* hand-rolled parser's exact behavior (flags recognized only before the
|
|
* positional member paths; both flags here are long-only, so the built
|
|
* optstring is empty and this is simply "+"). */
|
|
posix_optstring[0] = '+';
|
|
strcpy(posix_optstring + 1, optstring);
|
|
|
|
while ((opt = getopt_long(argc, argv, posix_optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case FLAGSPEC_LONG_ONLY_BASE + WALCOMPARE_OPT_REPORT:
|
|
report = 1;
|
|
break;
|
|
case FLAGSPEC_LONG_ONLY_BASE + WALCOMPARE_OPT_COMMITTED: {
|
|
char* end = NULL;
|
|
unsigned long v = strtoul(optarg, &end, 10);
|
|
if (end == optarg || *end != '\0') {
|
|
fprintf(stderr, "walcompare: bad --committed LSN '%s'\n",
|
|
optarg);
|
|
return -1;
|
|
}
|
|
committed = (uint64_t)v;
|
|
break;
|
|
}
|
|
case '?':
|
|
return flagspec_reject(
|
|
"walcompare", walcompare_flags, WALCOMPARE_NFLAGS,
|
|
argv[optind - 1],
|
|
"walcompare [--report] [--committed <N>] <memberA> "
|
|
"<memberB> [more...]");
|
|
}
|
|
}
|
|
|
|
int n = argc - optind;
|
|
if (n < 2) {
|
|
fprintf(stderr,
|
|
"usage: walcompare [--report] [--committed N] <memberA> "
|
|
"<memberB> [more...]\n");
|
|
return -1;
|
|
}
|
|
if (n > 16) {
|
|
fprintf(stderr, "walcompare: at most 16 members\n");
|
|
return -1;
|
|
}
|
|
|
|
walcompare_member_t members[16];
|
|
for (int i = 0; i < n; i++) {
|
|
members[i].path = argv[optind + i];
|
|
members[i].committed_lsn = committed; /* 0 => derive shared boundary */
|
|
}
|
|
|
|
walcompare_divergence_t div;
|
|
int rc = report ? walcompare_report(members, n, stdout, &div)
|
|
: walcompare(members, n, stdout, &div);
|
|
if (rc == WALCMP_OK)
|
|
return 0;
|
|
if (rc == WALCMP_DIVERGENT) {
|
|
fprintf(stderr,
|
|
"walcompare: FAIL — committed-prefix divergence at LSN %" PRIu64
|
|
"\n",
|
|
div.lsn);
|
|
return -1; /* nonzero session verdict */
|
|
}
|
|
fprintf(stderr, "walcompare: error: %s\n", strerror(-rc));
|
|
return -1;
|
|
}
|
|
|
|
/* snapverify <snap_file> [<snap_file> ...] — offline checkpoint-file
|
|
* structural validator for force-snapshot triage (bug-1389: OPCode_
|
|
* snapshotTransfer chunk-content corruption). Each path is a snap-*.capnp
|
|
* (locally generated) or snap-*.bin (received via ha_handle_snapshot_chunk)
|
|
* file. Read-only: walks the same layout persist_load_snapshot() parses
|
|
* without restoring anything, and on the first structural defect reports the
|
|
* exact byte offset and how many job/queue frames parsed cleanly before it —
|
|
* the diagnostic a bare "persist_load_snapshot failed rc=-5" log line
|
|
* doesn't give you. Returns nonzero if any path failed to verify. */
|
|
int cmd_snapverify(int argc, char* argv[])
|
|
{
|
|
if (argc < 2) {
|
|
fprintf(stderr, "usage: snapverify <snap_file> [more...]\n");
|
|
return -1;
|
|
}
|
|
|
|
int rc = 0;
|
|
for (int i = 1; i < argc; i++) {
|
|
snapverify_report_t rep;
|
|
int r = snapverify_path(argv[i], stdout, &rep);
|
|
if (r != 0)
|
|
rc = -1;
|
|
}
|
|
return rc;
|
|
}
|
|
|
|
/* snapdiff <source_file> <received_file> — byte-level checkpoint-file
|
|
* comparator for force-snapshot triage (bug-1389). `source_file` is the
|
|
* primary's own locally generated checkpoint (snap-*.capnp); `received_file`
|
|
* is the copy a standby assembled from the wire (snap-*.bin). Reports the
|
|
* first differing byte offset and whether it lands exactly on an
|
|
* OPCode_snapshotTransfer chunk boundary — the decisive signal for "the
|
|
* transport corrupted one chunk" vs "these were never the same content".
|
|
* Returns 0 if identical, nonzero otherwise. */
|
|
int cmd_snapdiff(int argc, char* argv[])
|
|
{
|
|
if (argc != 3) {
|
|
fprintf(stderr, "usage: snapdiff <source_file> <received_file>\n");
|
|
return -1;
|
|
}
|
|
|
|
snapdiff_report_t rep;
|
|
int rc = snapdiff_paths(argv[1], argv[2], stdout, &rep);
|
|
return (rc == 0) ? 0 : -1;
|
|
}
|
|
|
|
/* configcompare <member_dir> <member_dir> [more...] — cross-member
|
|
* active-configuration comparator (config-upgrade-snapshot task 8.4, Req
|
|
* 4.2). Each path is a member's `workdir/name` conf-root directory (the
|
|
* same directory persist.c's read_active_config_bytes() and config_repl.c's
|
|
* local_config_sha256() resolve `conf/current` under) — NOT a raw config
|
|
* file path. Recomputes each member's active-config sha256 fresh from its
|
|
* own on-disk bytes and reports whether all members match. Returns 0 if
|
|
* all members are present and identical, nonzero if any member is
|
|
* unreadable or divergent. */
|
|
int cmd_configcompare(int argc, char* argv[])
|
|
{
|
|
if (argc < 3) {
|
|
fprintf(stderr, "usage: configcompare <member_dir> <member_dir> [more...]\n");
|
|
return -1;
|
|
}
|
|
|
|
configcompare_report_t rep;
|
|
int rc = configcompare_paths((const char* const*)&argv[1], argc - 1, stdout, &rep);
|
|
return (rc == 0) ? 0 : -1;
|
|
}
|
|
|
|
/* --------------------------------------------------------------------
|
|
* source <file> -- in-session script sourcing (xce-test-cli-ux task 2.4,
|
|
* Requirements 13.1-13.6; 13.7's TTY-condition clarification relating
|
|
* `source` to the Confirmation Gate is a separate, later validation task
|
|
* and is NOT implemented here).
|
|
*
|
|
* Reuses rcfile_load() (rcfile.c/h, xce-test-rc-file spec) UNCHANGED as the
|
|
* actual line-reading/dispatch loop, passing preprocess_and_run as its
|
|
* runner -- the same per-line command dispatcher the interactive REPL loop
|
|
* and the startup rc-file loader already use. rcfile.c itself is not
|
|
* touched: its own missing-file-is-silent-success policy is correct for
|
|
* its one other caller (the startup rc file) but wrong for an explicit
|
|
* user command, so cmd_source layers its own missing-file check (access())
|
|
* and cycle detection (a small in-flight canonical-path stack) on top,
|
|
* entirely in this file -- per research.md's "source built directly on
|
|
* rcfile_load" decision.
|
|
*
|
|
* Cycle detection: source_push_path()/source_pop_path() maintain a static
|
|
* LIFO stack of realpath()-canonicalized in-flight source paths.
|
|
* source_push_path() scans the stack for the same canonical path already
|
|
* in flight (a real cycle -- including through symlinks or a differently-
|
|
* spelled relative path, since realpath() runs before the scan) and
|
|
* returns -ELOOP without pushing if found; otherwise it pushes and returns
|
|
* 0. cmd_source calls source_pop_path() on EVERY exit path once the push
|
|
* succeeded (success, or rcfile_load returning nonzero because a line
|
|
* inside the file failed under -e/abort-on-failure semantics) -- this is
|
|
* the "path stack popped exactly once per successful push, on every exit
|
|
* path" invariant design.md requires. When source_push_path() itself
|
|
* fails (cycle detected), nothing was pushed, so cmd_source returns
|
|
* immediately WITHOUT calling source_pop_path() -- there is nothing to
|
|
* pop.
|
|
*
|
|
* Abort/continue-on-error composition (Req 13.3): a failing line inside
|
|
* the sourced file makes preprocess_and_run (the runner) return 1 once
|
|
* g_continue_on_error is false, which is exactly rcfile_load's own
|
|
* "stop on nonzero runner return" mechanism -- no extra code is needed
|
|
* here for that part; it falls out of reusing rcfile_load() unchanged.
|
|
* cmd_source's own return value then translates rcfile_load's result
|
|
* (0 == every line ran without requesting an abort) into this codebase's
|
|
* negative-errno failure convention (nonzero -> a generic -1 failure,
|
|
* mirroring cmd_configcompare's/cmd_waldump's own "rc == 0 ? 0 : -1"
|
|
* pattern) rather than forwarding rcfile_load's raw positive "stop" signal
|
|
* verbatim: forwarding it unchanged would be silently treated as SUCCESS
|
|
* by an enclosing preprocess_and_run call (only rc<0 counts as failure
|
|
* there), which would incorrectly let an enclosing script's later lines
|
|
* keep running past a `source` that itself aborted under -e, and would
|
|
* break that composition for nested sourcing (source A containing
|
|
* "source B", where B aborts). Translating to a negative value here makes
|
|
* a failed `source` compose correctly at every nesting depth, exactly
|
|
* like any other failing command.
|
|
*
|
|
* Nested sourcing (Req 13.4): "source" is registered in every command
|
|
* table (see the four command_t[] tables below); a line inside a sourced
|
|
* file that itself reads "source <otherfile>" is dispatched by
|
|
* preprocess_and_run/execute_line exactly like any other line, so nested
|
|
* sourcing falls out of dispatch being generic. The path stack is a
|
|
* single process-wide stack (not scoped per active context), so a cycle
|
|
* is caught regardless of which context each nesting level was sourced
|
|
* from. */
|
|
#define SOURCE_MAX_DEPTH 64
|
|
|
|
static char* g_source_stack[SOURCE_MAX_DEPTH];
|
|
static int g_source_depth = 0;
|
|
|
|
/* Returns 0 and pushes `canonical_path` if it is not already in flight, or
|
|
* -ELOOP without pushing if it is (a cycle -- the same canonicalized path
|
|
* sourcing itself, directly or transitively). Returns -ENOMEM if the stack
|
|
* is exhausted (SOURCE_MAX_DEPTH levels of DISTINCT, non-cyclic nested
|
|
* sourcing) or a heap copy of the path could not be made -- either way,
|
|
* nothing is pushed, so the caller must not call source_pop_path(). */
|
|
static int source_push_path(const char* canonical_path)
|
|
{
|
|
int i;
|
|
|
|
for (i = 0; i < g_source_depth; i++) {
|
|
if (strcmp(g_source_stack[i], canonical_path) == 0) {
|
|
return -ELOOP;
|
|
}
|
|
}
|
|
|
|
if (g_source_depth >= SOURCE_MAX_DEPTH) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
g_source_stack[g_source_depth] = strdup(canonical_path);
|
|
if (g_source_stack[g_source_depth] == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
g_source_depth++;
|
|
return 0;
|
|
}
|
|
|
|
/* Pops the most recently pushed path (LIFO). A no-op if the stack is
|
|
* already empty (defensive; every real call site only calls this after a
|
|
* source_push_path() call that itself returned 0). */
|
|
static void source_pop_path(void)
|
|
{
|
|
if (g_source_depth <= 0) {
|
|
return;
|
|
}
|
|
|
|
g_source_depth--;
|
|
free(g_source_stack[g_source_depth]);
|
|
g_source_stack[g_source_depth] = NULL;
|
|
}
|
|
|
|
/* source <file> -- reads and executes each line of <file> as if it had
|
|
* been typed at the current prompt, in order (Req 13.1). A missing or
|
|
* unreadable file is rejected with an explicit stderr error BEFORE
|
|
* rcfile_load() is ever called, bypassing its silent-missing-file
|
|
* behavior entirely (Req 13.2) -- rcfile.c itself is unmodified. */
|
|
int cmd_source(int argc, char* argv[])
|
|
{
|
|
char canonical[PATH_MAX];
|
|
int rc;
|
|
|
|
if (argc != 2) {
|
|
fprintf(stderr, "usage: source <file>\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
if (access(argv[1], R_OK) != 0) {
|
|
fprintf(stderr, "source: %s: %s\n", argv[1], strerror(errno));
|
|
return -errno;
|
|
}
|
|
|
|
if (realpath(argv[1], canonical) == NULL) {
|
|
fprintf(stderr, "source: %s: %s\n", argv[1], strerror(errno));
|
|
return -errno;
|
|
}
|
|
|
|
rc = source_push_path(canonical);
|
|
if (rc == -ELOOP) {
|
|
fprintf(stderr,
|
|
"source: %s: cycle detected (file sources itself, directly "
|
|
"or transitively)\n",
|
|
canonical);
|
|
return rc;
|
|
} else if (rc != 0) {
|
|
fprintf(stderr, "source: %s: %s\n", canonical, strerror(-rc));
|
|
return rc;
|
|
}
|
|
|
|
rc = rcfile_load(argv[1], preprocess_and_run);
|
|
|
|
source_pop_path(); /* popped on EVERY exit path past the push above */
|
|
|
|
return (rc == 0) ? 0 : -1;
|
|
}
|
|
|
|
int cmd_up(int argc, char* argv[])
|
|
{
|
|
if (active_ctx->parent != NULL) {
|
|
int rc;
|
|
|
|
if ((rc = leave_ctx(active_ctx)) < 0) {
|
|
return rc;
|
|
}
|
|
|
|
active_ctx = active_ctx->parent;
|
|
}
|
|
|
|
return 0;
|
|
}
|
|
|
|
int cmd_worker(int argc, char* argv[])
|
|
{
|
|
int rc;
|
|
|
|
if ((rc = enter_ctx(&workerctx)) < 0) {
|
|
return rc;
|
|
}
|
|
|
|
active_ctx = &workerctx;
|
|
|
|
delaymgr_init(&delaymgr, 2000 /* precision_ms */, NULL);
|
|
delaymgr_start(&delaymgr);
|
|
delay_task(&delaymgr, 5, -1, (delay_task_entry_fn_t)worker_house_keeping,
|
|
&workerctx);
|
|
return 0;
|
|
}
|
|
|
|
int cmd_scheduler(int argc, char* argv[])
|
|
{
|
|
int rc;
|
|
|
|
if ((rc = enter_ctx(&schedulerctx)) < 0) {
|
|
return rc;
|
|
}
|
|
|
|
active_ctx = &schedulerctx;
|
|
return 0;
|
|
}
|
|
|
|
int cmd_client(int argc, char* argv[])
|
|
{
|
|
int rc;
|
|
|
|
if ((rc = enter_ctx(&clientctx)) < 0) {
|
|
return rc;
|
|
}
|
|
|
|
active_ctx = &clientctx;
|
|
return 0;
|
|
}
|
|
|
|
static int done = 0;
|
|
|
|
int cmd_quit(int argc, char* argv[])
|
|
{
|
|
leave_ctx(active_ctx);
|
|
|
|
done = 1;
|
|
return 0;
|
|
}
|
|
|
|
static const flagspec_t hello_flags[] = {
|
|
{ 'c', "count", "count", required_argument, 0, "1" },
|
|
{ 'm', "message", "message", required_argument, 0,
|
|
"\"Greeting from xce client\"" },
|
|
};
|
|
|
|
/* xce-test-cli-ux task 3.2: per-call-site heap block wrapping the blocking
|
|
* xcecli_hello() call for xcetest_run_interruptible() (interruptcall.h,
|
|
* tasks 1.4/1.5). `msg` is copied (strdup) rather than referenced, because
|
|
* `-m` can set it to `optarg`, which points into `argv[]` -- storage
|
|
* execute_line() frees (free_argv()) immediately after this command
|
|
* returns, including on the -EINTR/abandoned path, when a detached worker
|
|
* thread may still be reading it. See this file's other xcetest_run_
|
|
* interruptible() call sites for the same pattern applied to every other
|
|
* blocking RPC call below.
|
|
*
|
|
* Ownership contract (interruptcall.h): on a NON-EINTR return, fn() has
|
|
* already finished, so the call site below frees this block itself right
|
|
* after reading `rc`. On -EINTR, ownership has passed to the still-running
|
|
* worker thread -- the call site must not touch it again, so `hello_thunk`
|
|
* frees both `msg` and the block itself, unconditionally, once fn() has
|
|
* actually finished (this is safe on EITHER path, since exactly one of
|
|
* {call site, thunk} ever touches this block after the adapter call
|
|
* returns -- see xcetest_run_interruptible()'s own doc comment). A call
|
|
* abandoned via Ctrl-C therefore leaks nothing beyond the adapter's own
|
|
* documented worst case (an abandoned worker outliving the command by up
|
|
* to ~10s) -- deliberately simpler than reproducing interruptcall.c's own
|
|
* CAS-based ownership protocol per call site, which would be substantial
|
|
* duplicated complexity for a test client's ~19-odd RPC call sites. */
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
char* msg;
|
|
} hello_call_t;
|
|
|
|
static int hello_thunk(void* ctxp)
|
|
{
|
|
hello_call_t* call = (hello_call_t*)ctxp;
|
|
int rc = xcecli_hello(call->conn, call->msg);
|
|
|
|
free(call->msg);
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_hello(int argc, char* argv[])
|
|
{
|
|
int rc = 0;
|
|
int opt;
|
|
char* msg = "Greeting from xce client";
|
|
int count = 1;
|
|
struct option long_opts[HELLO_NFLAGS + 1];
|
|
char optstring[32];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(hello_flags, HELLO_NFLAGS, long_opts, optstring,
|
|
sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr, "xce_test: hello: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL)) != -1) {
|
|
switch (opt) {
|
|
case 'c': {
|
|
long n;
|
|
|
|
/* bug-1710: 0, negative or non-numeric sent nothing and still
|
|
* reported success */
|
|
if (cmdutil_parse_long(optarg, 1, INT_MAX, &n) < 0) {
|
|
fprintf(stderr, "hello: -c expects a count >= 1, got '%s'\n",
|
|
optarg);
|
|
return -EINVAL;
|
|
}
|
|
count = (int)n;
|
|
break;
|
|
}
|
|
case 'm':
|
|
msg = optarg;
|
|
break;
|
|
case '?':
|
|
return flagspec_reject("hello", hello_flags, HELLO_NFLAGS,
|
|
argv[optind - 1],
|
|
"hello [-c <count>] [-m <message>]");
|
|
}
|
|
}
|
|
|
|
break_out = 0;
|
|
while ((!break_out) && (count > 0)) {
|
|
hello_call_t* call = (hello_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
rc = -ENOMEM;
|
|
break;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->msg = strdup(msg);
|
|
|
|
rc = xcetest_run_interruptible(hello_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Ownership of `call` (and its owned msg copy) has passed to
|
|
* the still-running worker thread -- must not free or touch it
|
|
* again from here (interruptcall.h). Report the command as
|
|
* failed via the same errno/perror mechanism this function's
|
|
* other failure path would use, per Requirement 1.3. */
|
|
errno = EINTR;
|
|
perror("hello");
|
|
return rc;
|
|
}
|
|
|
|
count--;
|
|
}
|
|
|
|
return rc;
|
|
}
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
} shutdown_call_t;
|
|
|
|
static int shutdown_thunk(void* ctxp)
|
|
{
|
|
shutdown_call_t* call = (shutdown_call_t*)ctxp;
|
|
int rc = xcecli_shutdown(call->conn);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.3, Requirement 9.3: confirmation bypass. shutdown
|
|
* had no flags at all before this task (task 3.1's getopt migration
|
|
* explicitly left it out of scope for exactly that reason) -- this is the
|
|
* FIRST flagspec_t table it gets, added solely to carry --yes. Same
|
|
* short/long spelling as the other four destructive commands
|
|
* (Requirement 6.2). */
|
|
static const flagspec_t shutdown_flags[] = {
|
|
{ 'y', "yes", "yes", no_argument, 0, NULL },
|
|
};
|
|
|
|
int cmd_shutdown(int argc, char* argv[])
|
|
{
|
|
int rc;
|
|
int opt;
|
|
int yes = 0;
|
|
struct option long_opts[SHUTDOWN_NFLAGS + 1];
|
|
char optstring[16];
|
|
shutdown_call_t* call;
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(shutdown_flags, SHUTDOWN_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: shutdown: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'y':
|
|
yes = 1;
|
|
break;
|
|
case '?':
|
|
return flagspec_reject("shutdown", shutdown_flags,
|
|
SHUTDOWN_NFLAGS, argv[optind - 1],
|
|
"shutdown");
|
|
}
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.3, Requirement 9.5: shutdown is unconditionally
|
|
* destructive. See cmd_jobkill's comment above for why this call site
|
|
* only needs to check its own --yes flag. */
|
|
if (!yes && !confirm_destructive("shut down the xce controller")) {
|
|
fprintf(stderr, "shutdown: not run (declined)\n");
|
|
return 0;
|
|
}
|
|
|
|
call = (shutdown_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
|
|
rc = xcetest_run_interruptible(shutdown_thunk, call);
|
|
if (rc == -EINTR) {
|
|
errno = EINTR;
|
|
perror("shutdown");
|
|
return rc;
|
|
}
|
|
|
|
/* leave client context */
|
|
if (rc >= 0) {
|
|
execute_line("..");
|
|
}
|
|
|
|
return rc;
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.2: Case-B (output-bearing) call site. The block
|
|
* survives past the wrapped call (unlike the Case-A blocks above) because
|
|
* the caller needs to read `res`/`rc` back out of it -- but ONLY on a
|
|
* non-EINTR return, which interruptcall.h guarantees means fn() (the
|
|
* thunk) has already fully finished writing them. On -EINTR, the call
|
|
* site below never touches or frees this block again; the underlying
|
|
* xcecli_reslist() reply (and this block itself) is deliberately leaked
|
|
* in that case, matching every other call site's documented trade-off. */
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_resource_t** res;
|
|
int rc;
|
|
} reslist_call_t;
|
|
|
|
static int reslist_thunk(void* ctxp)
|
|
{
|
|
reslist_call_t* call = (reslist_call_t*)ctxp;
|
|
|
|
call->rc = xcecli_reslist(call->conn, &call->res);
|
|
|
|
return call->rc;
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.4, Requirement 5.1/5.2, 6.2: reslist's first-ever
|
|
* flag, --json (long-only -- no short form is required by the
|
|
* requirements and none is added here). Same long_opt spelling ("json")
|
|
* and concept ("json-output") as workerlist_flags/joblist_flags below --
|
|
* Requirement 6.2 mandates identical naming for the same concept across
|
|
* every command that has it. */
|
|
enum { RESLIST_OPT_JSON = 0 };
|
|
|
|
static const flagspec_t reslist_flags[] = {
|
|
{ 0, "json", "json-output", no_argument, 0, "human-readable table" },
|
|
};
|
|
|
|
int cmd_reslist(int argc, char* argv[])
|
|
{
|
|
int rc, i;
|
|
int opt;
|
|
int json_mode = 0;
|
|
xce_resource_t** res;
|
|
reslist_call_t* call;
|
|
struct option long_opts[RESLIST_NFLAGS + 1];
|
|
char optstring[8];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(reslist_flags, RESLIST_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: reslist: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case FLAGSPEC_LONG_ONLY_BASE + RESLIST_OPT_JSON:
|
|
json_mode = 1;
|
|
break;
|
|
case '?':
|
|
return flagspec_reject("reslist", reslist_flags, RESLIST_NFLAGS,
|
|
argv[optind - 1], "reslist [--json]");
|
|
}
|
|
}
|
|
|
|
call = (reslist_call_t*)calloc(1, sizeof(*call));
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
|
|
rc = xcetest_run_interruptible(reslist_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Same errno/perror mechanism, and the same "xce_reslist" label,
|
|
* as this function's own non-interrupt failure path just below. */
|
|
SET_ERROR(EINTR);
|
|
perror("xce_reslist");
|
|
return rc;
|
|
}
|
|
|
|
res = call->res;
|
|
free(call);
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xce_reslist", rc);
|
|
return rc;
|
|
}
|
|
|
|
if (json_mode) {
|
|
/* xce-test-cli-ux task 3.4, Requirement 5.1/5.3: emit the SAME
|
|
* fields the table below prints (Name, Flags, Interval,
|
|
* Description), unabbreviated, as one JSON object per resource.
|
|
* reslist_json_row() (listrow_json.h) owns the field mapping;
|
|
* listfmt_json.h (task 2.2) owns the array lifecycle/printing. */
|
|
cJSON* array = listfmt_json_begin();
|
|
|
|
if (array == NULL) {
|
|
free_Resource_list(rc, res);
|
|
return -ENOMEM;
|
|
}
|
|
|
|
for (i = 0; i < rc; i++) {
|
|
cJSON* row = listfmt_json_new_row(array);
|
|
|
|
reslist_json_row(row, res[i]->name, res[i]->flags,
|
|
res[i]->interval, res[i]->desc);
|
|
}
|
|
|
|
listfmt_json_finish(array);
|
|
free_Resource_list(rc, res);
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* Note the extra literal space before the Description specifier: unlike
|
|
* cmd_workerlist's all-left-justified text columns (which self-separate
|
|
* via their own trailing padding), Interval stays right-justified here,
|
|
* so without an explicit separator a short Interval value would run
|
|
* directly into the now-left-justified Description text. */
|
|
printf("%-10s%10s%10s %-64s\n", "Name", "Flags", "Interval",
|
|
"Description");
|
|
|
|
for (i = 0; i < rc; i++) {
|
|
/* bug-1711 (Requirement 11.4): prefix as hex ("0x%x"), matching this
|
|
* file's own state_raw convention (print_jobinfo), instead of an
|
|
* unlabeled bare %x that is ambiguous vs decimal. */
|
|
char flags_buf[16];
|
|
|
|
snprintf(flags_buf, sizeof(flags_buf), "0x%x", res[i]->flags);
|
|
|
|
printf("%-10s%10s%10d %-64s\n", res[i]->name, flags_buf,
|
|
res[i]->interval, res[i]->desc);
|
|
}
|
|
|
|
free_Resource_list(rc, res);
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* worker_status_str() -- extracted to joblist_display.c/.h (xce-test-cli-ux
|
|
* task 2.5 remediation): pure, dependency-light display formatting, no
|
|
* networking/REPL/global-state coupling, so it can be unit-tested outside
|
|
* of this file (see test_joblist_display.c), mirroring set_resval()'s own
|
|
* extraction to resval_typeresolve.c below. Call sites are unchanged --
|
|
* they now resolve through joblist_display.h's declaration. */
|
|
|
|
/* forward decl: defined near cmd_resval, below, which also uses it */
|
|
static void format_bytes_human(double bytes, char* buf, size_t bufsize);
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_worker_t** workers;
|
|
int rc;
|
|
} workerlist_call_t;
|
|
|
|
static int workerlist_thunk(void* ctxp)
|
|
{
|
|
workerlist_call_t* call = (workerlist_call_t*)ctxp;
|
|
|
|
call->rc = xcecli_workerlist(call->conn, &call->workers);
|
|
|
|
return call->rc;
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.4, Requirement 5.1/5.2, 6.2: workerlist's
|
|
* first-ever flag, --json -- IDENTICAL long_opt/concept spelling to
|
|
* reslist_flags/joblist_flags (Requirement 6.2). */
|
|
enum { WORKERLIST_OPT_JSON = 0 };
|
|
|
|
static const flagspec_t workerlist_flags[] = {
|
|
{ 0, "json", "json-output", no_argument, 0, "human-readable table" },
|
|
};
|
|
|
|
int cmd_workerlist(int argc, char* argv[])
|
|
{
|
|
int rc, i;
|
|
int opt;
|
|
int json_mode = 0;
|
|
xce_worker_t** workers;
|
|
workerlist_call_t* call;
|
|
struct option long_opts[WORKERLIST_NFLAGS + 1];
|
|
char optstring[8];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(workerlist_flags, WORKERLIST_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: workerlist: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case FLAGSPEC_LONG_ONLY_BASE + WORKERLIST_OPT_JSON:
|
|
json_mode = 1;
|
|
break;
|
|
case '?':
|
|
return flagspec_reject("workerlist", workerlist_flags,
|
|
WORKERLIST_NFLAGS, argv[optind - 1],
|
|
"workerlist [--json]");
|
|
}
|
|
}
|
|
|
|
call = (workerlist_call_t*)calloc(1, sizeof(*call));
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
|
|
rc = xcetest_run_interruptible(workerlist_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Same mechanism/label as this function's own failure path below. */
|
|
SET_ERROR(EINTR);
|
|
perror("xce_workerlist");
|
|
return rc;
|
|
}
|
|
|
|
workers = call->workers;
|
|
free(call);
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xce_workerlist", rc);
|
|
return rc;
|
|
}
|
|
|
|
if (json_mode) {
|
|
/* xce-test-cli-ux task 3.4, Requirement 5.1/5.3: emit the SAME
|
|
* fields the table below prints, unabbreviated, as one JSON
|
|
* object per worker -- MaxMem/MaxSwap/MaxTmp as their RAW byte
|
|
* values (not the table's humanized "64.0 MB"-style strings), see
|
|
* workerlist_json_row()'s own doc comment (listrow_json.h). */
|
|
cJSON* array = listfmt_json_begin();
|
|
|
|
if (array == NULL) {
|
|
free_Worker_list(rc, workers);
|
|
return -ENOMEM;
|
|
}
|
|
|
|
for (i = 0; i < rc; i++) {
|
|
cJSON* row = listfmt_json_new_row(array);
|
|
|
|
/* bug-1777: effective values, as the scheduler sees them */
|
|
workerlist_json_row(
|
|
row, workers[i]->name, worker_status_str(workers[i]->status),
|
|
workers[i]->attrs->hosttype, workers[i]->attrs->hostmodel,
|
|
workers[i]->attrs->cpufactor, workers[i]->attrs->maxcpus,
|
|
worker_effective_attr(workers[i], "maxmem",
|
|
workers[i]->attrs->maxmem),
|
|
worker_effective_attr(workers[i], "maxswap",
|
|
workers[i]->attrs->maxswap),
|
|
worker_effective_attr(workers[i], "maxtmp",
|
|
workers[i]->attrs->maxtmp),
|
|
workers[i]->attrs->ndisks);
|
|
}
|
|
|
|
listfmt_json_finish(array);
|
|
free_Worker_list(rc, workers);
|
|
|
|
return 0;
|
|
}
|
|
|
|
printf("%-20s%-8s%-25s%-25s%12s%12s%12s%12s%12s%12s\n", "HostName",
|
|
"Status", "HostType", "HostModel", "CPUfactor", "MaxCPU", "MaxMem",
|
|
"MaxSwap", "MaxTmp", "NDisks");
|
|
|
|
for (i = 0; i < rc; i++) {
|
|
/* MaxMem/MaxSwap/MaxTmp are stored/wired as raw bytes (see
|
|
* is_bytes_valued_resource) -- humanize for display only, every
|
|
* other column stays as-is. */
|
|
char maxmem_buf[64];
|
|
char maxswap_buf[64];
|
|
char maxtmp_buf[64];
|
|
|
|
/* bug-1777: the effective value (a hosts.yaml override included),
|
|
* which is what the scheduler uses, not the join-time attrs */
|
|
format_bytes_human(worker_effective_attr(workers[i], "maxmem",
|
|
workers[i]->attrs->maxmem),
|
|
maxmem_buf, sizeof(maxmem_buf));
|
|
format_bytes_human(worker_effective_attr(workers[i], "maxswap",
|
|
workers[i]->attrs->maxswap),
|
|
maxswap_buf, sizeof(maxswap_buf));
|
|
format_bytes_human(worker_effective_attr(workers[i], "maxtmp",
|
|
workers[i]->attrs->maxtmp),
|
|
maxtmp_buf, sizeof(maxtmp_buf));
|
|
|
|
printf("%-20s%-8s%-25s%-25s%12f%12d%12s%12s%12s%12d\n",
|
|
workers[i]->name, worker_status_str(workers[i]->status),
|
|
workers[i]->attrs->hosttype, workers[i]->attrs->hostmodel,
|
|
workers[i]->attrs->cpufactor, workers[i]->attrs->maxcpus,
|
|
maxmem_buf, maxswap_buf, maxtmp_buf,
|
|
workers[i]->attrs->ndisks);
|
|
}
|
|
|
|
free_Worker_list(rc, workers);
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* job_state_str()/format_job_time() -- extracted to joblist_display.c/.h
|
|
* (xce-test-cli-ux task 2.5 remediation), same rationale as
|
|
* worker_status_str() above: pure, dependency-light display formatting,
|
|
* unit-tested outside of this file in test_joblist_display.c. Call sites
|
|
* (below, and print_jobinfo's "state:" line) are unchanged. */
|
|
|
|
/* Shared, print-free joblist core for `cmd_joblist` and the `expect-jobs` /
|
|
* `expect-job` assertions (xce-test-e2e-primitives task 5.1, design.md:81,
|
|
* 116-117).
|
|
*
|
|
* Performs the single `xcecli_joblist` call (through the failover-aware client
|
|
* context) and returns the current job count (>= 0), or a negative errno-style
|
|
* value on failure. When `find_jobid != NULL`, `*contains` is set to 1 if a job
|
|
* whose `fmt_jobid(...)` string equals `find_jobid` is present, else 0; the
|
|
* `fmt_jobid` representation matches the one `cmd_joblist` prints and the one
|
|
* `cmd_jobsub` captures into `$jobid` (task 4.1), so `expect-job $jobid` lines
|
|
* up. When `print_table` is set, the human-readable table is printed (the
|
|
* `cmd_joblist` path). The job array is always freed internally.
|
|
*
|
|
* `json_mode` (xce-test-cli-ux task 3.4, Requirement 5.1/5.2): when set,
|
|
* emits the SAME fields `print_table` would have printed as one JSON
|
|
* object per job instead (joblist_json_row(), listrow_json.h) --
|
|
* mutually exclusive with `print_table` in practice: `cmd_joblist` (the
|
|
* only caller that ever sets either) passes exactly one of the two as
|
|
* true, never both; `expect-jobs`/`expect-job` always pass 0 for both.
|
|
*
|
|
* On error this returns the negative rc WITHOUT setting any auto-variable, so a
|
|
* failed joblist never advertises a fake `$jobcount` (Req 3.6). */
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_job_t** jobs;
|
|
int rc;
|
|
} joblist_call_t;
|
|
|
|
static int joblist_thunk(void* ctxp)
|
|
{
|
|
joblist_call_t* call = (joblist_call_t*)ctxp;
|
|
|
|
call->rc = xcecli_joblist(call->conn, &call->jobs);
|
|
|
|
return call->rc;
|
|
}
|
|
|
|
static int joblist_query(const char* find_jobid, int* contains, int print_table,
|
|
int quiet, int json_mode)
|
|
{
|
|
int rc, i;
|
|
xce_job_t** jobs;
|
|
cJSON* json_array = NULL;
|
|
joblist_call_t* call = (joblist_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (contains != NULL) {
|
|
*contains = 0;
|
|
}
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
|
|
rc = xcetest_run_interruptible(joblist_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Same mechanism/label as this function's own failure path below. */
|
|
if (!quiet) {
|
|
SET_ERROR(EINTR);
|
|
perror("xce_joblist");
|
|
}
|
|
return rc;
|
|
}
|
|
|
|
jobs = call->jobs;
|
|
free(call);
|
|
|
|
if (rc < 0) {
|
|
SET_ERROR(-rc);
|
|
/* `quiet` callers (e.g. the wait-primary poll, which probes repeatedly
|
|
* and EXPECTS failure until a primary serves) suppress the per-attempt
|
|
* error print so a normal failover wait does not spam the console. */
|
|
if (!quiet) {
|
|
cmdutil_rpc_error("xce_joblist", rc);
|
|
}
|
|
return rc;
|
|
}
|
|
|
|
if (print_table) {
|
|
printf("%-26s%-10s%-10s%-8s%-24s%-24s%-24s%12s%12s%30s\n", "JobId",
|
|
"User", "Queue", "State", "SubmitTime", "StartTime",
|
|
"FinishTime", "MinTasks", "MaxTasks", "Command");
|
|
} else if (json_mode) {
|
|
json_array = listfmt_json_begin();
|
|
|
|
if (json_array == NULL) {
|
|
for (i = 0; i < rc; i++) {
|
|
free_Job_ptr(&(jobs[i]));
|
|
}
|
|
free(jobs);
|
|
return -ENOMEM;
|
|
}
|
|
}
|
|
|
|
for (i = 0; i < rc; i++) {
|
|
char buf[128];
|
|
|
|
fmt_jobid(buf, sizeof(buf) - 1, jobs[i]->jobid);
|
|
|
|
if ((find_jobid != NULL) && (contains != NULL)
|
|
&& (strcmp(buf, find_jobid) == 0)) {
|
|
*contains = 1;
|
|
}
|
|
|
|
if (print_table) {
|
|
char submit_buf[32];
|
|
char start_buf[32];
|
|
char finish_buf[32];
|
|
|
|
printf("%-26s%-10s%-10s%-8s%-24s%-24s%-24s%12d%12d%30s\n", buf,
|
|
jobs[i]->submit->user, jobs[i]->submit->queue,
|
|
job_state_str(jobs[i]->state),
|
|
format_job_time(jobs[i]->queue_time, submit_buf,
|
|
sizeof(submit_buf)),
|
|
format_job_time(jobs[i]->start_time, start_buf,
|
|
sizeof(start_buf)),
|
|
format_job_time(jobs[i]->finish_time, finish_buf,
|
|
sizeof(finish_buf)),
|
|
jobs[i]->submit->mintasks, jobs[i]->submit->maxtasks,
|
|
jobs[i]->submit->start_info->cmdline);
|
|
} else if (json_mode) {
|
|
/* xce-test-cli-ux task 3.4, Requirement 5.1/5.3: the SAME
|
|
* fields the table above prints, unabbreviated, as one JSON
|
|
* object per job -- joblist_json_row() (listrow_json.h) owns
|
|
* the field mapping. */
|
|
cJSON* row = listfmt_json_new_row(json_array);
|
|
|
|
joblist_json_row(row, buf, jobs[i]->submit->user,
|
|
jobs[i]->submit->queue,
|
|
job_state_str(jobs[i]->state),
|
|
jobs[i]->queue_time, jobs[i]->start_time,
|
|
jobs[i]->finish_time, jobs[i]->submit->mintasks,
|
|
jobs[i]->submit->maxtasks,
|
|
jobs[i]->submit->start_info->cmdline);
|
|
}
|
|
}
|
|
|
|
if (json_mode) {
|
|
listfmt_json_finish(json_array);
|
|
}
|
|
|
|
for (i = 0; i < rc; i++) {
|
|
free_Job_ptr(&(jobs[i]));
|
|
}
|
|
|
|
free(jobs);
|
|
|
|
return rc;
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.4, Requirement 5.1/5.2, 6.2: joblist's first-ever
|
|
* flag, --json -- IDENTICAL long_opt/concept spelling to reslist_flags/
|
|
* workerlist_flags above (Requirement 6.2). */
|
|
enum { JOBLIST_OPT_JSON = 0 };
|
|
|
|
static const flagspec_t joblist_flags[] = {
|
|
{ 0, "json", "json-output", no_argument, 0, "human-readable table" },
|
|
};
|
|
|
|
int cmd_joblist(int argc, char* argv[])
|
|
{
|
|
int count;
|
|
int opt;
|
|
int json_mode = 0;
|
|
struct option long_opts[JOBLIST_NFLAGS + 1];
|
|
char optstring[8];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(joblist_flags, JOBLIST_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: joblist: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case FLAGSPEC_LONG_ONLY_BASE + JOBLIST_OPT_JSON:
|
|
json_mode = 1;
|
|
break;
|
|
case '?':
|
|
return flagspec_reject("joblist", joblist_flags, JOBLIST_NFLAGS,
|
|
argv[optind - 1], "joblist [--json]");
|
|
}
|
|
}
|
|
|
|
count = joblist_query(NULL, NULL, /*print_table=*/!json_mode,
|
|
/*quiet=*/0, json_mode);
|
|
|
|
if (count < 0) {
|
|
return count;
|
|
}
|
|
|
|
/* Capture $jobcount (Req 3.4): a successful joblist publishes the returned
|
|
* job count so a later `echo $jobcount` / `expect-jobs` substitutes it. On
|
|
* failure we returned above without touching g_last_jobcount, so a failed
|
|
* joblist never advertises a fake count (Req 3.6). */
|
|
g_last_jobcount = count;
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* `expect-jobs N` assertion (xce-test-e2e-primitives task 5.2, Req 5.1/5.2,
|
|
* design.md:116-117).
|
|
*
|
|
* Passes (returns 0) when the current job count equals N; otherwise prints the
|
|
* expected vs actual count and returns a negative errno-style value so
|
|
* preprocess_and_run records the session as failed (Req 1.1).
|
|
*
|
|
* A malformed argument (missing or non-numeric N) is itself an assertion
|
|
* failure, not a no-op. A failed joblist query is treated as a failure too: we
|
|
* cannot determine the count, so we never silently equate "query failed" with
|
|
* "zero jobs" (Req 5.2 reports actual; a failed query has no trustworthy
|
|
* actual). */
|
|
int cmd_expect_jobs(int argc, char* argv[])
|
|
{
|
|
long expected;
|
|
char* end;
|
|
int count;
|
|
|
|
if (argc != 2) {
|
|
fprintf(stderr, "expect-jobs <N>\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
errno = 0;
|
|
expected = strtol(argv[1], &end, 10);
|
|
|
|
if ((errno != 0) || (end == argv[1]) || (*end != '\0') || (expected < 0)) {
|
|
fprintf(stderr,
|
|
"expect-jobs: invalid count '%s' (expected a non-negative "
|
|
"integer)\n",
|
|
argv[1]);
|
|
return -EINVAL;
|
|
}
|
|
|
|
/* Print-free count via the shared helper. A negative rc means the query
|
|
* itself failed (e.g. no reachable master) — that is an assertion failure,
|
|
* not "0 jobs". */
|
|
count = joblist_query(NULL, NULL, 0, /*quiet=*/0, /*json_mode=*/0);
|
|
|
|
if (count < 0) {
|
|
fprintf(stderr,
|
|
"expect-jobs: could not determine job count (joblist query "
|
|
"failed)\n");
|
|
return count;
|
|
}
|
|
|
|
if (count != (int)expected) {
|
|
fprintf(stderr, "expect-jobs: expected %ld, actual %d\n", expected,
|
|
count);
|
|
return -EINVAL;
|
|
}
|
|
|
|
printf("expect-jobs: ok (%d)\n", count);
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* `expect-job <jobid>` assertion (xce-test-e2e-primitives task 5.3, Req 5.3/5.4,
|
|
* design.md cmd_expect_jobs / cmd_expect_job).
|
|
*
|
|
* Passes (returns 0) when a job whose `fmt_jobid` string equals argv[1] is
|
|
* present in the current job list; otherwise prints the missing id and returns a
|
|
* negative errno-style value so preprocess_and_run records the session as failed
|
|
* (Req 1.1).
|
|
*
|
|
* `$jobid` substitution is transparent: autovar substitution (task 3.1) replaces
|
|
* `$jobid` with the captured id string BEFORE this handler runs, so argv[1] is
|
|
* already the literal id. The membership test in joblist_query uses the SAME
|
|
* `fmt_jobid` representation that cmd_jobsub captured into `$jobid` (task 4.1),
|
|
* so `expect-job $jobid` matches the joblist entry.
|
|
*
|
|
* A missing argument is itself an assertion failure, not a no-op. A failed
|
|
* joblist query is also a failure: we cannot determine membership, so we report
|
|
* the query failure distinctly rather than silently equating it with "absent". */
|
|
int cmd_expect_job(int argc, char* argv[])
|
|
{
|
|
int contains;
|
|
int rc;
|
|
|
|
if (argc != 2) {
|
|
fprintf(stderr, "expect-job <jobid>\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
/* Print-free membership query via the shared helper. A negative rc means the
|
|
* query itself failed (e.g. no reachable master) — that is an assertion
|
|
* failure, distinct from "job absent". */
|
|
rc = joblist_query(argv[1], &contains, 0, /*quiet=*/0, /*json_mode=*/0);
|
|
|
|
if (rc < 0) {
|
|
fprintf(stderr,
|
|
"expect-job: could not determine membership of '%s' (joblist "
|
|
"query failed)\n",
|
|
argv[1]);
|
|
return rc;
|
|
}
|
|
|
|
if (!contains) {
|
|
fprintf(stderr, "expect-job: job %s not found\n", argv[1]);
|
|
return -ENOENT;
|
|
}
|
|
|
|
printf("expect-job: ok (%s)\n", argv[1]);
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* `wait-primary [timeout_ms]` (controller-status task 4.2, Req 9.1-9.4;
|
|
* originally xce-test-e2e-primitives task 6.1; design.md "cmd_wait_primary probe
|
|
* swap — State contract").
|
|
*
|
|
* Blocks until the AUTHORITATIVE controller status probe reaches the serving
|
|
* primary, or the timeout elapses. The probe is xcecli_ctrlstatus_primary: it
|
|
* drives active_ctx + libxce master-rotation/failover and returns 0 ONLY when a
|
|
* reply with responderRole == PRIMARY was obtained (it internally follows the
|
|
* redirect to the primary), or a typed negative when no primary answered within
|
|
* its budget (Req 9.1). A standby/hint answer is NOT success — the probe keeps
|
|
* polling until the primary itself answers (this avoids a false success on a
|
|
* standby's stale primary view after a primary death).
|
|
*
|
|
* On the first authoritative probe we print the served primary's identity and
|
|
* return 0 (Req 9.2). Otherwise we sleep a short poll interval and retry,
|
|
* re-checking a MONOTONIC deadline between attempts so the command returns
|
|
* within a bounded time of the timeout and never hangs indefinitely when there
|
|
* is no primary (Req 9.3). The deadline is derived from CLOCK_MONOTONIC, never
|
|
* wall-clock, so a clock step cannot skew it.
|
|
*
|
|
* The timeout is argv[1] in milliseconds when supplied, else
|
|
* WAIT_PRIMARY_DEFAULT_MS (Req 9.4). A malformed timeout argument is a usage
|
|
* error (return < 0) rather than a silent fallback to the default. */
|
|
#define WAIT_PRIMARY_DEFAULT_MS 30000L /* default max wait when arg omitted */
|
|
#define WAIT_PRIMARY_POLL_MS 200L /* sleep between probe attempts */
|
|
#define MS_PER_SEC 1000L
|
|
#define NS_PER_MS 1000000L
|
|
#define US_PER_MS 1000L
|
|
|
|
/* The client's configured log level (set in main from -v), so wait-primary can
|
|
* temporarily raise the threshold to silence libxce's per-attempt failover INFO
|
|
* spam during its poll and then RESTORE the user's chosen level. */
|
|
static char* g_active_loglevel = (char*)"info";
|
|
|
|
static long monotonic_ms(void)
|
|
{
|
|
struct timespec ts;
|
|
|
|
clock_gettime(CLOCK_MONOTONIC, &ts);
|
|
|
|
return ((long)ts.tv_sec * MS_PER_SEC) + (ts.tv_nsec / NS_PER_MS);
|
|
}
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_ctrlstatus_t* status;
|
|
int rc;
|
|
} ctrlstatus_primary_call_t;
|
|
|
|
static int ctrlstatus_primary_thunk(void* ctxp)
|
|
{
|
|
ctrlstatus_primary_call_t* call = (ctrlstatus_primary_call_t*)ctxp;
|
|
|
|
call->status = NULL;
|
|
call->rc = xcecli_ctrlstatus_primary(call->conn, &call->status);
|
|
|
|
return call->rc;
|
|
}
|
|
|
|
int cmd_wait_primary(int argc, char* argv[])
|
|
{
|
|
long timeout_ms = WAIT_PRIMARY_DEFAULT_MS;
|
|
long start_ms, elapsed_ms;
|
|
|
|
if (argc > 2) {
|
|
fprintf(stderr, "wait-primary [timeout_ms]\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
if (argc == 2) {
|
|
char* end;
|
|
|
|
errno = 0;
|
|
timeout_ms = strtol(argv[1], &end, 10);
|
|
|
|
if ((errno != 0) || (end == argv[1]) || (*end != '\0')
|
|
|| (timeout_ms < 0)) {
|
|
fprintf(stderr,
|
|
"wait-primary: invalid timeout '%s' (expected a "
|
|
"non-negative integer in ms)\n",
|
|
argv[1]);
|
|
return -EINVAL;
|
|
}
|
|
}
|
|
|
|
start_ms = monotonic_ms();
|
|
|
|
/* The poll probes repeatedly and EXPECTS failure until a primary serves, so
|
|
* it must not spam the console: xcecli_ctrlstatus_primary suppresses its own
|
|
* per-attempt output, and we raise the client log threshold for the duration
|
|
* of the loop to silence libxce's per-attempt "xcecli failover: attempt N"
|
|
* INFO logs — RESTORED to the user's level on every exit path. */
|
|
mylog_setlevel((char*)"warn");
|
|
|
|
/* Probe, then re-check the monotonic deadline between attempts. Each probe
|
|
* may itself take time (libxce failover + redirect-to-primary has its own
|
|
* internal retry budget); we re-read the clock after every probe returns so
|
|
* the overall loop honors the deadline and returns failure in bounded time. */
|
|
for (;;) {
|
|
xce_ctrlstatus_t* status;
|
|
int rc;
|
|
ctrlstatus_primary_call_t* call
|
|
= (ctrlstatus_primary_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
mylog_setlevel(g_active_loglevel);
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
|
|
rc = xcetest_run_interruptible(ctrlstatus_primary_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Abandon the whole wait-primary poll immediately (Requirement
|
|
* 1.1) rather than retrying -- ownership of `call` has passed
|
|
* to the still-running worker thread (interruptcall.h). */
|
|
mylog_setlevel(g_active_loglevel);
|
|
/* xce-test-cli-ux task 3.6, Requirement 3.1: an interrupted wait
|
|
* is a genuine command failure (rc == -EINTR here, same
|
|
* convention cmd_resval's EINTR path documents), so this goes
|
|
* to stderr like this command's other failure paths -- not
|
|
* grouped with its success message below anymore. */
|
|
fprintf(stderr, "wait-primary: interrupted after %ld ms\n",
|
|
monotonic_ms() - start_ms);
|
|
return rc;
|
|
}
|
|
|
|
status = call->status;
|
|
free(call);
|
|
|
|
/* Success ONLY on an authoritative reply from the serving primary
|
|
* (responderRole == PRIMARY, guaranteed by xcecli_ctrlstatus_primary).
|
|
* Print the served primary's identity (the responder IS the primary, so
|
|
* primaryName == responderName) and free the reply (Req 9.1, 9.2). */
|
|
if (rc == 0) {
|
|
const char* primary = (status != NULL)
|
|
? (NULLSTR(status->primaryName)
|
|
? status->responderName
|
|
: status->primaryName)
|
|
: NULL;
|
|
|
|
elapsed_ms = monotonic_ms() - start_ms;
|
|
mylog_setlevel(g_active_loglevel);
|
|
printf("wait-primary: primary '%s' serving after %ld ms\n",
|
|
NULLSTR(primary) ? "unknown" : primary, elapsed_ms);
|
|
xcecli_free_ctrlstatus(status);
|
|
return 0;
|
|
}
|
|
|
|
/* Non-authoritative / failed probe: nothing to free (the typed-failure
|
|
* contract leaves *status untouched), keep polling until the deadline. */
|
|
elapsed_ms = monotonic_ms() - start_ms;
|
|
|
|
if (elapsed_ms >= timeout_ms) {
|
|
mylog_setlevel(g_active_loglevel);
|
|
fprintf(stderr,
|
|
"wait-primary: no primary served within %ld ms (waited "
|
|
"%ld ms)\n",
|
|
timeout_ms, elapsed_ms);
|
|
return -ETIMEDOUT;
|
|
}
|
|
|
|
/* Don't oversleep past the deadline: cap the poll to the time left. */
|
|
{
|
|
long remaining_ms = timeout_ms - elapsed_ms;
|
|
long sleep_ms = (remaining_ms < WAIT_PRIMARY_POLL_MS)
|
|
? remaining_ms
|
|
: WAIT_PRIMARY_POLL_MS;
|
|
|
|
if (sleep_ms > 0) {
|
|
usleep((useconds_t)(sleep_ms * US_PER_MS));
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
/* Map a ctrl_role_t raw value (CtrlStatus.responderRole, see ha.h:60-63 and
|
|
* design.md "Schema — reply payload") to a display string. PRIMARY==0,
|
|
* STANDBY==1, INIT==2, ELECTING==3; anything else is reported verbatim so an
|
|
* unexpected role is never silently rendered as a known one. */
|
|
static const char* ctrlstatus_role_str(uint8_t role)
|
|
{
|
|
switch (role) {
|
|
case 0:
|
|
return "primary";
|
|
case 1:
|
|
return "standby";
|
|
case 2:
|
|
return "initializing";
|
|
case 3:
|
|
return "electing";
|
|
default:
|
|
return "unknown-role";
|
|
}
|
|
}
|
|
|
|
/* Map a CtrlPeer.status raw value (controller-status Req 13 delta,
|
|
* design.md "Design Delta — Requirement 13", Locked Decision 2 / schema
|
|
* delta) to a display string. UNKNOWN==0, OK==1, CATCHING_UP==2, DEAD==3;
|
|
* anything else is reported verbatim so an unexpected code is never
|
|
* silently rendered as a known one (mirrors ctrlstatus_role_str above). */
|
|
static const char* ctrlstatus_peer_status_str(uint8_t status)
|
|
{
|
|
switch (status) {
|
|
case 0:
|
|
return "unknown";
|
|
case 1:
|
|
return "ok";
|
|
case 2:
|
|
return "catching-up";
|
|
case 3:
|
|
return "dead";
|
|
default:
|
|
return "unknown-status";
|
|
}
|
|
}
|
|
|
|
/* `ctrlstatus` (controller-status task 4.1, Req 1.3/3.3/5.1/5.2/8.1/8.2/8.3,
|
|
* design.md "cmd_ctrlstatus (test client) — State contract").
|
|
*
|
|
* Issues the any-node status read (xcecli_ctrlstatus, is_write=false: whatever
|
|
* controller answers — primary or standby hint view — rides the configured
|
|
* failover/rotation) and prints the identity/health fields in human-readable
|
|
* form.
|
|
*
|
|
* On failure (no reachable controller etc.) it prints a typed error and returns
|
|
* the negative rc so the session records the command as failed and exits
|
|
* non-zero (Req 8.3). On success it prints the fields and returns 0 (Req 8.2). */
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_ctrlstatus_t* status;
|
|
int rc;
|
|
} ctrlstatus_call_t;
|
|
|
|
static int ctrlstatus_thunk(void* ctxp)
|
|
{
|
|
ctrlstatus_call_t* call = (ctrlstatus_call_t*)ctxp;
|
|
|
|
call->status = NULL;
|
|
call->rc = xcecli_ctrlstatus(call->conn, &call->status);
|
|
|
|
return call->rc;
|
|
}
|
|
|
|
int cmd_ctrlstatus(int argc, char* argv[])
|
|
{
|
|
xce_ctrlstatus_t* status = NULL;
|
|
int rc;
|
|
int i;
|
|
ctrlstatus_call_t* call;
|
|
|
|
(void)argv;
|
|
|
|
if (argc != 1) {
|
|
fprintf(stderr, "ctrlstatus\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
call = (ctrlstatus_call_t*)calloc(1, sizeof(*call));
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
|
|
rc = xcetest_run_interruptible(ctrlstatus_thunk, call);
|
|
if (rc != -EINTR) {
|
|
status = call->status;
|
|
free(call);
|
|
}
|
|
/* else: ownership of `call` has passed to the still-running worker
|
|
* thread (interruptcall.h) -- leave it untouched, `status` stays NULL,
|
|
* and rc (-EINTR) falls straight into the SAME generic error-reporting
|
|
* switch below that every other ctrlstatus failure already uses
|
|
* (Requirement 1.3: "consistent with how any other failing command is
|
|
* reported", not a special one-off). xce_strerror_r(EINTR, ...) below
|
|
* renders the standard "Interrupted system call" text via its errno
|
|
* fallback. */
|
|
|
|
if (rc < 0) {
|
|
char ebuf[128];
|
|
|
|
ebuf[0] = '\0';
|
|
xce_strerror_r(-rc, ebuf, sizeof(ebuf));
|
|
SET_ERROR(-rc);
|
|
|
|
switch (-rc) {
|
|
case E_XCE_NOMASTER:
|
|
fprintf(stderr, "ctrlstatus failed: no-master (%s)\n", ebuf);
|
|
break;
|
|
case E_XCE_NOT_READY:
|
|
fprintf(stderr, "ctrlstatus failed: not-ready (%s)\n", ebuf);
|
|
break;
|
|
case ETIMEDOUT:
|
|
fprintf(stderr, "ctrlstatus failed: timeout (%s)\n", ebuf);
|
|
break;
|
|
default:
|
|
fprintf(stderr, "ctrlstatus failed: %s\n", ebuf);
|
|
break;
|
|
}
|
|
|
|
return rc;
|
|
}
|
|
|
|
/* Current primary (Req 2.x): an empty primaryName means no elected/known
|
|
* primary — render the well-defined "unknown" (Req 1.3/2.2/2.3) rather than
|
|
* an empty or stale value. */
|
|
printf("Primary: %s\n",
|
|
NULLSTR(status->primaryName) ? "unknown" : status->primaryName);
|
|
|
|
/* Election term + time of the CURRENT primary (Req 12.1/12.3/12.4/12.6):
|
|
* gated on elected_at_ms itself, NOT on primaryName (design.md Locked
|
|
* Decision 3, corrected 2026-09-16 / task 7.6 live E2E finding).
|
|
* primaryName is unconditionally empty on every non-primary responder
|
|
* (ctrlstatus_fill only self-reports the primary's own name; a
|
|
* pre-existing Req 2 behavior, unrelated to this delta), so gating on it
|
|
* made electedTerm/electedAt read "unknown" on every standby forever,
|
|
* even once a real election had converged. elected_at_ms stays exactly 0
|
|
* until a real promotion stamps it (ha_do_promote, task 7.2) or a
|
|
* heartbeat carries a learned value from an actual primary (task 7.3) --
|
|
* unlike `term`, which can legitimately bump on mere vote/candidacy
|
|
* activity (raft term adoption) before any primary is actually elected,
|
|
* so `term == 0` would be the wrong gate. electedTerm is DERIVED
|
|
* client-side from the existing `term` field (no new wire field,
|
|
* design.md Locked Decision 1); electedAt renders `elected_at_ms` on the
|
|
* same clock/format as responderTime below, reusing
|
|
* ctrlstatus_format_time_ms (design.md Locked Decision 4). */
|
|
if (status->elected_at_ms == 0) {
|
|
printf("electedTerm: unknown\n");
|
|
printf("electedAt: unknown\n");
|
|
} else {
|
|
char etbuf[64];
|
|
|
|
printf("electedTerm: %llu\n", (unsigned long long)status->term);
|
|
if (ctrlstatus_format_time_ms(status->elected_at_ms, etbuf,
|
|
sizeof(etbuf)) != NULL) {
|
|
printf("electedAt: %s\n", etbuf);
|
|
} else {
|
|
printf("electedAt: unknown\n");
|
|
}
|
|
}
|
|
|
|
/* Answering node + role, framed as a property of the responder and kept
|
|
* DISTINCT from the authoritative Primary line above (Req 3.1/3.2/3.3): a
|
|
* transient responder role (electing/initializing) must not be misread as
|
|
* the cluster's primary state. */
|
|
printf("answered by %s (%s)\n",
|
|
NULLSTR(status->responderName) ? "unknown" : status->responderName,
|
|
ctrlstatus_role_str(status->responderRole));
|
|
|
|
printf("Cluster: %s\n",
|
|
NULLSTR(status->clusterName) ? "unknown" : status->clusterName);
|
|
|
|
/* Configured controller topology + count (Req 4.1/4.2). */
|
|
printf("Controllers (%u):\n", status->peerCount);
|
|
for (i = 0; i < status->peers.count; i++) {
|
|
xce_ctrlpeer_t* peer = status->peers.list[i];
|
|
|
|
if (peer == NULL) {
|
|
continue;
|
|
}
|
|
|
|
printf("\t%s\t%s:%u\t%s\n", NULLSTR(peer->name) ? "?" : peer->name,
|
|
NULLSTR(peer->host) ? "?" : peer->host, peer->port,
|
|
ctrlstatus_peer_status_str(peer->status));
|
|
}
|
|
|
|
/* Config version + committed config version (Req 5.1) and the content
|
|
* checksum as the hex string the controller already exposes (Req 5.2). */
|
|
printf("configVersion: %llu\n", (unsigned long long)status->configVersion);
|
|
printf("committedConfigVersion: %llu\n",
|
|
(unsigned long long)status->committedConfigVersion);
|
|
printf("configChecksum: %s\n",
|
|
NULLSTR(status->configChecksum) ? "unknown" : status->configChecksum);
|
|
|
|
/* HA health / progress (Req 6.x). */
|
|
printf("term: %llu\n", (unsigned long long)status->term);
|
|
printf("committedLsn: %llu\n", (unsigned long long)status->committedLsn);
|
|
printf("appliedLsn: %llu\n", (unsigned long long)status->appliedLsn);
|
|
printf("fenced: %s\n", status->fenced ? "true" : "false");
|
|
|
|
/* Responder snapshot timestamp (Req 11.2/11.3): render the responder's
|
|
* sample-time wall clock (epoch ms) in the SAME local-time
|
|
* "YYYY-MM-DD HH:MM:SS,mmm" form the controller logger emits, so it can be
|
|
* compared directly against ctrl-N.log. */
|
|
{
|
|
char tbuf[64];
|
|
if (ctrlstatus_format_time_ms(status->responder_time_ms, tbuf,
|
|
sizeof(tbuf)) != NULL) {
|
|
printf("responderTime: %s\n", tbuf);
|
|
} else {
|
|
printf("responderTime: unknown\n");
|
|
}
|
|
}
|
|
|
|
xcecli_free_ctrlstatus(status);
|
|
|
|
return 0;
|
|
}
|
|
|
|
static void print_jobinfo(xce_job_t* job)
|
|
{
|
|
char idstring[128];
|
|
|
|
if (job == NULL) {
|
|
return;
|
|
}
|
|
|
|
fmt_jobid(idstring, sizeof(idstring) - 1, job->jobid);
|
|
printf("JobID: %s\n", idstring);
|
|
printf("state: %s\n", job_state_str(job->state));
|
|
/* raft-apply-decoupling task 7.2: job_state_str's if/else-if chain checks
|
|
* JOB_STAT_RUN before JOB_STAT_EXIT, so a job that is simultaneously RUN
|
|
* and EXIT|PERR (e.g. a worker-reported FAIL received without a prior
|
|
* FINISH — apply_job_fail_transition ORs in EXIT|PERR without clearing
|
|
* RUN) prints as "RUN", silently hiding the EXIT/PERR bits from the
|
|
* pretty string. Print the raw bitmask too so a scriptable caller (the
|
|
* xce-test-ha factstream regression scenario's phantom-fact check) can
|
|
* test for a specific bit's presence unambiguously instead of relying on
|
|
* the display string's priority ordering. */
|
|
printf("state_raw: 0x%x\n", job->state);
|
|
printf("Queue: %s\n", job->submit->queue);
|
|
printf("User: %s\n", job->submit->user);
|
|
printf("Slots: %d,%d\n", job->submit->mintasks, job->submit->maxtasks);
|
|
printf("Command: %s\n", job->submit->start_info->cmdline);
|
|
|
|
if (job->pend_reasons != NULL) {
|
|
printf("Pend Reason:\n");
|
|
if (job->pend_reasons->main_reason != 0) {
|
|
printf("\tmain Reason: %d\n", job->pend_reasons->main_reason);
|
|
} else {
|
|
if (job->pend_reasons->nhost_reasons > 0) {
|
|
int i;
|
|
|
|
printf("\tHost Reasons:\n");
|
|
for (i = 0; i < job->pend_reasons->nhost_reasons; i++) {
|
|
printf("\t\t(%s, %d)\n",
|
|
job->pend_reasons->host_based_reasons[i]->host,
|
|
job->pend_reasons->host_based_reasons[i]->reason);
|
|
}
|
|
}
|
|
|
|
if (job->pend_reasons->nres_reasons > 0) {
|
|
int i;
|
|
|
|
printf("\tResource Reasons:\n");
|
|
for (i = 0; i < job->pend_reasons->nres_reasons; i++) {
|
|
printf("\t\t(%s, %s, %d)\n",
|
|
job->pend_reasons->res_based_reasons[i]->resource,
|
|
job->pend_reasons->res_based_reasons[i]->host,
|
|
job->pend_reasons->res_based_reasons[i]->reason);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
static const flagspec_t jobinfo_flags[] = {
|
|
{ 'j', "jobid", "jobid", required_argument, 1, NULL },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
uint64_t jobid;
|
|
xce_job_t* job;
|
|
int rc;
|
|
} jobinfo_call_t;
|
|
|
|
static int jobinfo_thunk(void* ctxp)
|
|
{
|
|
jobinfo_call_t* call = (jobinfo_call_t*)ctxp;
|
|
|
|
call->job = NULL;
|
|
call->rc = xcecli_jobinfo(call->conn, call->jobid, &call->job);
|
|
|
|
return call->rc;
|
|
}
|
|
|
|
int cmd_jobinfo(int argc, char* argv[])
|
|
{
|
|
int opt;
|
|
int ngiven = 0; /* -j values given */
|
|
int nfound = 0; /* lookups that succeeded */
|
|
int lastrc = -EINVAL;
|
|
struct option long_opts[JOBINFO_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(jobinfo_flags, JOBINFO_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: jobinfo: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
uint64_t jobid;
|
|
xce_job_t* job;
|
|
int rc = 0;
|
|
|
|
switch (opt) {
|
|
case 'j':
|
|
ngiven++;
|
|
parse_jobid(&jobid, optarg);
|
|
if (jobid == 0) {
|
|
/* bug-1709: a malformed jobid used to be skipped silently */
|
|
fprintf(stderr, "jobinfo: invalid jobid '%s'\n", optarg);
|
|
lastrc = -EINVAL;
|
|
continue;
|
|
}
|
|
|
|
{
|
|
jobinfo_call_t* call
|
|
= (jobinfo_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->jobid = jobid;
|
|
|
|
rc = xcetest_run_interruptible(jobinfo_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Requirement 1.3: abandon and fail the WHOLE command
|
|
* immediately, rather than continuing on to any
|
|
* further -j flags on this same invocation. Ownership
|
|
* of `call` has passed to the still-running worker
|
|
* thread (interruptcall.h) -- must not touch it
|
|
* again. */
|
|
errno = EINTR;
|
|
perror("xcecli_jobinfo");
|
|
return rc;
|
|
}
|
|
|
|
job = call->job;
|
|
free(call);
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xcecli_jobinfo", rc);
|
|
lastrc = rc;
|
|
continue;
|
|
}
|
|
|
|
print_jobinfo(job);
|
|
free_Job_ptr(&job);
|
|
nfound++;
|
|
}
|
|
|
|
break;
|
|
case '?':
|
|
return flagspec_reject("jobinfo", jobinfo_flags, JOBINFO_NFLAGS,
|
|
argv[optind - 1],
|
|
"jobinfo -j <jobid> [-j <jobid> ...]");
|
|
}
|
|
}
|
|
|
|
/* bug-1709: fail when there was nothing to look up or every lookup
|
|
* failed, so a script sees it in the exit code */
|
|
if (ngiven == 0) {
|
|
fprintf(stderr, "jobinfo: no jobid given (use -j <jobid>)\n");
|
|
return -EINVAL;
|
|
}
|
|
return (nfound > 0) ? 0 : lastrc;
|
|
}
|
|
|
|
static int fill_job_alloc(xce_job_alloc_t* jalloc, int nhosts,
|
|
host_task_t* hosts)
|
|
{
|
|
xce_host_alloc_t** halloc = NULL;
|
|
int i;
|
|
|
|
halloc = (xce_host_alloc_t**)calloc(nhosts, sizeof(xce_host_alloc_t*));
|
|
if (halloc == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
for (i = 0; i < nhosts; i++) {
|
|
xce_alloc_t* alloc;
|
|
xce_alloc_t** allocs;
|
|
|
|
alloc = (xce_alloc_t*)calloc(1, sizeof(*alloc));
|
|
allocs = (xce_alloc_t**)calloc(1, sizeof(*allocs));
|
|
halloc[i] = (xce_host_alloc_t*)calloc(1, sizeof(xce_host_alloc_t));
|
|
if ((halloc[i] == NULL) || (alloc == NULL) || (allocs == NULL)) {
|
|
/* FIXME: handle error */
|
|
return -ENOMEM;
|
|
}
|
|
|
|
alloc->resname = "slots";
|
|
alloc->reserve = "";
|
|
alloc->instid = -1;
|
|
alloc->value = (float)hosts[i].ntasks;
|
|
alloc->from = time(NULL);
|
|
alloc->duration = -1;
|
|
|
|
allocs[0] = alloc;
|
|
|
|
halloc[i]->nallocs = 1;
|
|
halloc[i]->allocs = allocs;
|
|
halloc[i]->host = hosts[i].host;
|
|
}
|
|
|
|
jalloc->nhosts = nhosts;
|
|
jalloc->allocs = halloc;
|
|
|
|
return 0;
|
|
}
|
|
static const flagspec_t jobrun_flags[] = {
|
|
{ 'm', "host", "host", required_argument, 0, "1 task; repeatable up to 5" },
|
|
{ 'j', "jobid", "jobid", required_argument, 1, NULL },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_job_dispatch_t jdispatch;
|
|
xce_job_alloc_t jalloc;
|
|
xce_job_alloc_t* allocs_arr[1];
|
|
int rc;
|
|
} jobrun_call_t;
|
|
|
|
static int jobrun_thunk(void* ctxp)
|
|
{
|
|
jobrun_call_t* call = (jobrun_call_t*)ctxp;
|
|
|
|
call->rc = xcecli_jobdispatch(call->conn, &call->jdispatch);
|
|
|
|
return call->rc;
|
|
}
|
|
|
|
int cmd_jobrun(int argc, char* argv[])
|
|
{
|
|
int rc;
|
|
int opt;
|
|
uint64_t jobid = 0;
|
|
host_task_t hosts[MAX_TASK_HOSTS];
|
|
int nhosts = 0;
|
|
char* host;
|
|
int count;
|
|
struct option long_opts[JOBRUN_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(jobrun_flags, JOBRUN_NFLAGS, long_opts, optstring,
|
|
sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: jobrun: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'm':
|
|
if (nhosts >= MAX_TASK_HOSTS) {
|
|
fprintf(stderr,
|
|
"too many run hosts, we only support max %d hosts\n",
|
|
MAX_TASK_HOSTS);
|
|
return -1;
|
|
}
|
|
|
|
{
|
|
char buf[256];
|
|
int prc = cmdutil_split_host_tasks(optarg, buf, sizeof(buf),
|
|
&count);
|
|
|
|
/* bug-1702: the host segment used to be strncpy'd into buf
|
|
* with no length check */
|
|
if (prc == -ENAMETOOLONG) {
|
|
fprintf(stderr,
|
|
"jobrun: host name in -m '%.32s...' is longer "
|
|
"than %zu characters\n",
|
|
optarg, sizeof(buf) - 1);
|
|
return -EINVAL;
|
|
}
|
|
if (prc < 0) {
|
|
fprintf(stderr,
|
|
"jobrun: -m expects <host>[:<ntasks>] with "
|
|
"ntasks >= 1, got '%s'\n",
|
|
optarg);
|
|
return -EINVAL;
|
|
}
|
|
host = strdup(buf);
|
|
}
|
|
|
|
hosts[nhosts].host = host;
|
|
hosts[nhosts].ntasks = count;
|
|
nhosts++;
|
|
break;
|
|
case 'j':
|
|
parse_jobid(&jobid, optarg);
|
|
if (jobid == 0) {
|
|
fprintf(stderr, "you need to specify a positive jobid with -j");
|
|
return -1;
|
|
}
|
|
break;
|
|
case '?':
|
|
return flagspec_reject(
|
|
"jobrun", jobrun_flags, JOBRUN_NFLAGS, argv[optind - 1],
|
|
"jobrun -j <jobid> [-m <host>[:<ntasks>]] [-m ...]");
|
|
}
|
|
}
|
|
|
|
if (jobid == 0) {
|
|
fprintf(stderr, "no jobid specified");
|
|
return -1;
|
|
}
|
|
|
|
{
|
|
jobrun_call_t* call = (jobrun_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
|
|
fill_job_alloc(&call->jalloc, nhosts, &hosts[0]);
|
|
call->jalloc.jobid = jobid;
|
|
call->allocs_arr[0] = &call->jalloc;
|
|
call->jdispatch.njobs = 1;
|
|
call->jdispatch.allocs = call->allocs_arr;
|
|
|
|
rc = xcetest_run_interruptible(jobrun_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Same mechanism/label as this function's own failure path
|
|
* below. */
|
|
SET_ERROR(EINTR);
|
|
perror("xce_jobdispatch");
|
|
return rc;
|
|
}
|
|
|
|
free(call);
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xce_jobdispatch", rc);
|
|
return rc;
|
|
}
|
|
|
|
#if 0
|
|
printf("job %s will be started on host: %s\n",
|
|
argv[1], halloc.host);
|
|
#endif
|
|
|
|
return 0;
|
|
}
|
|
|
|
static const flagspec_t jobsub_flags[] = {
|
|
{ 'u', "user", "user", required_argument, 1, NULL },
|
|
{ 'q', "queue", "queue", required_argument, 1, NULL },
|
|
{ 'R', "resreq", "resreq", required_argument, 0,
|
|
"\"type == any\"" },
|
|
{ 'c', "command-line", "command-line", required_argument, 1, NULL },
|
|
{ 'n', "ntasks", "ntasks", required_argument, 0, "1" },
|
|
{ 'x', "exdata", "exdata", required_argument, 0, NULL },
|
|
{ 'a', "array-size", "arraysize", required_argument, 0, "1" },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
job_submit_data_t submit;
|
|
xce_job_submit_t jobsubmit;
|
|
uint64_t jobid;
|
|
int rc;
|
|
} jobsub_call_t;
|
|
|
|
static int jobsub_thunk(void* ctxp)
|
|
{
|
|
jobsub_call_t* call = (jobsub_call_t*)ctxp;
|
|
|
|
call->rc = xcecli_jobsubmit(call->conn, &call->jobsubmit, &call->jobid);
|
|
|
|
return call->rc;
|
|
}
|
|
|
|
int cmd_jobsub(int argc, char* argv[])
|
|
{
|
|
int rc = 0;
|
|
job_submit_data_t submit;
|
|
xce_job_submit_t jobsubmit = { .submit = &submit };
|
|
int opt;
|
|
/* Initialize so a "rc >= 0 but *jobid never written" outcome (a non-clean
|
|
* ack such as a redirect/zero-row/not-ready surfaced as a non-negative rc)
|
|
* is detectable as an invalid (zero) JobId rather than printed as garbage
|
|
* (bug-786). */
|
|
uint64_t jobid = 0;
|
|
uint32_t min = 1, max = 1;
|
|
uint32_t arsize = 1;
|
|
char jobidStr[128];
|
|
struct option long_opts[JOBSUB_NFLAGS + 1];
|
|
char optstring[32];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
memset(&submit, 0x0, sizeof(submit));
|
|
submit.start_info
|
|
= (job_startinfo_t*)calloc(1, sizeof(*(submit.start_info)));
|
|
if (submit.start_info == NULL) {
|
|
perror("calloc");
|
|
return -ENOMEM;
|
|
}
|
|
|
|
if (flagspec_build_opts(jobsub_flags, JOBSUB_NFLAGS, long_opts, optstring,
|
|
sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: jobsub: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'u':
|
|
submit.user = strdup(optarg);
|
|
break;
|
|
case 'q':
|
|
submit.queue = strdup(optarg);
|
|
break;
|
|
case 'R':
|
|
submit.resreq = strdup(optarg);
|
|
break;
|
|
case 'c':
|
|
submit.start_info->cmdline = strdup(optarg);
|
|
break;
|
|
case 'x':
|
|
FREEUP(submit.start_info->exdata);
|
|
submit.start_info->exdata = strdup(optarg);
|
|
break;
|
|
case 'n':
|
|
/* bug-1703: the min part used to be strncpy'd into a 64-byte
|
|
* buffer with no length check */
|
|
if (cmdutil_parse_ntasks(optarg, &min, &max) < 0) {
|
|
fprintf(stderr,
|
|
"jobsub: -n expects <min>[,<max>] with 1 <= min <= "
|
|
"max, got '%.40s'\n",
|
|
optarg);
|
|
rc = -EINVAL;
|
|
goto cleanup;
|
|
}
|
|
break;
|
|
case 'a':
|
|
arsize = atoi(optarg);
|
|
if (arsize <= 0) {
|
|
arsize = 1;
|
|
}
|
|
break;
|
|
case '?':
|
|
rc = flagspec_reject(
|
|
"jobsub", jobsub_flags, JOBSUB_NFLAGS, argv[optind - 1],
|
|
"jobsub -u <user> -q <queue> -c <cmdline> [-R <resreq>] "
|
|
"[-n <min>[,<max>]] [-x <exdata>] [-a <arraysize>]");
|
|
goto cleanup;
|
|
}
|
|
}
|
|
|
|
if (NULLSTR(submit.user)) {
|
|
fprintf(stderr, "you need to specify job user with -u\n");
|
|
rc = -EINVAL;
|
|
goto cleanup;
|
|
}
|
|
|
|
if (NULLSTR(submit.queue)) {
|
|
fprintf(stderr, "you need to specify job queue with -q\n");
|
|
rc = -EINVAL;
|
|
goto cleanup;
|
|
}
|
|
|
|
if (NULLSTR(submit.start_info->cmdline)) {
|
|
fprintf(stderr, "you need to specify job command with -c\n");
|
|
rc = -EINVAL;
|
|
goto cleanup;
|
|
}
|
|
|
|
if (NULLSTR(submit.resreq)) {
|
|
submit.resreq = strdup("type == any");
|
|
}
|
|
|
|
submit.mintasks = min;
|
|
submit.maxtasks = max;
|
|
|
|
/* apiserverId=0 here let two unrelated jobs collide on the same jobid
|
|
* whenever the harness spawns a fresh xce_test process within the same
|
|
* wall-clock second (idgen_ctx_init resets sequence to 0 each time,
|
|
* and ts/tag/seq alone were then identical for both processes' first
|
|
* submission). Production apiserver instances never hit this: each
|
|
* holds one process-lifetime idgen_ctx_t (clib.go's sync.Once) and is
|
|
* assigned its own distinct apiserverId, so the jobid namespace is
|
|
* already partitioned by config.h's bit layout. Folding this process's
|
|
* pid into apiserverId gives each xce_test invocation its own slice of
|
|
* that same partitioning, making a same-second cross-process collision
|
|
* structurally impossible here too. */
|
|
idgen(&idctx, 0, (size_t)((uint64_t)getpid() & MAX_APISERVER_ID), arsize,
|
|
&(jobsubmit.jobid));
|
|
|
|
if (NULLSTR(submit.start_info->execuser)) {
|
|
submit.start_info->execuser = strdup(submit.user);
|
|
}
|
|
|
|
if (NULLSTR(submit.start_info->exechome)) {
|
|
submit.start_info->exechome = strdup("/tmp");
|
|
}
|
|
|
|
if (NULLSTR(submit.start_info->execcwd)) {
|
|
submit.start_info->execcwd = strdup("/tmp");
|
|
}
|
|
|
|
/* bug-1705: only when -x did not give one -- this used to overwrite
|
|
* (and leak) -x's exdata unconditionally */
|
|
if (cmdutil_default_exdata(&submit.start_info->exdata, getenv("PATH")) < 0) {
|
|
rc = -ENOMEM;
|
|
goto cleanup;
|
|
}
|
|
|
|
jobsubmit.array_size = arsize;
|
|
|
|
{
|
|
jobsub_call_t* call = (jobsub_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
rc = -ENOMEM;
|
|
goto cleanup;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->submit = submit;
|
|
call->jobsubmit = jobsubmit;
|
|
call->jobsubmit.submit = &call->submit;
|
|
call->jobid = 0;
|
|
|
|
rc = xcetest_run_interruptible(jobsub_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Ownership of `call` -- including the copy of `submit` (and
|
|
* the SAME heap-owned user/queue/resreq/start_info->* strings
|
|
* `submit` itself still references, by pointer value) embedded
|
|
* in it -- has passed to the still-running worker thread
|
|
* (interruptcall.h). Returning directly here, rather than via
|
|
* `goto cleanup`, is deliberate: the `cleanup:` label below
|
|
* frees those exact strings through the ORIGINAL `submit`
|
|
* local, which would race the worker's still-in-flight read
|
|
* of the SAME allocations through `call->submit`. Leaking
|
|
* `call` and every string it points to on this path is the
|
|
* same documented trade-off every other call site in this
|
|
* file makes on an abandoned call. */
|
|
errno = EINTR;
|
|
perror("jobsub");
|
|
return rc;
|
|
}
|
|
|
|
jobid = call->jobid;
|
|
free(call);
|
|
}
|
|
|
|
cleanup:
|
|
FREEUP(submit.user);
|
|
FREEUP(submit.queue);
|
|
FREEUP(submit.resreq);
|
|
FREEUP(submit.start_info->cmdline);
|
|
FREEUP(submit.start_info->exdata);
|
|
FREEUP(submit.start_info->execuser);
|
|
FREEUP(submit.start_info->exechome);
|
|
FREEUP(submit.start_info->execcwd);
|
|
FREEUP(submit.start_info);
|
|
|
|
/* Report success only on a clean ack that echoes back the EXACT JobId we
|
|
* submitted (bug-786, Req 7.1/7.2/7.3). xce_jobsubmit() writes *jobid only
|
|
* when the reply status is >= 0; XCECLI_FAILOVER_CALL surfaces a non-clean
|
|
* outcome as a negative rc (e.g. -E_XCE_NOT_READY on no-primary budget
|
|
* exhaustion, -E_XCE_REDIRECT / -E_XCE_NOMASTER on a standby, -ETIMEDOUT /
|
|
* -ENOTCONN on transport loss).
|
|
*
|
|
* The `jobid != 0` gate alone is NOT sufficient: on a failover, libxce's
|
|
* handle_sub_reply() returns -1 on a non-jobid reply but LEAVES the stale
|
|
* previous value in ctx->reply.u.jobid, and xceapi.c copies it out whenever
|
|
* status >= 0 — so a rejected submit can surface as rc >= 0 with a NON-ZERO
|
|
* stale-garbage jobid (the live soak captured `63-15--1-1015808[32751]`
|
|
* while the controller logged "handle_jobsubmit fail to lock job" and never
|
|
* committed it). We therefore require the returned jobid to EQUAL the JobId
|
|
* we generated locally and submitted: on a genuine clean ack the controller
|
|
* commits and echoes back exactly submit->jobid (handle_jobsubmit replies
|
|
* reply.data.jobid = job_submit() = base_id = submit->jobid; verified
|
|
* round-trip). `jobid != jobsubmit.jobid` covers BOTH the never-written
|
|
* (zero) case AND the stale non-zero garbage case in one uint64 compare. */
|
|
if (rc < 0) {
|
|
char ebuf[128];
|
|
|
|
ebuf[0] = '\0';
|
|
xce_strerror_r(-rc, ebuf, sizeof(ebuf));
|
|
SET_ERROR(-rc);
|
|
|
|
switch (-rc) {
|
|
case E_XCE_NOMASTER:
|
|
fprintf(stderr, "jobsubmit failed: no-master (%s)\n", ebuf);
|
|
break;
|
|
case E_XCE_REDIRECT:
|
|
fprintf(stderr, "jobsubmit failed: redirect (%s)\n", ebuf);
|
|
break;
|
|
case E_XCE_NOT_READY:
|
|
fprintf(stderr, "jobsubmit failed: not-ready (%s)\n", ebuf);
|
|
break;
|
|
case ETIMEDOUT:
|
|
fprintf(stderr, "jobsubmit failed: timeout (%s)\n", ebuf);
|
|
break;
|
|
default:
|
|
fprintf(stderr, "jobsubmit failed: %s\n", ebuf);
|
|
break;
|
|
}
|
|
|
|
return rc;
|
|
}
|
|
|
|
if (jobid != jobsubmit.jobid) {
|
|
/* Non-negative rc but the returned JobId does not match the one we
|
|
* submitted: not a clean ack. This rejects BOTH the never-written
|
|
* (jobid == 0) case AND the stale-garbage non-zero case left in the
|
|
* libxce reply union on a failover-rejected submit. Never print "ok"
|
|
* with an uninitialized/garbage id (Req 7.3). */
|
|
char wantStr[128];
|
|
char gotStr[128];
|
|
|
|
fmt_jobid(wantStr, sizeof(wantStr) - 1, jobsubmit.jobid);
|
|
fmt_jobid(gotStr, sizeof(gotStr) - 1, jobid);
|
|
fprintf(stderr,
|
|
"jobsubmit failed: no clean ack (controller did not confirm "
|
|
"the submitted JobId: wanted %s, got %s)\n",
|
|
wantStr, gotStr);
|
|
return -E_XCE_INTERNAL;
|
|
}
|
|
|
|
fmt_jobid(jobidStr, sizeof(jobidStr) - 1, array_jobid(jobid));
|
|
|
|
/* Capture $jobid (Req 3.2) using the SAME representation cmd_joblist prints
|
|
* for a job's JobId column — fmt_jobid(jobs[i]->jobid), the full jobid
|
|
* WITHOUT array_jobid() stripping — so a later `$jobid` substitution and
|
|
* `expect-job $jobid` compare equal to the joblist entry for this job. Only
|
|
* set on this clean-ack + valid-id path; never on a failure path, so an
|
|
* unset $jobid still fails per Req 3.6. */
|
|
fmt_jobid(g_last_jobid, sizeof(g_last_jobid) - 1, jobid);
|
|
|
|
printf("jobsubmit ok: JobId: %s\n", jobidStr);
|
|
|
|
return 0;
|
|
}
|
|
|
|
/*
|
|
* Resource names whose canonical stored/wire value is raw bytes (not a
|
|
* generic schema-driven unit system -- deliberately kept as a small,
|
|
* explicit, client-side list rather than a wire field, since only a
|
|
* handful of resources need this and a wire-level unit hint would have
|
|
* grown every Resource/ResourceVal message regardless of whether it's
|
|
* used). CLI display picks the best-fit human unit adaptively
|
|
* (format_bytes_human); the stored value is never converted. Covers every
|
|
* memory/disk-scale built-in host attribute and load index (maxmem, mem,
|
|
* tmp, maxswap, maxtmp) -- deliberately NOT maxcpus/ndisks (plain counts)
|
|
* or cpuf/r1m/r15s (unitless factors/averages).
|
|
*/
|
|
static int is_bytes_valued_resource(const char* name)
|
|
{
|
|
static const char* const BYTES_VALUED_RESOURCES[]
|
|
= { "maxmem", "mem", "tmp", "maxswap", "maxtmp" };
|
|
const int nres = (int)(sizeof(BYTES_VALUED_RESOURCES)
|
|
/ sizeof(BYTES_VALUED_RESOURCES[0]));
|
|
int i;
|
|
|
|
if (name == NULL) {
|
|
return 0;
|
|
}
|
|
|
|
for (i = 0; i < nres; i++) {
|
|
if (strcmp(name, BYTES_VALUED_RESOURCES[i]) == 0) {
|
|
return 1;
|
|
}
|
|
}
|
|
|
|
return 0;
|
|
}
|
|
|
|
/*
|
|
* Render a raw byte count as a human-readable string (e.g. "30.90 GB"),
|
|
* adaptively picking the smallest unit the value fits in. Only used for
|
|
* display -- the stored/wire value stays raw bytes always, this never
|
|
* feeds back into anything that's compared or persisted. A pure function
|
|
* so it's directly unit-testable without a live client connection.
|
|
*/
|
|
static void format_bytes_human(double bytes, char* buf, size_t bufsize)
|
|
{
|
|
static const char* const UNITS[] = { "B", "KB", "MB", "GB", "TB", "PB" };
|
|
const int nunits = (int)(sizeof(UNITS) / sizeof(UNITS[0]));
|
|
double val = bytes;
|
|
int i = 0;
|
|
|
|
while (val >= 1024.0 && i < nunits - 1) {
|
|
val /= 1024.0;
|
|
i++;
|
|
}
|
|
|
|
snprintf(buf, bufsize, "%.2f %s", val, UNITS[i]);
|
|
}
|
|
|
|
static const flagspec_t resval_flags[] = {
|
|
{ 'n', "resname", "resname", required_argument, 1, NULL },
|
|
{ 'm', "host", "host", required_argument, 0,
|
|
"required unless the resource is cluster-wide" },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
char* host;
|
|
char* res;
|
|
xce_resval_t** vals;
|
|
int rc;
|
|
} resval_call_t;
|
|
|
|
static int resval_thunk(void* ctxp)
|
|
{
|
|
resval_call_t* call = (resval_call_t*)ctxp;
|
|
|
|
call->vals = NULL;
|
|
call->rc = xcecli_resval(call->conn, call->host, call->res, &call->vals);
|
|
|
|
return call->rc;
|
|
}
|
|
|
|
int cmd_resval(int argc, char* argv[])
|
|
{
|
|
char* host = NULL;
|
|
char* res = NULL;
|
|
xce_resval_t** vals = NULL;
|
|
int opt;
|
|
int rc;
|
|
struct option long_opts[RESVAL_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(resval_flags, RESVAL_NFLAGS, long_opts, optstring,
|
|
sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: resval: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'n':
|
|
res = strdup(optarg);
|
|
break;
|
|
case 'm':
|
|
host = strdup(optarg);
|
|
break;
|
|
case '?':
|
|
rc = flagspec_reject("resval", resval_flags, RESVAL_NFLAGS,
|
|
argv[optind - 1],
|
|
"resval -n <resname> [-m <host>]");
|
|
FREEUP(host);
|
|
FREEUP(res);
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
{
|
|
resval_call_t* call = (resval_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
FREEUP(host);
|
|
FREEUP(res);
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->host = host;
|
|
call->res = res;
|
|
|
|
rc = xcetest_run_interruptible(resval_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Ownership of `call` -- including `host`/`res`, still
|
|
* embedded in it by pointer -- has passed to the still-
|
|
* running worker thread (interruptcall.h). Do NOT also
|
|
* FREEUP(host)/FREEUP(res) here: the SAME heap allocations
|
|
* are still referenced by call->host/call->res while that
|
|
* worker may still be reading them. Unlike this function's
|
|
* usual "always return 0" convention, an interrupted wait is
|
|
* reported as a genuine command failure (Requirement 1.3). */
|
|
errno = EINTR;
|
|
perror("resval");
|
|
return rc;
|
|
}
|
|
|
|
vals = call->vals;
|
|
free(call);
|
|
}
|
|
|
|
if (rc > 0) {
|
|
int i;
|
|
|
|
for (i = 0; i < rc; i++) {
|
|
xce_resval_t* val = vals[i];
|
|
|
|
printf("resource %s@%s:\n", val->name,
|
|
(val->host == NULL) ? "cluster" : val->host);
|
|
printf("value: ");
|
|
switch (val->val->which) {
|
|
case Value_u_svalue:
|
|
printf("%s\n", val->val->u.svalue);
|
|
break;
|
|
case Value_u_fvalue:
|
|
if (is_bytes_valued_resource(val->name)) {
|
|
char hbuf[64];
|
|
|
|
format_bytes_human(val->val->u.fvalue, hbuf,
|
|
sizeof(hbuf));
|
|
printf("%f (%s)\n", val->val->u.fvalue, hbuf);
|
|
} else {
|
|
printf("%f\n", val->val->u.fvalue);
|
|
}
|
|
break;
|
|
case Value_u_bvalue:
|
|
printf("%s\n", (val->val->u.bvalue == 0) ? "false" : "true");
|
|
break;
|
|
default:
|
|
printf(" <unknown type>\n");
|
|
break;
|
|
}
|
|
|
|
free_ResourceVal_ptr(&val);
|
|
}
|
|
|
|
FREEUP(vals);
|
|
} else {
|
|
printf("no resource value found in given host\n");
|
|
}
|
|
|
|
FREEUP(host);
|
|
FREEUP(res);
|
|
|
|
return 0;
|
|
}
|
|
|
|
static const flagspec_t jobkill_flags[] = {
|
|
{ 's', "signal", "signal", required_argument, 0, "SIGTERM" },
|
|
{ 'j', "jobid", "jobid", required_argument, 1, NULL },
|
|
/* xce-test-cli-ux task 3.3, Requirement 9.3: per-command confirmation
|
|
* bypass -- jobkill is unconditionally destructive (Requirement 9.5), so
|
|
* this flag's only effect is to skip the confirm_destructive() prompt
|
|
* below. Same short/long spelling used across all five destructive
|
|
* commands (Requirement 6.2 cross-command consistency). */
|
|
{ 'y', "yes", "yes", no_argument, 0, NULL },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_jobctrl_t jobctrl;
|
|
xce_signal_t siginfo;
|
|
} jobkill_call_t;
|
|
|
|
static int jobkill_thunk(void* ctxp)
|
|
{
|
|
jobkill_call_t* call = (jobkill_call_t*)ctxp;
|
|
int rc = xcecli_jobctrl(call->conn, &call->jobctrl);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_jobkill(int argc, char* argv[])
|
|
{
|
|
int signo;
|
|
uint64_t jobid = 0;
|
|
int opt;
|
|
int rc;
|
|
int yes = 0;
|
|
xce_jobctrl_t jobctrl;
|
|
xce_signal_t siginfo;
|
|
char jobidStr[64];
|
|
struct option long_opts[JOBKILL_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
signo = SIGTERM;
|
|
|
|
if (flagspec_build_opts(jobkill_flags, JOBKILL_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: jobkill: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 's': {
|
|
long n;
|
|
|
|
/* bug-1708: a non-numeric value became signal 0 and the kill
|
|
* was still sent and reported */
|
|
if (cmdutil_parse_long(optarg, 1, NSIG - 1, &n) < 0) {
|
|
fprintf(stderr,
|
|
"jobkill: -s expects a signal number 1-%d, got '%s'\n",
|
|
NSIG - 1, optarg);
|
|
return -EINVAL;
|
|
}
|
|
signo = (int)n;
|
|
break;
|
|
}
|
|
case 'j':
|
|
parse_jobid(&jobid, optarg);
|
|
break;
|
|
case 'y':
|
|
yes = 1;
|
|
break;
|
|
case '?':
|
|
return flagspec_reject("jobkill", jobkill_flags, JOBKILL_NFLAGS,
|
|
argv[optind - 1],
|
|
"jobkill -j <jobid> [-s <signo>]");
|
|
}
|
|
}
|
|
|
|
if (jobid == 0) {
|
|
fprintf(stderr, "you need to specify jobid with -j\n");
|
|
return -1;
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.3, Requirement 9.5: jobkill is unconditionally
|
|
* destructive -- no action-dependent check like qctrl/hostctrl below.
|
|
* confirm_destructive() already owns the session-toggle and TTY checks
|
|
* internally (confirm.h); this call site's only job is to short-circuit
|
|
* around it when jobkill's OWN --yes flag was given (confirm_destructive
|
|
* has no way to know about a per-command flag). A 0 return means the
|
|
* operator declined (or Ctrl-C'd the prompt, Requirement 9.9) -- report
|
|
* "not run" (a plain, non-negative return so the session verdict
|
|
* machinery in preprocess_and_run treats this as success, not failure --
|
|
* see confirm.h's return-contract note), distinct from a "failed" RPC. */
|
|
fmt_jobid(jobidStr, sizeof(jobidStr) - 1, jobid);
|
|
|
|
if (!yes) {
|
|
char desc[96];
|
|
|
|
snprintf(desc, sizeof(desc), "kill job %s", jobidStr);
|
|
if (!confirm_destructive(desc)) {
|
|
fprintf(stderr, "jobkill: not run (declined)\n");
|
|
return 0;
|
|
}
|
|
}
|
|
|
|
jobctrl.action = JobControl_u_siginfo;
|
|
jobctrl.jobid = jobid;
|
|
jobctrl.u.siginfo = &siginfo;
|
|
siginfo.signo = signo;
|
|
siginfo.force = 1;
|
|
|
|
{
|
|
jobkill_call_t* call = (jobkill_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->jobctrl = jobctrl;
|
|
call->siginfo = siginfo;
|
|
call->jobctrl.u.siginfo = &call->siginfo;
|
|
|
|
rc = xcetest_run_interruptible(jobkill_thunk, call);
|
|
if (rc == -EINTR) {
|
|
errno = EINTR;
|
|
perror("jobkill");
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xcecli_jobctrl", rc);
|
|
return rc;
|
|
}
|
|
|
|
printf("Job %s signaled\n", jobidStr);
|
|
|
|
return 0;
|
|
}
|
|
|
|
static const flagspec_t jobckpt_flags[] = {
|
|
{ 'k', "kill", "kill", no_argument, 0,
|
|
NULL },
|
|
{ 'j', "jobid", "jobid", required_argument, 1,
|
|
NULL },
|
|
{ 'm', "checkpoint-method", "checkpoint-method", required_argument, 0,
|
|
"\"default\"" },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_jobctrl_t jobctrl;
|
|
xce_ckpt_options_t ckptoption;
|
|
char* method; /* owned copy: -m's ckptoption.method may
|
|
* alias optarg (argv[]), which execute_line()
|
|
* frees right after this command returns --
|
|
* see hello_call_t's comment above for the
|
|
* same hazard/fix applied here. */
|
|
} jobckpt_call_t;
|
|
|
|
static int jobckpt_thunk(void* ctxp)
|
|
{
|
|
jobckpt_call_t* call = (jobckpt_call_t*)ctxp;
|
|
int rc = xcecli_jobctrl(call->conn, &call->jobctrl);
|
|
|
|
free(call->method);
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_jobckpt(int argc, char* argv[])
|
|
{
|
|
uint64_t jobid = 0;
|
|
int opt;
|
|
int rc;
|
|
xce_jobctrl_t jobctrl;
|
|
xce_ckpt_options_t ckptoption;
|
|
char jobidStr[64];
|
|
struct option long_opts[JOBCKPT_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 1;
|
|
opterr = 0;
|
|
ckptoption.kill = 0;
|
|
|
|
if (flagspec_build_opts(jobckpt_flags, JOBCKPT_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: jobckpt: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'k':
|
|
ckptoption.kill = 1;
|
|
break;
|
|
case 'j':
|
|
parse_jobid(&jobid, optarg);
|
|
break;
|
|
case 'm':
|
|
ckptoption.method = optarg;
|
|
break;
|
|
case '?':
|
|
return flagspec_reject("jobckpt", jobckpt_flags, JOBCKPT_NFLAGS,
|
|
argv[optind - 1],
|
|
"jobckpt -j <jobid> [-k] [-m <method>]");
|
|
}
|
|
}
|
|
|
|
if (jobid == 0) {
|
|
fprintf(stderr, "you need to specify jobid with -j\n");
|
|
return -1;
|
|
}
|
|
|
|
if (ckptoption.method == NULL) {
|
|
ckptoption.method = "default";
|
|
}
|
|
jobctrl.action = JobControl_u_ckptoption;
|
|
jobctrl.jobid = jobid;
|
|
jobctrl.u.ckptoption = &ckptoption;
|
|
|
|
{
|
|
jobckpt_call_t* call = (jobckpt_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->jobctrl = jobctrl;
|
|
call->ckptoption = ckptoption;
|
|
call->method = strdup(ckptoption.method);
|
|
call->ckptoption.method = call->method;
|
|
call->jobctrl.u.ckptoption = &call->ckptoption;
|
|
|
|
rc = xcetest_run_interruptible(jobckpt_thunk, call);
|
|
if (rc == -EINTR) {
|
|
errno = EINTR;
|
|
perror("jobckpt");
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xcecli_jobctrl", rc);
|
|
return rc;
|
|
}
|
|
|
|
fmt_jobid(jobidStr, sizeof(jobidStr) - 1, jobid);
|
|
printf("Job %s checkpointed\n", jobidStr);
|
|
|
|
return 0;
|
|
}
|
|
|
|
static const flagspec_t qctrl_flags[] = {
|
|
{ 'a', "activate", "activate", no_argument, 0, NULL },
|
|
{ 'i', "deactivate", "deactivate", no_argument, 0, NULL },
|
|
{ 'o', "open", "open", no_argument, 0, NULL },
|
|
{ 'c', "close", "close", no_argument, 0, NULL },
|
|
{ 'q', "queue", "queue", required_argument, 1, NULL },
|
|
/* xce-test-cli-ux task 3.3, Requirement 9.3: confirmation bypass.
|
|
* qctrl is destructive ONLY for -i/--deactivate and -c/--close
|
|
* (Requirement 9.5); -a/--activate and -o/--open are reversal/
|
|
* reactivation actions and never prompt. Same short/long spelling as
|
|
* the other four destructive commands (Requirement 6.2). */
|
|
{ 'y', "yes", "yes", no_argument, 0, NULL },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_queuectrl_t queuectrl;
|
|
} qctrl_call_t;
|
|
|
|
static int qctrl_thunk(void* ctxp)
|
|
{
|
|
qctrl_call_t* call = (qctrl_call_t*)ctxp;
|
|
int rc = xcecli_queuectrl(call->conn, &call->queuectrl);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_qctrl(int argc, char* argv[])
|
|
{
|
|
int opt;
|
|
int rc;
|
|
xce_queuectrl_t queuectrl;
|
|
/* bug-1704 fix: initialize to the QUEUE_DO_NONE sentinel (xce/types.h)
|
|
* and reject post-loop if it is still unset -- mirrors cmd_hostctrl's
|
|
* existing HOST_DO_NONE sentinel-and-reject pattern immediately below.
|
|
* Before this fix, `action` had no initializer and no post-loop check,
|
|
* so an invocation giving -q but none of -a/-i/-o/-c transmitted
|
|
* indeterminate stack memory as the queue action. */
|
|
int action = QUEUE_DO_NONE;
|
|
char* queue = NULL;
|
|
int yes = 0;
|
|
struct option long_opts[QCTRL_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(qctrl_flags, QCTRL_NFLAGS, long_opts, optstring,
|
|
sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: qctrl: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'a':
|
|
action = QUEUE_ACTIVE;
|
|
break;
|
|
case 'i':
|
|
action = QUEUE_INACTIVE;
|
|
break;
|
|
case 'o':
|
|
action = QUEUE_OPEN;
|
|
break;
|
|
case 'c':
|
|
action = QUEUE_CLOSE;
|
|
break;
|
|
case 'q':
|
|
FREEUP(queue);
|
|
queue = strdup(optarg);
|
|
break;
|
|
case 'y':
|
|
yes = 1;
|
|
break;
|
|
case '?':
|
|
FREEUP(queue);
|
|
return flagspec_reject("qctrl", qctrl_flags, QCTRL_NFLAGS,
|
|
argv[optind - 1],
|
|
"qctrl -q <queue> [-a] [-i] [-o] [-c]");
|
|
}
|
|
}
|
|
|
|
if (queue == NULL) {
|
|
fprintf(stderr, "you need to specify queue name with -q\n");
|
|
return -1;
|
|
}
|
|
|
|
/* bug-1704 fix (continued): reject rather than send an indeterminate
|
|
* action when none of -a/-i/-o/-c (nor their long forms) was given. */
|
|
if (action == QUEUE_DO_NONE) {
|
|
fprintf(stderr,
|
|
"you need to specify one of -a/--activate, -i/--deactivate, "
|
|
"-o/--open, or -c/--close\n");
|
|
free(queue);
|
|
return -1;
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.3, Requirement 9.5: qctrl is destructive ONLY
|
|
* for -i/--deactivate and -c/--close -- -a/--activate and -o/--open are
|
|
* reversal/reactivation actions and never prompt, regardless of short
|
|
* or long spelling. See cmd_jobkill's comment above for why this call
|
|
* site only needs to check its own --yes flag. */
|
|
if (((action == QUEUE_INACTIVE) || (action == QUEUE_CLOSE)) && !yes) {
|
|
char desc[160];
|
|
const char* verb = (action == QUEUE_CLOSE) ? "close" : "deactivate";
|
|
|
|
snprintf(desc, sizeof(desc), "%s queue %s", verb, queue);
|
|
if (!confirm_destructive(desc)) {
|
|
fprintf(stderr, "qctrl: not run (declined)\n");
|
|
free(queue);
|
|
return 0;
|
|
}
|
|
}
|
|
|
|
queuectrl.action = action;
|
|
queuectrl.queue = queue;
|
|
|
|
{
|
|
qctrl_call_t* call = (qctrl_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
free(queue);
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->queuectrl = queuectrl;
|
|
|
|
rc = xcetest_run_interruptible(qctrl_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Ownership of `call` -- including its embedded copy of
|
|
* `queue`'s pointer value -- has passed to the still-running
|
|
* worker thread. Do NOT also free(queue) here: the SAME
|
|
* allocation is still referenced by call->queuectrl.queue
|
|
* while that worker may still be reading it. */
|
|
errno = EINTR;
|
|
perror("qctrl");
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xcecli_queuectrl", rc);
|
|
free(queue); /* bug-1797: leaked on this path */
|
|
return rc;
|
|
}
|
|
|
|
printf("Operation on queue %s done\n", queue);
|
|
|
|
free(queue);
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* hostctrl -o|-c -h <hostname> — administratively open/close a host
|
|
* (xce_hostctrl_t{action, hostname} via xcecli_hostctrl). Mirrors cmd_qctrl's
|
|
* exact getopt shape; only -o/-c are wired (matching xce_hostctrl_req's own
|
|
* accepted action range, request.c — HOST_TAG/HOST_UNTAG exist in the enum
|
|
* but are rejected -EINVAL server-side today, so no flag is offered for them
|
|
* here). Used by the config-upgrade chaotic soak (task 8.6) to set a host's
|
|
* admin status before/around a config upgrade and assert it survives. */
|
|
static const flagspec_t hostctrl_flags[] = {
|
|
{ 'o', "open", "open", no_argument, 0, NULL },
|
|
{ 'c', "close", "close", no_argument, 0, NULL },
|
|
{ 'h', "host", "host", required_argument, 1, NULL },
|
|
/* xce-test-cli-ux task 3.3, Requirement 9.3: confirmation bypass.
|
|
* hostctrl is destructive ONLY for -c/--close (Requirement 9.5);
|
|
* -o/--open is a reactivation action and never prompts. Same short/long
|
|
* spelling as the other four destructive commands (Requirement 6.2). */
|
|
{ 'y', "yes", "yes", no_argument, 0, NULL },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_hostctrl_t hostctrl;
|
|
} hostctrl_call_t;
|
|
|
|
static int hostctrl_thunk(void* ctxp)
|
|
{
|
|
hostctrl_call_t* call = (hostctrl_call_t*)ctxp;
|
|
int rc = xcecli_hostctrl(call->conn, &call->hostctrl);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_hostctrl(int argc, char* argv[])
|
|
{
|
|
int opt;
|
|
int rc;
|
|
xce_hostctrl_t hostctrl;
|
|
int action = HOST_DO_NONE;
|
|
char* hostname = NULL;
|
|
int yes = 0;
|
|
struct option long_opts[HOSTCTRL_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(hostctrl_flags, HOSTCTRL_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: hostctrl: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'o':
|
|
action = HOST_OPEN;
|
|
break;
|
|
case 'c':
|
|
action = HOST_CLOSE;
|
|
break;
|
|
case 'h':
|
|
FREEUP(hostname);
|
|
hostname = strdup(optarg);
|
|
break;
|
|
case 'y':
|
|
yes = 1;
|
|
break;
|
|
case '?':
|
|
FREEUP(hostname);
|
|
return flagspec_reject("hostctrl", hostctrl_flags,
|
|
HOSTCTRL_NFLAGS, argv[optind - 1],
|
|
"hostctrl -h <hostname> (-o|-c)");
|
|
}
|
|
}
|
|
|
|
if (hostname == NULL) {
|
|
fprintf(stderr, "you need to specify host name with -h\n");
|
|
return -1;
|
|
}
|
|
|
|
if (action == HOST_DO_NONE) {
|
|
fprintf(stderr, "you need to specify -o (open) or -c (close)\n");
|
|
free(hostname);
|
|
return -1;
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.3, Requirement 9.5: hostctrl is destructive
|
|
* ONLY for -c/--close -- -o/--open is a reactivation action and never
|
|
* prompts, regardless of short or long spelling. See cmd_jobkill's
|
|
* comment above for why this call site only needs to check its own
|
|
* --yes flag. */
|
|
if ((action == HOST_CLOSE) && !yes) {
|
|
char desc[160];
|
|
|
|
snprintf(desc, sizeof(desc), "close host %s", hostname);
|
|
if (!confirm_destructive(desc)) {
|
|
fprintf(stderr, "hostctrl: not run (declined)\n");
|
|
free(hostname);
|
|
return 0;
|
|
}
|
|
}
|
|
|
|
hostctrl.action = action;
|
|
hostctrl.hostname = hostname;
|
|
|
|
{
|
|
hostctrl_call_t* call = (hostctrl_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
free(hostname);
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->hostctrl = hostctrl;
|
|
|
|
rc = xcetest_run_interruptible(hostctrl_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Ownership of `call` -- including its embedded copy of
|
|
* `hostname`'s pointer value -- has passed to the still-
|
|
* running worker thread. Do NOT also free(hostname) here. */
|
|
errno = EINTR;
|
|
perror("hostctrl");
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xcecli_hostctrl", rc);
|
|
free(hostname);
|
|
return rc;
|
|
}
|
|
|
|
printf("Operation on host %s done\n", hostname);
|
|
|
|
free(hostname);
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* upgradeconfig <cluster.bin> — submit a configuration upgrade
|
|
* (config-upgrade-snapshot task 8.6). Reads the given capnp-encoded
|
|
* ClusterConfig blob (the same format `xceadmin config build` produces and
|
|
* deploy-ha.sh installs as cluster.bin), computes its SHA-256 (OpenSSL
|
|
* SHA256(), matching config_repl.c's do_upgrade_config_with_bytes transit
|
|
* verification), and drives it through xcecli_upgradeconfig ->
|
|
* xce_upgradeconfig_req_raw (OP_upgradeConfig). Mirrors cmd_snapdiff's
|
|
* bounds-checked file-reading style. */
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
uint8_t sha256[SHA256_DIGEST_LENGTH];
|
|
uint8_t* bytes; /* ownership transferred here from cmd_upgradeconfig's
|
|
* local `bytes` -- the thunk frees it unconditionally,
|
|
* since the caller only needs `rc` back, not `bytes`
|
|
* itself, after the call (Case-A pattern). */
|
|
size_t len;
|
|
} upgradeconfig_call_t;
|
|
|
|
static int upgradeconfig_thunk(void* ctxp)
|
|
{
|
|
upgradeconfig_call_t* call = (upgradeconfig_call_t*)ctxp;
|
|
int rc = xcecli_upgradeconfig(
|
|
call->conn, call->sha256, sizeof(call->sha256), call->bytes,
|
|
call->len);
|
|
|
|
free(call->bytes);
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.3, Requirement 9.3: confirmation bypass.
|
|
* upgradeconfig had no flags at all before this task (task 3.1's getopt
|
|
* migration explicitly left it out of scope for exactly that reason,
|
|
* alongside shutdown) -- this is the FIRST flagspec_t table it gets, added
|
|
* solely to carry --yes; the <cluster.bin> path stays a plain positional
|
|
* argument (argv[optind] once getopt_long has consumed --yes). Same
|
|
* short/long spelling as the other four destructive commands
|
|
* (Requirement 6.2). */
|
|
static const flagspec_t upgradeconfig_flags[] = {
|
|
{ 'y', "yes", "yes", no_argument, 0, NULL },
|
|
};
|
|
|
|
int cmd_upgradeconfig(int argc, char* argv[])
|
|
{
|
|
int opt;
|
|
int yes = 0;
|
|
struct option long_opts[UPGRADECONFIG_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(upgradeconfig_flags, UPGRADECONFIG_NFLAGS,
|
|
long_opts, optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(
|
|
stderr,
|
|
"xce_test: upgradeconfig: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'y':
|
|
yes = 1;
|
|
break;
|
|
case '?':
|
|
return flagspec_reject("upgradeconfig", upgradeconfig_flags,
|
|
UPGRADECONFIG_NFLAGS, argv[optind - 1],
|
|
"upgradeconfig <cluster.bin>");
|
|
}
|
|
}
|
|
|
|
if ((argc - optind) != 1) {
|
|
fprintf(stderr, "usage: upgradeconfig <cluster.bin>\n");
|
|
return -1;
|
|
}
|
|
|
|
const char* path = argv[optind];
|
|
FILE* fp = fopen(path, "rb");
|
|
if (fp == NULL) {
|
|
fprintf(stderr, "upgradeconfig: cannot open %s: %s\n", path,
|
|
strerror(errno));
|
|
return -1;
|
|
}
|
|
|
|
if (fseek(fp, 0, SEEK_END) != 0) {
|
|
fprintf(stderr, "upgradeconfig: fseek failed on %s: %s\n", path,
|
|
strerror(errno));
|
|
fclose(fp);
|
|
return -1;
|
|
}
|
|
|
|
long fsz = ftell(fp);
|
|
if (fsz <= 0) {
|
|
fprintf(stderr, "upgradeconfig: %s is empty or ftell failed\n", path);
|
|
fclose(fp);
|
|
return -1;
|
|
}
|
|
rewind(fp);
|
|
|
|
uint8_t* bytes = (uint8_t*)malloc((size_t)fsz);
|
|
if (bytes == NULL) {
|
|
fprintf(stderr, "upgradeconfig: out of memory reading %s\n", path);
|
|
fclose(fp);
|
|
return -1;
|
|
}
|
|
|
|
size_t nread = fread(bytes, 1, (size_t)fsz, fp);
|
|
fclose(fp);
|
|
if (nread != (size_t)fsz) {
|
|
fprintf(stderr, "upgradeconfig: short read on %s (%zu of %ld bytes)\n",
|
|
path, nread, fsz);
|
|
free(bytes);
|
|
return -1;
|
|
}
|
|
|
|
uint8_t sha256[SHA256_DIGEST_LENGTH];
|
|
SHA256(bytes, (size_t)fsz, sha256);
|
|
|
|
/* xce-test-cli-ux task 3.3, Requirement 9.5: upgradeconfig is
|
|
* unconditionally destructive. See cmd_jobkill's comment above for why
|
|
* this call site only needs to check its own --yes flag. Deliberately
|
|
* placed after the file has already been read/hashed (a purely local,
|
|
* read-only step) and right before the network call itself. */
|
|
if (!yes) {
|
|
char desc[PATH_MAX + 64];
|
|
|
|
snprintf(desc, sizeof(desc),
|
|
"upgrade the cluster configuration from %s", path);
|
|
if (!confirm_destructive(desc)) {
|
|
fprintf(stderr, "upgradeconfig: not run (declined)\n");
|
|
free(bytes);
|
|
return 0;
|
|
}
|
|
}
|
|
|
|
upgradeconfig_call_t* call
|
|
= (upgradeconfig_call_t*)calloc(1, sizeof(*call));
|
|
int rc;
|
|
|
|
if (call == NULL) {
|
|
free(bytes);
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
memcpy(call->sha256, sha256, sizeof(sha256));
|
|
call->bytes = bytes; /* ownership transferred; do not free() locally */
|
|
call->len = (size_t)fsz;
|
|
|
|
rc = xcetest_run_interruptible(upgradeconfig_thunk, call);
|
|
/* -EINTR deliberately falls straight into the SAME generic error
|
|
* report below that every other upgradeconfig failure already uses
|
|
* (Requirement 1.3: "consistent with how any other failing command is
|
|
* reported") -- strerror(-rc) with rc == -EINTR renders the standard
|
|
* "Interrupted system call" text, no special-case branch needed. */
|
|
if (rc < 0) {
|
|
fprintf(stderr, "upgradeconfig: failed (rc=%d): %s\n", rc,
|
|
strerror(-rc));
|
|
return rc;
|
|
}
|
|
|
|
printf("Config upgrade submitted from %s (%ld bytes)\n", path, fsz);
|
|
|
|
return 0;
|
|
}
|
|
|
|
static const flagspec_t runtasks_flags[] = {
|
|
{ 'm', "host", "host", required_argument, 0,
|
|
"1 task; repeatable up to 5" },
|
|
{ 'u', "user", "user", required_argument, 0, NULL },
|
|
{ 'c', "command-line", "command-line", required_argument, 0, NULL },
|
|
{ 'x', "exdata", "exdata", required_argument, 0, NULL },
|
|
{ 'd', "dir", "dir", required_argument, 0, NULL },
|
|
{ 'j', "jobid", "jobid", required_argument, 0, NULL },
|
|
{ 't', "launch-type", "launch-type", required_argument, 0,
|
|
"\"mpi\" if >1 total tasks, else \"sequential\"" },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_taskrun_t taskrun;
|
|
xce_taskhost_t hosts[MAX_TASK_HOSTS];
|
|
xce_taskhost_t* taskhosts_arr[MAX_TASK_HOSTS];
|
|
} runtasks_call_t;
|
|
|
|
static int runtasks_thunk(void* ctxp)
|
|
{
|
|
runtasks_call_t* call = (runtasks_call_t*)ctxp;
|
|
int rc = xcecli_runtask(call->conn, &call->taskrun);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_runtasks(int argc, char* argv[])
|
|
{
|
|
xce_taskhost_t hosts[MAX_TASK_HOSTS];
|
|
size_t nhosts = 0;
|
|
xce_taskrun_t taskrun;
|
|
int opt;
|
|
int rc;
|
|
int count;
|
|
int totaltasks = 0;
|
|
char* user = NULL;
|
|
char* cmdline = NULL;
|
|
char* dir = NULL;
|
|
char* exdata = NULL;
|
|
char* host = NULL;
|
|
char* launch_type = NULL;
|
|
int i;
|
|
uint64_t jobid = 0;
|
|
struct option long_opts[RUNTASKS_NFLAGS + 1];
|
|
char optstring[32];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(runtasks_flags, RUNTASKS_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: taskrun: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'm':
|
|
if (nhosts >= MAX_TASK_HOSTS) {
|
|
fprintf(stderr,
|
|
"too many run hosts, just support %d run hosts\n",
|
|
MAX_TASK_HOSTS);
|
|
return -1;
|
|
}
|
|
|
|
if (strchr(optarg, (int)':') != NULL) {
|
|
char* p;
|
|
char buf[256];
|
|
|
|
p = strchr(optarg, (int)':');
|
|
count = atoi(p + 1);
|
|
memset(buf, 0x0, sizeof(buf));
|
|
strncpy(buf, optarg, (int)(p - optarg));
|
|
host = strdup(buf);
|
|
} else {
|
|
host = strdup(optarg);
|
|
count = 1;
|
|
}
|
|
|
|
hosts[nhosts].host = host;
|
|
hosts[nhosts].ntasks = count;
|
|
nhosts++;
|
|
totaltasks += count;
|
|
break;
|
|
case 'u':
|
|
if (user != NULL) {
|
|
free(user);
|
|
}
|
|
user = strdup(optarg);
|
|
break;
|
|
case 'c':
|
|
if (cmdline != NULL) {
|
|
free(cmdline);
|
|
}
|
|
cmdline = strdup(optarg);
|
|
break;
|
|
case 'x':
|
|
if (exdata != NULL) {
|
|
free(exdata);
|
|
}
|
|
exdata = strdup(optarg);
|
|
break;
|
|
case 'd':
|
|
if (dir != NULL) {
|
|
free(dir);
|
|
}
|
|
dir = strdup(optarg);
|
|
break;
|
|
case 'j':
|
|
parse_jobid(&jobid, optarg);
|
|
break;
|
|
case 't':
|
|
launch_type = strdup(optarg);
|
|
break;
|
|
case '?':
|
|
return flagspec_reject(
|
|
"taskrun", runtasks_flags, RUNTASKS_NFLAGS,
|
|
argv[optind - 1],
|
|
"taskrun [-j <jobid>] [-m <host>[:<n>]]... [-u <user>] "
|
|
"[-c <cmdline>] [-x <exdata>] [-d <dir>] [-t <launch_type>]");
|
|
}
|
|
}
|
|
|
|
for (i = 0; i < nhosts; i++) {
|
|
hosts[i].exechome = dir;
|
|
hosts[i].execcwd = dir;
|
|
hosts[i].execuser = user;
|
|
hosts[i].exdata = exdata;
|
|
hosts[i].cmdline = cmdline;
|
|
hosts[i].launch_type = launch_type;
|
|
if (hosts[i].launch_type == NULL) {
|
|
if (totaltasks > 1) {
|
|
hosts[i].launch_type = strdup("mpi");
|
|
} else {
|
|
hosts[i].launch_type = strdup("sequential");
|
|
}
|
|
}
|
|
}
|
|
|
|
{
|
|
runtasks_call_t* call = (runtasks_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
rc = -ENOMEM;
|
|
} else {
|
|
call->conn = active_ctx->ctx;
|
|
memcpy(call->hosts, hosts, sizeof(xce_taskhost_t) * nhosts);
|
|
|
|
for (i = 0; i < nhosts; i++) {
|
|
call->taskhosts_arr[i] = &call->hosts[i];
|
|
}
|
|
|
|
call->taskrun.taskhosts = call->taskhosts_arr;
|
|
call->taskrun.clientidx = -1;
|
|
call->taskrun.nhosts = nhosts;
|
|
call->taskrun.jobid = jobid;
|
|
|
|
/* On -EINTR, ownership of `call` (and its embedded copies of
|
|
* hosts[]'s dir/user/exdata/cmdline/launch_type pointer
|
|
* values) has passed to the still-running worker thread --
|
|
* runtasks_thunk() always frees `call` itself once fn()
|
|
* actually finishes, on either path, so there is nothing
|
|
* further to free here. */
|
|
rc = xcetest_run_interruptible(runtasks_thunk, call);
|
|
}
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xcecli_runtask", rc);
|
|
} else {
|
|
printf("run task ok\n");
|
|
}
|
|
|
|
return rc;
|
|
}
|
|
|
|
static int add_pendreason(xce_job_preason_t* preason, char* s)
|
|
{
|
|
char s1[MAX_LINE_LEN];
|
|
char* pcode = NULL;
|
|
char* phost = NULL;
|
|
char* pres = NULL;
|
|
char* p;
|
|
int code;
|
|
|
|
if ((preason == NULL) || (s == NULL)) {
|
|
fprintf(stderr, "bad parameter\n");
|
|
return -1;
|
|
}
|
|
|
|
if ((p = strrchr(s, (int)':')) == NULL) {
|
|
pcode = s;
|
|
} else {
|
|
pcode = p + 1;
|
|
memset(s1, 0x0, sizeof(s1));
|
|
strncpy(s1, s, p - s);
|
|
|
|
if ((p = strrchr(s1, (int)':')) == NULL) {
|
|
phost = s1;
|
|
} else {
|
|
*p = 0x0;
|
|
phost = p + 1;
|
|
pres = s1;
|
|
}
|
|
}
|
|
|
|
code = atoi(pcode);
|
|
|
|
return add_job_preason(preason, code, phost, pres);
|
|
}
|
|
|
|
static const flagspec_t setpend_flags[] = {
|
|
{ 'j', "jobid", "jobid", required_argument, 1, NULL },
|
|
{ 'p', "reason", "reason", required_argument, 1, NULL },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_job_preason_t jreasons;
|
|
xce_pendreason_t preason;
|
|
xce_pendreason_t* preasons_arr[1];
|
|
xce_preason_set_t reasons;
|
|
} setpend_call_t;
|
|
|
|
static int setpend_thunk(void* ctxp)
|
|
{
|
|
setpend_call_t* call = (setpend_call_t*)ctxp;
|
|
int rc = xcecli_set_pend_reason(call->conn, &call->reasons);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_setpend(int argc, char* argv[])
|
|
{
|
|
int opt;
|
|
xce_job_preason_t jreasons;
|
|
xce_pendreason_t preason = { .reasons = &jreasons };
|
|
xce_preason_set_t reasons
|
|
= { .nreasons = 1, .reasons = (xce_pendreason_t*[]) { &preason } };
|
|
int rc = 0;
|
|
struct option long_opts[SETPEND_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(setpend_flags, SETPEND_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: setpend: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
int rc = 0;
|
|
switch (opt) {
|
|
case 'j':
|
|
parse_jobid(&(preason.jobid), optarg);
|
|
break;
|
|
case 'p':
|
|
rc = add_pendreason(&jreasons, optarg);
|
|
if (rc < 0) {
|
|
perror("add_pendreason");
|
|
return rc;
|
|
}
|
|
break;
|
|
case '?':
|
|
return flagspec_reject(
|
|
"setpend", setpend_flags, SETPEND_NFLAGS, argv[optind - 1],
|
|
"setpend -j <jobid> -p <reason> [-p <reason> ...]");
|
|
}
|
|
}
|
|
|
|
if (preason.jobid == 0) {
|
|
fprintf(stderr, "no or incorrect jobid specified\n");
|
|
return -1;
|
|
}
|
|
|
|
if ((jreasons.main_reason == 0) && (jreasons.nhost_reasons == 0)
|
|
&& (jreasons.nres_reasons == 0)) {
|
|
fprintf(stderr, "no or incorrect pend reason set\n");
|
|
return -1;
|
|
}
|
|
|
|
{
|
|
setpend_call_t* call = (setpend_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
erase_job_preason(&jreasons);
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->jreasons = jreasons;
|
|
call->preason = preason;
|
|
call->preason.reasons = &call->jreasons;
|
|
call->preasons_arr[0] = &call->preason;
|
|
call->reasons = reasons;
|
|
call->reasons.reasons = call->preasons_arr;
|
|
|
|
rc = xcetest_run_interruptible(setpend_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* Ownership of `call` -- including the copy of `jreasons` (and
|
|
* its heap-owned host_based_reasons[]/res_based_reasons[]
|
|
* entries, still referenced by the SAME pointer values) -- has
|
|
* passed to the still-running worker thread. Do NOT also
|
|
* erase_job_preason(&jreasons) here: that would free those
|
|
* same allocations while the worker may still be reading
|
|
* them. */
|
|
errno = EINTR;
|
|
perror("setpend");
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xcecli_set_pend_reason", rc);
|
|
erase_job_preason(&jreasons);
|
|
return rc;
|
|
}
|
|
|
|
printf("ok\n");
|
|
|
|
return 0;
|
|
}
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
int role;
|
|
hostattrs_t attrs;
|
|
} join_worker_call_t;
|
|
|
|
static int join_worker_thunk(void* ctxp)
|
|
{
|
|
join_worker_call_t* call = (join_worker_call_t*)ctxp;
|
|
int rc = xcecli_join(call->conn, call->role, &call->attrs);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_join_worker(int argc, char* argv[])
|
|
{
|
|
int rc;
|
|
int role;
|
|
hostattrs_t attrs;
|
|
|
|
role = HostRole_worker;
|
|
|
|
/* xce-test-cli-ux task 2.6 remediation (bug-1557): the argument-parsing/
|
|
* defaulting block (including the maxmem default fix) is extracted into
|
|
* join_worker_args.c/.h so it can be unit-tested in isolation from this
|
|
* function's REPL/network state -- see join_worker_args.h for the full
|
|
* rationale. This is a structural extraction only; the parsing/defaulting
|
|
* behavior and everything from xcecli_join() onward are unchanged. */
|
|
rc = join_worker_parse_attrs(argc, argv, &attrs);
|
|
if (rc < 0) {
|
|
return rc;
|
|
}
|
|
|
|
{
|
|
join_worker_call_t* call
|
|
= (join_worker_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->role = role;
|
|
call->attrs = attrs;
|
|
|
|
rc = xcetest_run_interruptible(join_worker_thunk, call);
|
|
if (rc == -EINTR) {
|
|
SET_ERROR(EINTR);
|
|
perror("join");
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xce_join", rc);
|
|
return rc;
|
|
}
|
|
|
|
printf("host join ok\n");
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* jstart -j jobid [-p pid] — raft-apply-decoupling task 7.2: synthetic
|
|
* worker-fact-stream primitive. Mirrors cmd_jobfinish's existing pattern
|
|
* exactly (same getopt/parse_jobid/SET_ERROR shape) but drives the
|
|
* xcecli_notify_jobstart wire call that cmd_jobfinish's sibling
|
|
* xcecli_notify_jobfinish already exercises — libxce (client.c) already
|
|
* implements the full worker-fact API (jobstart/jobfinish/jobfail/...); this
|
|
* command was the missing REPL-level primitive for jobstart specifically.
|
|
* Drives the SAME production receipt path (on_job_start -> record_job_start_fact
|
|
* -> persist_submit_job_fact -> durable append -> commit -> apply pump ->
|
|
* apply_committed_entry()) that a real worker's report would, without
|
|
* requiring a real scheduler dispatch (on_job_start only requires the job to
|
|
* already exist in jobmgr, i.e. have been submitted). */
|
|
static const flagspec_t jobstart_flags[] = {
|
|
{ 'j', "jobid", "jobid", required_argument, 1, NULL },
|
|
{ 'p', "pid", "pid", required_argument, 0,
|
|
"this xce_test process's own pid" },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
job_pidinfo_t info;
|
|
unsigned int pids[1];
|
|
} jobstart_call_t;
|
|
|
|
static int jobstart_thunk(void* ctxp)
|
|
{
|
|
jobstart_call_t* call = (jobstart_call_t*)ctxp;
|
|
int rc = xcecli_notify_jobstart(call->conn, &call->info);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_jobstart(int argc, char* argv[])
|
|
{
|
|
int rc;
|
|
int opt;
|
|
uint64_t jobid = 0;
|
|
unsigned int pid = 0;
|
|
unsigned int pids[1];
|
|
job_pidinfo_t info;
|
|
struct option long_opts[JOBSTART_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
memset(&info, 0x0, sizeof(info));
|
|
|
|
if (flagspec_build_opts(jobstart_flags, JOBSTART_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: jstart: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'j':
|
|
parse_jobid(&jobid, optarg);
|
|
break;
|
|
case 'p':
|
|
pid = (unsigned int)atoi(optarg);
|
|
break;
|
|
case '?':
|
|
return flagspec_reject("jstart", jobstart_flags, JOBSTART_NFLAGS,
|
|
argv[optind - 1],
|
|
"jstart -j <jobid> [-p <pid>]");
|
|
}
|
|
}
|
|
|
|
if (jobid == 0) {
|
|
fprintf(stderr, " you need specify a positive jobid\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
pids[0] = (pid != 0) ? pid : (unsigned int)getpid();
|
|
info.jobid = jobid;
|
|
info.npids = 1;
|
|
info.pids = pids;
|
|
info.npgids = 0;
|
|
info.pgids = NULL;
|
|
info.exdata = NULL;
|
|
|
|
{
|
|
jobstart_call_t* call = (jobstart_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->info = info;
|
|
call->pids[0] = pids[0];
|
|
call->info.pids = call->pids;
|
|
|
|
rc = xcetest_run_interruptible(jobstart_thunk, call);
|
|
if (rc == -EINTR) {
|
|
SET_ERROR(EINTR);
|
|
perror("jstart");
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xcecli_jobstart", rc);
|
|
return rc;
|
|
}
|
|
|
|
printf("ok\n");
|
|
|
|
return 0;
|
|
}
|
|
|
|
static const flagspec_t jobfinish_flags[] = {
|
|
{ 'j', "jobid", "jobid", required_argument, 1, NULL },
|
|
{ 'c', "exit-code", "exit-code", required_argument, 0, "0" },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_jobfinish_t finish;
|
|
} jobfinish_call_t;
|
|
|
|
static int jobfinish_thunk(void* ctxp)
|
|
{
|
|
jobfinish_call_t* call = (jobfinish_call_t*)ctxp;
|
|
int rc = xcecli_notify_jobfinish(call->conn, &call->finish);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_jobfinish(int argc, char* argv[])
|
|
{
|
|
int rc;
|
|
int opt;
|
|
uint64_t jobid = 0;
|
|
int code = 0;
|
|
xce_jobfinish_t finish;
|
|
struct option long_opts[JOBFINISH_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
/* raft-apply-decoupling task 7.2: reset optind so repeated jfinish
|
|
* invocations within one xce_test process (the fact-stream soak scenario
|
|
* calls this many times in a single worker session) each re-parse their
|
|
* own argv from the start. Without this, getopt's optind carries over
|
|
* from the PRIOR command's parse position, silently skipping this
|
|
* command's own "-j jobid" on the second and later calls (jobid stays 0
|
|
* and the command errors out) — matches the reset convention most other
|
|
* cmd_ handlers in this file already follow (e.g. cmd_join_worker just
|
|
* above, cmd_jobsub, cmd_resupdate). */
|
|
optind = 0;
|
|
opterr = 0;
|
|
|
|
if (flagspec_build_opts(jobfinish_flags, JOBFINISH_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: jfinish: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'j':
|
|
parse_jobid(&jobid, optarg);
|
|
break;
|
|
case 'c':
|
|
code = atoi(optarg);
|
|
break;
|
|
case '?':
|
|
return flagspec_reject("jfinish", jobfinish_flags,
|
|
JOBFINISH_NFLAGS, argv[optind - 1],
|
|
"jfinish -j <jobid> [-c <exitcode>]");
|
|
}
|
|
}
|
|
|
|
if (jobid == 0) {
|
|
fprintf(stderr, " you need specify a positive jobid\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
finish.jobid = jobid;
|
|
finish.exit_code = code;
|
|
|
|
{
|
|
jobfinish_call_t* call = (jobfinish_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->finish = finish;
|
|
|
|
rc = xcetest_run_interruptible(jobfinish_thunk, call);
|
|
if (rc == -EINTR) {
|
|
SET_ERROR(EINTR);
|
|
perror("jfinish");
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xcecli_jobfinish", rc);
|
|
return rc;
|
|
}
|
|
|
|
printf("ok\n");
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* jfail -j jobid [-c code] [-s stage] — raft-apply-decoupling task 7.2:
|
|
* synthetic worker-fact-stream primitive, sibling of jstart/jfinish above.
|
|
* Drives xcecli_notify_jobfail -> on_job_fail -> record_job_fail_fact ->
|
|
* persist_submit_job_fact_ex (the _pad-discriminated path, task 6.1 round 2)
|
|
* -> apply_job_fail_transition() at apply time. Reuses the existing
|
|
* job_failinfo_t shape already exercised internally by this file's own
|
|
* notify_job_fail() helper (used by run_job() on a local exec failure). */
|
|
static const flagspec_t jobfail_flags[] = {
|
|
{ 'j', "jobid", "jobid", required_argument, 1, NULL },
|
|
{ 'c', "failure-code", "failure-code", required_argument, 0, "0" },
|
|
{ 's', "stage", "stage", required_argument, 0,
|
|
"JOB_EXEC_STAGE_POST_EXEC" },
|
|
};
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
job_failinfo_t failinfo;
|
|
} jobfail_call_t;
|
|
|
|
static int jobfail_thunk(void* ctxp)
|
|
{
|
|
jobfail_call_t* call = (jobfail_call_t*)ctxp;
|
|
int rc = xcecli_notify_jobfail(call->conn, &call->failinfo);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_jobfail(int argc, char* argv[])
|
|
{
|
|
int rc;
|
|
int opt;
|
|
uint64_t jobid = 0;
|
|
int code = 0;
|
|
int stage = JOB_EXEC_STAGE_POST_EXEC;
|
|
job_failinfo_t failinfo;
|
|
struct option long_opts[JOBFAIL_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
memset(&failinfo, 0x0, sizeof(failinfo));
|
|
|
|
if (flagspec_build_opts(jobfail_flags, JOBFAIL_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: jfail: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'j':
|
|
parse_jobid(&jobid, optarg);
|
|
break;
|
|
case 'c':
|
|
code = atoi(optarg);
|
|
break;
|
|
case 's':
|
|
stage = atoi(optarg);
|
|
break;
|
|
case '?':
|
|
return flagspec_reject(
|
|
"jfail", jobfail_flags, JOBFAIL_NFLAGS, argv[optind - 1],
|
|
"jfail -j <jobid> [-c <code>] [-s <stage>]");
|
|
}
|
|
}
|
|
|
|
if (jobid == 0) {
|
|
fprintf(stderr, " you need specify a positive jobid\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
failinfo.jobid = jobid;
|
|
failinfo.stage = stage;
|
|
failinfo.code = code;
|
|
failinfo.detail = "synthetic fact-stream failure (xce_test jfail)";
|
|
|
|
{
|
|
jobfail_call_t* call = (jobfail_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->failinfo = failinfo;
|
|
|
|
rc = xcetest_run_interruptible(jobfail_thunk, call);
|
|
if (rc == -EINTR) {
|
|
SET_ERROR(EINTR);
|
|
perror("jfail");
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xcecli_jobfail", rc);
|
|
return rc;
|
|
}
|
|
|
|
printf("ok\n");
|
|
|
|
return 0;
|
|
}
|
|
|
|
/*
|
|
* resource-value-verification task 2.2: the controller's own update RPC
|
|
* reply can never distinguish per-name accept/reject -- update_resval_list()
|
|
* (src/controller/worker.c:185-198) unconditionally returns success
|
|
* regardless of any per-item failure (design.md's "set_resval()/
|
|
* cmd_resupdate" component, Responsibilities & Constraints). To recover
|
|
* per-name accept/reject (Requirements 1.2, 1.3), cmd_resupdate() issues one
|
|
* follow-up xcecli_resval() query per submitted (resource, worker) pair --
|
|
* the exact client-library call resval/cmd_resval already use -- and
|
|
* compares the returned value against what was just sent.
|
|
*/
|
|
#define RESVAL_VERIFY_MAX_ATTEMPTS (5)
|
|
#define RESVAL_VERIFY_RETRY_MS (100L)
|
|
|
|
/* Compare a just-submitted value against a follow-up query's result. Both
|
|
* sides are xce_val_t (wire value union) -- a match requires the same
|
|
* `which` tag and an equal payload. */
|
|
static int resval_value_matches(xce_val_t* sent, xce_val_t* got)
|
|
{
|
|
if ((sent == NULL) || (got == NULL) || (sent->which != got->which)) {
|
|
return 0;
|
|
}
|
|
|
|
switch (sent->which) {
|
|
case Value_u_ivalue32:
|
|
return sent->u.ivalue32 == got->u.ivalue32;
|
|
case Value_u_uvalue32:
|
|
return sent->u.uvalue32 == got->u.uvalue32;
|
|
case Value_u_ivalue64:
|
|
return sent->u.ivalue64 == got->u.ivalue64;
|
|
case Value_u_uvalue64:
|
|
return sent->u.uvalue64 == got->u.uvalue64;
|
|
case Value_u_svalue:
|
|
return EQUAL_STRING(sent->u.svalue, got->u.svalue);
|
|
case Value_u_fvalue:
|
|
return float_equal(sent->u.fvalue, got->u.fvalue);
|
|
case Value_u_bvalue:
|
|
return (sent->u.bvalue == 0) == (got->u.bvalue == 0);
|
|
default:
|
|
return 0;
|
|
}
|
|
}
|
|
|
|
/*
|
|
* Query a just-submitted (name, host) pair back from the controller and
|
|
* print its per-name accepted/rejected verdict (Requirements 1.2, 1.3). A
|
|
* short bounded retry guards against a genuinely-registered resource whose
|
|
* value hasn't propagated to the query path yet (design.md's Implementation
|
|
* Notes: sub-second in practice for a controller-local query) -- NOT an
|
|
* arbitrary long poll.
|
|
*/
|
|
/* xce-test-cli-ux task 3.2: returns 0 when the verification loop below ran
|
|
* to its normal conclusion (accepted, or rejected after exhausting its
|
|
* retry budget), or -EINTR when the operator interrupted a still-in-flight
|
|
* follow-up query -- reusing resval_call_t/resval_thunk (defined above,
|
|
* next to cmd_resval, which drives the identical xcecli_resval() call).
|
|
* The caller (cmd_resupdate) propagates -EINTR up as its own return value
|
|
* (Requirement 1.3), stopping this verification pass immediately rather
|
|
* than continuing on to any further submitted (resource, worker) pairs. */
|
|
static int resupdate_report_one(char* name, char* host, xce_val_t* sent)
|
|
{
|
|
int attempt;
|
|
int ever_found = 0;
|
|
|
|
if (name == NULL) {
|
|
return 0;
|
|
}
|
|
|
|
for (attempt = 0; attempt < RESVAL_VERIFY_MAX_ATTEMPTS; attempt++) {
|
|
resval_call_t* call;
|
|
xce_resval_t** got;
|
|
int rc;
|
|
|
|
if (attempt > 0) {
|
|
usleep((useconds_t)(RESVAL_VERIFY_RETRY_MS * US_PER_MS));
|
|
}
|
|
|
|
call = (resval_call_t*)calloc(1, sizeof(*call));
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->host = host;
|
|
call->res = name;
|
|
|
|
rc = xcetest_run_interruptible(resval_thunk, call);
|
|
if (rc == -EINTR) {
|
|
errno = EINTR;
|
|
perror("resupdate");
|
|
return rc;
|
|
}
|
|
|
|
got = call->vals;
|
|
free(call);
|
|
|
|
if (rc > 0) {
|
|
int matched = resval_value_matches(sent, got[0]->val);
|
|
int i;
|
|
|
|
ever_found = 1;
|
|
|
|
for (i = 0; i < rc; i++) {
|
|
free_ResourceVal_ptr(&got[i]);
|
|
}
|
|
FREEUP(got);
|
|
|
|
if (matched) {
|
|
printf("%s: accepted\n", name);
|
|
return 0;
|
|
}
|
|
/* Found but the value differs -- could still be a stale value
|
|
* about to be overwritten by the just-sent update's propagation,
|
|
* so keep retrying within the bounded budget rather than giving
|
|
* up immediately. */
|
|
continue;
|
|
}
|
|
}
|
|
|
|
/* xce-test-cli-ux task 3.6, Requirement 3.1: unlike the "accepted" line
|
|
* above (legitimate per-resource outcome data on the success path), a
|
|
* "rejected" verdict reports that this specific update did NOT take
|
|
* effect -- a problem the operator/script needs to notice -- so it goes
|
|
* to stderr. This does NOT change cmd_resupdate()'s own return value:
|
|
* the caller still returns 0 overall even when one or more resources are
|
|
* reported rejected here (see this task's CONCERNS for why that return-
|
|
* value question is left as a follow-up, out of this stream-routing
|
|
* task's scope). */
|
|
if (ever_found) {
|
|
fprintf(stderr, "%s: rejected: value mismatch\n", name);
|
|
} else {
|
|
fprintf(stderr, "%s: rejected: unknown resource\n", name);
|
|
}
|
|
|
|
return 0;
|
|
}
|
|
|
|
/* Append one -n/-v pair to cmd_resupdate's vals[] (16 entries), taking
|
|
* ownership of `name`. A missing value gets the type's default with a
|
|
* warning. Returns 0, or -E2BIG when vals[] is full (the old code had no
|
|
* bound) or -ENOMEM; on error `name` is freed. */
|
|
static int resupdate_add(xce_resval_t* vals, int* nres, char* name,
|
|
const char* val, int type_override)
|
|
{
|
|
int rc;
|
|
|
|
if (*nres >= 16) {
|
|
fprintf(stderr, "resupdate: at most 16 resources per call\n");
|
|
free(name);
|
|
return -E2BIG;
|
|
}
|
|
if (val == NULL) {
|
|
fprintf(stderr,
|
|
"res `%s' has no value specified, default will be used\n",
|
|
name);
|
|
}
|
|
/* bug-1574: resval_fill() sets name/value for string resources
|
|
* (type, model) too; set_resval() left them uninitialized */
|
|
rc = resval_fill(&vals[*nres], name, val, type_override,
|
|
get_setting("myname"));
|
|
if (rc < 0) {
|
|
free(name);
|
|
return rc;
|
|
}
|
|
(*nres)++;
|
|
return 0;
|
|
}
|
|
|
|
static const flagspec_t resupdate_flags[] = {
|
|
{ 'n', "resname", "resname", required_argument, 1, NULL },
|
|
{ 'v', "value", "value", required_argument, 1, NULL },
|
|
{ 't', "value-type", "value-type", required_argument, 0,
|
|
"numeric unless overridden" },
|
|
};
|
|
|
|
/* Array sizes below are hardcoded to 16 to match cmd_resupdate's own
|
|
* MAX_N_RES -- that macro is #define'd inside cmd_resupdate's own body
|
|
* (this file's pre-existing convention for this one local constant), so
|
|
* it is not yet in scope at this point in the file. */
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
int nres;
|
|
xce_resval_t vals[16];
|
|
xce_resval_t* valptrs[16];
|
|
} resupdate_call_t;
|
|
|
|
static int resupdate_thunk(void* ctxp)
|
|
{
|
|
resupdate_call_t* call = (resupdate_call_t*)ctxp;
|
|
int rc = xcecli_update_resvals(call->conn, call->nres, call->valptrs);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_resupdate(int argc, char* argv[])
|
|
{
|
|
#define MAX_N_RES (16)
|
|
int nres;
|
|
xce_resval_t vals[MAX_N_RES];
|
|
char* resname = NULL;
|
|
char* resval_str = NULL;
|
|
int type_override = RESVAL_TYPE_UNKNOWN;
|
|
int opt;
|
|
int rc;
|
|
struct option long_opts[RESUPDATE_NFLAGS + 1];
|
|
char optstring[16];
|
|
|
|
optind = 0;
|
|
opterr = 0;
|
|
nres = 0;
|
|
memset(vals, 0, sizeof(vals));
|
|
|
|
if (flagspec_build_opts(resupdate_flags, RESUPDATE_NFLAGS, long_opts,
|
|
optstring, sizeof(optstring))
|
|
< 0) {
|
|
fprintf(stderr,
|
|
"xce_test: resupdate: internal error building flag table\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
while ((opt = getopt_long(argc, argv, optstring, long_opts, NULL))
|
|
!= -1) {
|
|
switch (opt) {
|
|
case 'n':
|
|
if (resname != NULL) {
|
|
rc = resupdate_add(vals, &nres, resname, resval_str,
|
|
type_override);
|
|
resname = NULL;
|
|
resval_str = NULL;
|
|
type_override = RESVAL_TYPE_UNKNOWN;
|
|
if (rc < 0) {
|
|
goto out;
|
|
}
|
|
}
|
|
|
|
resname = strdup(optarg);
|
|
break;
|
|
case 'v':
|
|
if (resname == NULL) {
|
|
fprintf(stderr,
|
|
"ignore res value `%s' as no res name found\n",
|
|
optarg);
|
|
} else if (resval_str != NULL) {
|
|
/* bug-1712: this case used to print "no res name found" */
|
|
fprintf(stderr,
|
|
"ignoring duplicate value `%s' for res `%s' (it "
|
|
"already has value `%s')\n",
|
|
optarg, resname, resval_str);
|
|
} else {
|
|
resval_str = optarg;
|
|
}
|
|
break;
|
|
case 't':
|
|
if (resname != NULL) {
|
|
if (strcasecmp(optarg, "numeric") == 0) {
|
|
type_override = Value_u_fvalue;
|
|
} else if (strcasecmp(optarg, "string") == 0) {
|
|
type_override = Value_u_svalue;
|
|
} else {
|
|
fprintf(
|
|
stderr,
|
|
"ignoring unrecognized -t value `%s' (expected "
|
|
"`numeric' or `string')\n",
|
|
optarg);
|
|
}
|
|
} else {
|
|
fprintf(stderr, "ignore -t `%s' as no res name found\n",
|
|
optarg);
|
|
}
|
|
break;
|
|
case '?':
|
|
rc = flagspec_reject(
|
|
"resupdate", resupdate_flags, RESUPDATE_NFLAGS,
|
|
argv[optind - 1],
|
|
"resupdate -n <resname> -v <value> [-t <numeric|string>] "
|
|
"[-n <resname> -v <value> [-t <numeric|string>] ...]");
|
|
FREEUP(resname);
|
|
goto out;
|
|
}
|
|
}
|
|
|
|
/* bug-1707: the trailing -n was dropped without a word when it had no
|
|
* -v; it now gets the same warning and default as a mid-list one */
|
|
if (resname != NULL) {
|
|
rc = resupdate_add(vals, &nres, resname, resval_str, type_override);
|
|
if (rc < 0) {
|
|
goto out;
|
|
}
|
|
}
|
|
|
|
if (nres == 0) {
|
|
fprintf(stderr,
|
|
"resupdate: no resource given (use -n <resname> -v <value>)\n");
|
|
rc = -EINVAL;
|
|
goto out;
|
|
}
|
|
|
|
{
|
|
resupdate_call_t* call = (resupdate_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
rc = -ENOMEM;
|
|
goto out;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->nres = nres;
|
|
memcpy(call->vals, vals, sizeof(xce_resval_t) * (size_t)nres);
|
|
{
|
|
int i;
|
|
for (i = 0; i < nres; i++) {
|
|
call->valptrs[i] = &call->vals[i];
|
|
}
|
|
}
|
|
|
|
rc = xcetest_run_interruptible(resupdate_thunk, call);
|
|
if (rc == -EINTR) {
|
|
/* the still-running worker thread owns `call`, whose vals[]
|
|
* point at the same names/values -- leave them allocated */
|
|
errno = EINTR;
|
|
perror("resupdate");
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
if (rc >= 0) {
|
|
int i;
|
|
|
|
printf("resource update ok!\n");
|
|
|
|
for (i = 0; i < nres; i++) {
|
|
int report_rc
|
|
= resupdate_report_one(vals[i].name, vals[i].host, vals[i].val);
|
|
|
|
if (report_rc == -EINTR) {
|
|
return report_rc; /* same ownership rule as above */
|
|
}
|
|
}
|
|
rc = 0;
|
|
} else {
|
|
/* the RPC's failure used to be printed and then reported as
|
|
* success (return 0) */
|
|
cmdutil_rpc_error("xcecli_update_resval", rc);
|
|
}
|
|
|
|
out:
|
|
for (int i = 0; i < nres; i++) {
|
|
free(vals[i].name);
|
|
resval_fill_release(&vals[i]);
|
|
}
|
|
return rc;
|
|
#undef MAX_N_RES
|
|
}
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
int role;
|
|
} join_sched_call_t;
|
|
|
|
static int join_sched_thunk(void* ctxp)
|
|
{
|
|
join_sched_call_t* call = (join_sched_call_t*)ctxp;
|
|
int rc = xcecli_join(call->conn, call->role, NULL);
|
|
|
|
free(call);
|
|
|
|
return rc;
|
|
}
|
|
|
|
int cmd_join_sched(int argc, char* argv[])
|
|
{
|
|
int role = HostRole_scheduler;
|
|
int rc = -1;
|
|
join_sched_call_t* call = (join_sched_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
call->role = role;
|
|
|
|
rc = xcetest_run_interruptible(join_sched_thunk, call);
|
|
if (rc == -EINTR) {
|
|
SET_ERROR(EINTR);
|
|
perror("join");
|
|
return rc;
|
|
}
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xce_join", rc);
|
|
return rc;
|
|
}
|
|
|
|
printf("scheduler join ok\n");
|
|
return 0;
|
|
}
|
|
|
|
static int print_reslist(xce_reslist_t reslist)
|
|
{
|
|
int i;
|
|
|
|
for (i = 0; i < reslist.count; i++) {
|
|
printf("name: %s\nflags: %d\ninterval: %d\ndesc: %s\n\n",
|
|
reslist.res[i]->name, reslist.res[i]->flags,
|
|
reslist.res[i]->interval, reslist.res[i]->desc);
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
static int print_queuelist(xce_queuelist_t qlist)
|
|
{
|
|
int i;
|
|
|
|
for (i = 0; i < qlist.count; i++) {
|
|
printf("name: %s\nstatus: %d\npriority: %d\n\n", qlist.queues[i]->queue,
|
|
qlist.queues[i]->status, qlist.queues[i]->priority);
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
static int print_workerlist(xce_workerlist_t wlist)
|
|
{
|
|
int i;
|
|
|
|
for (i = 0; i < wlist.count; i++) {
|
|
printf("name: %s\nstatus: %d\n\n", wlist.workers[i]->name,
|
|
wlist.workers[i]->status);
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
/* bug-1711 (Requirement 11.8): print the already-fetched group records
|
|
* (conf->uglist / conf->hglist) instead of the previous no-op stub, which
|
|
* always left the "User Groups"/"Host Groups" header followed by silent
|
|
* nothing -- indistinguishable from a genuinely empty cluster. Mirrors
|
|
* print_reslist/print_queuelist/print_workerlist's own one-record-per-block
|
|
* convention just above. */
|
|
static int print_grouplist(xce_group_configlist_t glist)
|
|
{
|
|
int i, j;
|
|
|
|
for (i = 0; i < glist.count; i++) {
|
|
printf("name: %s\nmembers:", glist.groups[i]->name);
|
|
for (j = 0; j < glist.groups[i]->members.count; j++) {
|
|
printf(" %s", glist.groups[i]->members.data[j]);
|
|
}
|
|
printf("\nex_members:");
|
|
for (j = 0; j < glist.groups[i]->ex_members.count; j++) {
|
|
printf(" %s", glist.groups[i]->ex_members.data[j]);
|
|
}
|
|
printf("\n\n");
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
/* bug-1711 (Requirement 11.8): print the already-fetched config records
|
|
* (conf->cfglist) instead of the previous no-op stub -- see print_grouplist
|
|
* above for the same fix applied to the "Configs" section. */
|
|
static int print_configlist(xce_configlist_t clist)
|
|
{
|
|
int i;
|
|
|
|
for (i = 0; i < clist.count; i++) {
|
|
printf("name: %s\nvalue: %s\n\n", clist.configs[i]->name,
|
|
clist.configs[i]->value);
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
typedef struct {
|
|
xcecli_ctx_t conn;
|
|
xce_clusterconf_t* conf;
|
|
int rc;
|
|
} getconf_call_t;
|
|
|
|
static int getconf_thunk(void* ctxp)
|
|
{
|
|
getconf_call_t* call = (getconf_call_t*)ctxp;
|
|
|
|
call->conf = NULL;
|
|
call->rc = xcecli_clusterconf(call->conn, &call->conf);
|
|
|
|
return call->rc;
|
|
}
|
|
|
|
int cmd_getconf(int argc, char* argv[])
|
|
{
|
|
xce_clusterconf_t* conf;
|
|
int rc;
|
|
getconf_call_t* call = (getconf_call_t*)calloc(1, sizeof(*call));
|
|
|
|
if (call == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
call->conn = active_ctx->ctx;
|
|
|
|
rc = xcetest_run_interruptible(getconf_thunk, call);
|
|
if (rc == -EINTR) {
|
|
SET_ERROR(EINTR);
|
|
perror("getconf");
|
|
return rc;
|
|
}
|
|
|
|
conf = call->conf;
|
|
free(call);
|
|
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xce_clusterconf", rc);
|
|
return rc;
|
|
}
|
|
|
|
printf("Cluster Name: %s\n", conf->name);
|
|
|
|
printf("\nResources\n===========\n");
|
|
print_reslist(conf->reslist);
|
|
|
|
printf("\nQueues\n========\n");
|
|
print_queuelist(conf->qlist);
|
|
|
|
printf("\nWorkers\n========\n");
|
|
print_workerlist(conf->wlist);
|
|
|
|
printf("\nUser Groups\n===========\n");
|
|
print_grouplist(conf->uglist);
|
|
|
|
printf("\nHost Groups\n============\n");
|
|
print_grouplist(conf->hglist);
|
|
|
|
printf("\nConfigs\n=======\n");
|
|
print_configlist(conf->cfglist);
|
|
|
|
free_ClusterConf(conf);
|
|
|
|
return 0;
|
|
}
|
|
#define KVAL(v) (((v) == NULL) ? "<not set>" : (v))
|
|
|
|
int cmd_set(int argc, char* argv[])
|
|
{
|
|
int i;
|
|
|
|
if (argc == 1) {
|
|
for (i = 0; active_ctx->settings[i].key; i++) {
|
|
printf("%-12s:%s default: %s\n", active_ctx->settings[i].key,
|
|
KVAL(active_ctx->settings[i].value),
|
|
KVAL(active_ctx->settings[i].defval));
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
/* errexit toggle (xce-test-e2e-primitives task 1.1), shell `-e`/`+e` style.
|
|
* Default is mode-dependent (batch `-e` abort-on-failure, interactive `+e`
|
|
* continue); these override it for the rest of the session. The session
|
|
* still exits non-zero if any command failed. Handled here ahead of the
|
|
* per-context key/value path. `continue-on-error` == `+e` (back-compat). */
|
|
if ((argc == 2) &&
|
|
((strcmp(argv[1], "+e") == 0) ||
|
|
(strcmp(argv[1], "continue-on-error") == 0))) {
|
|
g_continue_on_error = 1; /* +e: keep the read loop running on failure */
|
|
return 0;
|
|
}
|
|
if ((argc == 2) && (strcmp(argv[1], "-e") == 0)) {
|
|
g_continue_on_error = 0; /* -e: abort the read loop on first failure */
|
|
return 0;
|
|
}
|
|
|
|
/* Confirmation-skip toggle (xce-test-cli-ux task 3.3, Requirement
|
|
* 9.6/9.7), same shell-style on/off pair as `-e`/`+e` immediately
|
|
* above: `-y` enables skipping (assume yes to every destructive
|
|
* command's prompt, regardless of TTY state), `+y` restores the default
|
|
* (confirmation required). Wraps confirm_set_skip()/confirm_is_skip_set()
|
|
* (confirm.h, task 2.1), which the five destructive commands' own
|
|
* confirm_destructive() calls already consult internally. */
|
|
if ((argc == 2) && (strcmp(argv[1], "-y") == 0)) {
|
|
confirm_set_skip(1); /* -y: skip destructive-command confirmation */
|
|
return 0;
|
|
}
|
|
if ((argc == 2) && (strcmp(argv[1], "+y") == 0)) {
|
|
confirm_set_skip(0); /* +y: require confirmation again (default) */
|
|
return 0;
|
|
}
|
|
|
|
if (argc != 3) {
|
|
fprintf(stderr, "set <key> <value>\n");
|
|
return -EINVAL;
|
|
}
|
|
|
|
for (i = 0; active_ctx->settings[i].key; i++) {
|
|
if (strcmp(active_ctx->settings[i].key, argv[1]) == 0) {
|
|
if (active_ctx->settings[i].value != NULL) {
|
|
free(active_ctx->settings[i].value);
|
|
}
|
|
|
|
active_ctx->settings[i].value = strdup(argv[2]);
|
|
return 0;
|
|
}
|
|
}
|
|
|
|
fprintf(stderr, "Can not find setting key: %s\n", argv[1]);
|
|
return -EINVAL;
|
|
}
|
|
|
|
static int run_command(char* cmdline, pid_t* pid)
|
|
{
|
|
pid_t child;
|
|
|
|
child = fork();
|
|
|
|
*pid = child;
|
|
if (child < 0) {
|
|
*pid = -errno;
|
|
|
|
return -errno;
|
|
}
|
|
|
|
if (child == 0) {
|
|
execlp("/bin/sh", "/bin/sh", "-c", cmdline, NULL);
|
|
perror("execlp");
|
|
exit(-1);
|
|
}
|
|
|
|
return 0;
|
|
}
|
|
|
|
static int run_task(char* cmdline, pid_t* pid)
|
|
{
|
|
return run_command(cmdline, pid);
|
|
}
|
|
|
|
static int get_avail_run_job_slots()
|
|
{
|
|
int i;
|
|
|
|
for (i = 0; i < MAX_RUN_JOBS; i++) {
|
|
if (run_jobs[i].jobid == 0) {
|
|
return i;
|
|
}
|
|
}
|
|
|
|
return -1;
|
|
}
|
|
|
|
static int get_avail_run_task_slots()
|
|
{
|
|
int i;
|
|
|
|
for (i = 0; i < MAX_RUN_TASKS; i++) {
|
|
if (run_tasks[i].pid <= 0) {
|
|
return i;
|
|
}
|
|
}
|
|
|
|
return -1;
|
|
}
|
|
|
|
static int sched_on_jobs(int njobs, xce_job_t** jobs)
|
|
{
|
|
printf("%s %d jobs gotten\n", __func__, njobs);
|
|
|
|
return 0;
|
|
}
|
|
|
|
static int sched_on_workers(int nworkers, xce_worker_t** workers)
|
|
{
|
|
printf("%s %d workers gotten\n", __func__, nworkers);
|
|
|
|
return 0;
|
|
}
|
|
|
|
static void notify_job_fail(uint64_t jobid, int code)
|
|
{
|
|
/* clang-format off */
|
|
job_failinfo_t failinfo = {
|
|
.jobid = jobid,
|
|
.stage = JOB_EXEC_STAGE_MAIN,
|
|
.code = code,
|
|
.detail = "no detail information"
|
|
};
|
|
/*clang-format on*/
|
|
|
|
xcecli_notify_jobfail(active_ctx->ctx, &failinfo);
|
|
return;
|
|
}
|
|
|
|
static void notify_task_fail(uint64_t jobid,
|
|
int clientidx,
|
|
int taskidx,
|
|
int code) {
|
|
xce_task_failinfo_t failinfo = {
|
|
.jobid = jobid,
|
|
.host = get_setting("myname"),
|
|
.clientidx = clientidx,
|
|
.taskidx = taskidx,
|
|
.code = code,
|
|
.detail = NULL
|
|
};
|
|
|
|
xcecli_notify_taskfail(active_ctx->ctx, &failinfo);
|
|
}
|
|
|
|
static void notify_jobstart(uint64_t jobid,
|
|
int pid,
|
|
int pgid) {
|
|
/* clang-format off */
|
|
job_pidinfo_t pidinfo = {
|
|
.jobid = jobid,
|
|
.npids = 1,
|
|
.pids = (unsigned int []){(unsigned int)pid},
|
|
.npgids = 0,
|
|
.pgids = (unsigned int []){(unsigned int)pgid},
|
|
.exdata = NULL
|
|
};
|
|
/* clang-format on */
|
|
|
|
xcecli_notify_jobstart(active_ctx->ctx, &pidinfo);
|
|
|
|
return;
|
|
}
|
|
|
|
static void notify_task_start(uint64_t jobid, int clientidx, int taskidx,
|
|
int pid, int pgid)
|
|
{
|
|
xce_task_pidinfo_t pidinfo
|
|
= { .jobid = jobid,
|
|
.host = get_setting("myname"),
|
|
.clientidx = clientidx,
|
|
.taskidx = taskidx,
|
|
.npids = 1,
|
|
.pids = (unsigned int[]) { (unsigned int)pid },
|
|
.npgids = 1,
|
|
.pgids = (unsigned int[]) { (unsigned int)pgid },
|
|
.exdata = NULL };
|
|
|
|
xcecli_notify_taskstart(active_ctx->ctx, &pidinfo);
|
|
|
|
return;
|
|
}
|
|
|
|
static int run_task_remote(uint64_t jobid, job_startinfo_t* startinfo,
|
|
xce_launch_t* launchinfo)
|
|
{
|
|
xce_taskhost_t hosts[launchinfo->nhosts - 1];
|
|
xce_taskrun_t taskrun;
|
|
int i;
|
|
int rc;
|
|
|
|
taskrun.taskhosts = (xce_taskhost_t**)calloc(launchinfo->nhosts - 1,
|
|
sizeof(xce_taskhost_t*));
|
|
if (taskrun.taskhosts == NULL) {
|
|
return -ENOMEM;
|
|
}
|
|
|
|
taskrun.clientidx = -1;
|
|
|
|
for (i = 1; i < launchinfo->nhosts; i++) {
|
|
hosts[i - 1].exechome = startinfo->exechome;
|
|
hosts[i - 1].execcwd = startinfo->execcwd;
|
|
hosts[i - 1].cmdline = startinfo->cmdline;
|
|
hosts[i - 1].exdata = startinfo->exdata;
|
|
hosts[i - 1].execuser = startinfo->execuser;
|
|
hosts[i - 1].launch_type = startinfo->launch_type;
|
|
hosts[i - 1].host = launchinfo->hosts[i]->host;
|
|
hosts[i - 1].ntasks = launchinfo->hosts[i]->nranks;
|
|
|
|
if (hosts[i - 1].launch_type == NULL) {
|
|
/* use mpi as default */
|
|
hosts[i - 1].launch_type = "mpi";
|
|
}
|
|
taskrun.taskhosts[i - 1] = &hosts[i - 1];
|
|
}
|
|
|
|
taskrun.nhosts = launchinfo->nhosts - 1;
|
|
taskrun.jobid = jobid;
|
|
|
|
xcecli_unblock_tasks(active_ctx->ctx);
|
|
|
|
rc = xcecli_runtask(active_ctx->ctx, &taskrun);
|
|
if (rc < 0) {
|
|
fprintf(stderr, "run remote task failed\n");
|
|
}
|
|
|
|
FREEUP(taskrun.taskhosts);
|
|
|
|
return rc;
|
|
}
|
|
|
|
static int run_task_local(uint64_t jobid, job_startinfo_t* startinfo,
|
|
int ntasks)
|
|
{
|
|
pid_t pid;
|
|
char jobidStr[64];
|
|
|
|
pid = fork();
|
|
if (pid < 0) {
|
|
return pid;
|
|
}
|
|
|
|
if (pid == 0) {
|
|
int i;
|
|
pid_t pids[ntasks];
|
|
int rc = 0;
|
|
int nlive = 0;
|
|
|
|
/* disable detection of address sanitizer as
|
|
* the request structure will not be freed in
|
|
* child and we will exit directly
|
|
*/
|
|
setenv("ASAN_OPTIONS", "detect_leaks=false", 1);
|
|
|
|
for (i = 0; i < ntasks; i++) {
|
|
rc = run_command(startinfo->cmdline, &pids[i]);
|
|
if (rc >= 0) {
|
|
nlive++;
|
|
}
|
|
}
|
|
|
|
if (rc >= 0) {
|
|
rc = 0;
|
|
}
|
|
|
|
while (nlive > 0) {
|
|
sleep(1);
|
|
for (i = 0; i < ntasks; i++) {
|
|
int wstatus;
|
|
|
|
if (pids[i] <= 0) {
|
|
continue;
|
|
}
|
|
|
|
if (waitpid(pids[i], &wstatus, WNOHANG) <= 0) {
|
|
continue;
|
|
}
|
|
|
|
if (WIFEXITED(wstatus) || WIFSIGNALED(wstatus)) {
|
|
nlive--;
|
|
|
|
if (WIFSIGNALED(wstatus)) {
|
|
rc = 128 + WTERMSIG(wstatus);
|
|
} else {
|
|
if (WEXITSTATUS(wstatus) != 0) {
|
|
rc = WEXITSTATUS(wstatus);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
exit(rc);
|
|
}
|
|
|
|
fmt_jobid(jobidStr, sizeof(jobidStr) - 1, jobid);
|
|
printf("Job %s started, PID: %d\n", jobidStr, pid);
|
|
|
|
return pid;
|
|
}
|
|
|
|
int run_job(xce_job_exec_t* startmsg)
|
|
{
|
|
int rc = 0;
|
|
uint64_t jobid = startmsg->jobid;
|
|
job_startinfo_t* startinfo = startmsg->start_info;
|
|
xce_launch_t* launchinfo = startmsg->launchinfo;
|
|
|
|
printf("%s: exechome= %s\n", __func__,
|
|
(startinfo->exechome == NULL) ? "NULL" : startinfo->exechome);
|
|
printf("%s: execcwd= %s\n", __func__,
|
|
(startinfo->execcwd == NULL) ? "NULL" : startinfo->execcwd);
|
|
printf("%s: execuser= %s\n", __func__,
|
|
(startinfo->execuser == NULL) ? "NULL" : startinfo->execuser);
|
|
printf("%s: cmdline= %s\n", __func__,
|
|
(startinfo->cmdline == NULL) ? "NULL" : startinfo->cmdline);
|
|
printf("%s: exdata= %s\n", __func__,
|
|
(startinfo->exdata == NULL) ? "NULL" : startinfo->exdata);
|
|
printf("%s: launch_type= %s\n", __func__,
|
|
(startinfo->launch_type == NULL) ? "NULL" : startinfo->launch_type);
|
|
printf("%s: q_pre_exec= %s\n", __func__,
|
|
(startinfo->q_pre_exec == NULL) ? "NULL" : startinfo->q_pre_exec);
|
|
printf("%s: q_post_exec= %s\n", __func__,
|
|
(startinfo->q_post_exec == NULL) ? "NULL" : startinfo->q_post_exec);
|
|
printf("%s: job_pre_exec= %s\n", __func__,
|
|
(startinfo->job_pre_exec == NULL) ? "NULL"
|
|
: startinfo->job_pre_exec);
|
|
printf("%s: job_post_exec= %s\n", __func__,
|
|
(startinfo->job_post_exec == NULL) ? "NULL"
|
|
: startinfo->job_post_exec);
|
|
|
|
if (launchinfo->nhosts > 1) {
|
|
rc = run_task_remote(jobid, startinfo, launchinfo);
|
|
if (rc < 0) {
|
|
fprintf(stderr, "run_task_remote() fail, rc %d\n", rc);
|
|
return rc;
|
|
}
|
|
}
|
|
|
|
rc = run_task_local(jobid, startinfo, launchinfo->hosts[0]->nranks);
|
|
if (rc < 0) {
|
|
notify_job_fail(jobid, rc);
|
|
} else {
|
|
int idx;
|
|
int pid;
|
|
|
|
pid = rc;
|
|
idx = get_avail_run_job_slots();
|
|
if (idx < 0) {
|
|
printf("too many jobs run in this worker\n");
|
|
} else {
|
|
run_jobs[idx].jobid = jobid;
|
|
run_jobs[idx].pid = pid;
|
|
run_jobs[idx].ts = 0;
|
|
}
|
|
|
|
notify_jobstart(jobid, pid, pid);
|
|
}
|
|
|
|
return rc;
|
|
}
|
|
|
|
static int handle_startjob(xce_request_t* request)
|
|
{
|
|
return run_job(request->body.jobstartmsg);
|
|
}
|
|
|
|
static int handle_notify(xce_request_t* request)
|
|
{
|
|
xce_notify_data_t* data = request->body.notifymsg;
|
|
char jobidStr[64];
|
|
|
|
switch (data->event) {
|
|
case NotifyData_data_taskPidInfo:
|
|
fmt_jobid(jobidStr, sizeof(jobidStr) - 1, data->data.taskinfo->jobid);
|
|
printf("job %s host %s taskidx %d pid %d\n",
|
|
jobidStr, data->data.taskinfo->host,
|
|
data->data.taskinfo->taskidx, data->data.taskinfo->pids[0]);
|
|
break;
|
|
case NotifyData_data_taskFailInfo:
|
|
fmt_jobid(jobidStr, sizeof(jobidStr) - 1,
|
|
data->data.taskfailinfo->jobid);
|
|
printf("job %s host %s task %d error code %d\n",
|
|
jobidStr, data->data.taskfailinfo->host,
|
|
data->data.taskfailinfo->taskidx, data->data.taskfailinfo->code);
|
|
break;
|
|
case NotifyData_data_taskFinish:
|
|
fmt_jobid(jobidStr, sizeof(jobidStr) - 1,
|
|
data->data.taskfinish->jobid);
|
|
printf("job %s host %s task %d exit code %d\n",
|
|
jobidStr, data->data.taskfinish->host,
|
|
data->data.taskfinish->taskidx,
|
|
data->data.taskfinish->exit_code);
|
|
break;
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
static int job_kill(uint64_t jobid, int signo)
|
|
{
|
|
int i;
|
|
int killed = 0;
|
|
|
|
for (i = 0; i < MAX_RUN_JOBS; i++) {
|
|
if ((run_jobs[i].jobid == 0) || (run_jobs[i].jobid != jobid)) {
|
|
continue;
|
|
}
|
|
|
|
if (run_jobs[i].pid > 1) {
|
|
kill(run_jobs[i].pid, signo);
|
|
killed++;
|
|
}
|
|
}
|
|
|
|
return killed;
|
|
}
|
|
|
|
static int job_chkpnt(uint64_t jobid, char* method, int kill)
|
|
{
|
|
char buf[128];
|
|
char info[256];
|
|
|
|
printf("checkpoint job %ld by method %s\n", jobid, method);
|
|
sprintf(buf, "/tmp/%ld/ckpt_dir", jobid);
|
|
|
|
/* do some actions to checkpoint jobs */
|
|
|
|
sprintf(info,
|
|
"{\nhost_select=\"criu\"; \nrestart_command=\"cd %s; criu restore "
|
|
"-vvv\"}\n",
|
|
buf);
|
|
xcecli_notify_jobchkpt(active_ctx->ctx, jobid, info);
|
|
|
|
if (kill) {
|
|
job_kill(jobid, SIGTERM);
|
|
}
|
|
|
|
return 0;
|
|
}
|
|
|
|
static int handle_jobcontrol(xce_request_t* request)
|
|
{
|
|
uint64_t jobid;
|
|
int signo;
|
|
xce_jobctrl_t* jobctrl = request->body.jobctrlmsg;
|
|
int rc = 0;
|
|
|
|
switch (jobctrl->action) {
|
|
case JobControl_u_siginfo:
|
|
jobid = jobctrl->jobid;
|
|
signo = jobctrl->u.siginfo->signo;
|
|
rc = job_kill(jobid, signo);
|
|
break;
|
|
case JobControl_u_ckptoption:
|
|
rc = job_chkpnt(jobctrl->jobid, jobctrl->u.ckptoption->method,
|
|
jobctrl->u.ckptoption->kill);
|
|
break;
|
|
default:
|
|
rc = -EINVAL;
|
|
ERROR("unknown job control action %d", jobctrl->action);
|
|
break;
|
|
}
|
|
|
|
return rc;
|
|
}
|
|
|
|
static int handle_runtasks(xce_request_t* request)
|
|
{
|
|
xce_taskrun_t* runmsg = request->body.runtaskmsg;
|
|
int rc = 0;
|
|
int i = 0;
|
|
char jobidStr[64];
|
|
#if 0
|
|
job_startinfo_t startinfo;
|
|
#endif
|
|
|
|
if (runmsg->nhosts != 1) {
|
|
/* something wrong */
|
|
rc = -EINVAL;
|
|
|
|
notify_task_fail(runmsg->jobid, runmsg->clientidx, -1, rc);
|
|
return rc;
|
|
}
|
|
|
|
if (strcmp(runmsg->taskhosts[0]->host, get_setting("myname")) != 0) {
|
|
/* something wrong */
|
|
rc = -EINVAL;
|
|
|
|
notify_task_fail(runmsg->jobid, runmsg->clientidx, -1, rc);
|
|
return rc;
|
|
}
|
|
|
|
fmt_jobid(jobidStr, sizeof(jobidStr) - 1, runmsg->jobid);
|
|
printf("run task: jobid = %s\n", jobidStr);
|
|
printf("run task: user = %s\n", runmsg->taskhosts[0]->execuser);
|
|
printf("run task: cmdline = %s\n", runmsg->taskhosts[0]->cmdline);
|
|
printf("run task: exdata = %s\n", runmsg->taskhosts[0]->exdata);
|
|
printf("run task: ntasks =%d\n", runmsg->taskhosts[0]->ntasks);
|
|
|
|
#if 0
|
|
startinfo.exechome = runmsg->taskhosts[0]->exechome;
|
|
startinfo.execcwd = runmsg->taskhosts[0]->execcwd;
|
|
startinfo.execuser = runmsg->taskhosts[0]->execuser;
|
|
startinfo.cmdline = runmsg->taskhosts[0]->cmdline;
|
|
startinfo.exdata = runmsg->taskhosts[0]->exdata;
|
|
startinfo.launch_type = runmsg->taskhosts[0]->launch_type;
|
|
#endif
|
|
|
|
for (i = 0; i < runmsg->taskhosts[0]->ntasks; i++) {
|
|
int pid = 0;
|
|
int pgid = 0;
|
|
|
|
rc = run_task(runmsg->taskhosts[0]->cmdline, &pid);
|
|
if (rc < 0) {
|
|
notify_task_fail(runmsg->jobid, runmsg->clientidx, i, rc);
|
|
} else {
|
|
int idx;
|
|
|
|
printf("job %s task %d started, PID: %d\n", jobidStr, i, pid);
|
|
|
|
idx = get_avail_run_task_slots();
|
|
if (idx < 0) {
|
|
printf("too many running tasks\n");
|
|
} else {
|
|
run_tasks[idx].jobid = runmsg->jobid;
|
|
run_tasks[idx].taskid = i;
|
|
run_tasks[idx].clientidx = runmsg->clientidx;
|
|
run_tasks[idx].pid = pid;
|
|
run_tasks[idx].ts = 0;
|
|
}
|
|
notify_task_start(runmsg->jobid, runmsg->clientidx, i, pid, pgid);
|
|
}
|
|
}
|
|
|
|
return 0;
|
|
}
|
|
|
|
static int myreq_handler(xce_request_t* request, void* args)
|
|
{
|
|
int rc = 0;
|
|
|
|
switch (request->opcode) {
|
|
case OPCode_push:
|
|
if (request->body_which == Request_body_datamsg) {
|
|
dataset_t* datamsg = request->body.datamsg;
|
|
|
|
switch (datamsg->table) {
|
|
case XceData_data_schedjobs:
|
|
case XceData_data_jobs:
|
|
sched_on_jobs(datamsg->nrows, (xce_job_t**)datamsg->rows);
|
|
break;
|
|
case XceData_data_schedworkers:
|
|
case XceData_data_workers:
|
|
sched_on_workers(datamsg->nrows, (xce_worker_t**)datamsg->rows);
|
|
break;
|
|
default:
|
|
break;
|
|
}
|
|
}
|
|
break;
|
|
case OPCode_startjob:
|
|
rc = handle_startjob(request);
|
|
break;
|
|
case OPCode_notify:
|
|
rc = handle_notify(request);
|
|
break;
|
|
case OPCode_jobcontrol:
|
|
rc = handle_jobcontrol(request);
|
|
break;
|
|
case OPCode_runtasks:
|
|
rc = handle_runtasks(request);
|
|
break;
|
|
default:
|
|
break;
|
|
}
|
|
|
|
return rc;
|
|
}
|
|
|
|
static void check_run_jobs()
|
|
{
|
|
int i;
|
|
time_t now = time(NULL);
|
|
|
|
for (i = 0; i < MAX_RUN_JOBS; i++) {
|
|
int wstatus;
|
|
|
|
if (run_jobs[i].jobid == 0) {
|
|
continue;
|
|
}
|
|
|
|
if (run_jobs[i].pid <= 0) {
|
|
continue;
|
|
}
|
|
|
|
if (waitpid(run_jobs[i].pid, &wstatus, WNOHANG | WUNTRACED | WCONTINUED)
|
|
<= 0) {
|
|
/* fake job rusage */
|
|
int npids = 1;
|
|
int npgids = 0;
|
|
int nusage = 2;
|
|
/* bug-1735: match the real worker's collect_job_rusages()
|
|
* resnames (xcecd_cgroups.c), not the old never-matching
|
|
* "cputime"/"mem" strings. */
|
|
xce_rusage_t data[] = {
|
|
{ "cpu_time_sec", 100.0 },
|
|
{ "memory_rss", 200.0 }
|
|
};
|
|
xce_rusage_t* usages[] = { &(data[0]), &(data[1]) };
|
|
|
|
if (now - run_jobs[i].ts > RUSAGE_UPDATE_INTERVAL) {
|
|
xcecli_update_jusage(active_ctx->ctx, run_jobs[i].jobid, npids,
|
|
&(run_jobs[i].pid), npgids, NULL, nusage,
|
|
&usages[0]);
|
|
run_jobs[i].ts = now;
|
|
}
|
|
|
|
continue;
|
|
}
|
|
|
|
if (WIFSTOPPED(wstatus)) {
|
|
xcecli_notify_jobstop(active_ctx->ctx, run_jobs[i].jobid);
|
|
} else if (WIFCONTINUED(wstatus)) {
|
|
xcecli_notify_jobresume(active_ctx->ctx, run_jobs[i].jobid);
|
|
} else if (WIFEXITED(wstatus) || WIFSIGNALED(wstatus)) {
|
|
xce_jobfinish_t finish;
|
|
char jobidStr[64];
|
|
|
|
finish.jobid = run_jobs[i].jobid;
|
|
|
|
if (WIFEXITED(wstatus)) {
|
|
finish.exit_code = WEXITSTATUS(wstatus);
|
|
} else {
|
|
finish.exit_code = 128 + WTERMSIG(wstatus);
|
|
}
|
|
|
|
fmt_jobid(jobidStr, sizeof(jobidStr) - 1, finish.jobid);
|
|
printf("job %s exit code %d\n", jobidStr, finish.exit_code);
|
|
xcecli_notify_jobfinish(active_ctx->ctx, &finish);
|
|
run_jobs[i].jobid = 0;
|
|
run_jobs[i].pid = 0;
|
|
}
|
|
}
|
|
}
|
|
|
|
static void check_run_tasks()
|
|
{
|
|
int i;
|
|
time_t now = time(NULL);
|
|
|
|
for (i = 0; i < MAX_RUN_TASKS; i++) {
|
|
int wstatus;
|
|
|
|
if (run_tasks[i].jobid == 0) {
|
|
continue;
|
|
}
|
|
|
|
if (run_tasks[i].pid <= 0) {
|
|
continue;
|
|
}
|
|
|
|
if (waitpid(run_tasks[i].pid, &wstatus,
|
|
WNOHANG | WUNTRACED | WCONTINUED)
|
|
<= 0) {
|
|
/* fake job rusage */
|
|
int npids = 1;
|
|
int npgids = 0;
|
|
int nusage = 2;
|
|
/* bug-1735: match the real worker's collect_job_rusages()
|
|
* resnames (xcecd_cgroups.c), not the old never-matching
|
|
* "cputime"/"mem" strings. */
|
|
xce_rusage_t data[] = {
|
|
{ "cpu_time_sec", 100.0 },
|
|
{ "memory_rss", 200.0 }
|
|
};
|
|
xce_rusage_t* usages[] = { &(data[0]), &(data[1]) };
|
|
|
|
if (now - run_tasks[i].ts > RUSAGE_UPDATE_INTERVAL) {
|
|
xcecli_update_jusage(active_ctx->ctx, run_tasks[i].jobid, npids,
|
|
&(run_tasks[i].pid), npgids, NULL, nusage,
|
|
&usages[0]);
|
|
run_tasks[i].ts = now;
|
|
}
|
|
continue;
|
|
}
|
|
|
|
if (WIFEXITED(wstatus) || WIFSIGNALED(wstatus)) {
|
|
xce_task_finish_t finish = { .jobid = run_tasks[i].jobid,
|
|
.host = get_setting("myname"),
|
|
.taskidx = run_tasks[i].taskid,
|
|
.clientidx = run_tasks[i].clientidx,
|
|
.exit_code = 0 };
|
|
char jobidStr[64];
|
|
|
|
if (WIFEXITED(wstatus)) {
|
|
finish.exit_code = WEXITSTATUS(wstatus);
|
|
} else {
|
|
finish.exit_code = 128 + WTERMSIG(wstatus);
|
|
}
|
|
|
|
fmt_jobid(jobidStr, sizeof(jobidStr) - 1, finish.jobid);
|
|
printf("job %s task %d exit code %d\n", jobidStr,
|
|
finish.taskidx, finish.exit_code);
|
|
xcecli_notify_taskfinish(active_ctx->ctx, &finish);
|
|
run_tasks[i].jobid = 0;
|
|
run_tasks[i].pid = 0;
|
|
}
|
|
}
|
|
}
|
|
|
|
#define HEARTBEAT_INTERVAL (60)
|
|
|
|
static int worker_house_keeping(void* arg)
|
|
{
|
|
command_ctx_t* ctx = (command_ctx_t*)arg;
|
|
int rc = 0;
|
|
|
|
xce_use(ctx);
|
|
|
|
check_run_jobs();
|
|
check_run_tasks();
|
|
|
|
rc = xcecli_heartbeat(ctx->ctx, HEARTBEAT_INTERVAL);
|
|
if (rc < 0) {
|
|
cmdutil_rpc_error("xcecli_heartbeat", rc);
|
|
}
|
|
|
|
return rc;
|
|
}
|
|
|
|
int main(int argc, char* argv[])
|
|
{
|
|
struct arguments arguments;
|
|
char* buf;
|
|
char* logfile = "/tmp/xce_test.log";
|
|
char* loglevel = "info";
|
|
int logsize = 64 * 1024 * 1024;
|
|
int logcount = 10;
|
|
/* build-time-versioning task 3.2: xce_test's own --version/-V output
|
|
* (via argp_program_version) -- static so it outlives this point in
|
|
* main() (contract documented on xce_compose_version() in utils.c). */
|
|
static char version_buf[128];
|
|
|
|
memset(&arguments, 0x0, sizeof(arguments));
|
|
|
|
/* dirty-content-fingerprint task 3.2: xce_test is not a daemon -- it has
|
|
* no startup log line at all (see anatomy/build-time-versioning notes),
|
|
* so there is no second call site to keep divergent here (unlike
|
|
* controller/main.c and scheduler/src/main.c, which each keep a
|
|
* separate, untouched xce_compose_version() buffer for their startup
|
|
* INFO log per Req 3.2). version_buf can safely be composed directly
|
|
* via xce_compose_version_ex() to pick up the dirty_fingerprint suffix.
|
|
* Must run before argp_parse(): argp reads argp_program_version by
|
|
* pointer when --version/-V is processed. */
|
|
xce_compose_version_ex(version_buf, sizeof(version_buf), "xce v0.0.1-",
|
|
STR(VERSION_COMMIT_ID), STR(VERSION_BUILD_TIME),
|
|
STR(VERSION_DIRTY_FINGERPRINT));
|
|
argp_program_version = version_buf;
|
|
|
|
argp_parse(&argp, argc, argv, 0, 0, &arguments);
|
|
|
|
if (arguments.verbose != 0) {
|
|
loglevel = verbose_to_loglevel(arguments.verbose);
|
|
}
|
|
|
|
/* Remember the user's level so wait-primary can restore it after temporarily
|
|
* raising the threshold to silence its poll-loop failover spam. */
|
|
g_active_loglevel = loglevel;
|
|
|
|
mylog_init(logfile, loglevel, logsize, logcount);
|
|
|
|
/* xce-test-cli-ux task 1.3, Requirement 9.9: replaces the former bare
|
|
* `signal(SIGINT, sigbreak);` call. glibc's signal() installs with
|
|
* SA_RESTART SET (BSD semantics), which silently restarts a blocking
|
|
* slow syscall (read(), etc.) interrupted by SIGINT instead of
|
|
* returning EINTR -- see xcetest_interrupt.h's file header for the
|
|
* full rationale. xcetest_install_sigint() installs the same
|
|
* sigbreak() handler via sigaction() with SA_RESTART explicitly
|
|
* cleared, so a later blocking read (e.g. the confirmation prompt
|
|
* built on top of this task) reliably observes EINTR. */
|
|
xcetest_install_sigint(sigbreak);
|
|
|
|
memset(&run_jobs, 0x0, sizeof(run_jobs));
|
|
memset(&run_tasks, 0x0, sizeof(run_tasks));
|
|
|
|
idgen_ctx_init(&idctx);
|
|
|
|
/* bug-1420 follow-up: folding pid into apiserverId alone (4 bits, 16
|
|
* buckets) at the jsub call site still collides too often across this
|
|
* harness's rapid, per-invocation-fresh xce_test processes (confirmed
|
|
* empirically -- a 10-round chaotic soak still produced same-second
|
|
* duplicate jobids after that fix landed). idgen_ctx_init() always
|
|
* resets sequence to 0, and MAKE_JOBID's sequence field alone has 20
|
|
* bits (config.h N_SEQUENCE_BITS) of headroom -- two orders of
|
|
* magnitude more than the namespacing fields. Re-seed it here with a
|
|
* value that mixes pid (stable per process) with monotonic-clock
|
|
* nanoseconds (varies even across processes launched back-to-back in
|
|
* the same wall-clock second, unlike idgen's own second-resolution
|
|
* timestamp), masked to N_SEQUENCE_BITS so it can never bleed into the
|
|
* adjacent tag/timestamp bits MAKE_JOBID packs beside it. idgen()'s own
|
|
* ROUNDUP-to-N_ARRAY_IDX_BITS logic for array jobs still applies
|
|
* unchanged on top of this seed. */
|
|
{
|
|
struct timespec ts;
|
|
clock_gettime(CLOCK_MONOTONIC, &ts);
|
|
idctx.sequence = ((uint32_t)getpid() ^ (uint32_t)ts.tv_nsec)
|
|
& (uint32_t)UMAX_BITS(N_SEQUENCE_BITS);
|
|
}
|
|
|
|
/* errexit default, modelled on shell `-e`/`+e`, keyed on the stdin mode:
|
|
* batch (piped/redirected stdin) → `-e`: abort the read loop on the first
|
|
* failing command and exit non-zero (a script wants fail-fast + verdict).
|
|
* interactive (TTY) REPL → `+e`: a failed command prints its error and the
|
|
* loop CONTINUES (never exit on a typo).
|
|
* Override at any time with `set -e` / `set +e` (or `set continue-on-error`). */
|
|
g_interactive = isatty(STDIN_FILENO);
|
|
g_continue_on_error = g_interactive ? 1 : 0;
|
|
|
|
/* xce-test-cli-ux task 2.3, Requirement 7: register command-name tab
|
|
* completion, but ONLY for a genuinely interactive session -- design.md's
|
|
* explicit precondition on xcetest_complete(). A piped/redirected/
|
|
* scripted stdin never calls readline()'s completion machinery anyway
|
|
* (no terminal to press Tab on), but gating the registration itself
|
|
* (rather than relying on that) keeps the precondition explicit and
|
|
* matches this file's existing g_interactive-gated conventions. */
|
|
if (g_interactive) {
|
|
rl_attempted_completion_function = xcetest_complete;
|
|
}
|
|
|
|
/* xce-test-rc-file task 2.2: auto-load $XDG_CONFIG_HOME/xce_test/rc (or
|
|
* $HOME/.config/xce_test/rc) before the read loop starts, for both
|
|
* interactive and scripted sessions alike -- unless --norc was given.
|
|
* A missing/unresolvable rc file is silent (Req 1.3); a failing rc-file
|
|
* line is handled entirely by preprocess_and_run's existing abort/continue
|
|
* accounting (Req 3) -- an abort short-circuits straight to the same
|
|
* exit-code convention used at the bottom of this function, skipping the
|
|
* read loop entirely (Req 1.4/3.3). */
|
|
if (!arguments.norc) {
|
|
char* rcpath = rcfile_resolve_path();
|
|
|
|
if (rcpath != NULL) {
|
|
int rcrc = rcfile_load(rcpath, preprocess_and_run);
|
|
|
|
free(rcpath);
|
|
|
|
if (rcrc) {
|
|
return g_session_failed ? 1 : 0;
|
|
}
|
|
}
|
|
}
|
|
|
|
while (!done) {
|
|
char* s;
|
|
int rc;
|
|
|
|
prompt[0] = 0x0;
|
|
fill_prompt(active_ctx, prompt);
|
|
strcat(prompt, ">");
|
|
|
|
/* xce-test-cli-ux task 3.2, Requirement 1.2/1.4: reset break_out
|
|
* immediately before this specific readline() call, so the check
|
|
* just below attributes a NULL return to a fresh interrupt that
|
|
* happened DURING THIS call, never to a stale flag left over from
|
|
* an earlier interrupted command or an earlier interrupted
|
|
* readline() call -- mirrors cmd_hello's own pre-existing
|
|
* "break_out = 0;-then-loop" idiom just above it in this file. */
|
|
break_out = 0;
|
|
buf = readline(prompt);
|
|
|
|
if (buf == NULL) {
|
|
/* task 1.3's own flagged concern (tasks.md Implementation
|
|
* Notes): glibc/libedit's readline() returns NULL on BOTH a
|
|
* genuine EOF (Ctrl-D) AND a read interrupted by SIGINT
|
|
* (confirmed empirically -- xcetest_install_sigint() clears
|
|
* SA_RESTART, so the underlying read() is interrupted rather
|
|
* than transparently retried, and libedit's readline() itself
|
|
* does not retry across that EINTR; it just returns NULL,
|
|
* discarding whatever partial line was in its edit buffer).
|
|
* The two cases are NOT distinguishable from readline()'s
|
|
* return value alone -- break_out (freshly reset above) is
|
|
* what disambiguates them: a real Ctrl-C sets it via
|
|
* sigbreak() before readline() returns; a genuine EOF leaves
|
|
* it untouched. On the interrupt path, discard the (already
|
|
* libedit-discarded) input line and simply redisplay the
|
|
* prompt -- the session must never exit as a side effect of a
|
|
* single Ctrl-C keypress (Requirement 1.4). */
|
|
if (break_out) {
|
|
break_out = 0;
|
|
printf("\n"); /* fresh line: Ctrl-C doesn't itself echo one */
|
|
continue;
|
|
}
|
|
|
|
execute_line("quit");
|
|
continue;
|
|
}
|
|
|
|
if (strlen(buf) == 0) {
|
|
free(buf);
|
|
continue;
|
|
}
|
|
|
|
s = buf;
|
|
|
|
rc = preprocess_and_run(s);
|
|
free(buf);
|
|
if (rc) {
|
|
break; /* abort-on-failure requested loop stop */
|
|
}
|
|
}
|
|
|
|
return g_session_failed ? 1 : 0;
|
|
}
|