feat(xce-harness-component): relocate xce_test into standalone harness component

Move xce_test's 15 production sources + headers from src/controller/testsuite/
into a new src/harness/ component with its own meson.build, and strip every
now-dangling controller-side reference (11 controller_unit_test_src entries,
4 standalone test-binary blocks, 11 controller_test.c extern/RUN_TEST lines)
so the controller build is never left broken.

Fixes a broken relative-symlink bridge (xce_test_controller_link) added to
keep two controller-embedded tests (test_test_cli.c, test_xcetest_process_entry.c)
finding the relocated xce_test binary until task 2.1 moves them too.

Discovered and documented in design.md: four controller-domain regression
tests call functions directly from four of xce_test's own files, requiring
controller_unit_test to permanently recompile them from src/harness/ — a
real dependency neither the original discovery nor design audit caught.

controller_unit_test: 1228 Tests 0 Failures 0 Ignored (verified via the
Unity binary's own printed summary line, not just meson's aggregate count).

Task 1.1 of xce-harness-component.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-14 05:08:15 +00:00
parent b8ef82c1b2
commit cf55d691de
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
36 changed files with 550 additions and 180 deletions

View file

@ -0,0 +1,74 @@
#pragma once
/*
* configcompare.h — cross-member active-configuration comparator
* (config-upgrade-snapshot task 8.4, Requirement 4.2).
*
* Compares each cluster member's locally-held ACTIVE configuration content
* hash, recomputed FRESH from that member's own on-disk configuration file
* (never trusted from a cached/embedded tag), and reports whether every
* member agrees.
*
* A new sibling tool to walcompare, NOT an extension of it: walcompare's
* internal model is WAL-entry-indexed (lsn -> term, pcrc) -- an awkward fit
* for a single (tag, sha256) value per member with no LSN dimension
* (design.md "Regression Tooling"). Mirrors snapdiff_paths's exact
* signature/reporting shape instead.
*
* Offline/file-based: no live RPC to any cluster member, consistent with
* every other tool in this family (walcompare, snapdiff, snapverify).
*
* Strictly read-only: opens files O_RDONLY, mutates nothing.
*/
#include <stdint.h>
#include <stdio.h>
/* Bound on the number of members a single call can compare -- generous for
* any real cluster size this codebase targets; a caller with more members
* than this is almost certainly passing a malformed argument list. */
#define CONFIGCOMPARE_MAX_MEMBERS 32
typedef struct {
int n; /* member count passed in */
int read_rc[CONFIGCOMPARE_MAX_MEMBERS]; /* 0 = resolved+hashed ok,
* else -errno; entries at
* or past a short-circuit
* failure are unset (0)
* and must be ignored --
* see all_present */
uint8_t sha256[CONFIGCOMPARE_MAX_MEMBERS][32]; /* valid only where
* read_rc[i] == 0 */
int all_present; /* 1 iff every member (0..n-1) was resolved and
* hashed successfully */
int all_identical; /* 1 iff all_present && every member's sha256
* equals the first member's */
int first_divergent_index; /* -1 if all_identical or !all_present;
* else the first index (after the
* reference, member 0) whose hash
* differs from member 0's */
} configcompare_report_t;
/* Compare the active-configuration content hash of `n` members. Each
* `conf_paths[i]` is a member's `workdir/name` directory -- the SAME
* directory persist.c's `read_active_config_bytes()` and config_repl.c's
* `local_config_sha256()` both resolve `conf/current` (and the versioned
* config file it points at) under, matching where this spec's snapshot-embed
* step also reads from. NOT the raw `conf/current` file itself -- this
* function resolves `<path>/conf/current` and `<path>/conf/<versioned-file>`
* internally, exactly like those two functions do.
*
* Prints a human-readable report to `out` (may be NULL) and fills `*rep`
* (may be NULL). Returns:
* 0 if every member's config was readable and all hashes are identical
* 1 if every member's config was readable but at least one diverges
* (see `*rep->first_divergent_index` for which)
* <0 (negative errno) if `n < 2` (nothing to compare), `n` exceeds
* CONFIGCOMPARE_MAX_MEMBERS, or any member's config could not be
* resolved/read -- mirrors snapdiff_paths's early-return-on-open-
* failure style: the first unreadable member short-circuits the scan,
* and `*rep` is still filled with whatever was learned about the
* members checked before the failure, so the report can say which
* member and why. A member that could not be read is NOT evidence of
* matching OR mismatching configuration -- it is a distinct third
* outcome from "identical" and "divergent". */
int configcompare_paths(const char* const* conf_paths, int n, FILE* out,
configcompare_report_t* rep);

View file

@ -0,0 +1,101 @@
#pragma once
/*
* confirm.h -- destructive-command confirmation gate: prompts the operator
* on stderr before a destructive command runs, offers a session-scoped
* toggle to skip that prompt, and treats a Ctrl-C during the prompt the
* same as declining it (xce-test-cli-ux spec, task 2.1; Requirements
* 9.1-9.4, 9.6-9.9. Requirement 9.5 -- the five-command enumeration and
* each cmd_*'s own `--yes` bypass flag -- is task 3.3, NOT this module;
* this module is the generic, command-agnostic gate mechanism only).
*
* THE MECHANISM (see design.md's "Confirmation Gate" component and its
* "Destructive Command Confirmation Gate" flowchart):
* - confirm_destructive() is the gate itself. It internally checks, in
* order: the session-scoped skip toggle (confirm_is_skip_set()), then
* whether the top-level session is even interactive
* (xcetest_is_interactive(), see below). If either says "don't
* prompt", it returns 1 (proceed) immediately WITHOUT touching stdin
* or printing anything. Otherwise it prints a yes/no prompt to
* stderr and performs one blocking read of the answer from stdin.
* - confirm_destructive() deliberately has NO parameter for a
* per-command `--yes` bypass flag -- that flag is per-INVOCATION, not
* session state, so it cannot be folded into confirm_is_skip_set()'s
* persistent toggle without leaking into the next command. Task 3.3's
* five call sites (cmd_jobkill, cmd_shutdown, cmd_qctrl, cmd_hostctrl,
* cmd_upgradeconfig) are responsible for short-circuiting around
* confirm_destructive() entirely when their OWN --yes flag was given,
* e.g.:
* if (!opts.yes && !confirm_destructive("kill job 1234"))
* return NOT_RUN;
* (design.md's Implementation Notes describes this same three-way
* bypass -- toggle, TTY, --yes -- as one boolean expression each call
* site evaluates; confirm_destructive() itself owns the first two,
* the call site owns the third because only it knows its own flag.)
*
* RETURN CONTRACT (confirm_destructive): exactly two outcomes, matching
* design.md's Service Interface postcondition -- no third "error" case is
* defined for this function.
* 1 -- proceed: skip-toggle set, OR non-interactive session, OR the
* operator answered affirmatively ("y"/"yes", case-insensitive,
* surrounding whitespace ignored).
* 0 -- not run: the operator declined (anything else, including a bare
* Enter or unrecognized text), OR stdin hit EOF, OR the prompt's
* blocking read was interrupted by a real Ctrl-C/SIGINT (EINTR;
* Requirement 9.9 -- treated identically to a decline, never
* retried). The caller must report this as "not run", distinct
* from "failed" (Requirement 9.2/9.9).
*
* WHY THE PROMPT READ IS A PLAIN fgets(), NOT libedit's readline(): a
* y/n answer routed through readline() would pollute the session's
* command history and behave inconsistently when stdin isn't a TTY
* (design.md's Implementation Notes / research.md). It also would not
* reliably surface EINTR the way a bare blocking libc read does.
*
* WHY EINTR IS RELIABLE HERE: task 1.3 (xcetest_interrupt.h,
* xcetest_install_sigint()) replaced test.c's old `signal(SIGINT,
* sigbreak)` installation with an explicit sigaction() call with
* SA_RESTART cleared, once, at process startup. That is a PRECONDITION
* of this module, not something confirm_destructive() re-checks on every
* call -- see design.md's Service Interface preconditions.
*/
/* Prompts the operator (on stderr) to confirm a destructive action
* described by `action_description` (a short phrase completing a
* sentence like "About to <action_description>. Continue? [y/N] " --
* e.g. "kill job 1234"), unless the skip toggle is set or the session
* isn't interactive, in which case it returns 1 without printing
* anything or reading stdin. See the return contract above. */
int confirm_destructive(const char* action_description);
/* Sets/gets the session-scoped confirmation-skip toggle (Requirement
* 9.6/9.7), in the exact same style as test.c's existing g_continue_on_error
* `-e`/`+e` toggle: a plain `static int`, defaulting to 0 (confirmation
* required -- NOT skipped) at process start, flipped by `cmd_set` (task
* 3.3 wires the `set` command's new branch; this module only owns the
* flag itself). While set, confirm_destructive() proceeds without
* prompting regardless of TTY state, so a `source`d script (Requirement
* 13) can opt in to unattended destructive commands explicitly. */
void confirm_set_skip(int skip);
int confirm_is_skip_set(void);
/* Read-only accessor over test.c's private `g_interactive` flag (set once
* in main() from isatty(STDIN_FILENO), never changed after that) --
* mirrors xcetest_interrupted()'s accessor pattern over test.c's private
* `break_out` flag (xcetest_interrupt.h, task 1.3). Declared here, not in
* xcetest_interrupt.h, because confirm.c is its only consumer; g_interactive
* itself is unrelated to that header's SIGINT/EINTR concern.
*
* Per design.md's Confirmation Gate invariant, this is deliberately the
* TOP-LEVEL session's own stdin, not "is this line currently being read
* from a sourced file" -- Requirement 13.7 (a later task) depends on that
* distinction: a destructive command inside a `source`d file in an
* otherwise-interactive session must still prompt unless the skip toggle
* is set.
*
* Implemented in test.c. test.c itself is never linked into
* controller_unit_test (it pulls in libedit/cJSON/libxce/Cap'n Proto/
* OpenSSL and defines its own main() -- see xcetest_interrupt.h's header
* comment for the same reasoning applied to xcetest_interrupted()), so
* test_confirm.c supplies its own local fake definition, exactly as
* test_interruptcall.c already does for xcetest_interrupted(). */
int xcetest_is_interactive(void);

View file

@ -0,0 +1,154 @@
#pragma once
/*
* flagspec.h -- flag-descriptor table format, getopt_long-based parsing
* core, typo suggestion, and help-flag-block rendering for xce_test's
* command-line surface (xce-test-cli-ux spec, tasks 1.1/1.2; Requirements
* 2.1-2.4, 3.1, 6.1-6.4, 8.2, 10.1-10.3; the flagspec_t shape is a stable
* contract also relied on by task 3.1 (migrating every existing cmd_*
* command onto this table)).
*
* bug-1711's root cause was that xce_test's flag parsing had no shared
* helper at all: a plain getopt() loop per command, each with its own
* hand-written switch/default, drifted independently from its
* hand-authored USAGE_* string. This module gives every flag-accepting
* command ONE descriptor table to build its getopt_long() option table
* from, ONE shared rejection path (with typo suggestion) to call on any
* parse failure, and ONE renderer for the flag-listing portion of its
* help text, so the table is the single source of truth for all three.
*
* Task 1.1 declared/implemented flagspec_build_opts() and
* flagspec_reject() (without typo suggestion). THIS TASK (1.2) adds
* flagspec_render_help() and extends flagspec_reject()'s rejection
* message with a "did you mean" suggestion (Requirement 10) when exactly
* one flag in `spec` closely matches the unrecognized token.
*/
#include <getopt.h>
#include <stddef.h>
/* One flag's full description: its short/long spellings, the real-world
* concept it represents (for cross-command naming consistency, see
* design.md Requirement 6.2), whether getopt_long() expects a value, and
* documentation state consumed by later tasks (help rendering).
*
* Any two flagspec_t[] tables (i.e. any two commands) that describe the
* "same concept" (e.g. a target host, a job id) MUST use the identical
* long_opt string for it -- enforced by table review, not at runtime. */
typedef struct {
char short_opt; /* 0 if long-only */
const char* long_opt; /* stable, cross-command-consistent name;
* spelled WITHOUT leading dashes, e.g.
* "host", not "--host" */
const char* concept; /* e.g. "host", "jobid", "command-line";
* consumed by help rendering (task 1.2),
* not by flagspec_build_opts()/
* flagspec_reject() */
int has_arg; /* no_argument | required_argument, per
* <getopt.h>; this design uses no
* optional_argument flags anywhere */
int required; /* 1 if the command fails without it;
* enforcement is each command's own
* post-parse-loop check (see design.md's
* bug-1704/cmd_qctrl note), not this
* module's job */
const char* default_desc; /* human text describing the default when
* the flag is omitted, NULL if none;
* consumed by help rendering (task 1.2) */
} flagspec_t;
/* Synthetic getopt_long() `val` assigned to a long-only flag (short_opt
* == 0) at spec index i: FLAGSPEC_LONG_ONLY_BASE + i. Chosen safely above
* the full unsigned-char range (0-255) so it can never collide with a
* real short_opt value, with getopt_long()'s own -1 end-of-options
* sentinel, or with '?' (parse error). Every flagspec_t entry -- long-only
* or not -- therefore returns a value from getopt_long() that a caller's
* switch() can distinguish. */
#define FLAGSPEC_LONG_ONLY_BASE 0x100
/* Builds a getopt_long()-compatible option table and optstring from a
* flagspec_t descriptor table.
*
* `out_opts` must have room for at least n+1 entries -- this function
* appends the trailing all-zero sentinel entry getopt_long() requires to
* terminate the table at index n. `out_optstring` is filled with the
* short-option spec getopt_long()'s `optstring` argument expects: each
* flag with a nonzero short_opt contributes that character, followed by
* ':' when has_arg == required_argument; a long-only flag (short_opt ==
* 0) contributes nothing. The result is NUL-terminated.
*
* Invariant: the built optstring never begins with ':' -- this module
* deliberately does not use getopt_long()'s own leading-':' convention
* for distinguishing "missing argument" from "unrecognized option";
* flagspec_reject() makes that distinction itself by looking the
* offending token up in `spec`. The caller must set `opterr = 0` before
* its getopt_long() loop so libc's own diagnostic never doubles up with
* flagspec_reject()'s.
*
* Returns 0 on success. Returns a negative errno on failure:
* -EINVAL spec/out_opts/out_optstring is NULL, n == 0,
* out_optstring_cap == 0, or a spec entry is malformed
* (long_opt is NULL/empty, or has_arg is neither no_argument
* nor required_argument)
* -ENOSPC out_optstring_cap is too small to hold the built optstring
* (including its terminating NUL)
*
* On any failure, the contents of out_opts/out_optstring are unspecified
* -- the caller must not use them. */
int flagspec_build_opts(const flagspec_t* spec, size_t n,
struct option* out_opts, char* out_optstring,
size_t out_optstring_cap);
/* Shared rejection path for an unrecognized or malformed flag, called
* from a command's getopt_long() loop's `case '?':` arm (or right after
* the loop, for a bad_flag captured some other way).
*
* `bad_flag` is the literal command-line token that failed -- e.g. "-z"
* or "--bogus" -- ordinarily read as `argv[optind - 1]` right after
* getopt_long() returns '?'. flagspec_reject() looks `bad_flag` up
* against `spec` (matching a "-X" token by short_opt, a "--name" token by
* long_opt) to decide which case applies:
* - `bad_flag` matches no entry in `spec` ->
* "unrecognized flag" (Requirement 2.1)
* - `bad_flag` matches an entry with has_arg == required_argument ->
* "flag ... requires a value" (Requirement 2.2)
* - `bad_flag` matches an entry that takes no argument (getopt_long()
* still returned '?', e.g. "--flag=value" given for a no_argument
* long flag) -> "flag ... does not take a value"
* Either way, it prints the offending flag and `usage_text` (Requirement
* 2.4) to stderr only (Requirement 3.1 -- never stdout) and returns a
* negative value; it never partially executes the command.
*
* Typo suggestion (Requirement 10): when `bad_flag` matches no entry in
* `spec` (the "unrecognized flag" case above) and exactly one entry's
* short or long spelling closely matches it by edit distance (see
* flagspec.c's is_close_match()), the rejection message names that flag
* as the likely intended one (Requirement 10.1). When zero or more than
* one entry qualifies as a close match, no guess is made (Requirement
* 10.2) -- an ambiguous typo is treated the same as a typo that matches
* nothing at all. Suggestion is never attempted for the "requires a
* value"/"does not take a value" cases, since those already identify the
* exact intended flag.
*
* `usage_text` is expected to be the command's one-line usage summary
* (e.g. "jobkill -j <jobid> [-s <signo>]"), not its full help block; it
* may be NULL to omit the usage line entirely (defensive only -- every
* real call site always has one).
*
* Returns -EINVAL (a fixed, nonzero, negative-errno result) in every
* case. */
int flagspec_reject(const char* cmd_name, const flagspec_t* spec, size_t n,
const char* bad_flag, const char* usage_text);
/* Renders the flag-listing portion of a command's help text (Requirement
* 8.2) directly from `spec`: one line per flag, each stating its short
* form (when it has one), its long form, whether it is required or
* optional, and its default-value description (when `default_desc` is
* non-NULL). This is the flag-block ONLY -- the surrounding description,
* example, and related-command prose (Requirement 8.1/8.3/8.4) stay
* hand-authored, per design.md's "Flag-descriptor table as the single
* source of truth" decision.
*
* Returns a heap-allocated, NUL-terminated string the caller must free().
* Returns NULL if `spec` is NULL or n == 0, or if the underlying
* allocation fails; it never returns a partially-written buffer. */
char* flagspec_render_help(const flagspec_t* spec, size_t n);

View file

@ -0,0 +1,326 @@
#pragma once
/*
* interruptcall.h -- xcetest_run_interruptible(): a detached-worker-thread
* wrapper that lets a command make a blocking xce_* RPC call cancellably
* from the operator's Ctrl-C, without ever touching libxce's own blocking
* primitive (xce-test-cli-ux spec, task 1.4; Requirements 1.1, 1.2, 1.3,
* 1.4).
*
* ROOT CAUSE this module addresses: every blocking xce_* call
* (src/lib/xceapi.c) waits on pthread_cond_timedwait() with a fixed
* 10-second deadline (cbuf_deadline_10s/cbuf_timedget, src/lib/mylock.c) --
* a wait a signal-setting flag alone cannot interrupt (glibc's condvar
* wait retries internally). Modifying that shared libxce primitive is out
* of this spec's boundary (research.md's Architecture Pattern Evaluation:
* "Modify libxce's cbuf_timedget to poll a cancellation token" -- Rejected,
* out of boundary). The fix is entirely client-side: run the existing
* blocking call, UNMODIFIED, on a detached worker thread; have the calling
* thread poll xcetest_interrupted() (xcetest_interrupt.h, task 1.3) on a
* short interval instead of blocking on the call itself; on interrupt,
* abandon the wait and return immediately, leaving the worker thread to
* finish (or hit its own 10s deadline) on its own -- satisfying "leave the
* connection intact" (Requirement 1.5) and the design's "cancelling a
* client-side wait does not need to cancel the server-side request" note.
*
* THE HARD PART this module exists to get right: the worker thread and the
* calling thread both have a legitimate reason to touch the heap-allocated
* call state backing one call (the worker to run fn()/store its result;
* the caller to read that result) -- but at most ONE of them may free it,
* and it must be resolved without a lock the caller would have to block
* on (that would defeat the whole point of "abandon immediately"). See
* design.md's "Heap-Block Ownership Handoff" state diagram and
* research.md's "Decision: Interruptible RPC via detached worker thread
* with CAS-based ownership handoff" for the full rationale, including why
* pthread_cancel() was rejected (cancellation-safety of
* pthread_cond_timedwait()'s internal state is not established).
*
* PER-CONNECTION LOCK REGISTRY (task 1.5, Requirements 1.1, 1.5): an
* abandoned worker thread from task 1.4's mechanism above keeps running
* against whatever connection state `ctx` closes over. If the operator's
* very next command reuses that SAME connection, its own call would race
* the still-running abandoned worker for that connection's single-
* outstanding-reply state -- not a rare edge case, but the direct, likely
* consequence of Ctrl-C's own purpose (cancel, then immediately try
* something else). xcetest_run_interruptible() closes this by acquiring
* a per-connection lock -- looked up or lazily created in a small,
* process-lifetime registry keyed by whatever `xcetest_set_active_
* connection()` was last called with -- BEFORE spawning the worker, and
* holding it for the call's full lifetime, including through an
* abandonment (ownership of "release the lock" passes to the worker in
* that case, exactly mirroring the heap-block CAS handoff above). Lock
* acquisition is itself interruptible (the same break_out poll loop), so
* a second call waiting on a connection a prior abandoned call still
* holds can itself be Ctrl-C'd out of that wait, before ever acquiring
* the lock or spawning a worker of its own. See design.md's "Interruptible
* RPC Call and Ctrl-C Abandonment" sequence diagram and research.md's
* "Decision: Per-connection lock serializes overlapping calls on the
* Interruptible Call Adapter" for the full rationale.
*
* WHY THE LOCK IS A POSIX SEMAPHORE (sem_t), NOT A MUTEX: the handoff
* above is a genuine CROSS-THREAD release -- the REPL/caller thread
* acquires the lock before spawning the worker, but on the abandonment
* path it is the still-running WORKER thread that later releases it,
* once fn() finishes. A pthread_mutex_t (of ANY type, including the
* default PTHREAD_MUTEX_NORMAL) is the wrong primitive for this: POSIX
* defines pthread_mutex_t in terms of a single owning thread, and
* classifies pthread_mutex_unlock() from a non-owning thread as
* undefined behavior for every mutex type -- PTHREAD_MUTEX_ERRORCHECK/
* _RECURSIVE are simply *required* to detect and report that UB (hence
* the EPERM an earlier version of this code hit immediately), while
* NORMAL/DEFAULT are merely *permitted* not to detect it. POSIX does NOT
* separately declare cross-thread unlock well-defined for NORMAL; it
* just does not mandate an error return for it. A pthread_mutex_t-based
* fix that swaps ERRORCHECK for the default type therefore only trades a
* loud crash for a silent reliance on non-portable, POSIX-classified-UB
* behavior -- something that happens to hold on this project's actual
* target (glibc/NPTL's PTHREAD_MUTEX_NORMAL has had no ownership
* tracking for decades, and XCE's Nix-hermetic build is glibc/Linux-only)
* but is the wrong thing to build on when a fully-defined alternative
* exists at the same code complexity.
*
* sem_t has NO ownership concept at all: sem_wait()/sem_trywait()/
* sem_post() may be called from any thread regardless of which thread
* called which, with fully-defined POSIX semantics for exactly this
* cross-thread signal pattern. Each registry entry is therefore a binary
* semaphore (sem_init(&sem, 0, 1)) used as a mutual-exclusion lock:
* sem_trywait() takes the place of pthread_mutex_trylock() in the
* interruptible poll loop below (EAGAIN where trylock would have
* returned EBUSY), and sem_post() takes the place of
* pthread_mutex_unlock() at both release sites -- with no UB caveat on
* either.
*
* WHY THE REGISTRY IS A SINGLY-LINKED LIST, NOT A GROWABLE ARRAY: a
* sem_t* handed out by this registry may be cached inside a call block
* that a still-running abandoned worker thread holds onto indefinitely
* (up to ~10s, per the heap-block ownership handoff above). A growable
* array's realloc() can MOVE every previously-returned element the
* moment a later, unrelated connection is first seen -- silently
* invalidating that worker's pointer into stale/reused memory. A linked
* list's nodes are individually allocated and, per this registry's own
* "entries are never removed" contract, never freed or moved for the
* rest of the process's life, so a pointer into it stays valid forever
* once handed out.
*
* CONCURRENCY OF THE REGISTRY ITSELF: per design.md's Implementation
* Notes, "the registry's lookup/insert step needs no synchronization of
* its own -- a new registry entry is only ever created from the single
* REPL thread that issues commands" (detached worker threads never
* initiate a new top-level call; they only run the `fn` they were given
* and release the lock/free the block they were handed). The SEMAPHORE
* each entry owns, once handed out, is of course safe for concurrent
* wait/post from different threads -- that safety is sem_t's own job,
* not the registry's.
*/
#include <stdatomic.h>
/* A blocking call this module runs on a detached worker thread. `ctx` is
* whatever heap-owned state `fn` needs -- see xcetest_run_interruptible()'s
* preconditions below for the one hard rule about what `fn` may touch. */
typedef int (*xcetest_blocking_fn)(void* ctx);
/* The heap-allocated call block's lifecycle, resolved by exactly one
* atomic compare-and-swap regardless of which side "wins":
* RUNNING -- fn() has not yet returned, and no one has decided
* to abandon the call yet. Initial state.
* DONE_UNCLAIMED -- fn() returned and the worker's CAS from RUNNING to
* this state succeeded (i.e. the caller had not yet
* abandoned it). The CALLER is left owning the
* block: it reads rc and frees it.
* ABANDONED -- the caller's CAS from RUNNING to this state
* succeeded (i.e. it observed an interrupt before
* fn() finished). The WORKER is left owning the
* block: once fn() eventually returns, its own CAS
* attempt (RUNNING -> DONE_UNCLAIMED) fails against
* this state, and it frees the block itself instead.
* See design.md's "Heap-Block Ownership Handoff" stateDiagram-v2 -- this
* enum's three values are exactly its three states. */
typedef enum {
XCT_RUNNING,
XCT_DONE_UNCLAIMED,
XCT_ABANDONED
} xcetest_call_state_t;
/*
* Sets the connection-identity key that xcetest_run_interruptible() below
* reads INTERNALLY when a new call starts, to look up (or lazily create)
* that connection's entry in the per-connection lock registry (task 1.5,
* see the file header above). This is a simple settable-elsewhere,
* read-internally piece of process state -- deliberately mirroring how
* break_out/xcetest_interrupted() already works -- NOT a parameter added
* to xcetest_run_interruptible()'s own signature, which stays exactly
* `(fn, ctx)` per this task's design constraint.
*
* SEAM FOR A FUTURE CALLER (task 3.2): production code (test.c's cmd_*
* functions) does not exist inside this module and has no notion of
* "the currently active command_ctx_t's connection pointer" that this
* header can reach into directly -- test.c itself is never linked into
* this binary's unit tests (see this file's own header and
* xcetest_interrupt.h's). The intended real wiring is: before invoking
* xcetest_run_interruptible() for a blocking RPC, a cmd_* function (or a
* shared dispatch point above all of them) calls
* xcetest_set_active_connection(<that connection's own stable pointer,
* e.g. the command_ctx_t's connection field>). Any stable, comparable-by-
* identity pointer works as `conn_key` -- this module never dereferences
* it, only compares it by pointer equality against the registry's keys.
* Until task 3.2 wires that in, xce_test's real binary calls this
* exactly once, from wherever main()/the dispatcher sets up each
* command's active connection context.
*
* Concurrency: like break_out, callable only from the single REPL thread
* that issues top-level commands -- see the file header's "CONCURRENCY
* OF THE REGISTRY ITSELF" note. Passing the same key across unrelated
* connections would incorrectly serialize them against each other;
* passing a fresh/different key per connection is what keeps unrelated
* connections independent (Requirement 1.5, "leave ... its underlying
* connection intact").
*/
void xcetest_set_active_connection(void* conn_key);
/*
* Runs `fn(ctx)` on a detached worker thread while THIS (the calling)
* thread polls xcetest_interrupted() every ~50ms (design.md's Interruptible
* Call Adapter component), so a Ctrl-C observed during the wait abandons
* it immediately instead of blocking for fn()'s own, possibly much
* longer, duration.
*
* Before spawning that worker, this function first acquires -- also
* interruptibly, via the same ~50ms xcetest_interrupted() poll, but
* using sem_trywait() instead of a blocking sem_wait() -- the
* per-connection lock for whichever connection key was last passed to
* xcetest_set_active_connection() (task 1.5; see the file header's "PER-
* CONNECTION LOCK REGISTRY" section above). This serializes this call
* against any other in-flight or abandoned-but-still-running call on the
* same connection, so that connection's single-outstanding-reply state
* is never touched by two threads at once.
*
* Preconditions:
* - `fn` is non-NULL.
* - `fn` touches only what `ctx` itself heap-owns -- NEVER memory on
* this function's caller's own stack frame. This is the precondition
* that makes the -EINTR postcondition below safe: once ownership of
* `ctx` has passed to the still-running worker, the caller's own
* stack frame may have long since unwound.
*
* Postconditions:
* - Returns fn()'s own return value when fn() finishes before this
* thread's abandon-CAS wins the race (whether or not
* xcetest_interrupted() ever became true during the wait) -- see the
* header comment's "near-simultaneous" case below. On this path, the
* per-connection lock is released by this (the caller's) thread,
* after reading rc and freeing the block.
* - Returns -EINTR when this thread observes an interrupt WHILE
* WAITING FOR THE PER-CONNECTION LOCK -- i.e. before ever acquiring
* it. No worker is spawned for this call at all in this case, and
* nothing needs releasing, since this call never held the lock to
* begin with.
* - Returns -EINTR when this thread observes an interrupt AFTER
* acquiring the lock and spawning the worker, and its own CAS
* (RUNNING -> ABANDONED) wins the race. On this return, the caller
* MUST NOT touch `ctx` again -- ownership of both the call block AND
* the per-connection lock has passed to the still-running worker
* thread, which will finish fn(), free the call block, and release
* the lock on its own, asynchronously, with no further signal back
* to this caller.
* - Returns -EINVAL if `fn` is NULL (fn is never invoked in this case;
* no lock is acquired).
* - Returns a negative errno from the per-connection lock registry
* lookup (e.g. -ENOMEM), from sem_trywait() itself for an
* unexpected error other than EAGAIN, or from pthread_attr_init()/
* pthread_create() if the worker thread could not be set up or
* spawned at all; in every one of these cases fn() is never invoked,
* any call block already allocated is freed before returning, and
* the per-connection lock -- if this call had already acquired it --
* is released before returning.
*
* The near-simultaneous race (this thread notices the interrupt in the
* same window the worker finishes fn()) is resolved by the SAME single
* CAS deciding both sides at once: whichever of {this thread's
* RUNNING->ABANDONED attempt, the worker's RUNNING->DONE_UNCLAIMED
* attempt} executes first wins, and the loser observes that the state is
* no longer RUNNING and falls back to the OTHER side's outcome -- a
* caller who loses the abandon-CAS falls through to the normal
* completion path (reads rc, frees the block, releases the lock, returns
* rc) instead of returning -EINTR; a worker who loses the done-CAS frees
* the block and releases the lock itself instead of leaving them for the
* caller. Exactly one free() of the block, and exactly one release of
* the lock, are therefore reachable on every possible interleaving --
* never zero, never two.
*/
int xcetest_run_interruptible(xcetest_blocking_fn fn, void* ctx);
/*
* xcetest_wait_connection_idle (xce-test-cli-ux task 3.2 remediation round
* 1, bug-1795) -- blocks the calling thread until no worker thread
* (running or abandoned-but-still-running) is active on `conn_key`, per
* this module's own "at most one worker thread per connection" invariant,
* then returns immediately.
*
* WHY THIS EXISTS: xcetest_run_interruptible()'s per-connection lock
* registry above enforces that invariant BETWEEN calls on the same
* connection -- a second call on a connection an abandoned worker still
* holds waits (interruptibly) for that worker to finish. It does NOT by
* itself protect connection TEARDOWN: test.c's leave_ctx() (called by
* `up`/`quit`) tears down the connection object out from under an
* abandoned-but-still-running worker with no wait/join of its own,
* because leave_ctx() never calls xcetest_run_interruptible() -- it is
* connection teardown, not a new RPC. xcecli_wakeup() (called by
* leave_ctx() just before this function, best-effort) nudges that worker
* to unwind faster than its own ~10s deadline, but nothing forced
* leave_ctx() to actually wait for it -- a genuine, exploitable
* use-after-free window (any abandoned worker still reading/writing the
* connection object while leave_ctx()'s xcecli_stop()/xcecli_ctx_fini()
* frees it). This function closes that gap by reusing the SAME
* per-connection semaphore registry xcetest_run_interruptible() already
* builds and maintains, rather than inventing a second mechanism.
*
* CONTRACT: looks up (or lazily creates, via the exact same internal
* lookup xcetest_run_interruptible() itself uses) the sem_t for
* `conn_key`, does a PLAIN BLOCKING sem_wait() on it (deliberately NOT
* the interruptible sem_trywait()-poll variant used inside
* xcetest_run_interruptible() -- this is a narrow, safety-critical wait
* where correctness matters more than Ctrl-C responsiveness; it should
* resolve quickly in the ordinary case since the caller's own
* xcecli_wakeup() call already nudges the worker to finish sooner), then
* immediately sem_post()s it back before returning. It does NOT hold the
* semaphore locked on return -- the semantic this function provides is
* "prove that, at this instant, no worker is active on this connection,"
* not "hold this connection locked for some ongoing operation of the
* caller's own." A caller that needs the latter must acquire and hold the
* lock itself (which only xcetest_run_interruptible() currently does).
*
* A transient EINTR from sem_wait() itself (a signal delivered while
* blocked, distinct from `errno`-based break_out signal state) is
* retried internally rather than surfaced -- this function's whole
* purpose is to make the wait unconditional, not skippable.
*
* Safe to call with any `conn_key` previously passed to
* xcetest_set_active_connection() (or, for that matter, any key never
* seen before -- the lazy-create path simply creates a fresh, immediately
* uncontended semaphore for it and returns right away). Must be called
* from the same single REPL thread this module's other entry points are
* documented to run on (see this header's "CONCURRENCY OF THE REGISTRY
* ITSELF" note) -- it is not itself safe to call concurrently from
* multiple threads, though the semaphore it waits on is of course safe to
* be concurrently sem_post()'d by a worker thread while this function
* waits on it.
*/
void xcetest_wait_connection_idle(void* conn_key);
#ifdef XCE_TESTING
/*
* Test-only instrumentation, NOT part of the production API and never
* even declared unless XCE_TESTING is defined (i.e. never visible to the
* real xce_test binary): incremented (via atomic_fetch_add) by the worker
* thread immediately after it frees an ABANDONED call block.
*
* Exists solely so test_interruptcall.c's forced-abandonment test can
* confirm the abandoned worker's asynchronous cleanup genuinely completed
* before the test process exits, rather than merely hoping a fixed sleep
* was long enough -- a still-running detached worker racing a leak
* detector's exit-time check could otherwise either mask a real bug or
* (more likely, if the implementation above is correct) flag a false
* leak on an allocation that was always going to be freed, just not yet.
*/
extern _Atomic int xcetest_test_abandoned_free_count;
#endif

View file

@ -0,0 +1,81 @@
#ifndef JOBLIST_DISPLAY_H
#define JOBLIST_DISPLAY_H
/* joblist_display.h -- xce-test-cli-ux task 2.5 remediation (bug-1711 fixes)
*
* Extracted from test.c's worker_status_str()/job_state_str()/
* format_job_time(): pure, dependency-light display-formatting helpers, with
* NO networking, REPL, or global-session-state dependency (job_state_str's
* only state is a function-local `static` scratch buffer, used solely to
* render its own return value -- no cross-call or cross-session state).
* This isolation is what makes them unit-testable outside of test.c, which
* cannot be white-box #include'd into a Unity test (thousands of lines, its
* own main(), heavy libedit/network dependencies) -- same rationale as
* resval_typeresolve.c's extraction from set_resval() (resource-value-
* verification task 1.1) and misc.c's update_host_runusage()/
* aggregate_job_runusage() (bug-1735).
*
* test.c calls these directly at their existing call sites (cmd_workerlist,
* joblist_query, print_jobinfo) -- no behavior change, extraction only.
*/
#include <stddef.h> /* size_t */
#include <stdint.h> /* uint32_t */
#include <time.h> /* time_t */
/* Render a worker's status bitmask (xce_worker_t.status, XCE_WORKER_* bits
* in xce/types.h) as a short display string for cmd_workerlist's Status
* column.
*
* Priority order (first match wins, unchanged from the pre-extraction
* logic): CLOSED > UNAVAIL > LOCKEDU > LOCKEDW > BUSY > (status == 0) OK.
* Admin-set CLOSED intentionally takes priority over the dynamic UNAVAIL
* bit: a host closed via hostctrl that later disconnects would otherwise
* print only "UNAVAIL", masking the admin bit this check exists to verify
* (config-upgrade-snapshot soak, task 8.6).
*
* Any other nonzero bit combination not covered above returns "UNKNOWN"
* (Requirement 11.1 -- bug-1711's fix; previously misspelled "UNKOWN").
*
* Returns a string literal -- never a caller-owned buffer, safe to use
* across calls with no ownership/lifetime concerns. */
char* worker_status_str(uint32_t status);
/* Render a job's state bitmask (xce_job_t.state, JOB_STAT_* bits in
* xce/types.h) as a short display string for cmd_joblist's State column
* (joblist_query) and print_jobinfo's "state:" line.
*
* Priority order (first match wins, unchanged from the pre-extraction
* logic): PEND (+PSUSP) > RUN (+USUSP/+SSUSP/+EXIT) > EXIT > DONE > ERROR >
* UNKNOWN.
*
* Requirement 11.7 (bug-1711): a job that is simultaneously RUN and EXIT
* (e.g. a worker-reported FAIL received without a prior FINISH;
* apply_job_fail_transition ORs in EXIT|PERR without clearing RUN) is
* reported via a raw-state fallback "RUN(0x%x)", consistent with the
* fallback print_jobinfo already applies for that same ambiguity via its
* separate state_raw line -- checked AFTER USUSP/SSUSP in the same
* else-if chain, so it never masks either of those.
*
* Returns a string literal for every branch except the RUN+EXIT fallback,
* which is rendered into a function-local `static` scratch buffer valid
* until the next call (the same lifetime contract test.c relied on before
* extraction). */
char* job_state_str(int state);
/* Format a job time_t field (epoch seconds) using ctrlstatus_fmt.h's
* ctrlstatus_format_time_ms human-readable formatter (which takes epoch
* milliseconds) into `buf` (at least 24 bytes -- see
* ctrlstatus_format_time_ms's own size contract).
*
* A job time of 0 means the corresponding event has not happened yet (e.g.
* an unstarted job's start_time/finish_time) -- returns "-" rather than a
* misleading 1970-01-01 epoch date (Requirement 11.5). Also returns "-" if
* the underlying formatter itself fails (e.g. `buf`/`bufsize` too small).
*
* Returns `buf` on success (nonzero `t`), or a string literal ("-") for the
* zero/failure cases -- callers must not assume the return value is always
* `buf`. */
const char* format_job_time(time_t t, char* buf, size_t bufsize);
#endif /* JOBLIST_DISPLAY_H */

View file

@ -0,0 +1,68 @@
#ifndef JOIN_WORKER_ARGS_H
#define JOIN_WORKER_ARGS_H
/* join_worker_args.h -- xce-test-cli-ux task 2.6 remediation (bug-1557 fix)
*
* Extracted from test.c's cmd_join_worker(): the pure argument-parsing/
* defaulting block that populates a hostattrs_t (xce/types.h) from argc/argv,
* including the bug-1557 maxmem default fix. This block runs entirely before
* cmd_join_worker's only network call (xcecli_join()) and touches no
* active_ctx, REPL state, or any global beyond the standard `optind` reset
* every other cmd_ handler in this file already performs -- the same
* isolation rationale as joblist_display.c's extraction (task 2.5) and
* resval_typeresolve.c's (resource-value-verification task 1.1).
*
* test.c's cmd_join_worker() calls this directly in place of its old inline
* parsing block, then proceeds to xcecli_join() exactly as before -- no
* behavior change to the network-call phase, extraction only.
*/
#include <stddef.h> /* size_t */
#include "flagspec.h" /* flagspec_t */
#include "xce/types.h" /* hostattrs_t */
/* join's flagspec_t table, exposed so test.c's `join` command_t entry
* (worker context, cmd_join_worker) can wire it into the Help Renderer's
* generated flag block (xce-test-cli-ux task 3.5, Requirement 8.2, 12.2) --
* the same table join_worker_parse_attrs() itself parses against, so the
* rendered help and the actual parsing/defaulting behavior can never drift
* apart.
*
* JOIN_FLAGS_COUNT is a compile-time literal (not a `const size_t`
* variable) specifically so test.c's file-scope command_t table
* initializers -- which the C standard requires to be constant
* expressions -- can use it directly for a `.flagspec_count` field; an
* extern variable's value is only known at link time and is not a
* constant expression. Declaring `join_flags[]` here with this explicit
* bound, then defining it in join_worker_args.c with a plain `[]` and a
* 6-entry initializer, gets a free safety net: the two declarations of
* the same array must agree on size, so a future edit that adds/removes a
* flag without updating JOIN_FLAGS_COUNT fails the build immediately
* ("conflicting types") instead of silently reporting a wrong count.
* `JOIN_NFLAGS` (join_worker_args.c) stays an internal alias of this same
* constant, used to size join_worker_parse_attrs()'s own getopt_long()
* option table -- unchanged behavior, just no longer computed via
* sizeof(). */
#define JOIN_FLAGS_COUNT 6
extern const flagspec_t join_flags[JOIN_FLAGS_COUNT];
/* Parse join's CLI flags ("-f cpufactor -c maxcpus -m maxmem -s maxswap
* -t maxtmp -d ndisks") into `*attrs`, first defaulting every field
* (including the bug-1557-corrected maxmem default of 64 * 1024 * 1024
* bytes -- see the in-body comment in join_worker_args.c for the full
* bytes-vs-MB history) and then overriding any field whose flag was given.
*
* Resets `optind` to 1 before parsing, matching this file's own convention
* for repeated in-process command invocations (see cmd_jobfinish's comment
* on the same convention).
*
* Returns 0 on success. Returns -EINVAL on an unrecognized flag or a NULL
* `attrs`, leaving `*attrs` fully defaulted (and possibly partially
* overridden by whichever flags were already parsed before the
* unrecognized one) -- callers must not use `*attrs` after a nonzero
* return. On a parse error, this function itself prints the same usage
* string cmd_join_worker printed before extraction. */
int join_worker_parse_attrs(int argc, char* argv[], hostattrs_t* attrs);
#endif /* JOIN_WORKER_ARGS_H */

View file

@ -0,0 +1,54 @@
#pragma once
/*
* listfmt_json.h -- shared cJSON array/object plumbing for xce_test's
* listing commands' JSON output mode (xce-test-cli-ux spec, task 2.2;
* Requirements 5.1, 5.3).
*
* This module owns ONLY the JSON array/object lifecycle: start an array,
* append a new row object to it, add string/number fields to a row, then
* print + free the whole array. It does NOT know about workerlist's/
* joblist's/reslist's own field sets -- each command's existing row
* struct already exists and is stable (design.md's Simplification lens),
* so this library introduces no generic "record"/"row" abstraction beyond
* the bare cJSON object handle itself. Wiring this into cmd_workerlist/
* cmd_joblist/cmd_reslist (their field mapping, their --json flag) is
* task 3.4, not this one.
*
* cJSON is already linked into xce_test (struct2json_dep, per
* research.md) and already used for job exdata blobs (test.c's
* build_job_envs()) -- this module reuses that exact same cJSON API
* surface, not a different one.
*/
#include <cjson/cJSON.h>
/* Creates a new, empty JSON array meant to hold one object per output
* row. The caller passes the returned array to listfmt_json_new_row()
* once per row, then to listfmt_json_finish() exactly once when done.
* Returns NULL on allocation failure. */
cJSON* listfmt_json_begin(void);
/* Adds a string field `key`: `val` to `row`. `val` is copied in full,
* regardless of its length -- no field is ever truncated or dropped for
* being "too long" (Requirement 5.3). A NULL `val` is stored as an
* explicit JSON null rather than silently omitted or crashing. No-op if
* `row` or `key` is NULL. */
void listfmt_json_add_string(cJSON* row, const char* key, const char* val);
/* Adds a numeric field `key`: `val` to `row`. No-op if `row` or `key` is
* NULL. */
void listfmt_json_add_number(cJSON* row, const char* key, double val);
/* Creates a new, empty JSON object, appends it to `array`, and returns
* that object so the caller can then call listfmt_json_add_string()/
* listfmt_json_add_number() on it for each of this row's fields. Returns
* NULL (leaving `array` unmodified) if `array` is NULL or the new object
* could not be allocated. */
cJSON* listfmt_json_new_row(cJSON* array);
/* Prints `array` to stdout as JSON -- command-RESULT data, per
* Requirement 3's stdout/stderr discipline, so it is written to stdout
* only, never stderr -- and then frees the entire cJSON structure
* (`array` and every row/field it holds). `array` must not be
* referenced again after this call. No-op if `array` is NULL. */
void listfmt_json_finish(cJSON* array);

View file

@ -0,0 +1,113 @@
#pragma once
/*
* listrow_json.h -- pure per-command JSON-row field-mapping helpers for
* cmd_reslist/cmd_workerlist/joblist_query's new --json output mode
* (xce-test-cli-ux task 3.4, Requirements 5.1, 5.2, 5.3).
*
* Each function here takes one already-fetched record's fields --
* already extracted as plain scalars by the caller, no xce/types.h struct
* dependency, no RPC, no networking, no global session state -- and
* populates one JSON row object (from listfmt_json_new_row(), task 2.2)
* via listfmt_json.h's add_string()/add_number() primitives, mapping
* EVERY column the corresponding command's human-readable table prints
* (Requirement 5.1) with no truncation regardless of value length
* (Requirement 5.3).
*
* Extracted out of test.c (rather than inlined in cmd_reslist/
* cmd_workerlist/joblist_query alongside their RPC calls) specifically so
* this field-mapping logic is unit-testable in isolation with plain
* literal inputs -- test.c itself cannot be white-box #include'd into a
* Unity test (thousands of lines, its own main(), heavy libedit/network
* dependencies), and cmd_reslist/joblist_query's fetch+print bodies were
* already found "genuinely untestable without a live cluster" for the
* SAME reason during task 2.5's review. This mirrors joblist_display.c's
* extraction precedent (task 2.5) for worker_status_str()/job_state_str()/
* format_job_time() -- pure, dependency-light display formatting pulled
* out specifically to get real automated coverage.
*
* The caller (test.c) remains responsible for: calling xcecli_reslist()/
* xcecli_workerlist()/xcecli_joblist(), iterating the result, calling
* listfmt_json_begin()/listfmt_json_new_row() around each call here, and
* listfmt_json_finish() exactly once when done -- this module never
* touches the array lifecycle itself (listfmt_json.h already owns that,
* task 2.2, and is not modified by this task).
*/
#include <stdint.h>
#include <time.h>
#include <cjson/cJSON.h>
/* Maps cmd_reslist's table columns -- Name, Flags, Interval, Description,
* see test.c's printf("%-10s%10s%10d %-64s\n", ...) row -- onto `row`.
*
* `flags` is emitted as a JSON NUMBER under the key "flags" (the raw
* xce_resource_t.flags value), not the table's "0x%x"-formatted hex
* string (Requirement 11.4's hex prefix is a display-only convention for
* the human-readable table; the underlying integer value is identical
* either way) -- a number already carries the same information the
* table's Flags column shows, so nothing is lost relative to the table
* (Requirement 5.1), and a script consuming the JSON output gets a
* directly-usable integer instead of having to re-parse a "0x..." string.
*
* No-op if `row` is NULL (mirrors listfmt_json_add_string/number's own
* NULL-row no-op contract). */
void reslist_json_row(cJSON* row, const char* name, int flags, int interval,
const char* desc);
/* Maps cmd_workerlist's table columns -- HostName, Status, HostType,
* HostModel, CPUfactor, MaxCPU, MaxMem, MaxSwap, MaxTmp, NDisks -- onto
* `row`.
*
* Unlike the table (which humanizes MaxMem/MaxSwap/MaxTmp via
* format_bytes_human() for display, e.g. "64.0 MB"), this emits their RAW
* numeric byte values (`maxmem`/`maxswap`/`maxtmp`, matching hostattrs_t's
* own `double` field types in xce/types.h) as JSON numbers under keys
* "maxmem"/"maxswap"/"maxtmp" -- the same underlying value the table's
* humanized string represents, but directly script-consumable without
* re-parsing a unit suffix.
*
* `status_str` is the caller's worker_status_str(workers[i]->status)
* result (joblist_display.h) -- passed in rather than recomputed here, so
* this module has no dependency on xce/types.h's XCE_WORKER_* bit
* definitions.
*
* `cpufactor` is `double` (not truncated to float/int) so a genuine
* fractional value -- not just a power-of-two -- round-trips exactly
* through the JSON text (see test_listrow_json.c's fractional round-trip
* test, closing the gap task 2.2's Implementation Notes flagged: its own
* tests only exercised power-of-two doubles).
*
* No-op if `row` is NULL. */
void workerlist_json_row(cJSON* row, const char* name, const char* status_str,
const char* hosttype, const char* hostmodel,
double cpufactor, int maxcpus, double maxmem,
double maxswap, double maxtmp, int ndisks);
/* Maps joblist_query's table columns -- JobId, User, Queue, State,
* SubmitTime, StartTime, FinishTime, MinTasks, MaxTasks, Command -- onto
* `row`.
*
* `jobid_str` is the caller's fmt_jobid(...) result and `state_str` the
* caller's job_state_str(jobs[i]->state) result (joblist_display.h) --
* both passed in rather than recomputed here, keeping this module free of
* test.c's own fmt_jobid()/RPC dependencies. `state_str` is stored
* verbatim (including job_state_str()'s Requirement-11.7 raw-state
* fallback text for an ambiguous RUN+EXIT state, e.g. "RUN(0x38)"), so no
* information the table's State column carries is lost.
*
* Unlike the table (which formats SubmitTime/StartTime/FinishTime as
* human-readable text via format_job_time(), Requirement 11.5), this
* emits their RAW epoch-second values as JSON numbers under keys
* "submit_time"/"start_time"/"finish_time" -- 0 meaning "has not happened
* yet", the same convention format_job_time() itself uses for its "-"
* display -- the same underlying value the table's formatted string
* represents, directly script-consumable without re-parsing a date
* string.
*
* No-op if `row` is NULL. */
void joblist_json_row(cJSON* row, const char* jobid_str, const char* user,
const char* queue, const char* state_str,
time_t submit_time, time_t start_time,
time_t finish_time, uint32_t mintasks,
uint32_t maxtasks, const char* cmdline);

View file

@ -0,0 +1,39 @@
#pragma once
/*
* rcfile.h -- xce_test startup rc-file path resolution and line-loading
* (xce-test-rc-file spec, tasks 1.1/1.2).
*
* A standalone, dependency-light module following the same pattern as
* configcompare.c/h -- no dependency on test.c internals, linked into both
* the xce_test executable and controller_unit_test.
*
* Task 1.1 implemented path resolution (rcfile_resolve_path()). Task 1.2
* adds line-loading (rcfile_load()) with a pluggable runner callback.
* Wiring rcfile_load() into xce_test's own startup/main() is a later task
* in the same spec (2.2) and is out of scope here -- this module has no
* knowledge of xce_test commands, contexts, or client state.
*/
/* Resolves the rc-file path per the XDG Base Directory Specification:
* "$XDG_CONFIG_HOME/xce_test/rc" when XDG_CONFIG_HOME is set and non-empty,
* else "$HOME/.config/xce_test/rc" when HOME is set and non-empty. Returns
* a heap string the caller must free(), or NULL if neither variable yields
* a usable base directory. */
char* rcfile_resolve_path(void);
/* Opens `path` and calls `runner(line)` once per line (trailing newline
* stripped), stopping early if `runner` returns nonzero. Lines are read via
* getline() with no length cap -- a line is never truncated or split
* regardless of length.
*
* Returns:
* 0 - path is NULL, the file does not exist (ENOENT),
* the file could not be opened for another reason
* (a warning is printed to stderr; non-fatal), or
* every line ran and every runner call returned 0.
* nonzero (== runner's) - the runner requested an abort; no further lines
* are read.
*
* Preconditions: `runner` is non-NULL when `path` is non-NULL.
* Postconditions: the file handle is always closed before returning. */
int rcfile_load(const char* path, int (*runner)(char* line));

View file

@ -0,0 +1,53 @@
#ifndef RESVAL_TYPERESOLVE_H
#define RESVAL_TYPERESOLVE_H
/* resval_typeresolve.h -- resource-value-verification task 1.1
*
* Extracted from test.c's set_resval(): pure resource-name -> wire-type
* resolution logic, with NO networking, REPL, or global-state dependency
* (unlike set_resval() itself, which also allocates xce_val_t and tags
* .host via get_setting("myname") -- that stays in test.c since it needs
* REPL state). This isolation is what makes the logic unit-testable outside
* of test.c, which cannot be white-box #include'd into a Unity test (4298
* lines, its own main(), heavy libedit/network dependencies).
*
* The resolved type is a Value_u_which tag (controller.capnp.h) -- e.g.
* Value_u_fvalue for a numeric resource, Value_u_svalue for a string one.
*/
#include "controller.capnp.h" /* enum Value_u_which */
/* Sentinel: not a valid Value_u_which value. Used both as:
* - the RETURN value when `name` is not present in the known-resource
* table and no override applies;
* - the `type_override` INPUT value meaning "no override requested".
*/
#define RESVAL_TYPE_UNKNOWN (-1)
/* Resolve `name` to its Value_u_which wire type.
*
* `type_override`: pass a valid Value_u_which value (Value_u_fvalue,
* Value_u_svalue, ...) to request that type for a name not in the known-
* resource table, or RESVAL_TYPE_UNKNOWN for no override.
*
* Returns:
* - the table's type, for any of the 14 currently-known resource names
* (r15s, r1m, slots, ut, mem, maxcpus, ndisks, cpuf, maxswap, maxtmp,
* type, model, verilog, dyna) -- `type_override` is ignored for a known
* name, exactly as set_resval() behaves today.
* - `type_override`, for any name NOT in the table, when `type_override`
* is not RESVAL_TYPE_UNKNOWN (Requirement 1.1's `-t` escape hatch).
* - Value_u_fvalue (numeric), for any name NOT in the table when no
* override was requested (Requirement 1.1's default fallback).
* - RESVAL_TYPE_UNKNOWN, only for `name` == NULL.
*
* *** TASK 2.1 STATUS (fallback implemented) ***
* Task 1.1 was a byte-for-byte extraction of set_resval()'s existing
* resinfo[] table lookup, with the unrecognized-name fallback left
* unimplemented (RED tests in test_resval_typeresolve.c). Task 2.1 adds the
* numeric-by-default fallback and `type_override` handling described above
* -- behavior for all 14 known names remains byte-for-byte unchanged.
*/
int resval_resolve_type(const char* name, int type_override);
#endif /* RESVAL_TYPERESOLVE_H */

View file

@ -0,0 +1,43 @@
#pragma once
/*
* snapdiff.h — byte-level comparator for two checkpoint files
* (bug-1389 triage: OPCode_snapshotTransfer chunk-content corruption).
*
* Compares a locally-generated source checkpoint (snap-*.capnp) against a
* received one (snap-*.bin, ha_handle_snapshot_chunk) byte-for-byte, and
* reports the first divergent offset plus whether it lands exactly on an
* OPCode_snapshotTransfer chunk boundary (SNAPSHOT_CHUNK_SIZE, config_repl.c)
* — the decisive signal separating "chunk transport corrupted this specific
* chunk" from "the two files were never going to match" (different content
* entirely, unrelated to chunking).
*
* Strictly read-only: opens both files O_RDONLY, mutates nothing.
*/
#include <stdio.h>
#include <sys/types.h>
/* Mirrors config_repl.c's SNAPSHOT_CHUNK_SIZE (the OPCode_snapshotTransfer
* wire chunk size) — kept in sync manually; both are compile-time constants
* for the same wire protocol, not expected to change independently. */
#define SNAPDIFF_CHUNK_SIZE 65536
typedef struct {
off_t size1;
off_t size2;
int size_mismatch; /* 1 if size1 != size2 (diff_offset unset in that case) */
int identical; /* 1 if same size and byte-for-byte identical */
off_t diff_offset; /* -1 if identical or size_mismatch; else first
* differing byte offset */
int at_chunk_boundary; /* 1 if diff_offset is the first byte of a
* SNAPDIFF_CHUNK_SIZE-aligned chunk, i.e.
* diff_offset % SNAPDIFF_CHUNK_SIZE == 0 */
off_t chunk_index; /* diff_offset / SNAPDIFF_CHUNK_SIZE, valid only
* when diff_offset >= 0 */
} snapdiff_report_t;
/* Compare `path1` (source of truth) against `path2` (candidate, e.g. a
* received copy). Prints a human-readable report to `out` (may be NULL) and
* fills `*rep` (may be NULL). Returns 0 if identical, 1 if they differ (see
* `*rep` for detail), or a negative errno on open/fstat failure. */
int snapdiff_paths(const char* path1, const char* path2, FILE* out,
snapdiff_report_t* rep);

View file

@ -0,0 +1,78 @@
#pragma once
/*
* snapverify.h — read-only checkpoint-file structural inspector/validator
* (bug-1389 triage: OPCode_snapshotTransfer chunk-content corruption;
* extended by config-upgrade-snapshot task 8.4, Requirement 4.2, for the
* SNAPSHOT_FORMAT_VERSION=2 host + config-bytes sections and to decode --
* not merely report the presence of -- the header's cfgref tag/hash).
*
* Parses the on-disk checkpoint layout persist.c's load_snapshot() /
* load_snapshot_entities() use (design.md "Checkpoint File Layout") WITHOUT
* restoring any job/queue/host into a live cluster_t — a pure structural
* walk, so it needs no jobmgr/qmgr/workermgr context and is safe to run
* standalone against a snap-*.capnp (locally generated) or snap-*.bin
* (received via ha_handle_snapshot_chunk, ha.c) file on disk. This tool
* validates STRUCTURE only, never content: job/queue/host frame payloads
* and the config-bytes section are walked past via their length prefix,
* never capnp-decoded or interpreted.
*
* Strictly read-only: opens O_RDONLY, mutates nothing.
*/
#include <stdint.h>
#include <stdio.h>
#include <sys/types.h>
typedef struct {
uint64_t checkpoint_lsn;
int has_cfgref; /* config-ref tail present in the header
* (header[16] != 0) -- semantic presence,
* distinct from cfgref_tag/cfgref_sha below
* being decoded */
uint64_t cfgref_tag; /* header cfgref tag, decoded whenever the
* header extends to SNAP_HEADER_MAX bytes,
* REGARDLESS of has_cfgref -- meaningful
* only when has_cfgref is also 1 */
uint8_t cfgref_sha[32]; /* header cfgref sha256, same decode-
* unconditionally-but-gate-meaning-on-
* has_cfgref discipline as cfgref_tag */
int has_entity_section; /* fmt + job_count present past the header;
* 0 = legacy position-only checkpoint */
uint32_t format_version;
uint32_t job_count; /* as declared in the file (0 if legacy) */
uint32_t jobs_ok; /* job frames structurally walked before any failure */
uint32_t queue_count; /* as declared in the file (0 if legacy, or if the
* job-frame walk failed before reaching it) */
uint32_t queues_ok; /* queue frames structurally walked before any failure */
uint32_t host_count; /* as declared in the file (0 if legacy, or if an
* earlier section's walk failed before reaching
* it) */
uint32_t hosts_ok; /* host frames structurally walked before any failure */
int has_config_bytes_section; /* 1 iff the trailing config-bytes
* section's 4-byte length field
* itself was present and readable
* (i.e. the file extends past the
* host section at all) -- TRUE even
* when that length decodes to 0, and
* FALSE (not a failure) when the
* file simply ends right after the
* host section */
uint32_t config_bytes_len; /* decoded length of the config-bytes
* section; valid only when
* has_config_bytes_section is 1 */
off_t file_size;
off_t consumed; /* bytes structurally accounted for */
int ok; /* 1 = well-formed, consumed == file_size */
off_t fail_offset; /* -1 if ok; else the byte offset of the field that
* failed the structural check */
char fail_reason[160];
} snapverify_report_t;
/* Parse+validate `path` (a snap-*.capnp or snap-*.bin checkpoint file).
* Prints a human-readable report to `out` (may be NULL to suppress) and
* fills `*rep` (may be NULL). Returns 0 if the file is structurally
* well-formed to EOF, or the same rc class persist_load_snapshot() would
* return for the equivalent defect (-EINVAL bad magic / unrecognized
* format_version, -EIO short read or a length field that doesn't fit the
* remaining bytes, -errno on open/fstat failure) — so a "persist_load_snapshot
* failed rc=..." log line correlates directly against this tool's verdict. */
int snapverify_path(const char* path, FILE* out, snapverify_report_t* rep);

View file

@ -0,0 +1,105 @@
#pragma once
/*
* walcompare.h — cross-member WAL committed-prefix divergence detector
* (ha-term-safety R6.1/R6.2/R6.3).
*
* Reuses the canonical waldump.c WAL decoder (via waldump_path_visit) to index
* each member's log by `(lsn -> term, pcrc)`, then asserts that every member's
* COMMITTED prefix is identical by `(term, pcrc)` — not merely by LSN range.
*
* The decisive signal for bug-851: two members can report the same committed
* LSN over a contiguous, CRC-intact WAL yet hold DIFFERENT entries at the same
* LSN (same lsn, different term and/or payload). Comparing LSN ranges alone
* masks that; comparing `(term, pcrc)` per LSN surfaces it (R6.2).
*
* Strictly read-only — opens member WAL dirs O_RDONLY, mutates nothing. Safe to
* run against a live cluster's data dirs.
*/
#include <stdint.h>
#include <stdio.h>
/* A single member's WAL source and its committed boundary. */
typedef struct {
const char* path; /* WAL dir / data dir / seg-*.wal (waldump rules) */
uint64_t committed_lsn; /* highest committed LSN for this member; 0 =
* "derive" (use the shared boundary, see below) */
} walcompare_member_t;
/* Identity of the first divergent entry, filled on a WALCMP_DIVERGENT result. */
typedef struct {
uint64_t lsn; /* the diverging LSN within the committed prefix */
int n_members; /* members compared at this LSN */
uint64_t member_term[16]; /* per-member matched term at lsn */
uint32_t member_pcrc[16]; /* per-member payload CRC at lsn */
int member_present[16]; /* 1 if the member holds this LSN, else 0 */
int committed; /* 1 = divergence within the committed prefix
* (lsn <= committed boundary, a safety
* violation); 0 = uncommitted tail divergence */
} walcompare_divergence_t;
/* Result codes. */
enum {
WALCMP_OK = 0, /* all members' committed prefixes are identical */
WALCMP_DIVERGENT = 1, /* a committed-prefix divergence was found */
/* negative values are -errno (a load/usage error: missing WAL, bad args) */
};
/* Compare `n` members' committed WAL prefixes by `(lsn -> term, pcrc)`.
*
* Committed boundary: each member with `committed_lsn == 0` adopts the SHARED
* boundary = the minimum recovered tip (highest CRC-intact LSN) across all
* members, i.e. the largest prefix every member can be expected to hold. A
* member with an explicit `committed_lsn` uses that. An LSN in `[1..boundary]`
* is compared only against the members whose own boundary covers it; a member
* that is legitimately AHEAD (longer log) or behind the shared boundary is not
* flagged for the uncommitted tail it doesn't share (R6.1 is the committed
* prefix, not the full log).
*
* Per-member checkpoint floor (raft-ha-core-safety task 5.10a / ha-entity-
* snapshots task 7.1): a member that caught up via a full-checkpoint snapshot
* legitimately holds no raw WAL entries at or below its own installed
* checkpoint LSN — that range was restored via entity content, not WAL
* replay. Each member's floor is read independently from its own `snap/`
* sibling directory (0 if none installed, the unchanged pre-existing
* behavior); an LSN below a member's own floor is excluded from that
* member's comparison the same way an LSN above its committed_lsn already
* is — never flagged MISSING/divergent. This is a PER-MEMBER floor, never a
* shared/uniform value (unlike committed_lsn, which the caller may force to
* one shared value across all members).
*
* Legacy-zero term (R1.8) is mapped to WAL_LEGACY_TERM for matching, so two
* pre-feature entries are not spuriously divergent.
*
* On the FIRST divergent LSN, fills `*div` (when non-NULL) and prints a report
* to `out`, then returns WALCMP_DIVERGENT. Returns WALCMP_OK on identical
* committed prefixes, or a negative errno on a load/usage error. `out` may be
* NULL to suppress the report.
*/
int walcompare(const walcompare_member_t* members, int n, FILE* out,
walcompare_divergence_t* div);
/* Same comparison as walcompare(), but ALSO emits a machine-checkable,
* per-member report over the full log: one
*
* WALCMP member=<i> lsn=<L> term=<T> pcrc=<8hex> status=<present|MISSING|beyond> committed=<0|1>
*
* line per (member, LSN) for every LSN in [1..max-recovered-tip], where
* `committed=1` iff the LSN is within the committed boundary. Each divergence
* additionally emits a
*
* WALCMP_DIVERGENCE lsn=<L> class=<committed|uncommitted> compared=<c> present=<p>
*
* classification line. A `committed` divergence is fatal (verdict
* WALCMP_DIVERGENT, `div->committed = 1`); an `uncommitted`-tail divergence
* (above the committed boundary) is reported but NON-fatal — the verdict stays
* WALCMP_OK when the committed prefix is identical. Field order is fixed and
* the tokens are stable so the harness can grep them. `out` must be non-NULL.
*
* Backward-compatible: walcompare() keeps its original concise output (the
* `== walcompare ==` header, the human divergence block, and the final
* `walcompare: OK` verdict line) and additionally emits the machine
* classification at a committed divergence; only walcompare_report() prints the
* full per-LSN table and scans the uncommitted tail.
*/
int walcompare_report(const walcompare_member_t* members, int n, FILE* out,
walcompare_divergence_t* div);

View file

@ -0,0 +1,76 @@
#pragma once
/*
* waldump.h — read-only WAL segment inspector for failover/durability triage.
*
* Decodes the on-disk WAL (`wal_entry_hdr_t` + payload, see wal_types.h),
* lists every physical entry, decodes JOB_SUBMIT jobids via the Cap'n Proto
* Job schema, and — the decisive signal for the committed-job-loss bug —
* flags LSN gaps (a missing prefix, or any non-contiguous hole) and reports
* the CRC-chain "recovered tip" a real recovery would stop at.
*
* Strictly read-only: opens segments O_RDONLY, applies no entries, mutates no
* persist_t state. Safe to run against a live node's data dir.
*/
#include <stdint.h>
#include <stdio.h>
/* Aggregate result of scanning one path (a seg file, a wal/ dir, or a node
* data dir). All counts are over the physically-present, CRC-valid entries. */
typedef struct {
uint64_t first_lsn; /* lowest LSN seen (0 = no entries) */
uint64_t last_lsn; /* highest LSN seen */
uint64_t recovered_tip; /* highest LSN reachable via an intact CRC chain */
uint32_t n_entries; /* total entries scanned */
uint32_t n_submits; /* WAL_JOB_SUBMIT entries */
uint32_t n_gaps; /* LSN discontinuities (missing prefix counts as 1) */
uint32_t n_segments; /* segment files scanned */
uint32_t n_dup_submits; /* JOB_SUBMITs with same jobid as the immediately-
* preceding entry at an adjacent LSN (bug-845) */
uint32_t n_dup_identical; /* of n_dup_submits, those whose payload is
* byte-identical to the prior entry */
int crc_broken; /* nonzero if a CRC/chain break truncated a segment */
int leading_gap_explained; /* nonzero if the leading (prefix) gap's
* upper LSN is covered by a sibling
* data/snap/snap-<hex>.capnp file — i.e.
* normal checkpoint-triggered WAL trim
* (persist.c trim_wal_segments), not
* genuine loss. 0 if there was no leading
* gap, or no covering snapshot was found. */
} waldump_stats_t;
/* One physically-present, CRC-valid WAL entry, surfaced to a visitor so a
* caller (e.g. walcompare) can index the log by `(lsn -> term, pcrc)` without
* re-implementing the segment/CRC parse. `pcrc` is the payload-only CRC32 (0
* for an empty payload) — the same fingerprint waldump prints, isolating the
* blob from the header so a same-LSN content divergence is detectable. */
typedef struct {
uint64_t lsn;
uint64_t term; /* raw on-disk term (legacy 0 kept verbatim) */
uint32_t type; /* WAL entry type (wal_types.h) */
uint32_t payload_len; /* payload byte count */
uint32_t pcrc; /* CRC32 of payload only (0 if payload_len == 0) */
} waldump_entry_t;
/* Per-entry visitor. Invoked once per CRC-valid entry in ascending scan order
* (before the gap line is printed). Return 0 to continue, non-zero to stop the
* scan early. `ctx` is the caller's opaque pointer. */
typedef int (*waldump_visit_fn)(void* ctx, const waldump_entry_t* e);
/* Dump every WAL entry under `path` to `out` (human-readable) and fill
* `*stats` when non-NULL. `path` may be:
* - a single seg-*.wal file,
* - a directory holding seg-*.wal files (a wal/ dir), or
* - a node/data dir whose wal/ subdir (data/wal or wal) holds the segments.
* Returns 0 on success, or a negative errno on a hard I/O error (e.g. the
* path does not exist / no segments found). Gaps and CRC breaks are reported
* through `out`/`stats`, NOT via the return code. */
int waldump_path(const char* path, FILE* out, waldump_stats_t* stats);
/* Same as waldump_path, but also invokes `visit(ctx, &entry)` for every
* CRC-valid entry so a caller can collect the `(lsn -> term, pcrc)` index.
* `visit` may be NULL (then identical to waldump_path). `out` may be a sink
* the caller does not care about (e.g. a /dev/null FILE) — the human dump is
* still produced. The single canonical WAL decoder is reused; no second
* segment parser. */
int waldump_path_visit(const char* path, FILE* out, waldump_stats_t* stats,
waldump_visit_fn visit, void* ctx);

View file

@ -0,0 +1,84 @@
#pragma once
/*
* xcetest_interrupt.h -- tiny, storage-detail-hiding surface over xce_test's
* process-wide SIGINT handling (xce-test-cli-ux spec, task 1.3;
* Requirement 9.9).
*
* ROOT CAUSE this task fixes: test.c's main() used to install its SIGINT
* handler via the bare libc call `signal(SIGINT, sigbreak);`. glibc's
* signal() uses BSD semantics, which install the handler with SA_RESTART
* SET by default -- a slow syscall (read(), etc.) that is interrupted by
* the signal is transparently RESTARTED by the kernel instead of
* returning EINTR to the caller. That silently defeats Requirement 9.9:
* a later task's confirmation-prompt read (confirm.c) needs a real Ctrl-C
* to make its blocking read() return immediately with EINTR, not keep
* waiting for input as if nothing happened. See research.md's "Decision:
* Confirmation prompt's Ctrl-C-as-decline requires SA_RESTART cleared on
* SIGINT" for the full rationale and alternatives considered.
*
* THE FIX: xcetest_install_sigint() below installs the given handler via
* sigaction() with sa_flags = 0 (i.e. explicitly WITHOUT SA_RESTART) and
* an empty sa_mask. test.c's main() now calls
* `xcetest_install_sigint(sigbreak);` instead of the old signal() call.
*
* WHY THIS LIVES IN A HEADER, AS `static inline`, RATHER THAN AS A
* NORMAL EXPORTED FUNCTION DEFINED IN test.c: test.c pulls in a large
* dependency graph (libedit, cJSON, libxce, Cap'n Proto, OpenSSL) that
* this project's Unity-based `controller_unit_test` binary does not
* link, and test.c itself defines main() -- both make it impractical to
* link test.c into a small, fast, signal-level unit test. Defining the
* installer here means test.c's main() and this task's test
* (test_xcetest_interrupt.c) call the EXACT SAME code, not two
* hand-copies that could silently drift apart -- the test genuinely
* proves what main() actually installs.
*
* xcetest_interrupted() is a small read-only accessor over test.c's
* PRIVATE `static int break_out` flag (still declared/defined in test.c,
* unmodified in shape or storage -- only this accessor is new), so code
* outside test.c -- the Interruptible Call Adapter, `interruptcall.c`,
* task 1.4 -- can poll whether a SIGINT has been observed without
* depending on break_out's storage details.
*
* CONTRACT (task 1.4's implementer: read this before use):
* - Returns nonzero once sigbreak() has run (i.e. a SIGINT was
* delivered) since break_out was last 0; returns 0 otherwise.
* - This is a PURE READ -- calling it does NOT clear break_out as a
* side effect. It is safe to call any number of times from a polling
* loop (e.g. "poll break_out every fifty milliseconds", per
* design.md's Interruptible Call Adapter) without consuming the
* interrupt.
* - No reset/clear accessor is exposed yet. test.c itself already
* resets break_out to 0 at one existing call site (cmd_hello's
* repeat loop, immediately before that loop begins) by writing the
* static directly, since it has that access. If task 1.4's design
* needs an interruptible operation to start from a guaranteed-clear
* flag and it runs from OUTSIDE test.c, it will need a new
* xcetest_clear_interrupted()-style accessor -- none exists yet
* because no task through 1.3 needed one; do not assume one is
* silently available.
*/
#include <signal.h>
#include <string.h>
/* Installs `handler` as the process's SIGINT disposition via sigaction(),
* with sa_flags = 0 (i.e. explicitly WITHOUT SA_RESTART) and an empty
* sa_mask (no other signal is additionally blocked while `handler`
* runs). This is the one and only installation path for xce_test's
* SIGINT handling -- see the file header above for why a blocking slow
* syscall must observe EINTR rather than a silent restart. */
static inline void xcetest_install_sigint(void (*handler)(int))
{
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sa.sa_handler = handler;
sa.sa_flags = 0; /* explicitly WITHOUT SA_RESTART */
sigemptyset(&sa.sa_mask);
sigaction(SIGINT, &sa, NULL);
}
/* See "CONTRACT" in the file header above. Implemented in test.c, over
* its private `static int break_out`. */
int xcetest_interrupted(void);