Rongsong Shen
bd1f5175a1
Design review (NO-GO -> GO) and iterated requirements review until clean: - Controller refuses to start (non-zero exit, stderr, before any network I/O) when an env-supplied XCE_CONTROLLER_PEERS lacks its own name (1.6); new libxce xce_peers_find_identity() replaces both identity loops in controller/cluster.c so the check and my_priority/listen port agree. - deploy-ha.sh's generated scripts/xceadmin wrapper exports XCE_CONTROLLER_PEERS so an ambient value cannot redirect it (Req 4). - Opt-out is unsetting/emptying the variable (1.7); "No --peers" log lines say it was unset or empty (1.8); docs cover sudo/systemd not inheriting it (3.1). - Requirement 1 defines explicit candidate list / resolution point / own name so harness (set masters) is covered precisely; error messages name the value's source incl. validation failures (1.4). - xceadmin: name= stripping and a clean malformed-entry error apply to every --controller source; errors name the env var source (2.2, 2.8). - Test plan fixed against real code (stderr vs stdout, log texts, xcecd_virtual, --norc, free ports, failover order test). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
239 lines
14 KiB
Markdown
239 lines
14 KiB
Markdown
# Requirements Document
|
|
|
|
## Introduction
|
|
|
|
Operators and deploy/test scripts must retype the full controller
|
|
candidate list on every invocation of controller, scheduler and worker
|
|
(their `--peers` option) and of harness/xce_test (its `set masters`
|
|
setting) — there is no way to set a default once per session or
|
|
environment for these four peers-based components today. The candidate
|
|
list is optional for all four,
|
|
and leaving it out intentionally selects each component's existing
|
|
standalone/single-controller behavior (confirmed: the controller,
|
|
scheduler and worker startup logs already name this an intentional path,
|
|
not an error) — that behavior must keep working unchanged, apart from
|
|
added wording in its startup log line (1.8). The fifth tool, xceadmin,
|
|
already solved the equivalent problem for its own `--controller` option via
|
|
the `XCEADMIN_CONTROLLER` environment variable (bug-1566), and, separately,
|
|
xceadmin's `--controller` option can already accept a comma-separated list
|
|
of candidates and will try each in order until one is reachable (bug-1821),
|
|
rather than only a single target.
|
|
|
|
This feature introduces one new environment variable,
|
|
`XCE_CONTROLLER_PEERS`, read as a fallback default by all five components
|
|
(controller, scheduler, worker, harness/xce_test, and xceadmin) whenever
|
|
no explicit value is given (`--peers`, `set masters`, or `--controller`)
|
|
— and only then; an explicit value always wins, and the environment
|
|
variable is never used as the candidate list when one is given. Explicit `--peers` and
|
|
`set masters` values behave exactly as today, apart from the error wording
|
|
in 1.4; an explicit `--controller`
|
|
value gains the two parsing changes described under Adjacent expectations
|
|
(2.2, 2.8). xceadmin's existing `XCEADMIN_CONTROLLER` variable keeps
|
|
working as a lower-priority fallback beneath the new variable, unchanged
|
|
apart from the parsing changes in 2.2 and 2.8 (which apply to every source
|
|
of the `--controller` value).
|
|
|
|
For the controller specifically, `--peers` is not just a list of servers to
|
|
try: it defines the HA (Raft) cluster membership, and the controller finds
|
|
its own place in that list by matching its name. A value inherited from the
|
|
environment, rather than typed next to `--myname`, must therefore never let
|
|
a controller join a cluster it cannot identify itself in.
|
|
|
|
## Boundary Context
|
|
|
|
- **In scope**: A single new environment variable, `XCE_CONTROLLER_PEERS`,
|
|
providing the default connection target for controller, scheduler and
|
|
worker (via their existing `--peers` option), harness/xce_test (via its
|
|
existing `masters` setting) and xceadmin (via its existing `--controller`
|
|
option), used only when no explicit value is given.
|
|
- **Out of scope**:
|
|
- The apiserver (its controller connection comes from a configuration
|
|
file, not a command-line option) and xcecli's `--url` option (a
|
|
different target — the apiserver, not a controller). Both are
|
|
unaffected.
|
|
- Any change to `src/tools/deploy-ha.sh`'s generated daemon scripts,
|
|
which continue to pass `--peers` explicitly exactly as they do today.
|
|
One exception: the generated `scripts/xceadmin` wrapper selects its
|
|
deployment's controllers only through the `XCEADMIN_CONTROLLER`
|
|
environment variable, which the new variable outranks, so it also
|
|
exports `XCE_CONTROLLER_PEERS` (Requirement 4).
|
|
- Any change to what each component does today when no explicit value
|
|
is given and no environment variable applies either (today's
|
|
standalone/single-controller behavior is preserved, apart from the log
|
|
wording in 1.8).
|
|
- **Adjacent expectations**: xceadmin's existing in-order,
|
|
try-until-reachable failover for `--controller` (bug-1821) and its
|
|
existing `XCEADMIN_CONTROLLER` fallback (bug-1566) keep working as they
|
|
do today, apart from the parsing changes below. xceadmin's parsing of a `--controller`-shaped value changes in
|
|
two ways, whichever source the value comes from (an explicit
|
|
`--controller` included): a `name=` prefix on an entry is removed (2.2),
|
|
and a malformed entry is reported as an error rather than a crash (2.8).
|
|
Today an explicit `--controller ctrl-1=host:port` fails as an
|
|
unresolvable hostname; after this feature it reaches `host:port`.
|
|
|
|
## Requirements
|
|
|
|
### Requirement 1: Default candidate list for peers-based components
|
|
|
|
**Objective:** As an operator running controller, scheduler, worker, or
|
|
harness/xce_test, I want a default controller candidate list available via
|
|
an environment variable, so that I don't have to retype it on every
|
|
invocation while still being able to run any single invocation without it.
|
|
|
|
**Terms used below:**
|
|
- *Peers-based component*: controller, scheduler, worker, or
|
|
harness/xce_test.
|
|
- *Explicit candidate list*: for controller, scheduler and worker, the
|
|
`--peers` argument; for harness/xce_test, a `set masters ...` command,
|
|
whether typed interactively, in a script, or in its rc file.
|
|
- *Resolution point*: for controller, scheduler and worker, process
|
|
startup; for harness/xce_test, each connection attempt (`client`,
|
|
`worker` and similar commands), since `set masters` can run at any time
|
|
in a session.
|
|
- *Own name* (controller only): the `--myname` value, or the host name
|
|
the controller uses when `--myname` is not given.
|
|
|
|
#### Acceptance Criteria
|
|
1. When the peers-based component reaches its resolution point with an
|
|
explicit candidate list, the peers-based component shall use that list
|
|
and shall not use `XCE_CONTROLLER_PEERS` as its candidate list (it may
|
|
still read the variable, e.g. to show it as harness/xce_test's `masters`
|
|
default in `set` output).
|
|
2. When the peers-based component reaches its resolution point without an
|
|
explicit candidate list and `XCE_CONTROLLER_PEERS` is set to a
|
|
non-empty value, the peers-based component shall use that value as its
|
|
candidate list, exactly as if it had been given as an explicit
|
|
candidate list, except for the error-source wording in 1.4, the
|
|
controller's own-name check in 1.6, and, for harness/xce_test, that the
|
|
value is shown as the `masters` setting's default rather than its value
|
|
in `set` output.
|
|
3. When the peers-based component reaches its resolution point without an
|
|
explicit candidate list and `XCE_CONTROLLER_PEERS` is unset or empty,
|
|
the peers-based component shall behave exactly as it does today with no
|
|
explicit candidate list (controller: standalone single-node primary;
|
|
scheduler and worker: their single-controller path; harness/xce_test:
|
|
its built-in `masters` default), apart from the log wording in 1.8.
|
|
4. If the candidate list in use, from either an explicit candidate list
|
|
or `XCE_CONTROLLER_PEERS`, fails parsing or the component's own
|
|
validation of it (including the controller's rule that the number of
|
|
candidates must be odd), the peers-based component shall report an
|
|
error message that identifies whether the value came from the explicit
|
|
candidate list (`--peers`, or `set masters` for harness/xce_test) or
|
|
from the `XCE_CONTROLLER_PEERS` environment variable, and shall keep
|
|
today's failure behavior for that error.
|
|
5. The peers-based component's help/usage text for its candidate list
|
|
(`--peers` for controller, scheduler and worker; the `masters` setting
|
|
in harness/xce_test's `set` help) shall document `XCE_CONTROLLER_PEERS`
|
|
as an environment-variable fallback for it.
|
|
6. If the controller takes its candidate list from `XCE_CONTROLLER_PEERS`
|
|
and its own name matches no candidate in that list, the controller
|
|
shall exit with a non-zero status before opening any network
|
|
connection, writing an error message to stderr that states the
|
|
controller's own name, names `XCE_CONTROLLER_PEERS` as the source, and the two remedies: pass
|
|
`--myname` naming one of the candidates, or unset
|
|
`XCE_CONTROLLER_PEERS` to run standalone. When the candidate list comes
|
|
from an explicit `--peers` argument, the controller's identity handling
|
|
is unchanged.
|
|
7. The peers-based component shall treat an empty `XCE_CONTROLLER_PEERS`
|
|
the same as an unset one, so that unsetting or emptying the variable for
|
|
one invocation gives that invocation the behavior in 1.3; no separate
|
|
opt-out option is added, and the help text from 1.5 shall describe this
|
|
way of opting out.
|
|
8. When the controller, scheduler or worker starts with neither an
|
|
explicit `--peers` argument nor a non-empty `XCE_CONTROLLER_PEERS`, its
|
|
existing startup log line for that case shall also say that
|
|
`XCE_CONTROLLER_PEERS` was unset or empty, so an operator whose variable was
|
|
dropped (e.g. by `sudo`) can tell from the log why the standalone or
|
|
single-controller path was taken.
|
|
|
|
### Requirement 2: Default controller candidate list for xceadmin
|
|
|
|
**Objective:** As an xceadmin operator, I want `--controller` to fall back
|
|
to the same environment variable the other XCE components use, so that one
|
|
variable configures every tool's default connection target.
|
|
|
|
#### Acceptance Criteria
|
|
1. When an xceadmin subcommand that accepts `--controller` runs without an
|
|
explicit `--controller` option and `XCE_CONTROLLER_PEERS` is set to a
|
|
non-empty value, xceadmin shall derive its controller candidate list
|
|
from `XCE_CONTROLLER_PEERS`.
|
|
2. Where a candidate entry carries a `name=` prefix, xceadmin shall remove
|
|
the prefix before treating that entry as a controller candidate. This
|
|
applies to every source of the `--controller` value, an explicit
|
|
`--controller` included.
|
|
3. When xceadmin derives its controller candidate list from
|
|
`XCE_CONTROLLER_PEERS`, xceadmin shall attempt each resulting candidate
|
|
in the same order the variable lists them, moving to the next candidate
|
|
only when the current one is unreachable — the same behavior xceadmin
|
|
already provides for an explicit multi-candidate `--controller` value.
|
|
4. When an xceadmin subcommand that accepts `--controller` runs with an
|
|
explicit `--controller` option, xceadmin shall use that option's value
|
|
(parsed as in 2.2 and 2.8) and shall not consult `XCE_CONTROLLER_PEERS`
|
|
or `XCEADMIN_CONTROLLER`.
|
|
5. When an xceadmin subcommand that accepts `--controller` runs without an
|
|
explicit `--controller` option, and `XCE_CONTROLLER_PEERS` is unset or
|
|
empty, and `XCEADMIN_CONTROLLER` is set to a non-empty value, xceadmin
|
|
shall use `XCEADMIN_CONTROLLER` exactly as it does today, apart from
|
|
the parsing changes in 2.2 and 2.8.
|
|
6. When an xceadmin subcommand that accepts `--controller` runs without an
|
|
explicit `--controller` option, and both `XCE_CONTROLLER_PEERS` and
|
|
`XCEADMIN_CONTROLLER` are unset or empty, xceadmin shall fall back to
|
|
its existing hardcoded default controller target, unchanged.
|
|
7. The `--controller` option's help text shall document
|
|
`XCE_CONTROLLER_PEERS` as its preferred environment-variable fallback,
|
|
alongside the existing `XCEADMIN_CONTROLLER` mention.
|
|
8. If xceadmin cannot reach any controller candidate, or a candidate entry
|
|
is malformed (e.g. a non-numeric port), xceadmin shall exit with an
|
|
error message rather than an unhandled exception. When the
|
|
`--controller` value came from `XCE_CONTROLLER_PEERS` or
|
|
`XCEADMIN_CONTROLLER` rather than an explicit `--controller` option,
|
|
that message shall name the environment variable it came from. For a
|
|
malformed entry, whatever the source, the message shall quote the
|
|
malformed entry.
|
|
|
|
### Requirement 3: Documentation of the new environment variable
|
|
|
|
**Objective:** As an operator or future contributor, I want the project's
|
|
build/deployment documentation to describe `XCE_CONTROLLER_PEERS`, so that
|
|
I can discover and use it without reading source code.
|
|
|
|
#### Acceptance Criteria
|
|
1. The project's build/deployment steering documentation shall describe
|
|
`XCE_CONTROLLER_PEERS`: which components read it, its value format, its
|
|
precedence relative to each component's explicit value (`--peers`,
|
|
`set masters`, or `--controller`), how to opt out for one invocation
|
|
(1.7), the controller's own-name rule (1.6), and that it is not
|
|
inherited by a
|
|
process started through `sudo` (default `env_reset`) or by a systemd
|
|
service, with the way to pass it in each case (e.g.
|
|
`sudo --preserve-env=XCE_CONTROLLER_PEERS ...`, or `Environment=` in
|
|
the unit). This matters most for the production worker, which must run
|
|
as root.
|
|
2. The HA cluster deployment guidance shall note that
|
|
`XCE_CONTROLLER_PEERS` is available as an alternative to passing
|
|
`--peers`/`--controller` explicitly, that the generated daemon scripts
|
|
still pass `--peers` explicitly, and that the generated
|
|
`scripts/xceadmin` wrapper exports `XCE_CONTROLLER_PEERS` for its own
|
|
deployment (Requirement 4).
|
|
3. The multi-host verification guidance shall note the same availability
|
|
wherever it currently documents driving deployment via `--peers`,
|
|
without stating or implying any change to that verification's own
|
|
generated invocations.
|
|
|
|
### Requirement 4: Generated xceadmin wrapper is not shadowed by an ambient variable
|
|
|
|
**Objective:** As an operator who has `XCE_CONTROLLER_PEERS` set in my
|
|
shell for another cluster, I want a deployment's generated
|
|
`scripts/xceadmin` wrapper to keep reaching that deployment's own
|
|
controllers, so that the new variable does not bring back bug-1566's
|
|
wrong-controller symptom.
|
|
|
|
#### Acceptance Criteria
|
|
1. The `scripts/xceadmin` wrapper generated by `deploy-ha.sh` shall export
|
|
`XCE_CONTROLLER_PEERS` set to the deployment's own full `--peers`
|
|
candidate list before starting xceadmin, overriding any value inherited
|
|
from the caller's environment.
|
|
2. The generated wrapper shall keep exporting `XCEADMIN_CONTROLLER` exactly
|
|
as it does today.
|
|
3. An explicit `--controller` option passed to the generated wrapper shall
|
|
still take precedence over both exported variables.
|