update documents

src/docs/deployment.md

@@ -7,11 +7,11 @@ Covers building and running all four daemons (controller, worker, scheduler, api
 ```
 xceadmin (Python CLI)  --config push (TLS)-->  controller (C, HA cluster, 1..N nodes)
                                                      ^              ^
-                                    plaintext join   |              | plaintext join
+                                    TLS join         |              | TLS join
                                                      |              |
                                                  worker (C)     scheduler (C)
                                                      ^
-                                    CGO-linked libxce client (TCP, HA-aware)
+                                    CGO-linked libxce client (TCP, TLS, HA-aware)
                                                      |
                                                 apiserver (Go, REST) <-- xcecli (Go CLI, REST)
 ```
@@ -23,16 +23,23 @@ xceadmin (Python CLI)  --config push (TLS)-->  controller (C, HA cluster, 1..N n
 - **xcecli** — Go CLI, talks to `apiserver` over REST, never touches the controller directly.
 - **xceadmin** — Python CLI, the only tool that compiles and pushes *cluster configuration* (`cluster.bin`) to the controller. Separate concern from the four daemons above.
 
-Only controller↔controller HA replication traffic is TLS. Worker→controller, scheduler→controller, and apiserver→controller traffic is **plaintext**.
+**Every** connection into the controller is TLS-encrypted — controller↔controller HA replication, and every `libxce`-client connection (worker→controller, scheduler→controller, apiserver→controller, xceadmin→controller). The controller runs a single `iomgr` bound to one server `SSL_CTX` (`myapp.c`); `endpoint_accept()`/`endpoint_connect()` in `src/lib/netio.c` have no plaintext code path at all — every accepted or outgoing socket is wrapped in `SSL_new()` before use. What's actually missing is **client authentication and identity verification**, not encryption: TLS verification is `SSL_VERIFY_NONE` everywhere in this codebase (server and client contexts alike, no client certificates), so any of these connections buys you encryption-in-transit, not authentication of who's on the other end.
 
 ## Building
 
+Two different Nix entry points, for two different jobs — don't conflate them:
+
+- **`nix develop --command` — single-component dev builds.** `/xce-build` is not a separate build system; it's `nix develop ./src --command bash src/tools/xce-build.sh <component> <profile>` under the hood, i.e. the hermetic dev shell driving meson/ninja for exactly one component. This is the reproducible way to build just `controller`, just `worker`, etc.
+- **`nix build` — whole-stack production packaging.** A packaged Nix derivation, all-or-nothing (see Production Gaps item 6) — it always produces every daemon in one `result/`, never a single one.
+
 ```bash
-# Dev loop, per component (debug + ASAN, runs tests)
+# Dev loop, per component (debug + ASAN, runs tests) — wraps `nix develop ./src --command ...`
+/xce-build lib           # libxce only
 /xce-build controller
 /xce-build worker
 /xce-build scheduler
-/xce-build apiserver
+/xce-build apiserver      # also builds xcecli in the same pass (same meson step)
+/xce-build xcecli         # xcecli alone, if you don't need apiserver
 /xce-build xceadmin
 /xce-build              # all components
 
@@ -41,10 +48,10 @@ Only controller↔controller HA replication traffic is TLS. Worker→controller,
 #   debug            -> build.dbg/       -O0 -g3, no sanitizer, gdb-friendly
 #   release          -> build.release/   -O3, no sanitizer, build-only
 
-# Production (Nix)
+# Production packaging (Nix, whole stack only)
 nix build .#xcecore          # controller + worker + scheduler + apiserver + xceadmin + AppImage, one derivation
 nix build .#xcecore_debug    # same, debug + ASAN (this flake's default package)
-nix build .#xcecli           # xcecli only (separate Go-only derivation)
+nix build .#xcecli           # xcecli, as its own separate Nix *package* derivation (Go-only, no C toolchain) — `/xce-build xcecli` above gets you the same binary without going through `nix build` at all
 ```
 
 `nix build .#xcecore` is all-or-nothing — there is no per-daemon Nix target. It produces one `result/bin/` containing `xce_controller`, `xcecd` (+ `xcecd_worker`, `xcecd_virtual`), `xce_scheduler`, `xce_apiserver`, the `xceadmin` venv wrapper, and `xce` (the AppImage dispatcher script, see below). `nix build .#xcecli` is the one genuinely separate target.
@@ -96,8 +103,8 @@ xceadmin config push --dry-run cluster.bin --controller 127.0.0.1:9991
 
 ### Known limitations (as of this writing)
 
-- **`config push` against a real controller fails with a client-side decode crash**, live-verified against a freshly deployed single-node controller: the controller's `OPCode_upgradeConfig` dispatch is fine — `CONF_ENABLE_UPGRADE_CONFIG` defaults on, the earlier dispatch-gate polarity bug (bug-1468) was already fixed 2026-07-25, and the controller log shows the push genuinely committed, activated, and hash-verified server-side. The bug is in `xceadmin` itself: `xce_reply()` (`src/lib/request.c`) wraps every reply — including the upgradeConfig reply — in an outer `Request{opcode: OPCode_reply, body.replymsg: RequestReply{...}}` envelope, but `discovery.py`'s `_redirect_target()` decodes the raw wire bytes directly as a bare `RequestReply`, skipping the outer envelope. The resulting pointer-offset misalignment throws `capnp.lib.capnp.KjException: ... non-list pointer where text was expected` from `push_config()`. This affects every opcode reply routed through `follow_redirects`, not just upgradeConfig. Fix belongs in `xceadmin`, not the controller.
-- **`config push --dry-run`'s "OK" result is not meaningful evidence of anything**: `probe_config()`'s docstring itself notes `OPCode_probeConfigTag` has no dispatch case in `cluster.c`, so the "accepted" response is a client-side inference, not a real wire signal from the controller. Live-verified: dry-run returns exit 0 with no corresponding entry in the controller log at all.
+- **~~`config push` against a real controller fails with a client-side decode crash~~** — **Fixed 2026-07-30.** `decode_reply()` (`src/xceadmin/src/xceadmin/discovery.py:46-77`) now unwraps the outer `Request{opcode: OPCode_reply, body.replymsg: RequestReply{...}}` envelope before decoding the inner `RequestReply`. Both `_redirect_target()` and `push_config()` go through `decode_reply()`; the old bare-`RequestReply` decode path is gone. `decode_reply()` validates `body.which() == "replymsg"` and raises `ReplyDecodeError` on anything else. Live-verified against a real deployed controller.
+- **~~`config push --dry-run`'s "OK" result is not meaningful evidence of anything~~** — **Fixed 2026-08-09 (bug-1506/1507).** The controller gained a real `OPCode_probeConfigTag` handler (`config_repl.c:handle_probe_config_tag`) that returns the current `config_version_tag` through the standard `Request`/`RequestReply` envelope. `probe_config()` now wraps the probe in a `Request` envelope and goes through `follow_redirects` → `decode_reply` (same as `push_config`), so `accepted=True` means the controller actually replied with a valid tag, and `--dry-run` reports the real generation number from the cluster.
 - No `config diff`, no rollback, no version history command. `xceadmin` today has exactly two subcommands: `config build` and `config push [--dry-run]`.
 - No client authentication — the trust model is "same posture as controller-to-controller TLS," i.e. encryption without identity verification.
 
@@ -117,7 +124,7 @@ xce_controller \
 
 - Omit `--peers` for a single-node, non-HA controller.
 - `--workdir` holds per-node `conf/` (cluster config) and `data/` (WAL + checkpoint) subdirectories.
-- `--certdir` must contain `xce.crt`/`xce.key` — this is the **only** TLS material in the whole stack, used exclusively for controller↔controller HA traffic. Generate a self-signed pair for local/dev use:
+- `--certdir` must contain `xce.crt`/`xce.key` — this is the **only** TLS material in the whole stack, and the controller loads it into the single server `SSL_CTX` used for **every** inbound connection it accepts: controller↔controller HA traffic *and* every worker/scheduler/apiserver/xceadmin client join. Generate a self-signed pair for local/dev use:
   ```bash
   openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
     -keyout /var/lib/xce/certs/xce.key -out /var/lib/xce/certs/xce.crt -subj "/CN=xce"
@@ -163,7 +170,7 @@ xcecd -m controller-host -p 9990 --peers "ctrl-1=h1:9990,ctrl-2=h2:9990" -l /var
     log_dir: "/var/log/xce"
   ```
 - `--peers "[name=]host:port,..."` enables HA-aware controller discovery: the worker tries each candidate in turn (every 5s by default) until one accepts the join.
-- No TLS on this connection — worker↔controller traffic is plaintext.
+- Worker↔controller traffic is TLS-encrypted (via the shared `libxce` client, `xcecli_ctx_init()` → `xce_ctx_init()`), same posture as everything else in the stack: no client certificate, `SSL_VERIFY_NONE`.
 
 ## Deploying the Scheduler
 
@@ -174,7 +181,7 @@ xce_scheduler --peers "ctrl-1=controller-host:9990" -l /var/log/xce/scheduler.lo
 - No config file at all — pure CLI (`argp`).
 - **Gotcha**: the default controller target is hard-coded to `localhost:9990`. There is no `-m`/`-p` equivalent — the *only* way to point the scheduler at any other host or port, even a single non-HA controller, is `--peers`. Forgetting `--peers` silently targets `localhost:9990` regardless of where your controller actually is.
 - `-v` is repeatable for more verbose logging; default log file is `/tmp/xce_scheduler.log`.
-- No TLS — plaintext to the controller, same as the worker.
+- TLS-encrypted to the controller, same as the worker (same shared `libxce` client path).
 - Scheduling policies (fcfs, limit, parallel/sequential-job allocators) are compiled into the binary through the picker plugin interface (`builtin_fcfs` is the registered default), not dynamically loaded.
 
 ### Session recording (live cluster)
@@ -241,7 +248,7 @@ Both `app.yaml` and `logger.yaml` are JSON-Schema-validated at startup (fail-fas
 
 ### Relationship to the controller
 
-apiserver is a **separate OS process**, network-connected to the controller — not embedded, not in-process. It joins the controller's cluster protocol as `HostRole_apiserver` via a CGO-linked `libxce` client (the same join mechanism workers/schedulers use), configured entirely by `app.yaml`'s `backend:` block. If `backend.address` is empty, apiserver logs "running in stub mode" and serves HTTP, but every job/host/queue/cluster handler will fail. No TLS on this connection.
+apiserver is a **separate OS process**, network-connected to the controller — not embedded, not in-process. It joins the controller's cluster protocol as `HostRole_apiserver` via a CGO-linked `libxce` client (the same join mechanism workers/schedulers use), configured entirely by `app.yaml`'s `backend:` block. If `backend.address` is empty, apiserver logs "running in stub mode" and serves HTTP, but every job/host/queue/cluster handler will fail. This connection is TLS-encrypted, same as every other `libxce` client (no client certificate, `SSL_VERIFY_NONE`) — do not confuse it with apiserver's own separate HTTPS termination for its REST API, covered next.
 
 ### TLS
 
@@ -277,18 +284,18 @@ xceadmin config push cluster.bin --controller localhost:9991
 Be aware of these before treating anything above as a production runbook — they are honest gaps in this repo today, not documented-but-hidden features:
 
 1. **No process supervision is shipped.** No systemd unit, no Dockerfile, no docker-compose, no supervisord config exists anywhere in this repo for any of the four daemons. A systemd-unit *sketch* exists in `src/docs/config-replication-design.md` but has a wrong binary name (`xcecontroller` vs the real `xce_controller`) and specifies `Type=notify`, which the controller does not support (it never calls `sd_notify`) — using it as written will hang at startup and time out. If you adapt it, switch to `Type=simple` and fix the binary name.
-2. **`src/frontend/apiserver/config/auth.yaml` ships plaintext dev credentials checked into git**, including an `admin`/`admin` account and a named-user `cluster_admin` account. Treat all of them as exposed by default — don't ship the checked-in file as-is; copy `auth.yaml.example` and set real credentials/an external auth provider instead.
-3. **Only controller↔controller traffic is encrypted.** Worker→controller, scheduler→controller, and apiserver→controller are all plaintext, with no path to enable TLS on them today.
-4. **apiserver's own TLS uses a checked-in self-signed cert with no CA trust story**; `xcecli` hardcodes `InsecureSkipVerify: true` against it. A reverse-proxy-terminated-TLS model is sketched (nginx sample config) but not automated.
-5. **`xceadmin config push` doesn't work against a real cluster yet** (see xceadmin's Known Limitations above) — the controller side is fine; `xceadmin`'s `discovery.py` fails to unwrap the `Request` envelope around every reply, crashing on decode.
-6. **`nix build .#xcecore` is all-or-nothing** — there's no way to build just one daemon via Nix; use `/xce-build <component>` for per-component builds.
+2. **`src/frontend/apiserver/config/auth.yaml` ships plaintext dev credentials checked into git**: four accounts total — `admin`/`admin` and `op7x2k9`/`CHANGE_ME_dev_only` (both role `cluster_admin`), plus `user1`/`user1` and `user2`/`user2` (role `basic_user`). Treat all of them as exposed by default — don't ship the checked-in file as-is; copy `auth.yaml.example` and set real credentials/an external auth provider instead.
+3. **All controller-bound traffic is encrypted but not authenticated.** Controller↔controller HA replication and every `libxce` client (worker→controller, scheduler→controller, apiserver→controller, xceadmin→controller) run over TLS, but `SSL_VERIFY_NONE` is used throughout — no client certificates, no hostname/identity verification. This buys confidentiality and tamper-resistance in transit, not authentication of the peer; anyone who can reach the port and complete a TLS handshake is treated as trusted the moment the transport-level join succeeds.
+4. **apiserver's own TLS uses a checked-in self-signed cert with no CA trust story**; `xcecli` hardcodes `InsecureSkipVerify: true` against it. A reverse-proxy-terminated-TLS model is sketched (nginx sample config) but not automated. (This is apiserver's REST-facing HTTPS, separate from its TLS-encrypted `libxce` connection to the controller in item 3.)
+5. **`xceadmin config push` against a real cluster** — the decode-crash bug is fixed (2026-07-30, `decode_reply()` unwraps the `Request` envelope), but `--dry-run` still has no real controller-side handler (see Known Limitations item 2).
+6. **`nix build .#xcecore` (the packaged derivation) is all-or-nothing** — there's no per-daemon Nix package target. For a single component, use `/xce-build <component>`, which itself runs inside the Nix dev shell (`nix develop ./src --command ...`) rather than through `nix build`.
 
 ## Quick Reference
 
 | Daemon | Binary | Config | Controller-pointing flag | TLS |
 |---|---|---|---|---|
-| Controller | `xce_controller` | none (CLI only) | `--peers` (omit for single-node) | Yes, controller↔controller only |
-| Worker | `xcecd` | `worker.yaml` (or CLI) | `-m`/`-p` (config file wins) or `--peers` | No |
-| Scheduler | `xce_scheduler` | none (CLI only) | `--peers` (**required** even for a single non-default controller) | No |
-| apiserver | `xce_apiserver` | `config/{app,auth,rbac,logger}.yaml` | `app.yaml`'s `backend:` block | Optional, self-terminated |
+| Controller | `xce_controller` | none (CLI only) | `--peers` (omit for single-node) | Yes, all inbound (peers + clients), no client cert |
+| Worker | `xcecd` | `worker.yaml` (or CLI) | `-m`/`-p` (config file wins) or `--peers` | Yes, no client cert |
+| Scheduler | `xce_scheduler` | none (CLI only) | `--peers` (**required** even for a single non-default controller) | Yes, no client cert |
+| apiserver | `xce_apiserver` | `config/{app,auth,rbac,logger}.yaml` | `app.yaml`'s `backend:` block | To controller: yes, no client cert. Own REST API: optional, self-terminated |
 | xceadmin | `xceadmin` (Python) | none — takes YAML source dir | `--controller host:port` | Yes, no client cert |