diff --git a/.kiro/specs/admin-config-tool/tasks.md b/.kiro/specs/admin-config-tool/tasks.md index 541910c7..5dd0127b 100644 --- a/.kiro/specs/admin-config-tool/tasks.md +++ b/.kiro/specs/admin-config-tool/tasks.md @@ -39,7 +39,7 @@ - _Requirements: 5.1, 5.5, 7.4_ - _Boundary: xcecommon.schema_ -- [ ] 2.3 (P) Map validated YAML into cluster configuration fields +- [x] 2.3 (P) Map validated YAML into cluster configuration fields - Every field the cluster configuration binary format defines is either populated from validated YAML or deliberately left untouched, with no field silently dropped - List-based sections preserve the exact order they appeared in the YAML - No timestamp, hostname, or username value ever enters the mapped output @@ -148,3 +148,4 @@ - **1.3**: `src/xceadmin/src/xcecommon/schema_files/` (synced capnp + JSON schema copies) is a build artifact, regenerated every `xce-build.sh xceadmin` run — never hand-edit it, never commit it (`src/xceadmin/.gitignore` excludes it). - **2.1**: `check_undefined_refs` deliberately does not parse resource names out of `res_req`/`gpu_req` free-form LSF-style expression strings (e.g. `"mem>1000 rusage[mem=1000]"`) — design.md's `xcecommon.validation` component scopes this check to queue→host-group and limit→group/`resname` only. `limits.yaml`'s `projects` field and `scaler.yaml` name collisions are likewise out of scope — no `projects.yaml` catalog exists among the 7 config types, and Requirement 4.3 only enumerates queue/host/resource/group/limit names. `src/configs/hosts.yaml` has 2 pre-existing undefined refs (resource `mg`, host `h3`) that now surface as warnings — expected, not a regression; fixing the example fixture is outside this task's boundary. - **2.2**: The real `controller.capnp` names its reply struct `RequestReply`, not `Reply` as design.md's postcondition states — an inexact design pseudo-type, corrected empirically (verified directly against `src/schema/controller.capnp`, not assumed). pycapnp 2.0.0/capnproto 1.2.0 abort the whole process (uncaught C++ exception, not a Python exception) on a missing cross-file import or malformed `.capnp` content passed to `capnp.load()` — a bare `try/except` cannot catch this. `xcecommon.schema.load_schema()` works around it by pre-validating in an isolated subprocess before ever calling `capnp.load()` in the caller's own process, converting any abort into a clean `SchemaNotFoundError`. Any future code that calls `capnp.load()` directly (bypassing this loader) reintroduces the crash risk — always go through `xcecommon.schema.load_schema()`. +- **2.3**: `map_all_sections()` (a `SECTION_MAP`-driven orchestrator, not explicitly named in design.md's Service Interface but consistent with its Contracts) is the single call `xceadmin.build` (task 3.1) needs — it treats a missing YAML file identically to a present-but-empty section (byte-for-byte), satisfying Requirement 1.5 structurally. `target.init(field, 0)` and never touching an empty/absent list field produce *different* serialized bytes for the same logical "nothing here" value (verified empirically) — the mapper always picks "never touch" for byte-determinism (9.1). Calling `.to_bytes()` twice on the same capnp builder triggers a spurious "message already written" `UserWarning` even when no field was mutated between calls — capture the bytes once and reuse them rather than calling `.to_bytes()` again. diff --git a/src/xceadmin/src/xcecommon/mapping.py b/src/xceadmin/src/xcecommon/mapping.py new file mode 100644 index 00000000..681f5a92 --- /dev/null +++ b/src/xceadmin/src/xcecommon/mapping.py @@ -0,0 +1,456 @@ +"""YAML-section-dict -> ``ClusterConfig`` capnp field mapping for +xcecommon/xceadmin (design.md, ``xcecommon.mapping``; Requirements 1.1, +1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 5.3, 9.1, 9.2, 9.3). + +This module's only outbound dependency is ``xcecommon.schema`` (design.md, +Dependency Direction) -- it never imports ``xcecommon.validation``: every +function here assumes its ``dict`` inputs already passed +``xcecommon.validation``'s JSON Schema + business-rule checks. Re-validating +here would duplicate that boundary, not strengthen it. + +One declarative table, :data:`SECTION_MAP`, drives all 7 YAML files (8 +entries -- ``hosts.yaml`` is the one file that feeds two separate capnp +fields, ``hgroups`` and ``hostConfigs``; see the Data Models table in +design.md). Two mapper shapes cover every section: + +- :func:`map_scalar_section` -- the single flat ``xce.yaml`` section, 5 + top-level ``ClusterConfig`` scalar fields. +- :func:`map_list_section` -- the 7 remaining sections, each a YAML list + mapped to a capnp ``List(Struct)`` field, preserving YAML source order + exactly (never sorted -- Requirement 9.1's byte-determinism depends on + this). + +:func:`map_all_sections` is the SECTION_MAP-driven orchestrator: it is what +actually makes the declarative table load-bearing rather than an inert +constant, and is the single call a future ``xceadmin.build`` (task 3.1) +needs to populate an entire ``ClusterConfig`` builder from a +``schema_name -> parsed_yaml_dict`` mapping (the same shape +``xcecommon.validation.validate_yaml`` returns, keyed the same way +``check_reserved_names``/``check_duplicates``/``check_undefined_refs`` +already key their ``sections`` argument). + +Determinism (Requirements 9.1, 9.2, 9.3) +----------------------------------------- +- No wall-clock, hostname, or username value is read anywhere in this + module -- it imports nothing from the standard library's date/time, + networking-identity, or process-identity facilities, and + ``test_mapping.py`` asserts this statically as a regression guard. +- List order is preserved via ``zip(capnp_list, python_list)`` in + iteration order -- never ``sorted()``, never a ``set``. +- A YAML list that is absent (key missing) and one that is present but + empty (``[]``) are treated identically: neither ever calls + ``target.init(field, 0)``. This matters for byte-determinism -- + ``target.init(field, 0)`` and never touching ``field`` at all produce + *different* serialized bytes for the same logical "nothing here" value + (verified empirically against the real compiled schema), so this module + picks exactly one encoding (untouched) and uses it unconditionally. +- Building a fresh ``ClusterConfig`` message from the same input twice + (two independent ``new_message()`` calls run through the same mapping + calls) produces byte-identical ``to_bytes()`` output -- verified in + ``test_mapping.py``. + +Deliberately Unmapped Fields +----------------------------- +Every nested capnp field below is either populated from validated YAML or +intentionally left at its capnp default, for a specific, documented +reason -- never a silent drop. + +``ClusterConfig`` + ``configGeneration`` (@12) -- never touched (Requirement 1.9); the + primary controller is the sole authority for assigning it. + ``candidates``/``port``/hash-table-size fields/``xstrings`` do not + exist in the schema at all post ``clusterconfig-schema-split`` -- + there is nothing to exclude. + +``Resource`` (``resources.yaml``) + ``idx``, ``valtype``, ``ordtype``, ``flags``, ``lastReport`` are left + unset. The YAML ``type``/``order``/``consumable``/``release`` keys + would need to encode into ``valtype``/``ordtype``/``flags`` as + specific integers, but neither ``controller.capnp`` nor design.md + documents that encoding within this task's boundary -- inventing one + risks silently miswriting a value the controller then misinterprets, + strictly worse than shipping it unset. ``idx`` has no YAML source at + all (controller/runtime-assigned). ``lastReport`` is a runtime + timestamp field (epoch seconds of last report) -- excluded doubly by + Requirement 9.2's "no timestamp" rule. ``resourcemap`` has no + destination: the capnp fields it would map to (``distributions``, + ``hostBased``) are commented out of the ``Resource`` struct + (``controller.capnp:208-209``). + +``QueueConf`` (``queues.yaml``) + ``attrs`` (raw ``UInt32`` bitfield), ``status``, ``acceptIntervals``, + ``schedDelay`` are left unset -- no YAML source, and ``attrs``' bit + assignments are undocumented within this task's boundary (guessing + bit positions for ``backfill``/``exclusive``/``rerunable``/ + ``preemption`` would be worse than leaving the field unset). + ``gpu_req`` is left unmapped: ``QueueConf`` has exactly one + ``resreq`` Text field, already claimed by ``res_req``; concatenating + the two LSF-style expression strings would be a lossy, invented + format this task has no authority to define. ``description``, + ``nice``, ``fairshare``, ``cpulimit``, ``memlimit``, ``runlimit``, + ``runtimeest``, ``reservation``, ``preemption``, ``job_starter`` have + no corresponding ``QueueConf`` field at all. + +``HostConfig`` (``hosts.yaml`` hosts) + ``description`` has no corresponding field. ``mxj`` is set only when + YAML ``max_slot >= 0``; the schema's documented ``-1`` ("auto-detect") + sentinel is left unmapped because ``mxj`` is ``UInt32`` and cannot + represent a negative value (verified: assigning ``-1`` raises + ``capnp.KjException`` rather than wrapping) -- shipping ``mxj`` unset + (default ``0``) preserves "not sourced" instead of crashing the build + or silently wrapping to ``4294967295``. + +``GroupConfig`` used as an embedded ``Membership`` (``QueueConf.users``/ +``.hosts``, ``LimitConfig.users``/``.queues``/``.projects``/``.hosts``) + ``name`` is left unset -- the YAML ``Membership`` shape + (``{members, exclude_members}``) never carries a name for these + inline group references; only the top-level ``ugroups``/``hgroups`` + entries have a ``groupname`` to source ``name`` from. + +``TimeWindowTime`` (nested inside ``TimeWindow``/``dispatchWindow``/ +``runWindow``/``dispatchwindows``) + ``day`` is left at its capnp default (``0``) whenever the YAML time + spec omits a day prefix (bare ``HH`` or ``HH:MM`` form) -- capnp has + no "unset" sentinel distinct from ``0`` for a ``UInt32``, so an + omitted day and an explicit ``day: 0`` (Sunday) are indistinguishable + in the output. Documented, not a silent drop. + +``TimeWindowSeg.timezone`` + Left unset -- no timezone data exists anywhere in the YAML + ``TimeWindow`` shape (``start_time``/``end_time`` only). +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Literal, TypedDict + +from xcecommon.schema import load_schema + +if TYPE_CHECKING: + import capnp # noqa: F401 (type-hint only, see Service Interface below) + + +class SectionSpec(TypedDict): + """One entry of :data:`SECTION_MAP`. + + For the single ``kind="scalar"`` entry (``xce.yaml``), ``yaml_key`` + and ``capnp_field`` are unused placeholders -- :func:`map_scalar_section` + maps all 5 target fields directly via :data:`_XCE_SCALAR_FIELDS` + instead of a single ``yaml_key``/``capnp_field`` pair, since a flat + section maps many-keys-to-many-fields, not the list shape's + one-key-to-one-field. + """ + + yaml_file: str + yaml_key: str + capnp_field: str + kind: Literal["list", "scalar"] + + +# One entry per YAML-file/capnp-field pair (8 entries -- hosts.yaml feeds +# two capnp fields, see module docstring). This is the only place file/key +# names live; every mapper function below is generic over it. +SECTION_MAP: list[SectionSpec] = [ + {"yaml_file": "xce.yaml", "yaml_key": "", "capnp_field": "", "kind": "scalar"}, + {"yaml_file": "resources.yaml", "yaml_key": "resources", "capnp_field": "resources", "kind": "list"}, + {"yaml_file": "queues.yaml", "yaml_key": "queues", "capnp_field": "queues", "kind": "list"}, + {"yaml_file": "users.yaml", "yaml_key": "usergroups", "capnp_field": "ugroups", "kind": "list"}, + {"yaml_file": "hosts.yaml", "yaml_key": "hostgroups", "capnp_field": "hgroups", "kind": "list"}, + {"yaml_file": "hosts.yaml", "yaml_key": "hosts", "capnp_field": "hostConfigs", "kind": "list"}, + {"yaml_file": "scaler.yaml", "yaml_key": "scalers", "capnp_field": "hproviders", "kind": "list"}, + {"yaml_file": "limits.yaml", "yaml_key": "limits", "capnp_field": "limits", "kind": "list"}, +] + +# xce.yaml top-level key -> ClusterConfig scalar field name, in @N order. +_XCE_SCALAR_FIELDS: tuple[tuple[str, str], ...] = ( + ("cluster_name", "clusterName"), + ("heartbeat_interval", "heartbeatInterval"), + ("inactive_heartbeat_count", "inactiveHeartbeatCount"), + ("fetch_job_interval", "fetchJobInterval"), + ("netio_timeout", "netioTimeout"), +) + + +def new_cluster_config_builder() -> "capnp._DynamicStructBuilder": + """Convenience constructor: ``load_schema().ClusterConfig.new_message()``. + + Callers that already hold a builder (or the loaded schema module, for + building other message types alongside it) may call + :func:`map_all_sections`/:func:`map_list_section`/ + :func:`map_scalar_section` directly against their own builder instead + -- this is a convenience, not a required entry point. + """ + return load_schema().ClusterConfig.new_message() + + +def _parse_time_spec(spec: str) -> tuple[int | None, int, int]: + """Parse the ``[day:]HH[:MM]`` time format used by + ``TimeWindow``/``dispatch_window``/``run_window`` YAML entries + (schema pattern documented in ``hosts.schema.json``/ + ``queues.schema.json``). + + Returns ``(day, hour, minute)`` -- ``day`` is ``None`` when the input + has no day prefix (2-part ``HH:MM`` or bare ``HH`` form); callers + leave the capnp ``day`` field untouched in that case (see module + docstring, ``TimeWindowTime``). + """ + parts = spec.split(":") + if len(parts) == 3: + return int(parts[0]), int(parts[1]), int(parts[2]) + if len(parts) == 2: + return None, int(parts[0]), int(parts[1]) + return None, int(parts[0]), 0 + + +def _build_time_window(entries: list[dict] | None, target: "capnp._DynamicStructBuilder") -> None: + """Populate a ``TimeWindow`` struct's ``segs`` list from a YAML list of + ``{start_time, end_time}`` dicts (``dispatch_window``/``run_window`` + shape), preserving order (Requirement 9.1). + """ + if not entries: + return + segs = target.init("segs", len(entries)) + for seg, entry in zip(segs, entries): + start_day, start_hour, start_min = _parse_time_spec(entry["start_time"]) + end_day, end_hour, end_min = _parse_time_spec(entry["end_time"]) + if start_day is not None: + seg.open.day = start_day + seg.open.hour = start_hour + seg.open.min = start_min + if end_day is not None: + seg.close.day = end_day + seg.close.hour = end_hour + seg.close.min = end_min + + +def _build_threshold_list( + entries: list[dict] | None, target: "capnp._DynamicStructBuilder", field_name: str +) -> None: + """Populate a ``List(Threshold)`` field from a YAML ``sched_stop`` list + (``{resname, sched, stop}`` -> ``Threshold{name, sched, stop}``), + preserving order. + """ + if not entries: + return + thresholds = target.init(field_name, len(entries)) + for threshold, entry in zip(thresholds, entries): + threshold.name = entry["resname"] + if "sched" in entry: + threshold.sched = entry["sched"] + if "stop" in entry: + threshold.stop = entry["stop"] + + +def _build_config_var_list(entries: list[dict] | None, target: "capnp._DynamicStructBuilder") -> None: + """Populate ``HostConfig.resources`` (``List(ConfigVar)``) from a + YAML host's ``resource`` list (``{name, value}`` -> ``ConfigVar{name, + value}``, ``value`` stringified since ``ConfigVar.value`` is Text + while YAML allows a number or a string), preserving order. + """ + if not entries: + return + config_vars = target.init("resources", len(entries)) + for config_var, entry in zip(config_vars, entries): + config_var.name = entry["name"] + if "value" in entry: + config_var.value = str(entry["value"]) + + +def _build_membership_group(membership: dict | None, target: "capnp._DynamicStructBuilder") -> None: + """Populate an embedded ``GroupConfig`` (``members``/``exmembers`` + only -- ``name`` deliberately left unset, see module docstring) from + a YAML ``Membership`` dict (``{members, exclude_members}``). + """ + if not membership: + return + members = membership.get("members") or [] + if members: + capnp_members = target.init("members", len(members)) + for i, name in enumerate(members): + capnp_members[i] = name + excludes = membership.get("exclude_members") or [] + if excludes: + capnp_excludes = target.init("exmembers", len(excludes)) + for i, name in enumerate(excludes): + capnp_excludes[i] = name + + +def _build_named_group(item: dict, target: "capnp._DynamicStructBuilder") -> None: + """Populate a top-level ``GroupConfig`` (``ugroups``/``hgroups`` list + item) from a YAML ``HostGroup``/``UserGroup`` dict + (``{groupname, members, exclude_members}``). + """ + target.name = item["groupname"] + _build_membership_group(item, target) + + +def _build_resource(item: dict, target: "capnp._DynamicStructBuilder") -> None: + """Populate one ``Resource`` list item from a ``resources.yaml`` entry. + See module docstring, ``Resource``, for what is intentionally left + unmapped and why. + """ + target.name = item["name"] + if "description" in item: + target.desc = item["description"] + if "interval" in item: + target.interval = item["interval"] + + +def _build_queue(item: dict, target: "capnp._DynamicStructBuilder") -> None: + """Populate one ``QueueConf`` list item from a ``queues.yaml`` entry. + See module docstring, ``QueueConf``, for what is intentionally left + unmapped and why. + """ + target.name = item["name"] + if "priority" in item: + target.priority = item["priority"] + if "res_req" in item: + target.resreq = item["res_req"] + if "pre_exec" in item: + target.preExec = item["pre_exec"] + if "post_exec" in item: + target.postExec = item["post_exec"] + _build_membership_group(item.get("users"), target.users) + _build_membership_group(item.get("hosts"), target.hosts) + _build_threshold_list(item.get("sched_stop"), target, "thresholds") + _build_time_window(item.get("dispatch_window"), target.dispatchWindow) + _build_time_window(item.get("run_window"), target.runWindow) + + +def _build_host_config(item: dict, target: "capnp._DynamicStructBuilder") -> None: + """Populate one ``HostConfig`` list item from a ``hosts.yaml`` ``hosts`` + entry. See module docstring, ``HostConfig``, for what is intentionally + left unmapped and why. + """ + target.name = item["hostname"] + if "model" in item: + target.model = item["model"] + if "type" in item: + target.type = item["type"] + max_slot = item.get("max_slot") + if max_slot is not None and max_slot >= 0: + target.mxj = max_slot + _build_config_var_list(item.get("resource"), target) + _build_threshold_list(item.get("sched_stop"), target, "thresholds") + _build_time_window(item.get("dispatch_window"), target.dispatchwindows) + + +def _build_scaler_config(item: dict, target: "capnp._DynamicStructBuilder") -> None: + """Populate one ``ScalerConfig`` list item from a ``scaler.yaml`` + ``scalers`` entry -- a clean 1:1 field mapping, nothing left unmapped. + """ + target.name = item["name"] + target.type = item["type"] + if "conf_path" in item: + target.confPath = item["conf_path"] + target.scriptPath = item["script_path"] + if "script_option" in item: + target.scriptOption = item["script_option"] + + +def _build_limit_config(item: dict, target: "capnp._DynamicStructBuilder") -> None: + """Populate one ``LimitConfig`` list item from a ``limits.yaml`` + ``limits`` entry -- a clean field mapping, nothing left unmapped. + """ + target.name = item["name"] + _build_membership_group(item.get("users"), target.users) + _build_membership_group(item.get("queues"), target.queues) + _build_membership_group(item.get("projects"), target.projects) + _build_membership_group(item.get("hosts"), target.hosts) + target.val.resname = item["resname"] + target.val.value = item["value"] + if "percent" in item: + target.val.percentage = item["percent"] + if "per_user" in item: + target.perUser = item["per_user"] + if "per_queue" in item: + target.perQueue = item["per_queue"] + if "per_project" in item: + target.perProject = item["per_project"] + if "per_host" in item: + target.perHost = item["per_host"] + + +# capnp_field -> (YAML item dict, capnp list-item builder) -> None. +# Every list-shaped SECTION_MAP entry's capnp_field is a key here. +_LIST_ITEM_BUILDERS = { + "resources": _build_resource, + "queues": _build_queue, + "ugroups": _build_named_group, + "hgroups": _build_named_group, + "hostConfigs": _build_host_config, + "hproviders": _build_scaler_config, + "limits": _build_limit_config, +} + + +def map_list_section( + spec: SectionSpec, items: list[dict] | None, target: "capnp._DynamicStructBuilder" +) -> None: + """Populate ``target.`` as a capnp List sized to + ``len(items)``, preserving YAML source order exactly (Requirement + 9.1). + + A ``None`` or empty ``items`` is a no-op -- the field is never + ``init()``-ed with a zero length (see module docstring, Determinism). + + Preconditions: ``items`` already passed ``xcecommon.validation``. + Postconditions: only ``target.`` is touched; no other + field on ``target`` is modified. + """ + if not items: + return + builder_fn = _LIST_ITEM_BUILDERS[spec["capnp_field"]] + entries = target.init(spec["capnp_field"], len(items)) + for entry, item in zip(entries, items): + builder_fn(item, entry) + + +def map_scalar_section( + spec: SectionSpec, values: dict | None, target: "capnp._DynamicStructBuilder" # noqa: ARG001 +) -> None: + """Set each of the 5 ``xce.yaml``-sourced scalar ``ClusterConfig`` + fields directly from ``values`` (:data:`_XCE_SCALAR_FIELDS`). + + ``spec`` is accepted for interface symmetry with :func:`map_list_section` + but is otherwise unused here -- a flat section maps several + YAML-key-to-capnp-field pairs at once, not the single pair the list + shape's ``spec`` carries. + + A key absent from ``values`` leaves the corresponding capnp field + untouched (capnp default) rather than raising -- whether an absent + required key (e.g. ``cluster_name``) is acceptable is an + orchestration-level decision for a future ``xceadmin.build``, not + this function's concern. + + A ``None`` or empty ``values`` is a no-op. + """ + if not values: + return + for yaml_key, capnp_field in _XCE_SCALAR_FIELDS: + if yaml_key in values: + setattr(target, capnp_field, values[yaml_key]) + + +def map_all_sections(sections: dict[str, dict], target: "capnp._DynamicStructBuilder") -> None: + """Apply every :data:`SECTION_MAP` entry against ``sections`` onto + ``target`` (a ``ClusterConfig`` builder) -- the driver that makes + ``SECTION_MAP`` load-bearing. + + ``sections`` maps schema_name (a YAML file's base name with ``.yaml`` + stripped, e.g. ``"hosts"``) -> the parsed dict for that file, the same + shape ``xcecommon.validation.validate_yaml`` returns and + ``check_reserved_names``/``check_duplicates``/``check_undefined_refs`` + already consume. A schema_name absent from ``sections`` (missing YAML + file) is treated as an empty section -- identical output to an empty + section that was present (Requirement 1.5's "missing file" rule, + enforced structurally here rather than requiring the caller to + special-case it). + """ + for spec in SECTION_MAP: + schema_name = spec["yaml_file"].removesuffix(".yaml") + data = sections.get(schema_name) or {} + if spec["kind"] == "scalar": + map_scalar_section(spec, data, target) + else: + map_list_section(spec, data.get(spec["yaml_key"]), target) diff --git a/src/xceadmin/tests/test_mapping.py b/src/xceadmin/tests/test_mapping.py new file mode 100644 index 00000000..7e999334 --- /dev/null +++ b/src/xceadmin/tests/test_mapping.py @@ -0,0 +1,585 @@ +"""Unit tests for ``xcecommon.mapping`` (design.md, ``xcecommon.mapping``; +Requirements 1.1, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 5.3, 9.1, 9.2, 9.3). + +Pure unit tests against in-memory dicts (no file I/O) per task 2.3/5.1 -- +except where a test deliberately loads the real ``src/configs/*.yaml`` +fixtures to prove the mapper handles this repo's actual example configs, +not just synthetic minimal dicts (task 2.3's TDD approach explicitly asks +for both). + +These tests build real capnp ``ClusterConfig`` messages via +``xcecommon.schema.load_schema()`` -- the same loaded module ``mapping.py`` +targets -- so any drift between this module's hardcoded field names and the +real ``controller.capnp`` schema surfaces as an ``AttributeError``/ +``KjException`` here, not silently. +""" + +from __future__ import annotations + +import inspect +from pathlib import Path + +import pytest +import yaml + +from xcecommon import mapping +from xcecommon.schema import load_schema + +REPO_CONFIGS_DIR = Path(__file__).resolve().parents[2] / "configs" + + +@pytest.fixture +def cluster_config(): + """A fresh ``ClusterConfig`` builder -- never reused across tests, so + every test starts from an all-default message. + """ + return mapping.new_cluster_config_builder() + + +def _load_real_yaml(name: str) -> dict: + path = REPO_CONFIGS_DIR / name + with open(path, encoding="utf-8") as fh: + return yaml.safe_load(fh) + + +# --------------------------------------------------------------------------- +# SECTION_MAP shape +# --------------------------------------------------------------------------- + + +def test_section_map_has_eight_entries_one_per_yaml_capnp_field_pair(): + assert len(mapping.SECTION_MAP) == 8 + + +def test_section_map_hosts_yaml_feeds_two_distinct_capnp_fields(): + hosts_entries = [s for s in mapping.SECTION_MAP if s["yaml_file"] == "hosts.yaml"] + assert len(hosts_entries) == 2 + capnp_fields = {s["capnp_field"] for s in hosts_entries} + assert capnp_fields == {"hgroups", "hostConfigs"} + yaml_keys = {s["yaml_key"] for s in hosts_entries} + assert yaml_keys == {"hostgroups", "hosts"} + + +def test_section_map_covers_all_13_clusterconfig_fields_or_documents_exclusion(): + # 5 scalar (xce.yaml) + 7 list capnp_fields covers 12 of the 13 + # ClusterConfig fields; configGeneration (@12) is the one deliberately + # excluded field (Requirement 1.9), documented in the module docstring, + # not present in SECTION_MAP at all. + list_fields = {s["capnp_field"] for s in mapping.SECTION_MAP if s["kind"] == "list"} + assert list_fields == { + "resources", + "queues", + "ugroups", + "hgroups", + "hostConfigs", + "hproviders", + "limits", + } + scalar_entries = [s for s in mapping.SECTION_MAP if s["kind"] == "scalar"] + assert len(scalar_entries) == 1 + assert {f[1] for f in mapping._XCE_SCALAR_FIELDS} == { + "clusterName", + "heartbeatInterval", + "inactiveHeartbeatCount", + "fetchJobInterval", + "netioTimeout", + } + + +# --------------------------------------------------------------------------- +# map_scalar_section (xce.yaml) +# --------------------------------------------------------------------------- + + +def test_map_scalar_section_sets_all_5_fields(cluster_config): + spec = mapping.SECTION_MAP[0] + values = { + "cluster_name": "eda1", + "heartbeat_interval": 30, + "inactive_heartbeat_count": 3, + "fetch_job_interval": 10, + "netio_timeout": 30, + } + mapping.map_scalar_section(spec, values, cluster_config) + + assert cluster_config.clusterName == "eda1" + assert cluster_config.heartbeatInterval == 30 + assert cluster_config.inactiveHeartbeatCount == 3 + assert cluster_config.fetchJobInterval == 10 + assert cluster_config.netioTimeout == 30 + + +def test_map_scalar_section_leaves_missing_optional_keys_at_default(cluster_config): + spec = mapping.SECTION_MAP[0] + values = {"cluster_name": "eda1", "heartbeat_interval": 30, "inactive_heartbeat_count": 3} + mapping.map_scalar_section(spec, values, cluster_config) + + assert cluster_config.clusterName == "eda1" + assert cluster_config.fetchJobInterval == 0 + assert cluster_config.netioTimeout == 0 + + +@pytest.mark.parametrize("empty", [None, {}]) +def test_map_scalar_section_noop_on_empty_values(cluster_config, empty): + spec = mapping.SECTION_MAP[0] + mapping.map_scalar_section(spec, empty, cluster_config) + + assert cluster_config.clusterName == "" + assert cluster_config.heartbeatInterval == 0 + + +# --------------------------------------------------------------------------- +# map_list_section: resources.yaml -> resources +# --------------------------------------------------------------------------- + + +def _spec_for(capnp_field: str) -> mapping.SectionSpec: + for spec in mapping.SECTION_MAP: + if spec["capnp_field"] == capnp_field: + return spec + raise AssertionError(f"no SECTION_MAP entry for {capnp_field!r}") + + +def test_map_list_section_resources_populates_mapped_fields_only(cluster_config): + items = [ + {"name": "slots", "type": "numeric", "order": "decrease", "consumable": True, + "description": "job slot count", "interval": 0}, + {"name": "mem", "type": "numeric", "interval": 60, "description": "memory"}, + ] + mapping.map_list_section(_spec_for("resources"), items, cluster_config) + + assert len(cluster_config.resources) == 2 + r0, r1 = cluster_config.resources + assert r0.name == "slots" + assert r0.desc == "job slot count" + assert r0.interval == 0 + assert r1.name == "mem" + assert r1.interval == 60 + + # Deliberately-unmapped fields (see module docstring) stay at default. + assert r0.idx == 0 + assert r0.valtype == 0 + assert r0.ordtype == 0 + assert r0.flags == 0 + assert r0.lastReport == 0 + + +def test_map_list_section_noop_on_empty_or_missing_items(cluster_config): + mapping.map_list_section(_spec_for("resources"), [], cluster_config) + mapping.map_list_section(_spec_for("resources"), None, cluster_config) + assert len(cluster_config.resources) == 0 + + +# --------------------------------------------------------------------------- +# map_list_section: users.yaml/hosts.yaml -> ugroups/hgroups (named groups) +# --------------------------------------------------------------------------- + + +def test_map_list_section_ugroups_maps_groupname_members_exclude(cluster_config): + items = [{"groupname": "ug1", "members": ["user1", "user2"], "exclude_members": ["user3"]}] + mapping.map_list_section(_spec_for("ugroups"), items, cluster_config) + + g = cluster_config.ugroups[0] + assert g.name == "ug1" + assert list(g.members) == ["user1", "user2"] + assert list(g.exmembers) == ["user3"] + + +def test_map_list_section_hgroups_omits_exclude_members_when_absent(cluster_config): + items = [{"groupname": "hg1", "members": ["all"]}] + mapping.map_list_section(_spec_for("hgroups"), items, cluster_config) + + g = cluster_config.hgroups[0] + assert g.name == "hg1" + assert list(g.members) == ["all"] + assert list(g.exmembers) == [] + + +# --------------------------------------------------------------------------- +# map_list_section: hosts.yaml -> hostConfigs (the "feeds two fields" file) +# --------------------------------------------------------------------------- + + +def test_hosts_yaml_feeds_hgroups_and_hostconfigs_independently(cluster_config): + """The one file that feeds two separate output fields (design.md / + task 5.1) -- build both from the same parsed hosts.yaml dict and + assert each populated correctly and independently. + """ + hosts_data = { + "hosts": [{"hostname": "h1", "type": "linux", "max_slot": 4}], + "hostgroups": [{"groupname": "hg1", "members": ["all"]}], + } + mapping.map_list_section(_spec_for("hgroups"), hosts_data.get("hostgroups"), cluster_config) + mapping.map_list_section(_spec_for("hostConfigs"), hosts_data.get("hosts"), cluster_config) + + assert cluster_config.hgroups[0].name == "hg1" + assert cluster_config.hostConfigs[0].name == "h1" + assert cluster_config.hostConfigs[0].mxj == 4 + # Independence: touching hostConfigs must not disturb hgroups. + assert len(cluster_config.hgroups) == 1 + + +def test_map_list_section_host_config_full_field_mapping(cluster_config): + items = [ + { + "hostname": "default", + "type": "x86_64-linux-gnu", + "model": "x86_64-linux-gnu", + "max_slot": -1, + "description": "default host", + "resource": [{"name": "mg"}, {"name": "maxswap", "value": 10000}], + "sched_stop": [ + {"resname": "r1m", "sched": 3.5, "stop": 4.5}, + {"resname": "r15s", "sched": 4.0, "stop": 10.0}, + ], + "dispatch_window": [ + {"start_time": "9:00", "end_time": "18:00"}, + {"start_time": "21:00", "end_time": "7:00"}, + ], + }, + ] + mapping.map_list_section(_spec_for("hostConfigs"), items, cluster_config) + + h = cluster_config.hostConfigs[0] + assert h.name == "default" + assert h.type == "x86_64-linux-gnu" + assert h.model == "x86_64-linux-gnu" + # max_slot == -1 (auto-detect sentinel) cannot be represented by + # UInt32 mxj -- deliberately left at default (see module docstring). + assert h.mxj == 0 + + assert [r.name for r in h.resources] == ["mg", "maxswap"] + assert h.resources[0].value == "" + assert h.resources[1].value == "10000" + + assert [t.name for t in h.thresholds] == ["r1m", "r15s"] + assert h.thresholds[0].sched == 3.5 + assert h.thresholds[0].stop == 4.5 + + segs = h.dispatchwindows.segs + assert len(segs) == 2 + assert segs[0].open.hour == 9 + assert segs[0].open.min == 0 + assert segs[0].close.hour == 18 + assert segs[1].open.hour == 21 + assert segs[1].close.hour == 7 + + +def test_map_list_section_host_config_sets_mxj_when_max_slot_nonnegative(cluster_config): + items = [{"hostname": "worker2", "max_slot": 256}] + mapping.map_list_section(_spec_for("hostConfigs"), items, cluster_config) + + assert cluster_config.hostConfigs[0].mxj == 256 + + +def test_time_window_day_prefixed_spec_parses_day_hour_minute(cluster_config): + items = [ + { + "name": "night", + "priority": 30, + "run_window": [ + {"start_time": "5:19:00", "end_time": "1:8:30"}, + ], + } + ] + mapping.map_list_section(_spec_for("queues"), items, cluster_config) + + seg = cluster_config.queues[0].runWindow.segs[0] + assert seg.open.day == 5 + assert seg.open.hour == 19 + assert seg.open.min == 0 + assert seg.close.day == 1 + assert seg.close.hour == 8 + assert seg.close.min == 30 + + +# --------------------------------------------------------------------------- +# map_list_section: queues.yaml -> queues +# --------------------------------------------------------------------------- + + +def test_map_list_section_queue_full_field_mapping(cluster_config): + items = [ + { + "name": "night", + "description": "night queue", + "priority": 30, + "res_req": "mem>1000 rusage[mem=1000]", + "gpu_req": "num=1:mode=shared", + "sched_stop": [{"resname": "tmp", "sched": 2000, "stop": 1000}], + "dispatch_window": [{"start_time": "19:00", "end_time": "8:30"}], + "run_window": [{"start_time": "20:00", "end_time": "8:30"}], + "pre_exec": "/path/to/preexec.sh", + "post_exec": "/path/to/postexec.sh", + "users": {"members": ["ug1", "ug2"], "exclude_members": ["user3"]}, + "hosts": {"members": ["hgroup1", "hgroup2"], "exclude_members": ["host3"]}, + }, + ] + mapping.map_list_section(_spec_for("queues"), items, cluster_config) + + q = cluster_config.queues[0] + assert q.name == "night" + assert q.priority == 30 + assert q.resreq == "mem>1000 rusage[mem=1000]" + assert q.preExec == "/path/to/preexec.sh" + assert q.postExec == "/path/to/postexec.sh" + assert list(q.users.members) == ["ug1", "ug2"] + assert list(q.users.exmembers) == ["user3"] + assert q.users.name == "" # embedded Membership never carries a name + assert list(q.hosts.members) == ["hgroup1", "hgroup2"] + assert q.thresholds[0].name == "tmp" + assert q.thresholds[0].sched == 2000.0 + assert q.dispatchWindow.segs[0].open.hour == 19 + assert q.runWindow.segs[0].open.hour == 20 + + # Deliberately-unmapped fields: description, gpu_req, attrs, status, + # acceptIntervals, schedDelay have no destination or no data source. + assert q.attrs == 0 + assert q.status == 0 + assert q.acceptIntervals == 0 + assert q.schedDelay == 0 + + +def test_map_list_section_queue_minimal_required_fields_only(cluster_config): + items = [{"name": "xce", "priority": 30}] + mapping.map_list_section(_spec_for("queues"), items, cluster_config) + + q = cluster_config.queues[0] + assert q.name == "xce" + assert q.priority == 30 + assert q.resreq == "" + assert len(q.thresholds) == 0 + assert len(q.dispatchWindow.segs) == 0 + + +# --------------------------------------------------------------------------- +# map_list_section: scaler.yaml -> hproviders +# --------------------------------------------------------------------------- + + +def test_map_list_section_scaler_config_full_field_mapping(cluster_config): + items = [ + { + "name": "aws-asg", + "type": "script", + "conf_path": "/etc/xce/scalers/aws-asg.conf", + "script_path": "/opt/xce/scalers/aws_asg.sh", + "script_option": "--region us-east-1", + }, + {"name": "k8s-pool", "type": "script", "script_path": "/opt/xce/scalers/k8s_pool.sh"}, + ] + mapping.map_list_section(_spec_for("hproviders"), items, cluster_config) + + p0, p1 = cluster_config.hproviders + assert p0.name == "aws-asg" + assert p0.type == "script" + assert p0.confPath == "/etc/xce/scalers/aws-asg.conf" + assert p0.scriptPath == "/opt/xce/scalers/aws_asg.sh" + assert p0.scriptOption == "--region us-east-1" + + assert p1.name == "k8s-pool" + assert p1.confPath == "" + assert p1.scriptOption == "" + + +# --------------------------------------------------------------------------- +# map_list_section: limits.yaml -> limits +# --------------------------------------------------------------------------- + + +def test_map_list_section_limit_config_full_field_mapping(cluster_config): + items = [ + { + "name": "ug2_slots_per_user", + "users": {"members": ["ug2"], "exclude_members": ["user3"]}, + "resname": "slots", + "value": 4, + "per_user": True, + }, + { + "name": "night_mem_pct", + "queues": {"members": ["night"]}, + "resname": "mem", + "value": 50, + "percent": True, + }, + { + "name": "ug1_normal_slots", + "users": {"members": ["ug1"]}, + "queues": {"members": ["normal"]}, + "resname": "slots", + "value": 8, + }, + ] + mapping.map_list_section(_spec_for("limits"), items, cluster_config) + + l0, l1, l2 = cluster_config.limits + assert l0.name == "ug2_slots_per_user" + assert list(l0.users.members) == ["ug2"] + assert list(l0.users.exmembers) == ["user3"] + assert l0.val.resname == "slots" + assert l0.val.value == 4.0 + assert l0.val.percentage is False + assert l0.perUser is True + assert l0.perQueue is False + + assert l1.val.resname == "mem" + assert l1.val.percentage is True + + assert list(l2.users.members) == ["ug1"] + assert list(l2.queues.members) == ["normal"] + + +# --------------------------------------------------------------------------- +# Order preservation (Requirement 9.1) +# --------------------------------------------------------------------------- + + +def test_list_order_preserved_not_sorted(cluster_config): + items = [{"name": n} for n in ["zebra", "apple", "mango"]] + mapping.map_list_section(_spec_for("resources"), items, cluster_config) + + assert [r.name for r in cluster_config.resources] == ["zebra", "apple", "mango"] + + +def test_group_membership_order_preserved(cluster_config): + items = [{"groupname": "hg1", "members": ["h3", "h1", "h2"]}] + mapping.map_list_section(_spec_for("hgroups"), items, cluster_config) + + assert list(cluster_config.hgroups[0].members) == ["h3", "h1", "h2"] + + +# --------------------------------------------------------------------------- +# Determinism (Requirements 9.1, 9.2, 9.3) +# --------------------------------------------------------------------------- + + +def _build_full_sections() -> dict[str, dict]: + return { + "xce": { + "cluster_name": "eda1", + "heartbeat_interval": 30, + "inactive_heartbeat_count": 3, + "fetch_job_interval": 10, + "netio_timeout": 30, + }, + "resources": {"resources": [{"name": "slots", "type": "numeric"}, {"name": "mem", "type": "numeric"}]}, + "queues": {"queues": [{"name": "xce", "priority": 30, "users": {"members": ["all"]}}]}, + "users": {"usergroups": [{"groupname": "ug1", "members": ["user1", "user2"]}]}, + "hosts": { + "hosts": [{"hostname": "h1", "max_slot": 4}], + "hostgroups": [{"groupname": "hg1", "members": ["all"]}], + }, + "scaler": {"scalers": [{"name": "aws-asg", "type": "script", "script_path": "/x.sh"}]}, + "limits": {"limits": [{"name": "l1", "resname": "slots", "value": 10}]}, + } + + +def test_map_all_sections_building_same_input_twice_produces_identical_bytes(): + sections = _build_full_sections() + + target1 = mapping.new_cluster_config_builder() + mapping.map_all_sections(sections, target1) + bytes1 = target1.to_bytes() + + target2 = mapping.new_cluster_config_builder() + mapping.map_all_sections(sections, target2) + bytes2 = target2.to_bytes() + + assert bytes1 == bytes2 + + +def test_map_all_sections_missing_file_matches_present_empty_section(): + """A schema_name absent from `sections` (missing YAML file) must + produce byte-identical output to that section being present but + empty -- Requirement 1.5's rule, enforced structurally. + """ + sections_missing = { + "xce": {"cluster_name": "c", "heartbeat_interval": 1, "inactive_heartbeat_count": 1} + } + sections_present_empty = dict(sections_missing, resources={"resources": []}) + + target_missing = mapping.new_cluster_config_builder() + mapping.map_all_sections(sections_missing, target_missing) + + target_empty = mapping.new_cluster_config_builder() + mapping.map_all_sections(sections_present_empty, target_empty) + + assert target_missing.to_bytes() == target_empty.to_bytes() + + +def test_map_all_sections_populates_all_capnp_fields_and_skips_configgeneration(): + sections = _build_full_sections() + target = mapping.new_cluster_config_builder() + mapping.map_all_sections(sections, target) + + assert target.clusterName == "eda1" + assert len(target.resources) == 2 + assert len(target.queues) == 1 + assert len(target.ugroups) == 1 + assert len(target.hgroups) == 1 + assert len(target.hostConfigs) == 1 + assert len(target.hproviders) == 1 + assert len(target.limits) == 1 + # configGeneration (@12): controller-assigned, never touched (1.9). + assert target.configGeneration == 0 + + +def test_no_timestamp_hostname_or_username_source_anywhere_in_module(): + """Static regression guard for Requirement 9.2: no wall-clock, + hostname, or username source is ever imported or called by this + module. + """ + source = inspect.getsource(mapping) + forbidden_tokens = [ + "datetime", + "time.time", + "gethostname", + "getlogin", + "getuser", + "uuid.uuid", + "os.environ", + ] + for token in forbidden_tokens: + assert token not in source, f"forbidden non-deterministic source {token!r} found in mapping.py" + + +# --------------------------------------------------------------------------- +# Real src/configs/*.yaml fixtures +# --------------------------------------------------------------------------- + + +@pytest.mark.skipif(not REPO_CONFIGS_DIR.is_dir(), reason="src/configs/ fixtures not present") +def test_maps_real_repo_config_fixtures_without_error(): + sections = { + "xce": _load_real_yaml("xce.yaml"), + "resources": _load_real_yaml("resources.yaml"), + "queues": _load_real_yaml("queues.yaml"), + "users": _load_real_yaml("users.yaml"), + "hosts": _load_real_yaml("hosts.yaml"), + "scaler": _load_real_yaml("scaler.yaml"), + "limits": _load_real_yaml("limits.yaml"), + } + target = mapping.new_cluster_config_builder() + mapping.map_all_sections(sections, target) + + assert target.clusterName == "eda1" + assert len(target.resources) == len(sections["resources"]["resources"]) + assert len(target.queues) == len(sections["queues"]["queues"]) + assert len(target.hostConfigs) == len(sections["hosts"]["hosts"]) + assert len(target.hgroups) == len(sections["hosts"]["hostgroups"]) + assert len(target.limits) == len(sections["limits"]["limits"]) + + # Round-trip through to_bytes()/from_bytes() using the real loaded + # schema module -- catches drift between mapping.py's hardcoded field + # names and the real controller.capnp schema. + mod = load_schema() + target_bytes = target.to_bytes() + with mod.ClusterConfig.from_bytes(target_bytes) as reader: + assert reader.clusterName == "eda1" + assert reader.queues[3].name == "short" + + # Determinism against the real fixtures too. + target2 = mapping.new_cluster_config_builder() + mapping.map_all_sections(sections, target2) + assert target_bytes == target2.to_bytes()