feat(admin-config-tool): task 2.3 - xcecommon.mapping
Map validated YAML section dicts into ClusterConfig capnp field values via a declarative SECTION_MAP driving two generic mapper shapes (list/scalar). Preserves YAML source order, sources no timestamp/hostname/username, and is proven byte-deterministic against both synthetic and real src/configs/*.yaml fixtures.
This commit is contained in:
parent
dbf5c6c54d
commit
666a838fb2
3 changed files with 1043 additions and 1 deletions
|
|
@ -39,7 +39,7 @@
|
|||
- _Requirements: 5.1, 5.5, 7.4_
|
||||
- _Boundary: xcecommon.schema_
|
||||
|
||||
- [ ] 2.3 (P) Map validated YAML into cluster configuration fields
|
||||
- [x] 2.3 (P) Map validated YAML into cluster configuration fields
|
||||
- Every field the cluster configuration binary format defines is either populated from validated YAML or deliberately left untouched, with no field silently dropped
|
||||
- List-based sections preserve the exact order they appeared in the YAML
|
||||
- No timestamp, hostname, or username value ever enters the mapped output
|
||||
|
|
@ -148,3 +148,4 @@
|
|||
- **1.3**: `src/xceadmin/src/xcecommon/schema_files/` (synced capnp + JSON schema copies) is a build artifact, regenerated every `xce-build.sh xceadmin` run — never hand-edit it, never commit it (`src/xceadmin/.gitignore` excludes it).
|
||||
- **2.1**: `check_undefined_refs` deliberately does not parse resource names out of `res_req`/`gpu_req` free-form LSF-style expression strings (e.g. `"mem>1000 rusage[mem=1000]"`) — design.md's `xcecommon.validation` component scopes this check to queue→host-group and limit→group/`resname` only. `limits.yaml`'s `projects` field and `scaler.yaml` name collisions are likewise out of scope — no `projects.yaml` catalog exists among the 7 config types, and Requirement 4.3 only enumerates queue/host/resource/group/limit names. `src/configs/hosts.yaml` has 2 pre-existing undefined refs (resource `mg`, host `h3`) that now surface as warnings — expected, not a regression; fixing the example fixture is outside this task's boundary.
|
||||
- **2.2**: The real `controller.capnp` names its reply struct `RequestReply`, not `Reply` as design.md's postcondition states — an inexact design pseudo-type, corrected empirically (verified directly against `src/schema/controller.capnp`, not assumed). pycapnp 2.0.0/capnproto 1.2.0 abort the whole process (uncaught C++ exception, not a Python exception) on a missing cross-file import or malformed `.capnp` content passed to `capnp.load()` — a bare `try/except` cannot catch this. `xcecommon.schema.load_schema()` works around it by pre-validating in an isolated subprocess before ever calling `capnp.load()` in the caller's own process, converting any abort into a clean `SchemaNotFoundError`. Any future code that calls `capnp.load()` directly (bypassing this loader) reintroduces the crash risk — always go through `xcecommon.schema.load_schema()`.
|
||||
- **2.3**: `map_all_sections()` (a `SECTION_MAP`-driven orchestrator, not explicitly named in design.md's Service Interface but consistent with its Contracts) is the single call `xceadmin.build` (task 3.1) needs — it treats a missing YAML file identically to a present-but-empty section (byte-for-byte), satisfying Requirement 1.5 structurally. `target.init(field, 0)` and never touching an empty/absent list field produce *different* serialized bytes for the same logical "nothing here" value (verified empirically) — the mapper always picks "never touch" for byte-determinism (9.1). Calling `.to_bytes()` twice on the same capnp builder triggers a spurious "message already written" `UserWarning` even when no field was mutated between calls — capture the bytes once and reuse them rather than calling `.to_bytes()` again.
|
||||
|
|
|
|||
456
src/xceadmin/src/xcecommon/mapping.py
Normal file
456
src/xceadmin/src/xcecommon/mapping.py
Normal file
|
|
@ -0,0 +1,456 @@
|
|||
"""YAML-section-dict -> ``ClusterConfig`` capnp field mapping for
|
||||
xcecommon/xceadmin (design.md, ``xcecommon.mapping``; Requirements 1.1,
|
||||
1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 5.3, 9.1, 9.2, 9.3).
|
||||
|
||||
This module's only outbound dependency is ``xcecommon.schema`` (design.md,
|
||||
Dependency Direction) -- it never imports ``xcecommon.validation``: every
|
||||
function here assumes its ``dict`` inputs already passed
|
||||
``xcecommon.validation``'s JSON Schema + business-rule checks. Re-validating
|
||||
here would duplicate that boundary, not strengthen it.
|
||||
|
||||
One declarative table, :data:`SECTION_MAP`, drives all 7 YAML files (8
|
||||
entries -- ``hosts.yaml`` is the one file that feeds two separate capnp
|
||||
fields, ``hgroups`` and ``hostConfigs``; see the Data Models table in
|
||||
design.md). Two mapper shapes cover every section:
|
||||
|
||||
- :func:`map_scalar_section` -- the single flat ``xce.yaml`` section, 5
|
||||
top-level ``ClusterConfig`` scalar fields.
|
||||
- :func:`map_list_section` -- the 7 remaining sections, each a YAML list
|
||||
mapped to a capnp ``List(Struct)`` field, preserving YAML source order
|
||||
exactly (never sorted -- Requirement 9.1's byte-determinism depends on
|
||||
this).
|
||||
|
||||
:func:`map_all_sections` is the SECTION_MAP-driven orchestrator: it is what
|
||||
actually makes the declarative table load-bearing rather than an inert
|
||||
constant, and is the single call a future ``xceadmin.build`` (task 3.1)
|
||||
needs to populate an entire ``ClusterConfig`` builder from a
|
||||
``schema_name -> parsed_yaml_dict`` mapping (the same shape
|
||||
``xcecommon.validation.validate_yaml`` returns, keyed the same way
|
||||
``check_reserved_names``/``check_duplicates``/``check_undefined_refs``
|
||||
already key their ``sections`` argument).
|
||||
|
||||
Determinism (Requirements 9.1, 9.2, 9.3)
|
||||
-----------------------------------------
|
||||
- No wall-clock, hostname, or username value is read anywhere in this
|
||||
module -- it imports nothing from the standard library's date/time,
|
||||
networking-identity, or process-identity facilities, and
|
||||
``test_mapping.py`` asserts this statically as a regression guard.
|
||||
- List order is preserved via ``zip(capnp_list, python_list)`` in
|
||||
iteration order -- never ``sorted()``, never a ``set``.
|
||||
- A YAML list that is absent (key missing) and one that is present but
|
||||
empty (``[]``) are treated identically: neither ever calls
|
||||
``target.init(field, 0)``. This matters for byte-determinism --
|
||||
``target.init(field, 0)`` and never touching ``field`` at all produce
|
||||
*different* serialized bytes for the same logical "nothing here" value
|
||||
(verified empirically against the real compiled schema), so this module
|
||||
picks exactly one encoding (untouched) and uses it unconditionally.
|
||||
- Building a fresh ``ClusterConfig`` message from the same input twice
|
||||
(two independent ``new_message()`` calls run through the same mapping
|
||||
calls) produces byte-identical ``to_bytes()`` output -- verified in
|
||||
``test_mapping.py``.
|
||||
|
||||
Deliberately Unmapped Fields
|
||||
-----------------------------
|
||||
Every nested capnp field below is either populated from validated YAML or
|
||||
intentionally left at its capnp default, for a specific, documented
|
||||
reason -- never a silent drop.
|
||||
|
||||
``ClusterConfig``
|
||||
``configGeneration`` (@12) -- never touched (Requirement 1.9); the
|
||||
primary controller is the sole authority for assigning it.
|
||||
``candidates``/``port``/hash-table-size fields/``xstrings`` do not
|
||||
exist in the schema at all post ``clusterconfig-schema-split`` --
|
||||
there is nothing to exclude.
|
||||
|
||||
``Resource`` (``resources.yaml``)
|
||||
``idx``, ``valtype``, ``ordtype``, ``flags``, ``lastReport`` are left
|
||||
unset. The YAML ``type``/``order``/``consumable``/``release`` keys
|
||||
would need to encode into ``valtype``/``ordtype``/``flags`` as
|
||||
specific integers, but neither ``controller.capnp`` nor design.md
|
||||
documents that encoding within this task's boundary -- inventing one
|
||||
risks silently miswriting a value the controller then misinterprets,
|
||||
strictly worse than shipping it unset. ``idx`` has no YAML source at
|
||||
all (controller/runtime-assigned). ``lastReport`` is a runtime
|
||||
timestamp field (epoch seconds of last report) -- excluded doubly by
|
||||
Requirement 9.2's "no timestamp" rule. ``resourcemap`` has no
|
||||
destination: the capnp fields it would map to (``distributions``,
|
||||
``hostBased``) are commented out of the ``Resource`` struct
|
||||
(``controller.capnp:208-209``).
|
||||
|
||||
``QueueConf`` (``queues.yaml``)
|
||||
``attrs`` (raw ``UInt32`` bitfield), ``status``, ``acceptIntervals``,
|
||||
``schedDelay`` are left unset -- no YAML source, and ``attrs``' bit
|
||||
assignments are undocumented within this task's boundary (guessing
|
||||
bit positions for ``backfill``/``exclusive``/``rerunable``/
|
||||
``preemption`` would be worse than leaving the field unset).
|
||||
``gpu_req`` is left unmapped: ``QueueConf`` has exactly one
|
||||
``resreq`` Text field, already claimed by ``res_req``; concatenating
|
||||
the two LSF-style expression strings would be a lossy, invented
|
||||
format this task has no authority to define. ``description``,
|
||||
``nice``, ``fairshare``, ``cpulimit``, ``memlimit``, ``runlimit``,
|
||||
``runtimeest``, ``reservation``, ``preemption``, ``job_starter`` have
|
||||
no corresponding ``QueueConf`` field at all.
|
||||
|
||||
``HostConfig`` (``hosts.yaml`` hosts)
|
||||
``description`` has no corresponding field. ``mxj`` is set only when
|
||||
YAML ``max_slot >= 0``; the schema's documented ``-1`` ("auto-detect")
|
||||
sentinel is left unmapped because ``mxj`` is ``UInt32`` and cannot
|
||||
represent a negative value (verified: assigning ``-1`` raises
|
||||
``capnp.KjException`` rather than wrapping) -- shipping ``mxj`` unset
|
||||
(default ``0``) preserves "not sourced" instead of crashing the build
|
||||
or silently wrapping to ``4294967295``.
|
||||
|
||||
``GroupConfig`` used as an embedded ``Membership`` (``QueueConf.users``/
|
||||
``.hosts``, ``LimitConfig.users``/``.queues``/``.projects``/``.hosts``)
|
||||
``name`` is left unset -- the YAML ``Membership`` shape
|
||||
(``{members, exclude_members}``) never carries a name for these
|
||||
inline group references; only the top-level ``ugroups``/``hgroups``
|
||||
entries have a ``groupname`` to source ``name`` from.
|
||||
|
||||
``TimeWindowTime`` (nested inside ``TimeWindow``/``dispatchWindow``/
|
||||
``runWindow``/``dispatchwindows``)
|
||||
``day`` is left at its capnp default (``0``) whenever the YAML time
|
||||
spec omits a day prefix (bare ``HH`` or ``HH:MM`` form) -- capnp has
|
||||
no "unset" sentinel distinct from ``0`` for a ``UInt32``, so an
|
||||
omitted day and an explicit ``day: 0`` (Sunday) are indistinguishable
|
||||
in the output. Documented, not a silent drop.
|
||||
|
||||
``TimeWindowSeg.timezone``
|
||||
Left unset -- no timezone data exists anywhere in the YAML
|
||||
``TimeWindow`` shape (``start_time``/``end_time`` only).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Literal, TypedDict
|
||||
|
||||
from xcecommon.schema import load_schema
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import capnp # noqa: F401 (type-hint only, see Service Interface below)
|
||||
|
||||
|
||||
class SectionSpec(TypedDict):
|
||||
"""One entry of :data:`SECTION_MAP`.
|
||||
|
||||
For the single ``kind="scalar"`` entry (``xce.yaml``), ``yaml_key``
|
||||
and ``capnp_field`` are unused placeholders -- :func:`map_scalar_section`
|
||||
maps all 5 target fields directly via :data:`_XCE_SCALAR_FIELDS`
|
||||
instead of a single ``yaml_key``/``capnp_field`` pair, since a flat
|
||||
section maps many-keys-to-many-fields, not the list shape's
|
||||
one-key-to-one-field.
|
||||
"""
|
||||
|
||||
yaml_file: str
|
||||
yaml_key: str
|
||||
capnp_field: str
|
||||
kind: Literal["list", "scalar"]
|
||||
|
||||
|
||||
# One entry per YAML-file/capnp-field pair (8 entries -- hosts.yaml feeds
|
||||
# two capnp fields, see module docstring). This is the only place file/key
|
||||
# names live; every mapper function below is generic over it.
|
||||
SECTION_MAP: list[SectionSpec] = [
|
||||
{"yaml_file": "xce.yaml", "yaml_key": "", "capnp_field": "", "kind": "scalar"},
|
||||
{"yaml_file": "resources.yaml", "yaml_key": "resources", "capnp_field": "resources", "kind": "list"},
|
||||
{"yaml_file": "queues.yaml", "yaml_key": "queues", "capnp_field": "queues", "kind": "list"},
|
||||
{"yaml_file": "users.yaml", "yaml_key": "usergroups", "capnp_field": "ugroups", "kind": "list"},
|
||||
{"yaml_file": "hosts.yaml", "yaml_key": "hostgroups", "capnp_field": "hgroups", "kind": "list"},
|
||||
{"yaml_file": "hosts.yaml", "yaml_key": "hosts", "capnp_field": "hostConfigs", "kind": "list"},
|
||||
{"yaml_file": "scaler.yaml", "yaml_key": "scalers", "capnp_field": "hproviders", "kind": "list"},
|
||||
{"yaml_file": "limits.yaml", "yaml_key": "limits", "capnp_field": "limits", "kind": "list"},
|
||||
]
|
||||
|
||||
# xce.yaml top-level key -> ClusterConfig scalar field name, in @N order.
|
||||
_XCE_SCALAR_FIELDS: tuple[tuple[str, str], ...] = (
|
||||
("cluster_name", "clusterName"),
|
||||
("heartbeat_interval", "heartbeatInterval"),
|
||||
("inactive_heartbeat_count", "inactiveHeartbeatCount"),
|
||||
("fetch_job_interval", "fetchJobInterval"),
|
||||
("netio_timeout", "netioTimeout"),
|
||||
)
|
||||
|
||||
|
||||
def new_cluster_config_builder() -> "capnp._DynamicStructBuilder":
|
||||
"""Convenience constructor: ``load_schema().ClusterConfig.new_message()``.
|
||||
|
||||
Callers that already hold a builder (or the loaded schema module, for
|
||||
building other message types alongside it) may call
|
||||
:func:`map_all_sections`/:func:`map_list_section`/
|
||||
:func:`map_scalar_section` directly against their own builder instead
|
||||
-- this is a convenience, not a required entry point.
|
||||
"""
|
||||
return load_schema().ClusterConfig.new_message()
|
||||
|
||||
|
||||
def _parse_time_spec(spec: str) -> tuple[int | None, int, int]:
|
||||
"""Parse the ``[day:]HH[:MM]`` time format used by
|
||||
``TimeWindow``/``dispatch_window``/``run_window`` YAML entries
|
||||
(schema pattern documented in ``hosts.schema.json``/
|
||||
``queues.schema.json``).
|
||||
|
||||
Returns ``(day, hour, minute)`` -- ``day`` is ``None`` when the input
|
||||
has no day prefix (2-part ``HH:MM`` or bare ``HH`` form); callers
|
||||
leave the capnp ``day`` field untouched in that case (see module
|
||||
docstring, ``TimeWindowTime``).
|
||||
"""
|
||||
parts = spec.split(":")
|
||||
if len(parts) == 3:
|
||||
return int(parts[0]), int(parts[1]), int(parts[2])
|
||||
if len(parts) == 2:
|
||||
return None, int(parts[0]), int(parts[1])
|
||||
return None, int(parts[0]), 0
|
||||
|
||||
|
||||
def _build_time_window(entries: list[dict] | None, target: "capnp._DynamicStructBuilder") -> None:
|
||||
"""Populate a ``TimeWindow`` struct's ``segs`` list from a YAML list of
|
||||
``{start_time, end_time}`` dicts (``dispatch_window``/``run_window``
|
||||
shape), preserving order (Requirement 9.1).
|
||||
"""
|
||||
if not entries:
|
||||
return
|
||||
segs = target.init("segs", len(entries))
|
||||
for seg, entry in zip(segs, entries):
|
||||
start_day, start_hour, start_min = _parse_time_spec(entry["start_time"])
|
||||
end_day, end_hour, end_min = _parse_time_spec(entry["end_time"])
|
||||
if start_day is not None:
|
||||
seg.open.day = start_day
|
||||
seg.open.hour = start_hour
|
||||
seg.open.min = start_min
|
||||
if end_day is not None:
|
||||
seg.close.day = end_day
|
||||
seg.close.hour = end_hour
|
||||
seg.close.min = end_min
|
||||
|
||||
|
||||
def _build_threshold_list(
|
||||
entries: list[dict] | None, target: "capnp._DynamicStructBuilder", field_name: str
|
||||
) -> None:
|
||||
"""Populate a ``List(Threshold)`` field from a YAML ``sched_stop`` list
|
||||
(``{resname, sched, stop}`` -> ``Threshold{name, sched, stop}``),
|
||||
preserving order.
|
||||
"""
|
||||
if not entries:
|
||||
return
|
||||
thresholds = target.init(field_name, len(entries))
|
||||
for threshold, entry in zip(thresholds, entries):
|
||||
threshold.name = entry["resname"]
|
||||
if "sched" in entry:
|
||||
threshold.sched = entry["sched"]
|
||||
if "stop" in entry:
|
||||
threshold.stop = entry["stop"]
|
||||
|
||||
|
||||
def _build_config_var_list(entries: list[dict] | None, target: "capnp._DynamicStructBuilder") -> None:
|
||||
"""Populate ``HostConfig.resources`` (``List(ConfigVar)``) from a
|
||||
YAML host's ``resource`` list (``{name, value}`` -> ``ConfigVar{name,
|
||||
value}``, ``value`` stringified since ``ConfigVar.value`` is Text
|
||||
while YAML allows a number or a string), preserving order.
|
||||
"""
|
||||
if not entries:
|
||||
return
|
||||
config_vars = target.init("resources", len(entries))
|
||||
for config_var, entry in zip(config_vars, entries):
|
||||
config_var.name = entry["name"]
|
||||
if "value" in entry:
|
||||
config_var.value = str(entry["value"])
|
||||
|
||||
|
||||
def _build_membership_group(membership: dict | None, target: "capnp._DynamicStructBuilder") -> None:
|
||||
"""Populate an embedded ``GroupConfig`` (``members``/``exmembers``
|
||||
only -- ``name`` deliberately left unset, see module docstring) from
|
||||
a YAML ``Membership`` dict (``{members, exclude_members}``).
|
||||
"""
|
||||
if not membership:
|
||||
return
|
||||
members = membership.get("members") or []
|
||||
if members:
|
||||
capnp_members = target.init("members", len(members))
|
||||
for i, name in enumerate(members):
|
||||
capnp_members[i] = name
|
||||
excludes = membership.get("exclude_members") or []
|
||||
if excludes:
|
||||
capnp_excludes = target.init("exmembers", len(excludes))
|
||||
for i, name in enumerate(excludes):
|
||||
capnp_excludes[i] = name
|
||||
|
||||
|
||||
def _build_named_group(item: dict, target: "capnp._DynamicStructBuilder") -> None:
|
||||
"""Populate a top-level ``GroupConfig`` (``ugroups``/``hgroups`` list
|
||||
item) from a YAML ``HostGroup``/``UserGroup`` dict
|
||||
(``{groupname, members, exclude_members}``).
|
||||
"""
|
||||
target.name = item["groupname"]
|
||||
_build_membership_group(item, target)
|
||||
|
||||
|
||||
def _build_resource(item: dict, target: "capnp._DynamicStructBuilder") -> None:
|
||||
"""Populate one ``Resource`` list item from a ``resources.yaml`` entry.
|
||||
See module docstring, ``Resource``, for what is intentionally left
|
||||
unmapped and why.
|
||||
"""
|
||||
target.name = item["name"]
|
||||
if "description" in item:
|
||||
target.desc = item["description"]
|
||||
if "interval" in item:
|
||||
target.interval = item["interval"]
|
||||
|
||||
|
||||
def _build_queue(item: dict, target: "capnp._DynamicStructBuilder") -> None:
|
||||
"""Populate one ``QueueConf`` list item from a ``queues.yaml`` entry.
|
||||
See module docstring, ``QueueConf``, for what is intentionally left
|
||||
unmapped and why.
|
||||
"""
|
||||
target.name = item["name"]
|
||||
if "priority" in item:
|
||||
target.priority = item["priority"]
|
||||
if "res_req" in item:
|
||||
target.resreq = item["res_req"]
|
||||
if "pre_exec" in item:
|
||||
target.preExec = item["pre_exec"]
|
||||
if "post_exec" in item:
|
||||
target.postExec = item["post_exec"]
|
||||
_build_membership_group(item.get("users"), target.users)
|
||||
_build_membership_group(item.get("hosts"), target.hosts)
|
||||
_build_threshold_list(item.get("sched_stop"), target, "thresholds")
|
||||
_build_time_window(item.get("dispatch_window"), target.dispatchWindow)
|
||||
_build_time_window(item.get("run_window"), target.runWindow)
|
||||
|
||||
|
||||
def _build_host_config(item: dict, target: "capnp._DynamicStructBuilder") -> None:
|
||||
"""Populate one ``HostConfig`` list item from a ``hosts.yaml`` ``hosts``
|
||||
entry. See module docstring, ``HostConfig``, for what is intentionally
|
||||
left unmapped and why.
|
||||
"""
|
||||
target.name = item["hostname"]
|
||||
if "model" in item:
|
||||
target.model = item["model"]
|
||||
if "type" in item:
|
||||
target.type = item["type"]
|
||||
max_slot = item.get("max_slot")
|
||||
if max_slot is not None and max_slot >= 0:
|
||||
target.mxj = max_slot
|
||||
_build_config_var_list(item.get("resource"), target)
|
||||
_build_threshold_list(item.get("sched_stop"), target, "thresholds")
|
||||
_build_time_window(item.get("dispatch_window"), target.dispatchwindows)
|
||||
|
||||
|
||||
def _build_scaler_config(item: dict, target: "capnp._DynamicStructBuilder") -> None:
|
||||
"""Populate one ``ScalerConfig`` list item from a ``scaler.yaml``
|
||||
``scalers`` entry -- a clean 1:1 field mapping, nothing left unmapped.
|
||||
"""
|
||||
target.name = item["name"]
|
||||
target.type = item["type"]
|
||||
if "conf_path" in item:
|
||||
target.confPath = item["conf_path"]
|
||||
target.scriptPath = item["script_path"]
|
||||
if "script_option" in item:
|
||||
target.scriptOption = item["script_option"]
|
||||
|
||||
|
||||
def _build_limit_config(item: dict, target: "capnp._DynamicStructBuilder") -> None:
|
||||
"""Populate one ``LimitConfig`` list item from a ``limits.yaml``
|
||||
``limits`` entry -- a clean field mapping, nothing left unmapped.
|
||||
"""
|
||||
target.name = item["name"]
|
||||
_build_membership_group(item.get("users"), target.users)
|
||||
_build_membership_group(item.get("queues"), target.queues)
|
||||
_build_membership_group(item.get("projects"), target.projects)
|
||||
_build_membership_group(item.get("hosts"), target.hosts)
|
||||
target.val.resname = item["resname"]
|
||||
target.val.value = item["value"]
|
||||
if "percent" in item:
|
||||
target.val.percentage = item["percent"]
|
||||
if "per_user" in item:
|
||||
target.perUser = item["per_user"]
|
||||
if "per_queue" in item:
|
||||
target.perQueue = item["per_queue"]
|
||||
if "per_project" in item:
|
||||
target.perProject = item["per_project"]
|
||||
if "per_host" in item:
|
||||
target.perHost = item["per_host"]
|
||||
|
||||
|
||||
# capnp_field -> (YAML item dict, capnp list-item builder) -> None.
|
||||
# Every list-shaped SECTION_MAP entry's capnp_field is a key here.
|
||||
_LIST_ITEM_BUILDERS = {
|
||||
"resources": _build_resource,
|
||||
"queues": _build_queue,
|
||||
"ugroups": _build_named_group,
|
||||
"hgroups": _build_named_group,
|
||||
"hostConfigs": _build_host_config,
|
||||
"hproviders": _build_scaler_config,
|
||||
"limits": _build_limit_config,
|
||||
}
|
||||
|
||||
|
||||
def map_list_section(
|
||||
spec: SectionSpec, items: list[dict] | None, target: "capnp._DynamicStructBuilder"
|
||||
) -> None:
|
||||
"""Populate ``target.<spec['capnp_field']>`` as a capnp List sized to
|
||||
``len(items)``, preserving YAML source order exactly (Requirement
|
||||
9.1).
|
||||
|
||||
A ``None`` or empty ``items`` is a no-op -- the field is never
|
||||
``init()``-ed with a zero length (see module docstring, Determinism).
|
||||
|
||||
Preconditions: ``items`` already passed ``xcecommon.validation``.
|
||||
Postconditions: only ``target.<capnp_field>`` is touched; no other
|
||||
field on ``target`` is modified.
|
||||
"""
|
||||
if not items:
|
||||
return
|
||||
builder_fn = _LIST_ITEM_BUILDERS[spec["capnp_field"]]
|
||||
entries = target.init(spec["capnp_field"], len(items))
|
||||
for entry, item in zip(entries, items):
|
||||
builder_fn(item, entry)
|
||||
|
||||
|
||||
def map_scalar_section(
|
||||
spec: SectionSpec, values: dict | None, target: "capnp._DynamicStructBuilder" # noqa: ARG001
|
||||
) -> None:
|
||||
"""Set each of the 5 ``xce.yaml``-sourced scalar ``ClusterConfig``
|
||||
fields directly from ``values`` (:data:`_XCE_SCALAR_FIELDS`).
|
||||
|
||||
``spec`` is accepted for interface symmetry with :func:`map_list_section`
|
||||
but is otherwise unused here -- a flat section maps several
|
||||
YAML-key-to-capnp-field pairs at once, not the single pair the list
|
||||
shape's ``spec`` carries.
|
||||
|
||||
A key absent from ``values`` leaves the corresponding capnp field
|
||||
untouched (capnp default) rather than raising -- whether an absent
|
||||
required key (e.g. ``cluster_name``) is acceptable is an
|
||||
orchestration-level decision for a future ``xceadmin.build``, not
|
||||
this function's concern.
|
||||
|
||||
A ``None`` or empty ``values`` is a no-op.
|
||||
"""
|
||||
if not values:
|
||||
return
|
||||
for yaml_key, capnp_field in _XCE_SCALAR_FIELDS:
|
||||
if yaml_key in values:
|
||||
setattr(target, capnp_field, values[yaml_key])
|
||||
|
||||
|
||||
def map_all_sections(sections: dict[str, dict], target: "capnp._DynamicStructBuilder") -> None:
|
||||
"""Apply every :data:`SECTION_MAP` entry against ``sections`` onto
|
||||
``target`` (a ``ClusterConfig`` builder) -- the driver that makes
|
||||
``SECTION_MAP`` load-bearing.
|
||||
|
||||
``sections`` maps schema_name (a YAML file's base name with ``.yaml``
|
||||
stripped, e.g. ``"hosts"``) -> the parsed dict for that file, the same
|
||||
shape ``xcecommon.validation.validate_yaml`` returns and
|
||||
``check_reserved_names``/``check_duplicates``/``check_undefined_refs``
|
||||
already consume. A schema_name absent from ``sections`` (missing YAML
|
||||
file) is treated as an empty section -- identical output to an empty
|
||||
section that was present (Requirement 1.5's "missing file" rule,
|
||||
enforced structurally here rather than requiring the caller to
|
||||
special-case it).
|
||||
"""
|
||||
for spec in SECTION_MAP:
|
||||
schema_name = spec["yaml_file"].removesuffix(".yaml")
|
||||
data = sections.get(schema_name) or {}
|
||||
if spec["kind"] == "scalar":
|
||||
map_scalar_section(spec, data, target)
|
||||
else:
|
||||
map_list_section(spec, data.get(spec["yaml_key"]), target)
|
||||
585
src/xceadmin/tests/test_mapping.py
Normal file
585
src/xceadmin/tests/test_mapping.py
Normal file
|
|
@ -0,0 +1,585 @@
|
|||
"""Unit tests for ``xcecommon.mapping`` (design.md, ``xcecommon.mapping``;
|
||||
Requirements 1.1, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 5.3, 9.1, 9.2, 9.3).
|
||||
|
||||
Pure unit tests against in-memory dicts (no file I/O) per task 2.3/5.1 --
|
||||
except where a test deliberately loads the real ``src/configs/*.yaml``
|
||||
fixtures to prove the mapper handles this repo's actual example configs,
|
||||
not just synthetic minimal dicts (task 2.3's TDD approach explicitly asks
|
||||
for both).
|
||||
|
||||
These tests build real capnp ``ClusterConfig`` messages via
|
||||
``xcecommon.schema.load_schema()`` -- the same loaded module ``mapping.py``
|
||||
targets -- so any drift between this module's hardcoded field names and the
|
||||
real ``controller.capnp`` schema surfaces as an ``AttributeError``/
|
||||
``KjException`` here, not silently.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import inspect
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from xcecommon import mapping
|
||||
from xcecommon.schema import load_schema
|
||||
|
||||
REPO_CONFIGS_DIR = Path(__file__).resolve().parents[2] / "configs"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def cluster_config():
|
||||
"""A fresh ``ClusterConfig`` builder -- never reused across tests, so
|
||||
every test starts from an all-default message.
|
||||
"""
|
||||
return mapping.new_cluster_config_builder()
|
||||
|
||||
|
||||
def _load_real_yaml(name: str) -> dict:
|
||||
path = REPO_CONFIGS_DIR / name
|
||||
with open(path, encoding="utf-8") as fh:
|
||||
return yaml.safe_load(fh)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SECTION_MAP shape
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_section_map_has_eight_entries_one_per_yaml_capnp_field_pair():
|
||||
assert len(mapping.SECTION_MAP) == 8
|
||||
|
||||
|
||||
def test_section_map_hosts_yaml_feeds_two_distinct_capnp_fields():
|
||||
hosts_entries = [s for s in mapping.SECTION_MAP if s["yaml_file"] == "hosts.yaml"]
|
||||
assert len(hosts_entries) == 2
|
||||
capnp_fields = {s["capnp_field"] for s in hosts_entries}
|
||||
assert capnp_fields == {"hgroups", "hostConfigs"}
|
||||
yaml_keys = {s["yaml_key"] for s in hosts_entries}
|
||||
assert yaml_keys == {"hostgroups", "hosts"}
|
||||
|
||||
|
||||
def test_section_map_covers_all_13_clusterconfig_fields_or_documents_exclusion():
|
||||
# 5 scalar (xce.yaml) + 7 list capnp_fields covers 12 of the 13
|
||||
# ClusterConfig fields; configGeneration (@12) is the one deliberately
|
||||
# excluded field (Requirement 1.9), documented in the module docstring,
|
||||
# not present in SECTION_MAP at all.
|
||||
list_fields = {s["capnp_field"] for s in mapping.SECTION_MAP if s["kind"] == "list"}
|
||||
assert list_fields == {
|
||||
"resources",
|
||||
"queues",
|
||||
"ugroups",
|
||||
"hgroups",
|
||||
"hostConfigs",
|
||||
"hproviders",
|
||||
"limits",
|
||||
}
|
||||
scalar_entries = [s for s in mapping.SECTION_MAP if s["kind"] == "scalar"]
|
||||
assert len(scalar_entries) == 1
|
||||
assert {f[1] for f in mapping._XCE_SCALAR_FIELDS} == {
|
||||
"clusterName",
|
||||
"heartbeatInterval",
|
||||
"inactiveHeartbeatCount",
|
||||
"fetchJobInterval",
|
||||
"netioTimeout",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# map_scalar_section (xce.yaml)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_map_scalar_section_sets_all_5_fields(cluster_config):
|
||||
spec = mapping.SECTION_MAP[0]
|
||||
values = {
|
||||
"cluster_name": "eda1",
|
||||
"heartbeat_interval": 30,
|
||||
"inactive_heartbeat_count": 3,
|
||||
"fetch_job_interval": 10,
|
||||
"netio_timeout": 30,
|
||||
}
|
||||
mapping.map_scalar_section(spec, values, cluster_config)
|
||||
|
||||
assert cluster_config.clusterName == "eda1"
|
||||
assert cluster_config.heartbeatInterval == 30
|
||||
assert cluster_config.inactiveHeartbeatCount == 3
|
||||
assert cluster_config.fetchJobInterval == 10
|
||||
assert cluster_config.netioTimeout == 30
|
||||
|
||||
|
||||
def test_map_scalar_section_leaves_missing_optional_keys_at_default(cluster_config):
|
||||
spec = mapping.SECTION_MAP[0]
|
||||
values = {"cluster_name": "eda1", "heartbeat_interval": 30, "inactive_heartbeat_count": 3}
|
||||
mapping.map_scalar_section(spec, values, cluster_config)
|
||||
|
||||
assert cluster_config.clusterName == "eda1"
|
||||
assert cluster_config.fetchJobInterval == 0
|
||||
assert cluster_config.netioTimeout == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("empty", [None, {}])
|
||||
def test_map_scalar_section_noop_on_empty_values(cluster_config, empty):
|
||||
spec = mapping.SECTION_MAP[0]
|
||||
mapping.map_scalar_section(spec, empty, cluster_config)
|
||||
|
||||
assert cluster_config.clusterName == ""
|
||||
assert cluster_config.heartbeatInterval == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# map_list_section: resources.yaml -> resources
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _spec_for(capnp_field: str) -> mapping.SectionSpec:
|
||||
for spec in mapping.SECTION_MAP:
|
||||
if spec["capnp_field"] == capnp_field:
|
||||
return spec
|
||||
raise AssertionError(f"no SECTION_MAP entry for {capnp_field!r}")
|
||||
|
||||
|
||||
def test_map_list_section_resources_populates_mapped_fields_only(cluster_config):
|
||||
items = [
|
||||
{"name": "slots", "type": "numeric", "order": "decrease", "consumable": True,
|
||||
"description": "job slot count", "interval": 0},
|
||||
{"name": "mem", "type": "numeric", "interval": 60, "description": "memory"},
|
||||
]
|
||||
mapping.map_list_section(_spec_for("resources"), items, cluster_config)
|
||||
|
||||
assert len(cluster_config.resources) == 2
|
||||
r0, r1 = cluster_config.resources
|
||||
assert r0.name == "slots"
|
||||
assert r0.desc == "job slot count"
|
||||
assert r0.interval == 0
|
||||
assert r1.name == "mem"
|
||||
assert r1.interval == 60
|
||||
|
||||
# Deliberately-unmapped fields (see module docstring) stay at default.
|
||||
assert r0.idx == 0
|
||||
assert r0.valtype == 0
|
||||
assert r0.ordtype == 0
|
||||
assert r0.flags == 0
|
||||
assert r0.lastReport == 0
|
||||
|
||||
|
||||
def test_map_list_section_noop_on_empty_or_missing_items(cluster_config):
|
||||
mapping.map_list_section(_spec_for("resources"), [], cluster_config)
|
||||
mapping.map_list_section(_spec_for("resources"), None, cluster_config)
|
||||
assert len(cluster_config.resources) == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# map_list_section: users.yaml/hosts.yaml -> ugroups/hgroups (named groups)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_map_list_section_ugroups_maps_groupname_members_exclude(cluster_config):
|
||||
items = [{"groupname": "ug1", "members": ["user1", "user2"], "exclude_members": ["user3"]}]
|
||||
mapping.map_list_section(_spec_for("ugroups"), items, cluster_config)
|
||||
|
||||
g = cluster_config.ugroups[0]
|
||||
assert g.name == "ug1"
|
||||
assert list(g.members) == ["user1", "user2"]
|
||||
assert list(g.exmembers) == ["user3"]
|
||||
|
||||
|
||||
def test_map_list_section_hgroups_omits_exclude_members_when_absent(cluster_config):
|
||||
items = [{"groupname": "hg1", "members": ["all"]}]
|
||||
mapping.map_list_section(_spec_for("hgroups"), items, cluster_config)
|
||||
|
||||
g = cluster_config.hgroups[0]
|
||||
assert g.name == "hg1"
|
||||
assert list(g.members) == ["all"]
|
||||
assert list(g.exmembers) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# map_list_section: hosts.yaml -> hostConfigs (the "feeds two fields" file)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_hosts_yaml_feeds_hgroups_and_hostconfigs_independently(cluster_config):
|
||||
"""The one file that feeds two separate output fields (design.md /
|
||||
task 5.1) -- build both from the same parsed hosts.yaml dict and
|
||||
assert each populated correctly and independently.
|
||||
"""
|
||||
hosts_data = {
|
||||
"hosts": [{"hostname": "h1", "type": "linux", "max_slot": 4}],
|
||||
"hostgroups": [{"groupname": "hg1", "members": ["all"]}],
|
||||
}
|
||||
mapping.map_list_section(_spec_for("hgroups"), hosts_data.get("hostgroups"), cluster_config)
|
||||
mapping.map_list_section(_spec_for("hostConfigs"), hosts_data.get("hosts"), cluster_config)
|
||||
|
||||
assert cluster_config.hgroups[0].name == "hg1"
|
||||
assert cluster_config.hostConfigs[0].name == "h1"
|
||||
assert cluster_config.hostConfigs[0].mxj == 4
|
||||
# Independence: touching hostConfigs must not disturb hgroups.
|
||||
assert len(cluster_config.hgroups) == 1
|
||||
|
||||
|
||||
def test_map_list_section_host_config_full_field_mapping(cluster_config):
|
||||
items = [
|
||||
{
|
||||
"hostname": "default",
|
||||
"type": "x86_64-linux-gnu",
|
||||
"model": "x86_64-linux-gnu",
|
||||
"max_slot": -1,
|
||||
"description": "default host",
|
||||
"resource": [{"name": "mg"}, {"name": "maxswap", "value": 10000}],
|
||||
"sched_stop": [
|
||||
{"resname": "r1m", "sched": 3.5, "stop": 4.5},
|
||||
{"resname": "r15s", "sched": 4.0, "stop": 10.0},
|
||||
],
|
||||
"dispatch_window": [
|
||||
{"start_time": "9:00", "end_time": "18:00"},
|
||||
{"start_time": "21:00", "end_time": "7:00"},
|
||||
],
|
||||
},
|
||||
]
|
||||
mapping.map_list_section(_spec_for("hostConfigs"), items, cluster_config)
|
||||
|
||||
h = cluster_config.hostConfigs[0]
|
||||
assert h.name == "default"
|
||||
assert h.type == "x86_64-linux-gnu"
|
||||
assert h.model == "x86_64-linux-gnu"
|
||||
# max_slot == -1 (auto-detect sentinel) cannot be represented by
|
||||
# UInt32 mxj -- deliberately left at default (see module docstring).
|
||||
assert h.mxj == 0
|
||||
|
||||
assert [r.name for r in h.resources] == ["mg", "maxswap"]
|
||||
assert h.resources[0].value == ""
|
||||
assert h.resources[1].value == "10000"
|
||||
|
||||
assert [t.name for t in h.thresholds] == ["r1m", "r15s"]
|
||||
assert h.thresholds[0].sched == 3.5
|
||||
assert h.thresholds[0].stop == 4.5
|
||||
|
||||
segs = h.dispatchwindows.segs
|
||||
assert len(segs) == 2
|
||||
assert segs[0].open.hour == 9
|
||||
assert segs[0].open.min == 0
|
||||
assert segs[0].close.hour == 18
|
||||
assert segs[1].open.hour == 21
|
||||
assert segs[1].close.hour == 7
|
||||
|
||||
|
||||
def test_map_list_section_host_config_sets_mxj_when_max_slot_nonnegative(cluster_config):
|
||||
items = [{"hostname": "worker2", "max_slot": 256}]
|
||||
mapping.map_list_section(_spec_for("hostConfigs"), items, cluster_config)
|
||||
|
||||
assert cluster_config.hostConfigs[0].mxj == 256
|
||||
|
||||
|
||||
def test_time_window_day_prefixed_spec_parses_day_hour_minute(cluster_config):
|
||||
items = [
|
||||
{
|
||||
"name": "night",
|
||||
"priority": 30,
|
||||
"run_window": [
|
||||
{"start_time": "5:19:00", "end_time": "1:8:30"},
|
||||
],
|
||||
}
|
||||
]
|
||||
mapping.map_list_section(_spec_for("queues"), items, cluster_config)
|
||||
|
||||
seg = cluster_config.queues[0].runWindow.segs[0]
|
||||
assert seg.open.day == 5
|
||||
assert seg.open.hour == 19
|
||||
assert seg.open.min == 0
|
||||
assert seg.close.day == 1
|
||||
assert seg.close.hour == 8
|
||||
assert seg.close.min == 30
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# map_list_section: queues.yaml -> queues
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_map_list_section_queue_full_field_mapping(cluster_config):
|
||||
items = [
|
||||
{
|
||||
"name": "night",
|
||||
"description": "night queue",
|
||||
"priority": 30,
|
||||
"res_req": "mem>1000 rusage[mem=1000]",
|
||||
"gpu_req": "num=1:mode=shared",
|
||||
"sched_stop": [{"resname": "tmp", "sched": 2000, "stop": 1000}],
|
||||
"dispatch_window": [{"start_time": "19:00", "end_time": "8:30"}],
|
||||
"run_window": [{"start_time": "20:00", "end_time": "8:30"}],
|
||||
"pre_exec": "/path/to/preexec.sh",
|
||||
"post_exec": "/path/to/postexec.sh",
|
||||
"users": {"members": ["ug1", "ug2"], "exclude_members": ["user3"]},
|
||||
"hosts": {"members": ["hgroup1", "hgroup2"], "exclude_members": ["host3"]},
|
||||
},
|
||||
]
|
||||
mapping.map_list_section(_spec_for("queues"), items, cluster_config)
|
||||
|
||||
q = cluster_config.queues[0]
|
||||
assert q.name == "night"
|
||||
assert q.priority == 30
|
||||
assert q.resreq == "mem>1000 rusage[mem=1000]"
|
||||
assert q.preExec == "/path/to/preexec.sh"
|
||||
assert q.postExec == "/path/to/postexec.sh"
|
||||
assert list(q.users.members) == ["ug1", "ug2"]
|
||||
assert list(q.users.exmembers) == ["user3"]
|
||||
assert q.users.name == "" # embedded Membership never carries a name
|
||||
assert list(q.hosts.members) == ["hgroup1", "hgroup2"]
|
||||
assert q.thresholds[0].name == "tmp"
|
||||
assert q.thresholds[0].sched == 2000.0
|
||||
assert q.dispatchWindow.segs[0].open.hour == 19
|
||||
assert q.runWindow.segs[0].open.hour == 20
|
||||
|
||||
# Deliberately-unmapped fields: description, gpu_req, attrs, status,
|
||||
# acceptIntervals, schedDelay have no destination or no data source.
|
||||
assert q.attrs == 0
|
||||
assert q.status == 0
|
||||
assert q.acceptIntervals == 0
|
||||
assert q.schedDelay == 0
|
||||
|
||||
|
||||
def test_map_list_section_queue_minimal_required_fields_only(cluster_config):
|
||||
items = [{"name": "xce", "priority": 30}]
|
||||
mapping.map_list_section(_spec_for("queues"), items, cluster_config)
|
||||
|
||||
q = cluster_config.queues[0]
|
||||
assert q.name == "xce"
|
||||
assert q.priority == 30
|
||||
assert q.resreq == ""
|
||||
assert len(q.thresholds) == 0
|
||||
assert len(q.dispatchWindow.segs) == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# map_list_section: scaler.yaml -> hproviders
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_map_list_section_scaler_config_full_field_mapping(cluster_config):
|
||||
items = [
|
||||
{
|
||||
"name": "aws-asg",
|
||||
"type": "script",
|
||||
"conf_path": "/etc/xce/scalers/aws-asg.conf",
|
||||
"script_path": "/opt/xce/scalers/aws_asg.sh",
|
||||
"script_option": "--region us-east-1",
|
||||
},
|
||||
{"name": "k8s-pool", "type": "script", "script_path": "/opt/xce/scalers/k8s_pool.sh"},
|
||||
]
|
||||
mapping.map_list_section(_spec_for("hproviders"), items, cluster_config)
|
||||
|
||||
p0, p1 = cluster_config.hproviders
|
||||
assert p0.name == "aws-asg"
|
||||
assert p0.type == "script"
|
||||
assert p0.confPath == "/etc/xce/scalers/aws-asg.conf"
|
||||
assert p0.scriptPath == "/opt/xce/scalers/aws_asg.sh"
|
||||
assert p0.scriptOption == "--region us-east-1"
|
||||
|
||||
assert p1.name == "k8s-pool"
|
||||
assert p1.confPath == ""
|
||||
assert p1.scriptOption == ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# map_list_section: limits.yaml -> limits
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_map_list_section_limit_config_full_field_mapping(cluster_config):
|
||||
items = [
|
||||
{
|
||||
"name": "ug2_slots_per_user",
|
||||
"users": {"members": ["ug2"], "exclude_members": ["user3"]},
|
||||
"resname": "slots",
|
||||
"value": 4,
|
||||
"per_user": True,
|
||||
},
|
||||
{
|
||||
"name": "night_mem_pct",
|
||||
"queues": {"members": ["night"]},
|
||||
"resname": "mem",
|
||||
"value": 50,
|
||||
"percent": True,
|
||||
},
|
||||
{
|
||||
"name": "ug1_normal_slots",
|
||||
"users": {"members": ["ug1"]},
|
||||
"queues": {"members": ["normal"]},
|
||||
"resname": "slots",
|
||||
"value": 8,
|
||||
},
|
||||
]
|
||||
mapping.map_list_section(_spec_for("limits"), items, cluster_config)
|
||||
|
||||
l0, l1, l2 = cluster_config.limits
|
||||
assert l0.name == "ug2_slots_per_user"
|
||||
assert list(l0.users.members) == ["ug2"]
|
||||
assert list(l0.users.exmembers) == ["user3"]
|
||||
assert l0.val.resname == "slots"
|
||||
assert l0.val.value == 4.0
|
||||
assert l0.val.percentage is False
|
||||
assert l0.perUser is True
|
||||
assert l0.perQueue is False
|
||||
|
||||
assert l1.val.resname == "mem"
|
||||
assert l1.val.percentage is True
|
||||
|
||||
assert list(l2.users.members) == ["ug1"]
|
||||
assert list(l2.queues.members) == ["normal"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Order preservation (Requirement 9.1)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_list_order_preserved_not_sorted(cluster_config):
|
||||
items = [{"name": n} for n in ["zebra", "apple", "mango"]]
|
||||
mapping.map_list_section(_spec_for("resources"), items, cluster_config)
|
||||
|
||||
assert [r.name for r in cluster_config.resources] == ["zebra", "apple", "mango"]
|
||||
|
||||
|
||||
def test_group_membership_order_preserved(cluster_config):
|
||||
items = [{"groupname": "hg1", "members": ["h3", "h1", "h2"]}]
|
||||
mapping.map_list_section(_spec_for("hgroups"), items, cluster_config)
|
||||
|
||||
assert list(cluster_config.hgroups[0].members) == ["h3", "h1", "h2"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Determinism (Requirements 9.1, 9.2, 9.3)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_full_sections() -> dict[str, dict]:
|
||||
return {
|
||||
"xce": {
|
||||
"cluster_name": "eda1",
|
||||
"heartbeat_interval": 30,
|
||||
"inactive_heartbeat_count": 3,
|
||||
"fetch_job_interval": 10,
|
||||
"netio_timeout": 30,
|
||||
},
|
||||
"resources": {"resources": [{"name": "slots", "type": "numeric"}, {"name": "mem", "type": "numeric"}]},
|
||||
"queues": {"queues": [{"name": "xce", "priority": 30, "users": {"members": ["all"]}}]},
|
||||
"users": {"usergroups": [{"groupname": "ug1", "members": ["user1", "user2"]}]},
|
||||
"hosts": {
|
||||
"hosts": [{"hostname": "h1", "max_slot": 4}],
|
||||
"hostgroups": [{"groupname": "hg1", "members": ["all"]}],
|
||||
},
|
||||
"scaler": {"scalers": [{"name": "aws-asg", "type": "script", "script_path": "/x.sh"}]},
|
||||
"limits": {"limits": [{"name": "l1", "resname": "slots", "value": 10}]},
|
||||
}
|
||||
|
||||
|
||||
def test_map_all_sections_building_same_input_twice_produces_identical_bytes():
|
||||
sections = _build_full_sections()
|
||||
|
||||
target1 = mapping.new_cluster_config_builder()
|
||||
mapping.map_all_sections(sections, target1)
|
||||
bytes1 = target1.to_bytes()
|
||||
|
||||
target2 = mapping.new_cluster_config_builder()
|
||||
mapping.map_all_sections(sections, target2)
|
||||
bytes2 = target2.to_bytes()
|
||||
|
||||
assert bytes1 == bytes2
|
||||
|
||||
|
||||
def test_map_all_sections_missing_file_matches_present_empty_section():
|
||||
"""A schema_name absent from `sections` (missing YAML file) must
|
||||
produce byte-identical output to that section being present but
|
||||
empty -- Requirement 1.5's rule, enforced structurally.
|
||||
"""
|
||||
sections_missing = {
|
||||
"xce": {"cluster_name": "c", "heartbeat_interval": 1, "inactive_heartbeat_count": 1}
|
||||
}
|
||||
sections_present_empty = dict(sections_missing, resources={"resources": []})
|
||||
|
||||
target_missing = mapping.new_cluster_config_builder()
|
||||
mapping.map_all_sections(sections_missing, target_missing)
|
||||
|
||||
target_empty = mapping.new_cluster_config_builder()
|
||||
mapping.map_all_sections(sections_present_empty, target_empty)
|
||||
|
||||
assert target_missing.to_bytes() == target_empty.to_bytes()
|
||||
|
||||
|
||||
def test_map_all_sections_populates_all_capnp_fields_and_skips_configgeneration():
|
||||
sections = _build_full_sections()
|
||||
target = mapping.new_cluster_config_builder()
|
||||
mapping.map_all_sections(sections, target)
|
||||
|
||||
assert target.clusterName == "eda1"
|
||||
assert len(target.resources) == 2
|
||||
assert len(target.queues) == 1
|
||||
assert len(target.ugroups) == 1
|
||||
assert len(target.hgroups) == 1
|
||||
assert len(target.hostConfigs) == 1
|
||||
assert len(target.hproviders) == 1
|
||||
assert len(target.limits) == 1
|
||||
# configGeneration (@12): controller-assigned, never touched (1.9).
|
||||
assert target.configGeneration == 0
|
||||
|
||||
|
||||
def test_no_timestamp_hostname_or_username_source_anywhere_in_module():
|
||||
"""Static regression guard for Requirement 9.2: no wall-clock,
|
||||
hostname, or username source is ever imported or called by this
|
||||
module.
|
||||
"""
|
||||
source = inspect.getsource(mapping)
|
||||
forbidden_tokens = [
|
||||
"datetime",
|
||||
"time.time",
|
||||
"gethostname",
|
||||
"getlogin",
|
||||
"getuser",
|
||||
"uuid.uuid",
|
||||
"os.environ",
|
||||
]
|
||||
for token in forbidden_tokens:
|
||||
assert token not in source, f"forbidden non-deterministic source {token!r} found in mapping.py"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Real src/configs/*.yaml fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.skipif(not REPO_CONFIGS_DIR.is_dir(), reason="src/configs/ fixtures not present")
|
||||
def test_maps_real_repo_config_fixtures_without_error():
|
||||
sections = {
|
||||
"xce": _load_real_yaml("xce.yaml"),
|
||||
"resources": _load_real_yaml("resources.yaml"),
|
||||
"queues": _load_real_yaml("queues.yaml"),
|
||||
"users": _load_real_yaml("users.yaml"),
|
||||
"hosts": _load_real_yaml("hosts.yaml"),
|
||||
"scaler": _load_real_yaml("scaler.yaml"),
|
||||
"limits": _load_real_yaml("limits.yaml"),
|
||||
}
|
||||
target = mapping.new_cluster_config_builder()
|
||||
mapping.map_all_sections(sections, target)
|
||||
|
||||
assert target.clusterName == "eda1"
|
||||
assert len(target.resources) == len(sections["resources"]["resources"])
|
||||
assert len(target.queues) == len(sections["queues"]["queues"])
|
||||
assert len(target.hostConfigs) == len(sections["hosts"]["hosts"])
|
||||
assert len(target.hgroups) == len(sections["hosts"]["hostgroups"])
|
||||
assert len(target.limits) == len(sections["limits"]["limits"])
|
||||
|
||||
# Round-trip through to_bytes()/from_bytes() using the real loaded
|
||||
# schema module -- catches drift between mapping.py's hardcoded field
|
||||
# names and the real controller.capnp schema.
|
||||
mod = load_schema()
|
||||
target_bytes = target.to_bytes()
|
||||
with mod.ClusterConfig.from_bytes(target_bytes) as reader:
|
||||
assert reader.clusterName == "eda1"
|
||||
assert reader.queues[3].name == "short"
|
||||
|
||||
# Determinism against the real fixtures too.
|
||||
target2 = mapping.new_cluster_config_builder()
|
||||
mapping.map_all_sections(sections, target2)
|
||||
assert target_bytes == target2.to_bytes()
|
||||
Loading…
Add table
Add a link
Reference in a new issue