Dev 9850 remove list boilerplate

changelog.md

@@ -12,6 +12,7 @@
   * `EnergySource.phases`
   * `PowerElectronicsConnection.phases`
 * ContactDetails are now Identifiable and no longer have default id generation. The constructor now requires a string. The `id` field is deprecated, to be replaced with mrid.
+* Removing items with backref from a containing object's list automatically clears the backref field on the item - it is not accessible after removal.
 
 ### New Features
 * Added a `lint` tox environment that runs `ruff check .` to enforce code quality standards. The test environments now depend on lint passing first, so CI will fail if any new lint violations are introduced.
@@ -23,6 +24,8 @@
 * Added `skip_install = true` to the lint environment to avoid installing package dependencies that aren't needed for linting.
 * Added E402 noqa comments for intentional mid-file imports used to avoid circular dependency issues in `dataclassy/dataclass.py`, `context_value_computer.py`, `queue_condition.py`, `direction_logger.py`, and `test_network_trace.py`.
 * Switched from using `dataclassy` to native dataclasses
+* Every UML relationship is now a custom list wrapper that takes no memory and has a reference to the backing field - usage interface is nearly identical to lists
+* UML relationships can now automatically set backref values where needed (eg `Cut.ac_line_segment`). They also clear the backref field when the item is removed.
 
 ### Fixes
 * added python3.14 to compatibility list.

src/zepben/ewb/__init__.py

@@ -127,6 +127,7 @@
 
 from zepben.ewb.model.cim.iec61970.base.core.ac_dc_terminal import *
 from zepben.ewb.model.cim.iec61970.base.core.base_voltage import *
+from zepben.ewb.model.cim.iec61970.base.core.terminal import *
 from zepben.ewb.model.cim.iec61970.base.core.conducting_equipment import *
 from zepben.ewb.model.cim.iec61970.base.core.connectivity_node import *
 from zepben.ewb.model.cim.iec61970.base.core.connectivity_node_container import *
@@ -144,7 +145,6 @@
 from zepben.ewb.model.cim.iec61970.base.core.power_system_resource import *
 from zepben.ewb.model.cim.iec61970.base.core.sub_geographical_region import *
 from zepben.ewb.model.cim.iec61970.base.core.substation import *
-from zepben.ewb.model.cim.iec61970.base.core.terminal import *
 
 from zepben.ewb.model.cim.iec61970.base.diagramlayout.diagram import *
 from zepben.ewb.model.cim.iec61970.base.diagramlayout.diagram_object import *

src/zepben/ewb/boilerplate/MANIFESTO.md

@@ -6,21 +6,21 @@ Yes, this is not your usual manifest
 
 We have removed the dataclassy library from the SDK, migrating to using the actual Python dataclass decorator. This was motivated by lack of ongoing support for dataclassy, as well as no-one on the team wanting to deal with its mess of magic code.
 
-Our Python version of CIM has some very specific requirements leading to some unorthodox decisions (eg custom inits). Whatever your "why does it work this way" is, it has likely been considered, and many are provided below.
+Our Python version of CIM has some very specific requirements leading to some unorthodox decisions (eg custom inits). Whatever your "why does it work this way" is, it has likely been considered, and many are answered below.
 
 This document is to serve as a guide in case you need to modify the SDK and are unfamiliar with how things are implemented around here. The following are some major decisions we have made and how they influence your potential contribution.
 
 ## Slots
 
-Due to the immense size of grid models we are dealing with, every byte of memory matters, lest you fry your RAM with 120% load. By default, Python stores its values as a dictionary, which takes up hella space. Slots exist to fix that - a `__slots__` class variable describes the fixed memory layout of a class, making it rigid and a lot more compact. Every single one of our CIM classes uses slots, and all the new ones should too. `@dataclass` with slots enabled saves us specifying them explicitly at the cost of transparency - every field of such a class is a descriptor, and overwrites don't always work the way you expect them to. (For example, `@property` overwrite of a static field in a superclass is completely ignored by Python).
+Due to the immense size of grid models we are dealing with, every byte of memory matters, lest you fry your RAM with 120% load. By default, Python stores instance values as a dictionary, which takes up hella space. Slots exist to fix that - a `__slots__` class variable describes the fixed attribute layout of a class, and through some dark magic (static instance-hashed attribute dictionaries) removes a lot of the memory overhead. Every single one of our CIM classes uses slots, and all the new ones should too. `@dataclass` with slots enabled saves us specifying them explicitly at the cost of transparency - every field of such a class is a descriptor, and overwrites don't always work the way you expect them to. (For example, `@property` overwrite of a static field in a superclass is completely ignored by Python).
 
 Our chosen approach is delegating as much functionality as we can to the native Python `@dataclass`, which lets you rely on its documentation and AI's knowledge of its quirks. It might be weird, but it is weird in consistent ways. No more reading a random biologist's library to figure out why your ConnectivityNode just did a barrel roll! 
 
 ## Dataclass
 
 We have chosen to use dataclasses simply because we have a lot of attributes, and any non-dataclass Python init requires a ridiculous amount of code duplication. This allows us to make use of its slot construction, as well as default values for fields that don't need to be passed in the init.
 
-Unfortunately, dataclass generates custom `__init__` functions for the classes, but ONLY if you don't have an existing init already. That's why slightly tweaking class instantiation is quite complicated (requiring a custom decorator to rebuild the class from the original and the dataclass combined - I tried this). Instead, we went for a "clean slate" base class init approach, which, while stripping some of the dataclass functionality, is a lot more transparent when it explodes in your face. 
+Unfortunately, dataclass generates custom `__init__` functions for the classes, but ONLY if you don't have an existing init already. That's why slightly tweaking class instantiation is quite complicated (requiring a custom decorator to rebuild the class from the original and the dataclass combined - I tried this). Instead, we went for a "clean slate" base class init approach (explained below), which, while stripping some of the dataclass functionality, is a lot more transparent when it explodes in your face.
 
 ## Instantiation
 
@@ -38,16 +38,40 @@ Since `@dataclass` generates an init for every subclass without one, the `@zb_da
 
 Positional args are very hard to get consistent with the use of `@dataclass` - to know the order, you must parse the inheritance tree. `@property` values are not included in dataclass inits by default, meaning that adding them as an argument would create arbitrary ordering, and that is a readability nightmare. Moreover, we have custom inits in subclasses, making maintaining the arg order functionally impossible.
 
-That is why, from this version onwards, any argument that is not the object's `mrid` (or equivalent identifier) must be passed as a keyword argument (`MyClass("mrid", 42)` bad, `MyClass("mrid", thing=42)` good).
+That is why, from this version onwards, any argument that is not the object's `mrid` (or equivalent identifier) must be passed as a keyword argument (`MyClass("mRID", 42)` bad, `MyClass("mRID", thing=42)` good).
 
 While it would be possible to bring args back if all subclass inits are removed (probable future feature), it is a terrible idea. Do not do it unless you are very sure of yourself.
 
 ## Eq, Hash, Str, Repr
 
-CIM classes need to be homogenous in their behaviour. Normally, dataclasses define these four methods on subclass level, but we need them to be propagated all the way up to `Identifiable` (or any overriding children such as `NameType`), where `__eq__` and `__hash__` both asses mrid equality, and `__str__` and `__rer__` represent the class as `<ClassName>{<mrid>}` eg `Terminal{terminal_mrid}`. 
+CIM classes need to be homogenous in their behaviour. Normally, dataclasses define these four methods on subclass level, but we need them to be propagated all the way up to `Identifiable` (or any overriding children such as `NameType`), where `__eq__` and `__hash__` both asses memory reference equality, and `__str__` and `__repr__` represent the class as `<ClassName>{<mRID>}` eg `Terminal{terminal_mRID}`. 
 
 ## Descriptors
 
-Properties are bulky and limited in functionality. There is a more general python concept called a Descriptor. This is any class implementing `__get__` and/or (usually and) `__set__` methods, which overwrite accessing a class member provided they are attached to the class and not its instance. This allows us to, for example, implement internally nullable fields.
+Properties are bulky and limited in functionality. There is a more general python concept called a Descriptor. This is any class implementing `__get__` and/or (usually and) `__set__` methods, which overwrite accessing a class member (provided they are attached to the class and not its instance). This allows us to, for example, implement internally nullable fields.
 
-These fields technically require no space in the object, since they access other values internally, but `@dataclass` assigns a slot to anything with a type. Obviously, if you want type hinting on member access, descriptor fields need to be typed. `@remove_descriptor_annotations` decorator takes care of that by stripping the types off of anything defining the aforementioned methods before it gets passed to a dataclass. Since this happens at runtime, IDE type-checking still gets the types, and `@dataclass` is no longer confused about which fields should be created.
+These fields technically require no space in the object, since they access other values internally, but `@dataclass` assigns a slot to anything with a type. Obviously, if you want type hinting on member access, descriptor fields need to be typed. `@remove_descriptor_annotations` decorator (ordinarily part of `@zb_dataclass`) takes care of that by stripping the types off of anything defining the aforementioned methods before it gets passed to a dataclass. Since this happens at runtime, IDE type-checking still gets the types, and `@dataclass` is no longer confused about which fields should be created.
+
+## Lists
+
+Since CIM is UML-based, there are a lot of one-to-many and many-to-many class relationships, represented with lists. Most such relationships are usually empty. Python empty list is 56 bytes, whereas `None` is 16, so we want the lists to be nullable.
+
+Consequently, we have created a hierarchy of nullable descriptors backed by private fields. Each descriptor implements a list-like interface, allowing the user to interact with the relations normally, blissfully unaware of the nullable insanity going on under the hood.
+
+Due to the high diversity of the relationship behaviours, there is a lot of optional functionality shared by some of these lists:
+
+- *mRID collision checks*: for mRID collections, we perform an mRID lookup when an item is added. If another item comes back, we check if the identities match. If they do, we disregard the addition. If they don't, we throw an appropriate error.
+- *backfill*: some items contained in lists link back to the owner item. In that case, we perform a backfill to do this automatically. We also ensure that the item is not already linked to another owner. 
+- *validation*: some lists are created with validation lambdas, calling a validator function on the owner itself. This is useful when additional validation is required, eg sequence number matching.
+- *sorting*: passing a sorting lambda to ordered collections automatically re-sorts the list after an item is added. 
+
+Here are the concrete implementations of the lists that are used in the SDK:
+
+- `LazyList`: A nullable list of objects that has validation and sorting built in. Typically used for non-`Identifiable` relationships.
+- `LazyIndexList`: `LazyList` with index-based insertion and deletion, for strictly ordered relationships such as diagram points.
+
+The remaining collections all implement `MridCollection` - an interface that declares functionality of interaction with a collection of `Identifiable` objects, mainly focusing on `get_by_mRID`, `append`, `remove`, and `clear` methods. This allows us to have a common way of interacting with inter-object relationships, while hiding the underlying implementation (eg `list` vs `dict`)
+
+- `LazyMridList`: A `LazyList` with an mRID check and backfill added. `O(N)` mRID lookup.
+- `MridList`: An mRID collection backed by a non-nullable list. Useful for relationships that are most likely to be filled (eg `ConnectivityNode.terminals`). `O(N)` mRID lookup.
+- `LazyMridMap`: A nullable map of Identifiable objects keyed on their mRID. Supports mRID checks and `O(1)` mRID lookup. Does not support sorting (duh). Useful for large-scale collections (for which the lookup speedup is significant) 

src/zepben/ewb/boilerplate/__init__.py

@@ -2,4 +2,3 @@
 #  This Source Code Form is subject to the terms of the Mozilla Public
 #  License, v. 2.0. If a copy of the MPL was not distributed with this
 #  file, You can obtain one at https://mozilla.org/MPL/2.0/.
-

src/zepben/ewb/boilerplate/backed_descriptor.py

@@ -2,18 +2,10 @@
 #  This Source Code Form is subject to the terms of the Mozilla Public
 #  License, v. 2.0. If a copy of the MPL was not distributed with this
 #  file, You can obtain one at https://mozilla.org/MPL/2.0/.
-#  Copyright 2026 Zeppelin Bend Pty Ltd
-#  This Source Code Form is subject to the terms of the Mozilla Public
-#  License, v. 2.0. If a copy of the MPL was not distributed with this
-#  file, You can obtain one at https://mozilla.org/MPL/2.0/.
 
 from __future__ import annotations
 
 from dataclasses import Field
-from typing import TypeVar
-
-T = TypeVar("T", bound=type)
-
 
 
 class BackedDescriptor:
@@ -24,89 +16,28 @@ class BackedDescriptor:
     """
 
     def __init__(self, private_field):
-        if not isinstance(private_field, Field):
-            raise TypeError("private_field parameter of the Descriptor constructor has to be an instance of dataclass Field.")
-        self.private_field: Field = private_field
+        if not isinstance(private_field, Field) and not isinstance(private_field, BackedDescriptor):
+            raise TypeError(f"private_field parameter of the Descriptor constructor has to be an instance of dataclass Field, instead is {private_field}")
+        self.private_field: Field | 'BackedDescriptor' = private_field
+        self._backing_name: str | None = None
         self.name = None
 
     def __set_name__(self, owner, name):
-        self.name = name
+        if name is None:
+            return
+        self.__name__ = self.name = name
+        if not self._backing_name:
+            self._backing_name = self.private_field.name
 
     def __get__(self, instance, _):
-        return getattr(instance, self.private_field.name)
+        if instance is None:
+            return self
+        return getattr(instance, self._backing_name)
 
     def __set__(self, instance, value):
-        return setattr(instance, self.private_field.name, value)
-
-
-
-def remove_descriptor_annotations(cls: T) -> T:
-    """
-    Remove annotations for class attributes that are data descriptors.
-
-    This is intended for descriptor attributes that should remain class-level
-    descriptors (classes defining ``__get__``/``__set__`` methods, controlling attribute access),
-    rather than becoming dataclass fields/slots. Dataclasses decide
-    which fields to create from ``__annotations__``. By removing annotations for
-    descriptor-backed attributes before ``@dataclass`` runs, those attributes are
-    left alone and can continue to behave as descriptors.
-
-    Use this decorator below ``@dataclass`` so that it is called first. Python
-    applies decorators from the bottom up::
-
-        @dataclass
-        @remove_descriptor_annotations
-        class MyClass:
-            _x: int = field(default=0)
-
-            x: int = MyDescriptor("_x")
-
-    In the example above, ``x`` is annotated, but its class value is a descriptor.
-    Without ``remove_descriptor_annotations``, ``@dataclass`` would treat ``x`` as
-    a dataclass field and may try to include it in generated fields, slots, init,
-    repr, etc. With this decorator, the annotation for ``x`` is removed before
-    dataclass processing, while normal fields such as ``_x`` are left intact.
-
-    Dataclass ``Field`` instances are themselves descriptors and are thus skipped.
-    In the example above, the slot for ``_x`` is still created.
-
-    Before decoration::
-
-        MyClass.__annotations__ == {
-            "_x": int,
-            "x": int,
-        }
-
-    After ``remove_descriptor_annotations`` runs::
-
-        MyClass.__annotations__ == {
-            "_x": int,
-        }
-
-    The class is then passed to ``@dataclass`` with only real dataclass fields
-    remaining in ``__annotations__``.
-    """
-    # Get editable annotations
-    original_annotations = dict(getattr(cls, "__annotations__", {}))
-    tweaked_annotations = dict(original_annotations)
-
-    # Get the current values of class fields. Before @dataclass is run, we see Descriptor instances here
-    cls_dict = vars(cls)
-
-    for name in list(tweaked_annotations):
-        try:
-            value = cls_dict[name]
-            # Skip dataclass fields - they need annotations
-            if isinstance(value, Field):
-                continue
-            # Any descriptor needs to implement get or set - most likely both.
-            if hasattr(value, "__get__") or hasattr(value, "__set__"):
-                tweaked_annotations.pop(name)
-        # Values without defaults will error out - definitely not descriptors
-        except KeyError:
-            pass
+        if self._backing_name is None:
+            raise ValueError(f"Descriptor {self} is not yet aware of the supporting field - `__set__` cannot be called")
+        return setattr(instance, self._backing_name, value)
 
-    # Update annotations on the class
-    cls.__annotations__ = tweaked_annotations
 
-    return cls
+Alias = BackedDescriptor