EIDOS Loader¶

Diverges from the code — 2 finding(s) · 19d408be · 2026-08-25

Both first-pass findings survive adversarial verification: the KM YAML's canonical_operations names three functions (load_graph, invalidate_cache, get_cached_graph) that exist nowhere in the codebase — confirmed by direct grep of backend/ — while the real canonical entry points load_instance (line 549) and refresh_instance (line 558) are correctly documented elsewhere on the same page; sources.files also points at a nonexistent backend/eidos_loader.py rather than the real backend/persistence/eidos_loader.py.

Divergences from the code — details

Sev	Where	Doc says	Code does	Evidence
HIGH	canonical_operations	canonical_operations declares this engine's operations as load_graph() -> effective_graph, invalidate_cache(), and get_cached_graph() -> effective_graph \| None.	Confirmed absent: `def load_graph` and `get_cached_graph` produce zero matches anywhere under backend/. The only `invalidate_cache()` in the repo is an unrelated function in backend/domain/relation_catalog.py:244. The real canonical entry points in backend/persistence/eidos_loader.py are `load_instance(instance_path)` at line 549 and `refresh_instance(instance)` at line 558 (both explicitly labeled 'Canonical entry point' in the module and correctly described in the compiled doc's own Key Functions section). Cache invalidation is instead driven externally via mutation_store.invalidate() (backend/persistence/mutation_store.py:637).	`backend/persistence/eidos_loader.py:549 (load_instance), :558 (refresh_instance); backend/domain/relation_catalog.py:244 (unrelated invalidate_cache)`
MEDIUM	sources.files	sources.files lists the code file as backend/eidos_loader.py.	Confirmed absent via filesystem check: no file at backend/eidos_loader.py. The module lives at backend/persistence/eidos_loader.py, which the compiled doc page itself correctly cites throughout (Layer: persistence, `from persistence import eidos_loader` in its code examples).	`backend/eidos_loader.py (absent); actual file backend/persistence/eidos_loader.py`

Layer: persistence

Purpose and Responsibilities¶

The eidos_loader module is the single source of truth for the EffectiveGraph — the fully merged, indexed, in-memory representation of everything stored in a .eidos ZIP file together with any mutations layered on top by the mutation store. Every other backend module that needs node data, relation data, or tree structure calls into this module; none of them touch the ZIP file directly.

What this engine owns: - Loading and parsing the .eidos ZIP container (manifest, snapshot, optional relations.eidr, optional make.eidm) - Building and maintaining the module-level _cache dict (the EffectiveGraph) - Constructing all performance indexes: ref_index, ref_by_uid, parent_of, children_of, rels_by_node, search_corpus - Detecting cross-worker cache staleness via the sentinel counter and triggering a full rebuild when needed - Applying surgical cache patches (patch_add_relation, patch_remove_relation) that mutation_store uses to avoid a full rebuild after a single write - Exposing the canonical engine entry points load_instance and refresh_instance

What this engine does NOT own: - Persisting mutations — that is mutation_store's responsibility - Merging the overlay into the base graph — that is domain.graph_merge.merge_overlay's responsibility - IAM / access-control filtering — that is server.py's responsibility - The inverted search index beyond the flat search_corpus — domain.search_index._build_search_index builds the inverted index lazily; get_search_index just stores it in _cache after the first call

Cache Lifecycle¶

The EffectiveGraph lives in the module-level _cache dict. It is populated lazily on the first request after process start, and is rebuilt whenever the sentinel counter changes.

Process start
    _cache = {}   _cache_sentinel_mtime = 0.0
        │
        ▼  (first request)
    _load()
      ├─ open EIDOS_FILE ZIP
      ├─ parse manifest + snapshot.json
      ├─ parse optional relations.eidr + make.eidm
      ├─ build nodes, relations, trees
      ├─ build parent_of, children_of, ref_index, ref_by_uid, rels_by_node
      ├─ call graph_merge.merge_overlay()
      ├─ build search_corpus
      ├─ populate _cache
      └─ _cache_sentinel_mtime = _sentinel_mtime()

        │
        ▼  (mutation committed by any worker)
    mutation_store.invalidate()
      ├─ _cache.clear()           ← this worker's cache cleared immediately
      └─ touch _CACHE_SENTINEL    ← signals all other workers

        │
        ▼  (next request on any worker)
    _load() warm-cache check:
      _sentinel_mtime() ≠ _cache_sentinel_mtime  → _cache.clear() → full rebuild
      _sentinel_mtime() == _cache_sentinel_mtime → return _cache  (O(1) fast path)

The sentinel value is a monotonic integer counter, not a real file modification time. The name _cache_sentinel_mtime is kept for back-compatibility with existing tests that patch it. Any increment — even two mutations within the same wall-clock second — causes a cache miss, avoiding the one-second resolution problem of filesystem st_mtime.

Key Functions¶

`seeded_rel_id`¶

def seeded_rel_id(from_id: str, to_id: str, predicate: str) -> str:

Derives a stable, deterministic UUID-5 for a relation using key_derivation.NAMESPACE_SEEDED_RELATION as the namespace.

Parameter	Type	Meaning
`from_id`	`str`	UUID of the source node
`to_id`	`str`	UUID of the target node
`predicate`	`str`	The relation predicate string, e.g. `"hasPart"`

Returns: str — a UUID-5 string that is stable across reloads and process restarts.

This value is stored in the "source" field of every relation dict in _cache["relations"]. It is also used as the tombstone key in mutation_store — if you delete a relation, its seeded_rel_id is recorded as a tombstone and filtered out during the next merge_overlay call.

FROZEN — the namespace constant _EIDOS_NS must never change. Changing it invalidates all existing base-graph tombstones.

rel_uuid = seeded_rel_id(
    "a1b2c3d4-...",
    "e5f6a7b8-...",
    "hasPart"
)
# → e.g. "7c9f1234-..."  — same value every run

`_sentinel_mtime`¶

def _sentinel_mtime() -> float:

Returns the current cache version counter from mutation_store._read_cache_version. Returns 0.0 when the sentinel file is absent (fresh install, no mutations yet). Despite the name, this is an integer counter, not a real mtime.

Called on every request inside _load() to detect cross-worker staleness. The overhead is one integer read; do not try to skip it.

`_load` (internal)¶

def _load() -> dict:

The engine's core function. All public accessors call it as their first step. It returns _cache on a warm hit (O(1)), or fully rebuilds it from disk otherwise.

Rebuild sequence: 1. Open EIDOS_FILE as a zipfile.ZipFile 2. Read manifest.json → locate snapshot_file 3. Parse snapshot.json → nodes_raw, relations_raw, trees_raw, domains, propagation_rules 4. Parse optional relations.eidr → rule_labels, all_rules 5. Parse optional make.eidm → instance_props_by_ref 6. Flatten nodes_raw into the nodes dict (see Data Structures below) 7. Deduplicate and classify relations_raw + external-domain rules 8. Build trees, parent_of, children_of 9. Build ref_index and ref_by_uid via key_derivation.index_ref + _build_ref 10. Apply instance_props_by_ref to nodes via ref_index lookup 11. Build rels_by_node 12. Call graph_merge.merge_overlay to layer mutations on top 13. Build search_corpus via _build_search_corpus (must happen after step 12). Field filtering is owned by the canonical denylist flattener domain.search_index._flatten_indexable (_CORPUS_DENIED_FIELDS) — the loader carries no flattener of its own (the legacy _corpus_flatten blocklist was removed, BL-FE-080 / hardening #511) 14. Identify domain_root_uid and domain_root_path 15. Populate _cache and record _cache_sentinel_mtime

Returns: dict — the _cache dict (see Data Structures below). Callers must treat this as read-only.

Exceptions: Raises zipfile.BadZipFile if EIDOS_FILE is corrupt or missing. Optional inner files (relations.eidr, make.eidm) are silently skipped if absent.

`get_nodes`¶

def get_nodes() -> dict:

Returns the full node map {uid: node_dict} from the EffectiveGraph.

from persistence import eidos_loader

nodes = eidos_loader.get_nodes()
node = nodes.get("a1b2c3d4-0000-0000-0000-000000000001")
print(node["label"])  # → "Electrical System"

`get_node`¶

def get_node(uid: str) -> dict | None:

Returns a single node dict by UUID, or None if not found.

node = eidos_loader.get_node("a1b2c3d4-...")
if node is None:
    raise HTTPException(404, "Node not found")

`get_relations`¶

def get_relations() -> list:

Returns the full flat list of all relation dicts in the EffectiveGraph. Each dict has the shape described in Data Structures below.

`get_trees`¶

def get_trees() -> list:

Returns the list of tree structure dicts. Each dict has root, type, predicate, and children (a {parent_id: [child_ids]} map). These are the raw tree structures; for traversal, prefer get_children and get_parent.

`get_children`¶

def get_children(node_id: str) -> list[str]:

Returns the direct child UUIDs of node_id across all trees. If the node has no children (or does not exist), returns [].

The docstring: "Return direct children of a node across all trees."

kids = eidos_loader.get_children("parent-uuid-...")
for child_uid in kids:
    child = eidos_loader.get_node(child_uid)
    print(child["label"])

`get_parent`¶

def get_parent(node_id: str) -> str | None:

Returns the parent UUID of node_id, or None if the node is a root (domain root or WORLD). When a node appears in multiple trees, parent_of stores only the first tree's parent ("first tree wins").

`get_node_relations`¶

def get_node_relations(node_id: str) -> list:

Returns all relations that have node_id as either from or to. Returns [] for unknown nodes.

relations = eidos_loader.get_node_relations("some-uid-...")
outgoing = [r for r in relations if r["from"] == "some-uid-..."]
incoming = [r for r in relations if r["to"]   == "some-uid-..."]

`get_node_ref`¶

def get_node_ref(uid: str) -> str | None:

Returns the dotted-label path (external ref) for a node, e.g. "&Disciplines.ELECT.CTP10". Returns None if the UID is not in the graph.

The docstring: "Return the external_ref path for a node (dotted label path from root)."

`get_node_by_ref`¶

def get_node_by_ref(ref: str) -> dict | None:

Resolves a dotted ref path (e.g. "+FFL.CTP10.TA37") to a node dict. Delegates to key_derivation.resolve_ref for prefix-tolerant matching. Returns None if not found.

The docstring: "Resolve an external_ref path (e.g. '+FFL.CTP10.TA37') to a node."

node = eidos_loader.get_node_by_ref("&Disciplines.ELECT")
# Works even if the stored key has a different leading prefix

`get_propagation_rules`¶

def get_propagation_rules() -> list:

Returns the propagation_rules list from the base-graph snapshot only. User-defined rules are no longer stored here — they live in the shared rulebook (BL-ARCH-003). This list is consumed by the cross-domain propagation path.

`get_all_rules`¶

def get_all_rules() -> list:

Returns the full all_rules list parsed from relations.eidr. These are the structured rule objects that define relation predicates, source patterns, and target patterns (including external-domain cross-references).

`get_search_corpus`¶

def get_search_corpus() -> dict:

Returns {uid: lowercased_text_blob} — a flat full-text representation of every node, used by the /search endpoint. The corpus is built after the overlay merge so that overlay-only nodes are included. Fields on the content denylist (domain.search_index._CORPUS_DENIED_FIELDS — document body, OCR, webpage/html, attachments, timestamps) are excluded, and values longer than _MAX_INDEXABLE_VALUE_LEN (500 chars) are dropped regardless of field name, so document content is never searchable (BL-FE-080 privacy invariant; hardening #511 + #530).

`get_search_index`¶

def get_search_index() -> dict:

Returns the inverted search index for Global Search v2 (BL-FE-080). Built lazily on the first call and stored in _cache["search_index"], so it is invalidated together with the rest of the EffectiveGraph.

The docstring: "Built once per _load() cache cycle and stored alongside search_corpus in the same _cache dict — so it shares the sentinel-mtime invalidation path. The first call after a cache miss pays the build cost; subsequent calls hit the cached index."

`get_tree_name_by_id`¶

def get_tree_name_by_id() -> dict:

Returns {tree_root_uid: display_name}. The display name is derived from the root node's label; falls back to formalType, then to the root UID, then to "Unknown".

`get_domain_prefix`¶

def get_domain_prefix() -> str:

Returns the leading non-alphanumeric characters from the domain root node's label. Used by remote domains to advertise their ref prefix without hardcoding it.

Examples: root label "&Disciplines" → "&", "-FFL" → "-", "FFL" → "".

`patch_remove_relation`¶

def patch_remove_relation(rel_uuid: str) -> None:

Surgically removes one relation from the warm cache without triggering a full rebuild. If the cache is cold, this is a no-op (the relation simply will not appear when the cache next loads).

rel_uuid is the value stored in relation["source"] — a seeded_rel_id string.

Called by mutation_store immediately after writing a tombstone, as a performance optimization (T10 precursor). Do not call from application code.

`patch_add_relation`¶

def patch_add_relation(rel: dict) -> None:

Surgically appends one relation dict to the warm cache. No-op if the cache is cold.

rel must have at least "from" and optionally "to" keys matching the structure described in Data Structures. Called by mutation_store after writing a new overlay relation.

`load_instance`¶

def load_instance(instance_path: str) -> dict:

Canonical entry point (GCF contract 02.01 / engine_registry.yaml). Sets EIDOS_FILE to Path(instance_path), clears _cache and _cache_sentinel_mtime, then calls _load().

Use this when switching the active .eidos file at runtime (e.g. test setup, tenant switching).

Returns: the newly built _cache dict.

graph = eidos_loader.load_instance("/data/tenants/acme/acme.eidos")
print(f"Loaded {len(graph['nodes'])} nodes")

`refresh_instance`¶

def refresh_instance(instance: str) -> dict:

Canonical entry point. Clears the cache and reloads from the current EIDOS_FILE (the instance parameter is accepted for interface uniformity but not used to change the file path). Use this to force a rebuild after an external change.

Returns: the newly built _cache dict.

Data Structures¶

`_cache` — the EffectiveGraph dict¶

The module-level _cache is populated by _load() and returned by every public accessor. Its keys and value types:

_cache: dict = {
    "nodes":            dict[str, dict],   # uid → node_dict
    "relations":        list[dict],        # all relation dicts
    "trees":            list[dict],        # tree structure dicts
    "parent_of":        dict[str, str],    # child_uid → parent_uid
    "children_of":      dict[str, list[str]],  # parent_uid → [child_uids]
    "ref_index":        dict[str, str],    # dotted_path → uid
    "ref_by_uid":       dict[str, str],    # uid → dotted_path
    "rels_by_node":     dict[str, list[dict]],  # uid → [relation_dicts]
    "search_corpus":    dict[str, str],    # uid → lowercased_text_blob
    "tree_name_by_id":  dict[str, str],    # tree_root_uid → display_name
    "domains":          list[dict],        # raw domains list from snapshot
    "propagation_rules": list[dict],       # base-graph propagation rules
    "all_rules":        list[dict],        # parsed from relations.eidr
    "domain_root_uid":  str | None,        # uid of this backend's domain root
    "domain_root_path": str,               # dotted path of domain root
    # set lazily on first get_search_index() call:
    "search_index":     dict | None,       # inverted index for search v2
}

Node dict (`node_dict`)¶

Built in _load() from nodes_raw. Shape after the overlay merge:

{
    "id":          str,          # UUID (same as the key in nodes dict)
    "label":       str,          # display label; falls back to uid if absent
    "class":       str,          # ontological class
    "treeId":      str,          # UUID of the tree this node was declared in
    "productType": str | None,
    "created":     str | None,   # ISO-8601 timestamp
    "modified":    str | None,   # ISO-8601 timestamp
    "properties":  dict,         # key→value pairs from attrs.properties
    "engineering": list[dict],   # engineering/instance property entries
}

The "_attrs" key is a transient loader artifact that is present during the _load() call but is consumed and removed by _build_search_corpus. Never access "_attrs" from outside eidos_loader.py.

Relation dict¶

{
    "from":      str,            # source node UUID
    "to":        str,            # target node UUID (may be external placeholder)
    "predicate": str,            # resolved predicate label, e.g. "hasPart"
    "source":    str,            # seeded_rel_id — stable UUID for this relation
    # present only for cross-domain relations:
    "external_domain": str,      # e.g. "signal"
    "external_ref":    str,      # dotted path in the remote domain
}

Tree dict¶

{
    "root":      str,            # declaredRootUuid
    "type":      str,            # formalType
    "predicate": str,            # hierarchyPredicate, e.g. "hasPart"
    "children":  dict[str, list[str]],  # parent_uid → [child_uids]
}

`ref_index` and `ref_by_uid`¶

ref_index maps a dotted label path (the external ref) to a node UUID. It stores two keys per node: the canonical path (e.g. "&Disciplines.ELECT") and an alternate stripped-and-lowercased root variant (e.g. "disciplines.ELECT"), enabling cross-domain ref lookups that tolerate prefix mismatches. The alternate key is inserted by key_derivation.index_ref.

ref_by_uid is the inverse: UUID → canonical dotted path.

The dotted path is built by _build_ref, which walks parent_of from the node to the root, collects labels, reverses them, and joins with ".". All segments except the first have their leading non-alphanumeric prefix stripped by _strip_prefix.

`parent_of` and `children_of`¶

parent_of maps a child UUID to its parent UUID. When a node appears in multiple trees, only the first tree's parent is stored ("first tree wins"). The WORLD node ("00000000-0000-0000-0000-000000000000") has no entry in parent_of. The domain root node of each backend also has no entry in parent_of.

children_of maps a parent UUID to a deduplicated list of child UUIDs across all trees.

`rels_by_node`¶

Maps a UUID to every relation dict where that UUID appears as either "from" or "to". A relation therefore appears in two buckets: once for the source node and once for the target node.

`instance_props_by_ref`¶

An intermediate dict built during _load() before it is merged into nodes. It maps a dotted ref string from make.eidm to a list of engineering property entries:

instance_props_by_ref: dict[str, list[dict]] = {
    "SYS.ELECT.CTP10": [
        {"property": "Voltage", "value": "24", "unit": "V", "group": "Electrical"}
    ]
}

Entries are matched to nodes via ref_index. On a miss the loader tries a stripped-and-lowercased root variant. Unmatched refs are logged at DEBUG level with a sample ref_index key to help diagnose mismatches.

Design Constraints¶

Always call through the public accessor functions. Never access _cache directly from outside this module. The module-level variable is an implementation detail; its dict reference may be replaced by _cache.clear() + _cache.update(...) during a rebuild.
Never mutate the return values of get_nodes(), get_relations(), etc. All accessors return live references into _cache. Mutating them will corrupt the EffectiveGraph for all concurrent requests in the same worker process. If you need to filter or transform, copy first.
EIDOS_FILE must exist before any accessor is called. The default is resolved at import time from $EIDOS_FILE (env var) or <module_dir>/../data.eidos. In tests, use load_instance to point to a fixture file before calling any accessor.
Do not call patch_add_relation or patch_remove_relation from application code. These are low-level hooks for mutation_store. Calling them out of sequence will leave the cache in an inconsistent state until the next full rebuild.
Build the search corpus only after the overlay merge. _build_search_corpus is called at the very end of _load() because overlay-only nodes are not present in nodes until graph_merge.merge_overlay runs. The comment in the source notes this explicitly: "Order matters: overlay nodes only exist in nodes after merge_overlay runs (BL-FE-080 demo bug 2026-05-28 — earlier build saw only the base nodes, so 99% of overlay-only projects returned 'No results' for every query)."
Never look up the domain root via ref_index using EIDOS_DOMAIN. The environment variable name and the node's data-file path are independently defined and need not match. Always use the tree-traversal pattern (find the unique non-WORLD node absent from parent_of), as documented in graph-model.md and implemented in _load().
seeded_rel_id namespace is frozen. _EIDOS_NS must never be changed. It is sourced from key_derivation.NAMESPACE_SEEDED_RELATION. Any change invalidates all existing tombstones in the mutation store.

Common Pitfalls¶

1. Accessing `_cache` directly¶

# WRONG — _cache may be empty (cold), or its dict may be replaced during rebuild
from persistence import eidos_loader
node = eidos_loader._cache["nodes"].get(uid)

# CORRECT
node = eidos_loader.get_node(uid)

2. Mutating the returned dict or list¶

# WRONG — modifies the live cache in place
nodes = eidos_loader.get_nodes()
nodes["fake-uid"] = {...}

# CORRECT — work on a copy
import copy
nodes_copy = dict(eidos_loader.get_nodes())

3. Expecting `parent_of` to contain the WORLD node¶

WORLD ("00000000-0000-0000-0000-000000000000") has no entry in parent_of. The domain root node also has no entry. Code that walks up the tree by following parent_of must explicitly stop when get_parent returns None.

uid = some_uid
path = []
while uid is not None:
    node = eidos_loader.get_node(uid)
    path.append(node["label"])
    uid = eidos_loader.get_parent(uid)
path.reverse()

4. Using `EIDOS_DOMAIN` to find the domain root in `ref_index`¶

# WRONG — env var and data path are independently named
import os
root_uid = eidos_loader._load()["ref_index"].get(os.environ["EIDOS_DOMAIN"])

# CORRECT — use the precomputed value from the cache
root_uid = eidos_loader._load()["domain_root_uid"]

5. Assuming `_attrs` exists on a node after loading¶

_attrs is a transient key placed on base-style nodes during _load() and consumed (via node.pop("_attrs", None)) by _build_search_corpus before the cache is stored. After _load() returns, _attrs is absent from every node. Overlay-style nodes never have it at all.

6. Assuming `_sentinel_mtime` returns a real timestamp¶

The function name is kept for back-compatibility, but it returns a monotonic integer counter from mutation_store._read_cache_version. Code that compares this value must use equality (==), not a greater-than check.

7. Calling `get_search_index` before `get_search_corpus` is warm¶

get_search_index calls _load() internally, so it forces the cache warm automatically. However, the inverted index is built lazily inside get_search_index — it is not created by _load() itself. The "search_index" key is absent from _cache until the first get_search_index() call. Code that reads _cache.get("search_index") directly will see None on a fresh load.

8. Multi-tree nodes and `parent_of`¶

When a node belongs to more than one tree, only the first tree's parent is stored in parent_of. This is intentional ("first tree wins"). If your logic depends on a node's position in a specific tree, use get_trees() and traverse the "children" map of the relevant tree directly rather than relying on parent_of.

9. Unmatched `make.eidm` instance properties¶

If engineering properties from make.eidm do not appear on nodes after loading, the most likely cause is a ref path mismatch between the make.eidm dotted path and the path in ref_index. The loader logs the first unmatched ref at DEBUG level together with a sample ref_index key:

instance_properties_sample  example="SYS.ELECT.CTP10"  ref_sample="&SYS.ELECT.CTP10"

The leading prefix (&, -, etc.) or case difference is the usual culprit. The loader tries a stripped-and-lowercased root fallback automatically, but only for the root segment — if intermediate segments differ, the lookup will still fail.

EIDOS Loader¶

Purpose and Responsibilities¶

Cache Lifecycle¶

Key Functions¶

seeded_rel_id¶

_sentinel_mtime¶

_load (internal)¶

get_nodes¶

get_node¶

get_relations¶

get_trees¶

get_children¶

get_parent¶

get_node_relations¶

get_node_ref¶

get_node_by_ref¶

get_propagation_rules¶

get_all_rules¶

get_search_corpus¶

get_search_index¶

get_tree_name_by_id¶

get_domain_prefix¶

patch_remove_relation¶

patch_add_relation¶

load_instance¶

refresh_instance¶