Effective Graph¶

Verified — minor divergences from the code — 2 finding(s) · 19d408be · 2026-08-25

Code claims are largely accurate and verified with high precision — merge_overlay's exact line-134 citation for nodes.update(overlay["nodes"]), the tombstone/node-tombstone filtering snippets, the _cache dict keys/lifecycle (invalidate/_clear_caches/_bump_cache_version/patch_remove_relation), and get_effective_graph() all match backend/persistence/eidos_loader.py, backend/domain/graph_merge.py, and backend/persistence/mutation_store.py exactly. Two minor divergences found.

Divergences from the code — details

Sev	Where	Doc says	Code does	Evidence
LOW	sources.files	KM entity YAML sources.files lists the code file as `backend/eidos_loader.py`.	No such file exists; the loader now lives at `backend/persistence/eidos_loader.py` (the compiled doc page itself correctly cites this path throughout, only the YAML source list is stale).	`backend/persistence/eidos_loader.py`
LOW	The Merge Rule > Overlay wins on conflict (relations extend, not replace)	Doc quotes the overlay-relations-extend step as `relations = relations + overlay["relations"]`.	The actual code is `relations = relations + overlay_rels`, where `overlay_rels` is `overlay["relations"]` filtered to drop any relation whose `from`/`to` endpoint is in `node_tombstones` (Contract 06 §3.11 R37-R39) — i.e. overlay relations touching a tombstoned node ARE filtered before the concat, which the quoted snippet does not show.	`backend/domain/graph_merge.py:176-181`

What EffectiveGraph Is¶

The EffectiveGraph is the single in-memory data structure that the entire backend queries for all read operations. It is produced by merging two independent layers:

The base graph — the read-only data parsed from the .eidos ZIP file (the snapshot, optional relations.eidr, optional make.eidm). This layer is immutable at runtime; the ZIP is never modified.
The overlay — the mutable layer written by the mutation engine into mutations/nodes/*.jsonld, mutations/relations/*.jsonld, mutations/tombstones/*.jsonld, and mutations/node_tombstones/*.jsonld. This layer accumulates additions, edits, and soft-deletes.

The EffectiveGraph is not stored on disk. It lives in the _cache dict in backend/persistence/eidos_loader.py and is rebuilt from scratch whenever the cache is stale. Every public accessor (get_nodes, get_node, get_relations, get_node_relations, etc.) calls _load() as its first step, which either returns the warm dict in O(1) or triggers a full rebuild.

Why it exists instead of reading raw files¶

The base .eidos ZIP is immutable. It cannot be written without replacing the file wholesale, which would require a crash-safe copy-on-write process for every single mutation. Instead, mutations land in the mutations/ directory as individual atomic files (each written via temp → fsync → rename per CCR Rule 5). The EffectiveGraph is the runtime join of these two layers.

This separation provides four guarantees:

Crash safety: a partial write in mutations/ cannot corrupt the base. The worst case is a single malformed overlay file, which the loader skips silently.
Determinism: rebuilding the graph from the same files always produces the same result. There is no in-place accumulation of state.
Restart safety: the mutations directory survives process restarts; no in-memory state needs to be flushed to disk.
Single authoritative view: no caller ever queries the ZIP directly or scans the mutations directory for individual lookups. Everything goes through _load().

The Merge Rule¶

Overlay wins on conflict¶

When the same node UUID appears in both the base graph (nodes_raw from snapshot.json) and the overlay (mutations/nodes/<uuid>.jsonld), the overlay entry fully replaces the base entry for that UUID. This is implemented in graph_merge.merge_overlay (backend/domain/graph_merge.py) at line 134:

nodes.update(overlay["nodes"])

dict.update unconditionally overwrites keys that already exist. After this call, nodes[uid] is the overlay node — the base node is gone from the effective view for the lifetime of this cache. Two fields receive special handling before the update: if the overlay node does not carry an "engineering" key, the base node's engineering list is copied in; likewise for "productType". This preserves inherited values for fields that the overlay author did not explicitly touch.

For relations, the overlay does not replace base relations. It extends them. merge_overlay appends overlay relations to the already-filtered base list:

relations = relations + overlay["relations"]

A base relation and an overlay relation with the same logical pair (from, to, predicate) will therefore both appear in _cache["relations"] unless the base one was tombstoned first.

Tombstones: what they are and how they hide base data¶

Tombstones are soft-delete records. They are never applied by deleting a file from the base ZIP; they are applied during the merge by filtering the base data before it enters the EffectiveGraph.

Relation tombstones (mutations/tombstones/<uuid>.jsonld) remove a seeded base relation from the effective view. The UUID stored in each tombstone file matches the source field of the target relation — a stable UUID-5 derived from (from, to, predicate) via seeded_rel_id. During merge_overlay:

tombstones: set = overlay.get("tombstones", set())
if tombstones:
    relations = [r for r in relations if r.get("_overlay") or r.get("source") not in tombstones]

Any base relation whose source UUID is in the tombstone set is dropped. Overlay relations are tagged _overlay=True and are never filtered by the relation tombstone set — you cannot tombstone your own overlay relation this way.

Node tombstones (mutations/node_tombstones/<uuid>.jsonld) implement BL-DM-001. They remove a node and every relation that touches it from the EffectiveGraph. The removal cascade is comprehensive: the node is deleted from nodes, all relations with that node as from or to are removed from relations and from rels_by_node, and the node's entries in parent_of, children_of, and the two ref indexes are cleaned up:

for uid in list(nodes):
    if uid in node_tombstones:
        del nodes[uid]
relations = [r for r in relations
             if r.get("from") not in node_tombstones
             and r.get("to") not in node_tombstones]
...
for nid in node_tombstones:
    parent_of.pop(nid, None)
    children_of.pop(nid, None)
    ref = ref_by_uid.pop(nid, None)
    if ref is not None:
        key_derivation.unindex_ref(ref_index, ref, nid)

A critical ordering rule (BL-FE-037) prevents a node from being re-introduced by its own overlay file after a node tombstone is applied. At the very top of merge_overlay, the overlay's nodes dict is filtered to exclude any UUID that appears in node_tombstones before the nodes.update() call. The tombstone file and the overlay node file both remain on disk — the layered history is preserved — but the effective view sees neither.

EffectiveGraph Data Structure¶

The EffectiveGraph is the _cache dict populated by _load() in backend/persistence/eidos_loader.py. All public accessors return references into this dict. Callers must treat every returned value as read-only.

Key	Type	Description
`nodes`	`dict[str, dict]`	Primary node map: UUID → node dict. Includes both base and surviving overlay nodes; tombstoned nodes are absent.
`relations`	`list[dict]`	Flat list of all effective relation dicts. Base relations minus tombstoned ones, plus overlay relations.
`trees`	`list[dict]`	Tree structure dicts, one per declared tree. Each has `root` (UUID), `type`, `predicate`, and `children` (`{parent_uid: [child_uids]}`).
`parent_of`	`dict[str, str]`	Child UUID → parent UUID. Only the first tree's parent is stored when a node spans multiple trees. The WORLD node and domain roots have no entry.
`children_of`	`dict[str, list[str]]`	Parent UUID → list of direct child UUIDs, deduplicated across all trees.
`ref_index`	`dict[str, str]`	Dotted label path → UUID. Stores two keys per node: the canonical path (e.g. `"&Disciplines.ELECT"`) and a prefix-stripped lowercase variant for cross-domain lookups.
`ref_by_uid`	`dict[str, str]`	UUID → canonical dotted label path. Inverse of `ref_index`.
`rels_by_node`	`dict[str, list[dict]]`	UUID → all relation dicts where that UUID is `from` or `to`. A relation appears in two buckets: one for the source node, one for the target node.
`search_corpus`	`dict[str, str]`	UUID → lowercased full-text blob for that node. Built after the overlay merge so overlay-only nodes are included.
`tree_name_by_id`	`dict[str, str]`	Tree root UUID → display name. Falls back from label to `formalType` to root UID to `"Unknown"`.
`domains`	`list[dict]`	Raw domains list from `snapshot.json`.
`propagation_rules`	`list[dict]`	Base-graph propagation rules from the snapshot.
`all_rules`	`list[dict]`	Structured rule objects parsed from `relations.eidr`.
`domain_root_uid`	`str \\| None`	UUID of this backend's domain root node.
`domain_root_path`	`str`	Canonical dotted ref path of the domain root.
`search_index`	`dict \\| None`	Inverted search index (BL-FE-080). Absent until the first `get_search_index()` call; populated lazily and then lives alongside the rest of `_cache` until the next invalidation.

Node dict shape¶

Each value in nodes has these fields after the overlay merge:

Field	Type	Notes
`id`	`str`	UUID, same as the dict key
`label`	`str`	Display label
`class`	`str`	Ontological class
`treeId`	`str`	UUID of the tree the node was declared in
`productType`	`str \\| None`	Product classification
`parent`	`str \\| None`	Overlay nodes only: parent UUID at load time
`properties`	`dict`	Key-value properties including description
`engineering`	`list[dict]`	Engineering/instance property entries
`created`	`str \\| None`	ISO-8601 timestamp
`modified`	`str \\| None`	ISO-8601 timestamp
`_overlay`	`bool`	Present and `True` on overlay nodes only
`rev`	`int`	Monotonic revision counter; `0` on base nodes

The _attrs key is a transient loader artifact consumed by _build_search_corpus before the cache is stored. It is absent from every node after _load() returns.

Relation dict shape¶

Field	Type	Notes
`from`	`str`	Source node UUID
`to`	`str`	Target node UUID
`predicate`	`str`	Resolved predicate label, e.g. `"hasPart"`
`source`	`str`	Stable UUID-5 (`seeded_rel_id`) for this relation
`_overlay`	`bool`	Present and `True` on overlay relations only
`external_domain`	`str`	Cross-domain relations only
`external_ref`	`str`	Cross-domain relations only
`inherit_source`	`bool`	Overlay relations: whether to inherit source

Lifecycle¶

When EffectiveGraph is valid¶

The EffectiveGraph is valid from the moment _load() completes until any mutation is committed. The warm-cache check inside _load() reads the monotonic counter from mutations/_cache_version (via mutation_store._read_cache_version) and compares it against the value recorded when the cache was last built (_cache_sentinel_mtime). If they match, the cached dict is returned immediately.

When it becomes stale¶

Every successful mutation committed through mutation_store calls invalidate(), which calls _clear_caches(). This does two things:

Calls eidos_loader._cache.clear() — the in-process cache is empty immediately on this worker.
Calls _bump_cache_version() — atomically increments the counter in mutations/_cache_version using temp → rename, so no concurrent read of the file sees a torn write.

Because the counter is monotonically increasing and resolution-independent, any other worker that served from a warm cache will detect a mismatch on its next _load() call — even if two invalidations happen within the same wall-clock second. This is the cross-process coherence guarantee described in BL-ARCH-004.

How the cache-coherence workflow keeps it fresh¶

The sequence for any write operation is:

1. mutation_store.write_lock() acquired
2. Write overlay files atomically (temp → fsync → rename per CCR Rule 5)
3. mutation_store.invalidate() called
   a. eidos_loader._cache.clear()                   ← this worker's in-memory cache gone
   b. bump_version_counter(_cache_sentinel)          ← cross-process signal written
4. write_lock() released

5. Next request (any worker):
   _load() reads counter → mismatch → _cache.clear() → full rebuild from ZIP + mutations/

For the common case of a single relation deletion, mutation_store passes rel_uuid to invalidate(), which calls eidos_loader.patch_remove_relation(rel_uuid) instead of clearing the whole cache. This surgical patch removes exactly one relation from the warm cache without a full reload. The cross-process counter is still bumped, so other workers rebuild from scratch on their next request.

Concrete Example¶

Suppose the .eidos ZIP contains three nodes and two relations:

Base graph (from snapshot.json):

nodes:
  uid-A  label="Electrical System"  class="System"
  uid-B  label="Battery"            class="Component"  parent=uid-A
  uid-C  label="Charger"            class="Component"  parent=uid-A

relations:
  source=rel-1  from=uid-A  to=uid-B  predicate="hasPart"
  source=rel-2  from=uid-A  to=uid-C  predicate="hasPart"

parent_of:   { uid-B: uid-A, uid-C: uid-A }
children_of: { uid-A: [uid-B, uid-C] }

After some user activity, the overlay layer contains:

Overlay (mutations/ directory):

nodes/uid-B.jsonld        — edited: label="High-Voltage Battery"
nodes/uid-D.jsonld        — new node: label="Controller"  parent=uid-A
tombstones/rel-2.jsonld   — rel-2 (the Charger relation) was deleted

merge_overlay processes these in order:

Step 1 — relation tombstones applied:

relations = [rel-1]          ← rel-2 dropped (source=rel-2 is in tombstones)
rels_by_node[uid-C] filtered — rel-2 removed

Step 2 — no node tombstones in this example.

Step 3 — overlay nodes merged:

nodes[uid-B].update(overlay[uid-B])  → label="High-Voltage Battery"
nodes[uid-D] = overlay[uid-D]        → new entry added

Step 4 — structural indexes extended for uid-D:

parent_of[uid-D]      = uid-A
children_of[uid-A]    = [uid-B, uid-C, uid-D]
trees[0]["children"][uid-A] includes uid-D
ref_by_uid[uid-D]     = "ElectricalSystem.Controller"
ref_index["ElectricalSystem.Controller"] = uid-D

Resulting EffectiveGraph:

nodes:
  uid-A  label="Electrical System"      (base, unchanged)
  uid-B  label="High-Voltage Battery"   (overlay won — label updated)
  uid-C  label="Charger"                (base, still present)
  uid-D  label="Controller"             (overlay-only)

relations:
  rel-1  from=uid-A  to=uid-B  predicate="hasPart"   (base, surviving)
  (rel-2 is absent — tombstoned)
  (any overlay relations for uid-D would appear here)

parent_of:    { uid-B: uid-A, uid-C: uid-A, uid-D: uid-A }
children_of:  { uid-A: [uid-B, uid-C, uid-D] }

Notice that uid-C (the Charger node) is still in nodes — deleting the relation to it does not delete the node itself. To remove uid-C from the EffectiveGraph you would need a node tombstone.

Common Mistakes¶

Holding a reference to EffectiveGraph across a write¶

_load() returns a reference to the live _cache dict. After a mutation, mutation_store.invalidate() calls _cache.clear(), which empties that same dict object in place. Any variable holding a reference to _cache (or to a sub-dict extracted from it) now points at empty or stale data.

# WRONG — snapshot taken before a write, used after
graph = mutation_store.get_effective_graph()
await some_mutation_engine.create_node(...)       # invalidate() fires inside
node = graph["nodes"].get(new_uid)                # graph["nodes"] is now empty

# CORRECT — re-fetch after the mutation completes
await some_mutation_engine.create_node(...)
graph = mutation_store.get_effective_graph()       # fresh load
node = graph["nodes"].get(new_uid)

The same hazard applies to sub-references:

# WRONG
nodes = eidos_loader.get_nodes()     # live reference into _cache
await some_write(...)                # _cache.clear() fires
label = nodes[uid]["label"]          # KeyError — dict was cleared

Always call the accessor again after any await that could cross a mutation boundary.

Assuming EffectiveGraph is the on-disk state¶

The EffectiveGraph is a derived, in-memory view. Its contents can differ from the on-disk files in two ways:

Tombstoned data is absent from the EffectiveGraph but present on disk. A node tombstone file at mutations/node_tombstones/<uuid>.jsonld and the corresponding overlay node file at mutations/nodes/<uuid>.jsonld both remain on disk after a delete. The EffectiveGraph hides both, but the files are intentionally preserved to maintain the layer-stack history ("existed, then deleted"). Code that reads mutations/nodes/ directly to list nodes will see the deleted node; code that calls eidos_loader.get_nodes() will not.
Overlay files are not yet applied to the base ZIP. The base ZIP is never modified at runtime. If you read a node's raw data from snapshot.json inside the ZIP, you will see the pre-mutation state even if the overlay has edited it. Always use eidos_loader.get_node(uid) — never parse the ZIP directly in application code.

Mutating the returned dicts and lists¶

All accessors return live references into _cache. Modifying them in place corrupts the EffectiveGraph for all concurrent requests on the same worker:

# WRONG — inserts a fake node into the live cache
nodes = eidos_loader.get_nodes()
nodes["fake-uid"] = {"label": "ghost"}

# CORRECT — copy before transforming
nodes_copy = dict(eidos_loader.get_nodes())
nodes_copy["fake-uid"] = {"label": "ghost"}

The same applies to relations (use list(eidos_loader.get_relations()) for a shallow copy) and to any nested dict extracted from a node (dict(node) for a copy of the node dict).

Using the EffectiveGraph to infer tombstone history¶

Because tombstones hide data from the EffectiveGraph, you cannot determine from _cache alone whether a UUID was deleted or never existed. Both cases result in eidos_loader.get_node(uid) returning None. If you need to distinguish "deleted" from "never existed", read the mutations/node_tombstones/ directory directly — the tombstone file's presence is the authoritative record.