Effective Graph¶

Verified against the code · 4996d1ceb770fb4545edebc5fd140792bfb23830 · 2026-10-20

956 D1b-4 (2026-10-20): 2 finding(s) fixed — every accessor/lifecycle/cache reference now names eidos_loader.current_graph() (eidos_loader.py:123) instead of the removed *load(); nodes.update at graph_merge.py:152, overlay_rels at 194-199; malformed overlay files are logged (load_overlay**_parse_failed) and skipped, not silent. Re-verified against main 4996d1ce. #935 S1 (2026-10-21): line pins moved (eidos_loader.py:191, graph_merge.py:214-219). Checked against archon/issue-935. #935 S7 (2026-10-22): the base layer is described as empty since S5a; the cache fields domains, propagation_rules and all_rules are always []; a rebuild reads mutations/ only. Checked against eidos_loader.py at a11d2636.¶

What EffectiveGraph Is¶

The EffectiveGraph is the single in-memory data structure that the entire backend queries for all read operations. It is produced by merging two independent layers:

The base graph — historically the read-only data parsed from the .eidos ZIP file (the snapshot, optional relations.eidr, optional make.eidm). Since #935 S5a the backend never reads a .eidos file: the base is empty, the domain root is an overlay node and the tree declaration comes from the runtime domains.json generated from config/domains.yaml (see the EIDOS Loader page). The base-graph descriptions below explain the merge rules, which are unchanged; a converted domain simply has no base data for them to act on.
The overlay — the mutable layer written by the mutation engine into mutations/nodes/*.jsonld, mutations/relations/*.jsonld, mutations/tombstones/*.jsonld, and mutations/node_tombstones/*.jsonld. This layer accumulates additions, edits, and soft-deletes.

The EffectiveGraph is not stored on disk. It lives in the _cache dict in backend/persistence/eidos_loader.py and is rebuilt from scratch whenever the cache is stale. Every public accessor (get_nodes, get_node, get_relations, get_node_relations, etc.) calls current_graph() (backend/persistence/eidos_loader.py:193) as its first step, which either returns the warm dict in O(1) or triggers a full rebuild.

Why it exists instead of reading raw files¶

The retired .eidos base was an immutable ZIP. It could not be written without replacing the file wholesale, which would have required a crash-safe copy-on-write process for every single mutation. That is why mutations land in the mutations/ directory as individual atomic files (each written via temp → fsync → rename per CCR Rule 5). The EffectiveGraph is the runtime join of these two layers.

This separation provides four guarantees:

Crash safety: a partial write in mutations/ cannot corrupt the base. The worst case is a single malformed overlay file, which load_overlay() logs as a structured warning (e.g. load_overlay_node_parse_failed) and skips.
Determinism: rebuilding the graph from the same files always produces the same result. There is no in-place accumulation of state.
Restart safety: the mutations directory survives process restarts; no in-memory state needs to be flushed to disk.
Single authoritative view: no caller ever queries the ZIP directly or scans the mutations directory for individual lookups. Everything goes through current_graph().

The Merge Rule¶

Overlay wins on conflict¶

When the same node UUID appears in both the base graph (nodes_raw from snapshot.json) and the overlay (mutations/nodes/<uuid>.jsonld), the overlay entry fully replaces the base entry for that UUID. This is implemented in graph_merge.merge_overlay (backend/domain/graph_merge.py) at line 152:

nodes.update(overlay["nodes"])

dict.update unconditionally overwrites keys that already exist. After this call, nodes[uid] is the overlay node — the base node is gone from the effective view for the lifetime of this cache. Two fields receive special handling before the update: if the overlay node does not carry an "engineering" key, the base node's engineering list is copied in; likewise for "productType". This preserves inherited values for fields that the overlay author did not explicitly touch.

For relations, the overlay does not replace base relations. It extends them. merge_overlay first filters overlay relations through node_tombstones (Contract 06 §3.11 R37-R39 — cascade-delete protection: an overlay-added edge pointing at a tombstoned node is dropped from the composite view), then appends the filtered list to the already-tombstoned base relations:

# backend/domain/graph_merge.py:214-219
overlay_rels = [
    r for r in overlay["relations"]
    if r.get("from") not in node_tombstones
    and r.get("to") not in node_tombstones
] if node_tombstones else overlay["relations"]
relations = relations + overlay_rels

A base relation and an overlay relation with the same logical pair (from, to, predicate) will therefore both appear in _cache["relations"] unless the base one was tombstoned first. The rels_by_node index is extended from overlay_rels, not raw overlay["relations"], so it never points at a tombstoned endpoint.

Tombstones: what they are and how they hide base data¶

Tombstones are soft-delete records. They are never applied by deleting a file from the base ZIP; they are applied during the merge by filtering the base data before it enters the EffectiveGraph.

Relation tombstones (mutations/tombstones/<uuid>.jsonld) remove a seeded base relation from the effective view. The UUID stored in each tombstone file matches the source field of the target relation — a stable UUID-5 that a retired .eidos base relation got from (from, to, predicate) via seeded_rel_id, which now lives only in tools/migration/eidos_archive.py. During merge_overlay:

tombstones: set = overlay.get("tombstones", set())
if tombstones:
    relations = [r for r in relations if r.get("_overlay") or r.get("source") not in tombstones]

Any base relation whose source UUID is in the tombstone set is dropped. Overlay relations are tagged _overlay=True and are never filtered by the relation tombstone set — you cannot tombstone your own overlay relation this way.

Node tombstones (mutations/node_tombstones/<uuid>.jsonld) implement BL-DM-001. They remove a node and every relation that touches it from the EffectiveGraph. The removal cascade is comprehensive: the node is deleted from nodes, all relations with that node as from or to are removed from relations and from rels_by_node, and the node's entries in parent_of, children_of, and the two ref indexes are cleaned up:

for uid in list(nodes):
    if uid in node_tombstones:
        del nodes[uid]
relations = [r for r in relations
             if r.get("from") not in node_tombstones
             and r.get("to") not in node_tombstones]
...
for nid in node_tombstones:
    parent_of.pop(nid, None)
    children_of.pop(nid, None)
    ref = ref_by_uid.pop(nid, None)
    if ref is not None:
        key_derivation.unindex_ref(ref_index, ref, nid)

A critical ordering rule (BL-FE-037) prevents a node from being re-introduced by its own overlay file after a node tombstone is applied. At the very top of merge_overlay, the overlay's nodes dict is filtered to exclude any UUID that appears in node_tombstones before the nodes.update() call. The tombstone file and the overlay node file both remain on disk — the layered history is preserved — but the effective view sees neither.

EffectiveGraph Data Structure¶

The EffectiveGraph is the _cache dict populated by current_graph() in backend/persistence/eidos_loader.py. All public accessors return references into this dict. Callers must treat every returned value as read-only.

Key	Type	Description
`nodes`	`dict[str, dict]`	Primary node map: UUID → node dict. Includes both base and surviving overlay nodes; tombstoned nodes are absent.
`relations`	`list[dict]`	Flat list of all effective relation dicts. Base relations minus tombstoned ones, plus overlay relations.
`trees`	`list[dict]`	Tree structure dicts, one per declared tree. Each has `root` (UUID), `type`, `predicate`, and `children` (`{parent_uid: [child_uids]}`).
`parent_of`	`dict[str, str]`	Child UUID → parent UUID. Only the first tree's parent is stored when a node spans multiple trees. The WORLD node and domain roots have no entry.
`children_of`	`dict[str, list[str]]`	Parent UUID → list of direct child UUIDs, deduplicated across all trees.
`ref_index`	`dict[str, str]`	Dotted label path → UUID. Stores two keys per node: the canonical path (e.g. `"&Disciplines.ELECT"`) and a prefix-stripped lowercase variant for cross-domain lookups.
`ref_by_uid`	`dict[str, str]`	UUID → canonical dotted label path. Inverse of `ref_index`.
`rels_by_node`	`dict[str, list[dict]]`	UUID → all relation dicts where that UUID is `from` or `to`. A relation appears in two buckets: one for the source node, one for the target node.
`search_corpus`	`dict[str, str]`	UUID → lowercased full-text blob for that node. Built after the overlay merge so overlay-only nodes are included.
`tree_name_by_id`	`dict[str, str]`	Tree root UUID → display name. Falls back from label to `formalType` to root UID to `"Unknown"`.
`domains`	`list[dict]`	Always `[]` since #935 S5a; kept for the cache shape (only a `.eidos` snapshot ever filled it).
`propagation_rules`	`list[dict]`	Always `[]` since #935 S5a (formerly the snapshot's base-graph propagation rules).
`all_rules`	`list[dict]`	Always `[]` since #935 S5a (formerly parsed from `relations.eidr`).
`domain_root_uid`	`str \\| None`	UUID of this backend's domain root node.
`domain_root_path`	`str`	Canonical dotted ref path of the domain root.
`search_index`	`dict \\| None`	Inverted search index (BL-FE-080). Absent until the first `get_search_index()` call; populated lazily and then lives alongside the rest of `_cache` until the next invalidation.

Node dict shape¶

Each value in nodes has these fields after the overlay merge:

Field	Type	Notes
`id`	`str`	UUID, same as the dict key
`label`	`str`	Display label
`class`	`str`	Ontological class
`treeId`	`str`	UUID of the tree the node was declared in
`productType`	`str \\| None`	Product classification
`parent`	`str \\| None`	Overlay nodes only: parent UUID at load time
`properties`	`dict`	Key-value properties including description
`engineering`	`list[dict]`	Engineering/instance property entries
`created`	`str \\| None`	ISO-8601 timestamp
`modified`	`str \\| None`	ISO-8601 timestamp
`_overlay`	`bool`	Present and `True` on overlay nodes only
`rev`	`int`	Monotonic revision counter; `0` on base nodes

The _attrs key is a transient loader artifact consumed by _build_search_corpus before the cache is stored. It is absent from every node after current_graph() returns.

Relation dict shape¶

Field	Type	Notes
`from`	`str`	Source node UUID
`to`	`str`	Target node UUID
`predicate`	`str`	Resolved predicate label, e.g. `"hasPart"`
`source`	`str`	Stable id of this relation (overlay id; a retired `.eidos` base relation had a UUID-5 `seeded_rel_id`)
`_overlay`	`bool`	Present and `True` on overlay relations only
`external_domain`	`str`	Cross-domain relations only
`external_ref`	`str`	Cross-domain relations only
`inherit_source`	`bool`	Overlay relations: whether to inherit source

Lifecycle¶

When EffectiveGraph is valid¶

The EffectiveGraph is valid from the moment current_graph() completes until any mutation is committed. The warm-cache check inside current_graph() reads the monotonic counter from mutations/_cache_version (via mutation_store._read_cache_version) and compares it against the value recorded when the cache was last built (_cache_sentinel_mtime). If they match, the cached dict is returned immediately.

When it becomes stale¶

Every successful mutation committed through mutation_store calls invalidate(), which calls _clear_caches(). This does two things:

Calls eidos_loader._cache.clear() — the in-process cache is empty immediately on this worker.
Calls _bump_cache_version() — atomically increments the counter in mutations/_cache_version using temp → rename, so no concurrent read of the file sees a torn write.

Because the counter is monotonically increasing and resolution-independent, any other worker that served from a warm cache will detect a mismatch on its next current_graph() call — even if two invalidations happen within the same wall-clock second. This is the cross-process coherence guarantee described in BL-ARCH-004.

How the cache-coherence workflow keeps it fresh¶

The sequence for any write operation is:

1. mutation_store.write_lock() acquired
2. Write overlay files atomically (temp → fsync → rename per CCR Rule 5)
3. mutation_store.invalidate() called
   a. eidos_loader._cache.clear()                   ← this worker's in-memory cache gone
   b. bump_version_counter(_cache_sentinel)          ← cross-process signal written
4. write_lock() released

5. Next request (any worker):
   current_graph() reads counter → mismatch → _cache.clear() → full rebuild from mutations/ (overlay only)

For the common case of a single relation deletion, mutation_store passes rel_uuid to invalidate(), which calls eidos_loader.patch_remove_relation(rel_uuid) instead of clearing the whole cache. This surgical patch removes exactly one relation from the warm cache without a full reload. The cross-process counter is still bumped, so other workers rebuild from scratch on their next request.

Concrete Example¶

Suppose the .eidos ZIP contains three nodes and two relations:

Base graph (from snapshot.json):

nodes:
  uid-A  label="Electrical System"  class="System"
  uid-B  label="Battery"            class="Component"  parent=uid-A
  uid-C  label="Charger"            class="Component"  parent=uid-A

relations:
  source=rel-1  from=uid-A  to=uid-B  predicate="hasPart"
  source=rel-2  from=uid-A  to=uid-C  predicate="hasPart"

parent_of:   { uid-B: uid-A, uid-C: uid-A }
children_of: { uid-A: [uid-B, uid-C] }

After some user activity, the overlay layer contains:

Overlay (mutations/ directory):

nodes/uid-B.jsonld        — edited: label="High-Voltage Battery"
nodes/uid-D.jsonld        — new node: label="Controller"  parent=uid-A
tombstones/rel-2.jsonld   — rel-2 (the Charger relation) was deleted

merge_overlay processes these in order:

Step 1 — relation tombstones applied:

relations = [rel-1]          ← rel-2 dropped (source=rel-2 is in tombstones)
rels_by_node[uid-C] filtered — rel-2 removed

Step 2 — no node tombstones in this example.

Step 3 — overlay nodes merged:

nodes[uid-B].update(overlay[uid-B])  → label="High-Voltage Battery"
nodes[uid-D] = overlay[uid-D]        → new entry added

Step 4 — structural indexes extended for uid-D:

parent_of[uid-D]      = uid-A
children_of[uid-A]    = [uid-B, uid-C, uid-D]
trees[0]["children"][uid-A] includes uid-D
ref_by_uid[uid-D]     = "ElectricalSystem.Controller"
ref_index["ElectricalSystem.Controller"] = uid-D

Resulting EffectiveGraph:

nodes:
  uid-A  label="Electrical System"      (base, unchanged)
  uid-B  label="High-Voltage Battery"   (overlay won — label updated)
  uid-C  label="Charger"                (base, still present)
  uid-D  label="Controller"             (overlay-only)

relations:
  rel-1  from=uid-A  to=uid-B  predicate="hasPart"   (base, surviving)
  (rel-2 is absent — tombstoned)
  (any overlay relations for uid-D would appear here)

parent_of:    { uid-B: uid-A, uid-C: uid-A, uid-D: uid-A }
children_of:  { uid-A: [uid-B, uid-C, uid-D] }

Notice that uid-C (the Charger node) is still in nodes — deleting the relation to it does not delete the node itself. To remove uid-C from the EffectiveGraph you would need a node tombstone.

Common Mistakes¶

Holding a reference to EffectiveGraph across a write¶

current_graph() returns a reference to the live _cache dict. After a mutation, mutation_store.invalidate() calls _cache.clear(), which empties that same dict object in place. Any variable holding a reference to _cache (or to a sub-dict extracted from it) now points at empty or stale data.

# WRONG — snapshot taken before a write, used after
graph = mutation_store.get_effective_graph()
await some_mutation_engine.create_node(...)       # invalidate() fires inside
node = graph["nodes"].get(new_uid)                # graph["nodes"] is now empty

# CORRECT — re-fetch after the mutation completes
await some_mutation_engine.create_node(...)
graph = mutation_store.get_effective_graph()       # fresh load
node = graph["nodes"].get(new_uid)

The same hazard applies to sub-references:

# WRONG
nodes = eidos_loader.get_nodes()     # live reference into _cache
await some_write(...)                # _cache.clear() fires
label = nodes[uid]["label"]          # KeyError — dict was cleared

Always call the accessor again after any await that could cross a mutation boundary.

Assuming EffectiveGraph is the on-disk state¶

The EffectiveGraph is a derived, in-memory view. Its contents can differ from the on-disk files in two ways:

Tombstoned data is absent from the EffectiveGraph but present on disk. A node tombstone file at mutations/node_tombstones/<uuid>.jsonld and the corresponding overlay node file at mutations/nodes/<uuid>.jsonld both remain on disk after a delete. The EffectiveGraph hides both, but the files are intentionally preserved to maintain the layer-stack history ("existed, then deleted"). Code that reads mutations/nodes/ directly to list nodes will see the deleted node; code that calls eidos_loader.get_nodes() will not.
Raw overlay files are not the effective state. Superseded and tombstoned files stay on disk, and merge_overlay normalises what it reads. Always use eidos_loader.get_node(uid) — never parse mutations/ directly in application code.

Mutating the returned dicts and lists¶

All accessors return live references into _cache. Modifying them in place corrupts the EffectiveGraph for all concurrent requests on the same worker:

# WRONG — inserts a fake node into the live cache
nodes = eidos_loader.get_nodes()
nodes["fake-uid"] = {"label": "ghost"}

# CORRECT — copy before transforming
nodes_copy = dict(eidos_loader.get_nodes())
nodes_copy["fake-uid"] = {"label": "ghost"}

The same applies to relations (use list(eidos_loader.get_relations()) for a shallow copy) and to any nested dict extracted from a node (dict(node) for a copy of the node dict).

Using the EffectiveGraph to infer tombstone history¶

Because tombstones hide data from the EffectiveGraph, you cannot determine from _cache alone whether a UUID was deleted or never existed. Both cases result in eidos_loader.get_node(uid) returning None. If you need to distinguish "deleted" from "never existed", read the mutations/node_tombstones/ directory directly — the tombstone file's presence is the authoritative record.