Skip to content

Overlay

Verified against the code · 4996d1ceb770fb4545edebc5fd140792bfb23830 · 2026-10-20

956 D1b-4 (2026-10-20): 3 finding(s) fixed — the merge caller is eidos_loader.current_graph(); malformed overlay files are logged via load_overlay_{node,relation,tombstone,node_tombstone}_parse_failed (mutation_store.py:357-437) and then skipped; nodes.update at graph_merge.py:152, engineering/productType carry-over 146-151, DomainStore.write_atomic 246-260. Re-verified against main 4996d1ce. #935 S1 (2026-10-21): line pin moved (nodes.update at graph_merge.py:167). Checked against archon/issue-935. #935 S7 (2026-10-22): the page describes the overlay-only model (Effective Graph = overlay minus tombstones; root and WORLD are overlay files); the sparse-patch-on-an-immutable-base model and the Immutability Contract section are marked as history; the write-ahead-layer wording is dropped. Checked against eidos_loader.py and graph_merge.py at a11d2636. #1102 S8a (2026-10-22): an eidos:UnresolvedRef may carry eidos:node (explicit placeholder, never derived), and the loader reads an IRI-form eidos:source as its bare uuid (infra.iri.relation_source_id). Checked against target_ref.py and mutation_store.py on the #1102 branch (base 538bb015).

What the Overlay Is

The overlay is the mutable layer of the EIDOS data model and, since #935 S5a, its only store. It is a directory of JSON-LD files that holds every node and relation of the domain — the domain root and WORLD included — plus tombstone markers for entities that have been deleted. The Effective Graph — the merged view that API callers and the frontend consume — is built from the overlay alone.

The mental model is the overlay-only store:

Effective Graph = overlay (minus tombstoned entities)

The backend never reads a .eidos file: the root and WORLD are overlay files, written by bootstrap, or by python -m tools.migration convert for a domain that still held them in data.eidos. The merge in graph_merge.py still starts from a base graph, but that base is empty, so "overlay wins over the base on identical IDs" no longer selects between two stores.

History (before #935 S5a). The overlay was a sparse patch layer on top of an immutable base .eidos file: it held only the nodes and relations created or modified since the base was last published, plus tombstones, and everything else was read from the base. The base/overlay wording further down this page describes that historical merge, whose code path is unchanged.

The overlay lives at backend/mutations/ and is governed by backend/persistence/mutation_store.py. The merge that combines the base with the overlay is performed by backend/domain/graph_merge.py, called by eidos_loader.current_graph().

What the overlay contains

The overlay directory holds four kinds of data relevant to the merge:

  • New and updated nodes — every node created at runtime, plus any base node that was subsequently modified. These live in mutations/nodes/.
  • New relations — every relation created at runtime. These live in mutations/relations/.
  • Relation tombstones — soft-delete markers for seeded relations. These live in mutations/tombstones/.
  • Node tombstones — soft-delete markers for nodes. These live in mutations/node_tombstones/.

There are additional subdirectories (ownership/, idempotency/, audit/, snapshots/, pending/) that support other subsystems but are not part of the overlay merge path.

On-Disk Layout

mutations/
  nodes/              one .jsonld file per created or updated node
  relations/          one .jsonld file per created relation
  tombstones/         one .jsonld file per soft-deleted seeded relation
  node_tombstones/    one .jsonld file per soft-deleted node
  ownership.jsonld    all ownership assignments (single shared file)
  idempotency/        one .jsonld file per committed idempotency key
  snapshots/          one .jsonld file per snapshot
  audit/              append-only audit log, one file per calendar day
  pending/            short-lived preview commands (one .json per preview)
  _cache_version      monotonic integer counter for cross-process cache invalidation

File naming convention

Every node and relation file is named after its UUID (UUID v5):

mutations/nodes/{uuid}.jsonld
mutations/relations/{uuid}.jsonld
mutations/tombstones/{uuid}.jsonld
mutations/node_tombstones/{uuid}.jsonld

The filename stem must equal the UUID carried on the envelope's @id (https://id.ontoteq.com/{uuid}), read via infra.iri.id_from_envelope. load_overlay() detects and logs discrepancies between the filename and the embedded id (Contract 06-R28), skipping any file where they do not match. For pre-WP1 files that still carry only eidos:id, id_from_envelope falls back to that key — the reader accepts both shapes.

What a node overlay entry looks like

A file at mutations/nodes/abc-123.jsonld has this shape (WP1 #950, WP2 #951):

{
  "@context": {"eidos": "https://ontoteq.com/ns/eidos#", "rdfs": "http://www.w3.org/2000/01/rdf-schema#", "skos": "http://www.w3.org/2004/02/skos/core#"},
  "@id": "https://id.ontoteq.com/abc-123",
  "@type": "eidos:OverlayNode",
  "rdfs:label": "My Component",
  "skos:notation": "MyComponent",
  "eidos:nodeType": "Component",
  "eidos:treeId": "tree-xyz",
  "eidos:parent": "parent-456",
  "eidos:productType": "hardware",
  "eidos:properties": {"description": "A new component added at runtime"},
  "eidos:created": "2026-06-01T10:00:00+00:00",
  "eidos:updatedAt": "2026-06-15T09:30:00+00:00",
  "eidos:createdBy": "user@example.com",
  "eidos:rev": 3
}

The skos:notation field (WP2 #951) is the node's per-segment ref-path anchor. It is written by mutation_engine.build_overlay_node and mutation_engine.build_overlay_update_node using the canonical generator infra.key_derivation.derive_slug_segment(label, is_root=...). The full dotted ref path is assembled from the chain of stored segments; a pre-WP2 overlay file that still lacks skos:notation falls back to deriving the segment from rdfs:label at read time (no file rewrite), so existing ref paths are byte-stable.

Writers emit @id (via infra.iri.node_iri) and rdfs:label; eidos:id and eidos:label are no longer written. The reader keys on @id through id_from_envelope, which falls back to eidos:id only for pre-WP1 overlay files; _normalize_node() likewise reads rdfs:label first and falls back to eidos:label on legacy input.

After reading this file, _normalize_node() in mutation_store.py converts the JSON-LD fields into the flat dict format used throughout the codebase:

JSON-LD field Flat dict key Description
@id (legacy fallback: eidos:id) id UUID of this node, parsed via id_from_envelope
rdfs:label (legacy fallback: eidos:label) label Display label
eidos:nodeType class Node type (compat key with base format)
eidos:treeId treeId Tree this node belongs to
eidos:parent parent UUID of the parent node
eidos:productType productType Product classification
eidos:properties properties Arbitrary key-value pairs; description lives here
eidos:created created ISO timestamp of creation
eidos:updatedAt modified ISO timestamp of last modification
eidos:createdBy createdBy Identity of creator
eidos:rev rev Monotonic version token; absent on legacy/base nodes resolves to 0
eidos:engineering engineering Optional list of engineering data dicts
skos:notation notation Per-node ref-path segment (WP2 #951); absent on pre-WP2 overlay files — reader derives from label as a fallback
(always set) _overlay Boolean True — marks this as an overlay-originated node

The _overlay: True marker is important: merge_overlay() uses it in tombstone filtering to distinguish overlay relations (which are never tombstoned by tombstones) from base relations (which are).

What a relation overlay entry looks like

A file at mutations/relations/def-456.jsonld has this shape (WP4b #965):

{
  "@context": {"eidos": "https://ontoteq.com/ns/eidos#", "rdfs": "http://www.w3.org/2000/01/rdf-schema#", "skos": "http://www.w3.org/2004/02/skos/core#", "eidos:target": {"@id": "https://ontoteq.com/ns/eidos#target", "@type": "@id"}},
  "@id": "https://id.ontoteq.com/def-456",
  "@type": "eidos:OverlayRelation",
  "eidos:source": "abc-123",
  "eidos:target": "https://id.ontoteq.com/ghi-789",
  "eidos:relationType": "implements",
  "eidos:inheritSource": false
}

Writers emit the IRI on @id (via infra.iri.relation_iri); eidos:id is no longer written. The reader keys on @id through id_from_envelope and falls back to eidos:id only for pre-WP1 overlay files.

eidos:target — one field, three status forms (WP4b #965)

Before WP4b, a relation carried its target across four separate keys (eidos:toNodeId, eidos:externalDomain, eidos:externalTargetRef, eidos:unresolvedTargetRef) and the reader status-dispatched on which were set. WP4b collapses that into a single eidos:target field whose shape encodes the status directly. The writer (build_overlay_relation) emits exactly one of these three shapes; the reader (read_target_ref_jsonld) status-dispatches on shape alone and returns a typed TargetRef.

  • In-model — target is a node in this domain. eidos:target is the node's canonical IRI (string), declared with @type: @id in the @context so a JSON-LD processor expands it as a node reference:

    "eidos:target": "https://id.ontoteq.com/ghi-789"

  • External — target lives in another domain. eidos:target is an eidos:ExternalRef node carrying the domain and the ref. The placeholder UUID is NOT written as an IRI — writing it as a target string would read back as a broken in-model relation to a node that does not exist. The reader re-derives the placeholder from (eidos:source, eidos:ref, eidos:relationType) via derive_relation_uuid_for_unresolved_target:

    "eidos:target": {
  "@type": "eidos:ExternalRef",
  "eidos:domain": "signals",
  "eidos:ref": "+SIG.CH01.PT100"
}

An eidos:ExternalRef may carry just eidos:domain with an empty or absent eidos:ref — the cross-domain migration path writes this shape when it cannot ask the remote instance for a ref at write time (documents-migration DOC_A). The reader returns an external TargetRef whose ref is empty and whose placeholder UUID is empty unless eidos:node carries the real remote id.

eidos:node — the real remote node IRI — is embedded in the eidos:ExternalRef whenever a real remote target UUID is known AND the derived placeholder would NOT round-trip to it. That is: either the ref is empty (no placeholder can be derived at all — the DOC_A case) or the real targetId differs from the placeholder derived from (sourceId, externalTargetRef, relationType) — the UI-created cross-domain path. When eidos:node is present the reader parses the IRI back to a bare UUID and uses it as the placeholder; only the Excel-imported path where targetId equals the derived placeholder omits eidos:node (the reader re-derives the same placeholder on read):

```json
"eidos:target": {
  "@type": "eidos:ExternalRef",
  "eidos:domain": "signals",
  "eidos:ref": "+SIG.CH01.PT100",
  "eidos:node": "https://id.ontoteq.com/<real-remote-uuid>"
}
```

  • Unresolved — target is a deferred LOCAL reference that will bind when the matching node is created. eidos:target is an eidos:UnresolvedRef node carrying the ref only:

    "eidos:target": {
  "@type": "eidos:UnresolvedRef",
  "eidos:ref": "local.DEFERRED.X"
}

An eidos:UnresolvedRef may also carry eidos:node (#1102 S8a): the migration import writes it when a legacy file's eidos:toNodeId was set beside eidos:unresolvedTargetRef, and the reader returns that uuid. Without eidos:node the placeholder is empty — nothing is derived for an unresolved target.

The @context declares the two ref node types (eidos:UnresolvedRef, eidos:ExternalRef) and the two ref-node payload keys (eidos:domain, eidos:ref) so every shape round-trips through a JSON-LD processor.

Malformed targets fail typed, never silently. An unparseable IRI, an eidos:UnresolvedRef with no eidos:ref, an eidos:ExternalRef missing eidos:domain, an unknown @type on a target node, or an unparseable eidos:node IRI inside an eidos:ExternalRef or eidos:UnresolvedRef, raises InvalidTargetRefError. The overlay loader catches that error, logs a structured warning (load_overlay_relation_bad_target), and skips the offending file so one bad relation never crashes the whole load. The two WP4b-era catch sites in mutations_router (delete preview and heal scan) likewise log a structured warning and continue, never leaking the exception text into an API response.

Legacy target form refused (#1102 S8b). Overlay files written before WP4b carried the four pre-WP4b target keys (eidos:toNodeId, eidos:externalDomain, eidos:externalTargetRef, eidos:unresolvedTargetRef) instead of eidos:target. The migration import (tools.migration, #1102 S8a) rewrites them into eidos:target; since S8b read_target_ref_jsonld no longer reads them. A relation with any of the four keys and no eidos:target raises LegacyFormRefusedError (an InvalidIRIError subclass, infra.iri) naming tools.migration, and the overlay load fails on it instead of dropping the file. tools.migration is the one remaining reader of the form (infra.target_ref.legacy_target_ref).

What a tombstone entry looks like

A relation tombstone at mutations/tombstones/def-456.jsonld (WP1 #950):

{
  "@context": {"eidos": "https://ontoteq.com/ns/eidos#", "rdfs": "http://www.w3.org/2000/01/rdf-schema#", "skos": "http://www.w3.org/2004/02/skos/core#"},
  "@type": "eidos:RelationTombstone",
  "@id": "https://id.ontoteq.com/def-456",
  "eidos:created": "2026-06-20T14:00:00+00:00",
  "eidos:createdBy": "user@example.com"
}

A node tombstone at mutations/node_tombstones/abc-123.jsonld (WP1 #950):

{
  "@context": {"eidos": "https://ontoteq.com/ns/eidos#", "rdfs": "http://www.w3.org/2000/01/rdf-schema#", "skos": "http://www.w3.org/2004/02/skos/core#"},
  "@type": "eidos:NodeTombstone",
  "@id": "https://id.ontoteq.com/abc-123",
  "eidos:deletedAt": "2026-06-20T15:00:00+00:00",
  "eidos:deletedBy": "user@example.com",
  "eidos:auditRef": "https://id.ontoteq.com/some-audit-id"
}

Both tombstone writers now emit @id only; the reader keys on @id via id_from_envelope, falling back to eidos:id for pre-WP1 tombstone files.

Legacy urn ids refused (#1102 S8b). Overlay files written before WP1 carried @id, eidos:source and audit-reference values in the old urn:eidos:(node|relation|audit):{uuid} form. The migration import (tools.migration, #946 / #1102 S8a) rewrites them to https://id.ontoteq.com/{uuid}. Since S8b infra.iri.from_iri, id_from_envelope and relation_source_id refuse a urn:eidos:node: / urn:eidos:relation: value with LegacyFormRefusedError naming tools.migration (an eidos:id duplicate does not answer it), and load_overlay fails naming the file. The urn:eidos:audit:{uuid} form is still read (the append-only audit log keeps it), and the empty legacy audit ref urn:eidos:audit: stays the #1071 exception.

Overlay vs Base — The Immutability Contract (history)

History. This section describes the base .eidos + overlay model retired by #935 S5a. Today there is no base: the overlay is the only store and the invariant below reduces to "Effective Graph = overlay".

The base .eidos file was a ZIP archive that was never modified at runtime; eidos_loader loaded it read-only until #935 S5a retired that reader (a legacy file is now never opened, and convert archives it). The reasons for this constraint are practical and safety-related:

  1. ZIP files are not append-friendly. Modifying a ZIP in place requires rewriting the central directory, which is not atomic. A crash mid-rewrite corrupts the entire base.
  2. The base represents a published snapshot. Mutating it would destroy the ability to reconstruct the exact state that was reviewed and released.
  3. The overlay is the crash-safe write path. Every write to the overlay uses the atomic pattern described below. A crash can leave a partial .tmp file but can never corrupt the base or an existing overlay file.

The invariant is:

Effective Graph = immutable base + mutable overlay, where overlay wins on identical IDs.

This means: any node or relation in the base can be superseded at runtime by writing a file with the same ID to mutations/nodes/ or mutations/relations/. The base entry is not deleted; it is simply shadowed. If the overlay entry is later removed or its ID no longer appears in the overlay, the base entry becomes visible again.

A developer who misunderstands this and tries to write directly into the .eidos ZIP, or who removes an overlay file expecting "undo" semantics without considering the merge order, will get incorrect Effective Graph output. The right way to suppress a base node is to write a node tombstone, not to remove or modify the base.

Writing to the Overlay

All writes go through write_atomic(data: dict, path: Path) in DomainStore (also available as the module-level wrapper mutation_store.write_atomic).

write_atomic — the temp/fsync/rename pattern

def write_atomic(self, data: dict, path: Path) -> None:
Parameter Type Description
data dict JSON-serialisable data to write
path Path Absolute destination path inside mutations/

The function implements CCR Rule 5: temp file → fsync → atomic rename. In detail:

  1. A tempfile.mkstemp() call creates a uniquely named .tmp file in the same directory as path. Using the same directory is essential: os.replace() (step 4) requires that the source and destination be on the same filesystem.
  2. The data is written as indented JSON with json.dump(..., ensure_ascii=False, indent=2) and the file handle is flushed with f.flush().
  3. os.fsync(f.fileno()) is called. This forces the kernel to push the data to the storage device before the rename. Without fsync, a crash after the rename could leave a zero-byte or partially written file that appears complete to os.replace.
  4. os.replace(tmp, path) performs an atomic rename. On POSIX systems, this is guaranteed to be atomic by the kernel: any reader of path sees either the complete old content or the complete new content — never a partial write.
  5. If any step fails, the .tmp file is deleted with contextlib.suppress(OSError) and the original exception is re-raised. The destination file is never corrupted.

The method is serialised by self._write_lock (a threading.Lock), which prevents two coroutines in the same process from interleaving writes to the same path. For the single shared files (ownership.jsonld) an additional cross-process advisory fcntl lock is acquired by the callers before calling write_atomic.

What happens if a write is interrupted

If the process crashes between the fsync and the os.replace, the only artefact is an orphaned .tmp file in the same directory as the intended target. The target file (if it already existed) is unchanged. The .tmp file is inert: load_overlay() globs *.jsonld and will never pick up *.tmp files. The next successful write to that path will replace the .tmp or the surviving old file, whichever applies.

Reading from the Overlay

load_overlay() — what it assembles

def load_overlay(self) -> dict:

Returns:

{
    "nodes":           {uuid: normalized_node_dict, ...},
    "relations":       [rel_dict, ...],
    "tombstones":      {uuid, ...},        # set of soft-deleted seeded relation UUIDs
    "node_tombstones": {uuid, ...},        # set of soft-deleted node UUIDs
}

The function scans four directories in order:

  1. mutations/nodes/*.jsonld — each file is parsed, UUID-checked (id_from_envelope reads @id; falls back to eidos:id on pre-WP1 files), and passed through _normalize_node(). The result is keyed by UUID in the nodes dict.
  2. mutations/relations/*.jsonld — each file is parsed, UUID-checked the same way, and the relevant fields (eidos:source, eidos:target, eidos:relationType, etc.) are extracted into a flat relation dict with _overlay: True. The runtime from is the bare source uuid: an eidos:source still in a node-IRI form (https://id.ontoteq.com/… or …/node/…) is read as its uuid through infra.iri.relation_source_id (#1102 S8a), so a migration import that writes the bare uuid leaves the effective graph unchanged. The target is routed through read_target_ref_jsonld so one of the three WP4b (#965) shapes — in-model IRI, eidos:ExternalRef, or eidos:UnresolvedRef — resolves to the runtime (to, external_domain, external_ref, unresolved_ref) fields the merger expects. A malformed new-format target raises InvalidTargetRefError; the loader logs a structured load_overlay_relation_bad_target warning and skips the offending file so one bad relation never crashes the whole overlay load.
  3. mutations/tombstones/*.jsonld — each file's UUID (via id_from_envelope) is added to the tombstones set.
  4. mutations/node_tombstones/*.jsonld — each file's UUID (via id_from_envelope) is added to the node_tombstones set.

Malformed files (JSON parse errors, missing fields) are logged and skipped: each scan loop catches Exception, emits a structured warning (load_overlay_node_parse_failed, load_overlay_relation_parse_failed, load_overlay_tombstone_parse_failed, load_overlay_node_tombstone_parse_failed — mutation_store.py:357-437) naming the file, and continues. This is intentional: a single corrupted overlay file must not prevent the rest of the overlay from loading. The one exception is a file in a legacy form (#1102 S8b) — a urn:eidos:node: / urn:eidos:relation: @id or eidos:source, or a relation with the four pre-WP4b target keys and no eidos:target: the loop re-raises LegacyFormRefusedError prefixed with the file (relations/<uuid>.jsonld: …), naming tools.migration, so the whole load fails instead of dropping the file. At the API the module-level mutation_store.load_overlay() surfaces it as the typed LegacyOverlayFileRefusedError (HTTP 500, legacy_overlay_file_refused): the client message is generic, and the file, the refusal and tools.migration ride on the domain_error log record only (log_fields file, reason, migration_tool). A legacy id nested elsewhere in a loaded file (an inheritedFrom, a relation origin/target, an id-index lookup) is not read either: the read helpers re-raise LegacyFormRefusedError instead of returning "" or the raw string. The migration import converts such files; the backend no longer reads them. The UUID-mismatch case additionally carries rule="C-01" (load_overlay_uuid_mismatch / load_overlay_rel_uuid_mismatch) so it can be found in audit tooling.

Caching and cross-process invalidation

load_overlay() caches its result in self._overlay_cache. The cache is valid as long as the monotonic counter in mutations/_cache_version matches the value recorded when the cache was last built (self._overlay_cache_sentinel_mtime). Every call to invalidate() bumps the counter via bump_version_counter(), which uses os.replace internally (same atomic-rename guarantee). Because the counter is stored on disk, other uvicorn worker processes see the change on their next load_overlay() call and discard their own warm caches. This replaces the old st_mtime comparison, which had whole-second resolution and could miss two invalidations in the same second.

Concrete Example

Suppose the base .eidos file contains this node:

{
  "id": "node-001",
  "label": "Engine Mount",
  "class": "Mechanical",
  "treeId": "tree-A",
  "properties": {"description": "Original description from base"}
}

A user updates the label and description at runtime. The system writes mutations/nodes/node-001.jsonld (WP1 #950):

{
  "@context": {"eidos": "https://ontoteq.com/ns/eidos#", "rdfs": "http://www.w3.org/2000/01/rdf-schema#", "skos": "http://www.w3.org/2004/02/skos/core#"},
  "@id": "https://id.ontoteq.com/node-001",
  "@type": "eidos:OverlayNode",
  "rdfs:label": "Engine Mount (Rev B)",
  "eidos:nodeType": "Mechanical",
  "eidos:treeId": "tree-A",
  "eidos:properties": {"description": "Updated description after engineering review"},
  "eidos:updatedAt": "2026-06-23T08:00:00+00:00",
  "eidos:rev": 1
}

When eidos_loader.current_graph() rebuilds the cache, it:

  1. Starts from the base graph — empty since #935 S5a; in the historical base+overlay setup shown here nodes["node-001"] held the original data.
  2. Calls load_overlay(). The overlay returns nodes = {"node-001": <normalized overlay entry>}.
  3. Passes both to merge_overlay(). Inside merge_overlay(), line 152: nodes.update(overlay["nodes"]) — the overlay entry for "node-001" overwrites the base entry.

The Effective Graph now contains:

{
    "id": "node-001",
    "label": "Engine Mount (Rev B)",
    "class": "Mechanical",
    "treeId": "tree-A",
    "properties": {"description": "Updated description after engineering review"},
    "modified": "2026-06-23T08:00:00+00:00",
    "rev": 1,
    "_overlay": True,
    # engineering and productType are preserved from base (see graph_merge.py lines 146-151)
}

The base file is untouched. If mutations/nodes/node-001.jsonld were deleted (do not do this directly — use a node tombstone), the next load would show the original base entry again.

Tombstones

Why tombstones exist instead of deleting files

The overlay follows an additive invariant: files are never deleted from it. Instead, a separate tombstone file is written to signal deletion. This design has two motivations:

  1. Audit trail. The layer stack of mutations/nodes/{uuid}.jsonld + mutations/node_tombstones/{uuid}.jsonld tells the full history: "this entity existed, then was deleted." Deleting the overlay file would erase that evidence.
  2. Atomic semantics. Deleting a file and ensuring that no concurrent reader sees a half-deleted state is harder than writing a new file. write_atomic gives the same crash-safe guarantee for tombstones as for all other overlay writes.

write_node_tombstone — soft-deleting a node

def write_node_tombstone(
    self, node_id: str, deleted_by: str = "", audit_id: str = ""
) -> None:
Parameter Type Description
node_id str UUID of the node to soft-delete
deleted_by str Identity of the user requesting deletion
audit_id str Reference to the audit log entry for this deletion

Writes a eidos:NodeTombstone document to mutations/node_tombstones/{node_id}.jsonld via write_atomic. The node's overlay file at mutations/nodes/{node_id}.jsonld (if any) is intentionally left on disk. The operation is idempotent: re-writing the same tombstone is an overwrite with no duplicate-file error.

write_tombstone — soft-deleting a seeded relation

def write_tombstone(self, rel_uuid: str, created_by: str = "") -> None:
Parameter Type Description
rel_uuid str UUID of the seeded relation to soft-delete
created_by str Identity of the user requesting deletion

Writes a eidos:RelationTombstone document to mutations/tombstones/{rel_uuid}.jsonld via write_atomic. Only base (seeded) relations use this mechanism. Overlay relations (those in mutations/relations/) are deleted by removing their file — but again, do not do this manually; use the appropriate API endpoint which manages invalidation.

How merge_overlay handles tombstones

merge_overlay() in graph_merge.py applies tombstones in a strict order to prevent re-introduction of deleted entities:

Step 1 (BL-FE-037): Before any merge work, node_tombstones are used to filter overlay["nodes"]. If a node has both an overlay file and a node tombstone, the overlay entry is removed from the working copy of the overlay dict. This prevents the subsequent nodes.update() call from silently resurrecting a deleted node. The original _overlay_cache dict is not mutated; a shallow copy is made (overlay = dict(overlay)) so the cache remains valid.

Step 2: Relation tombstones are applied to the base relations list and to rels_by_node. The filter keeps overlay relations (identified by r.get("_overlay")) because overlay relations have a separate deletion path and are never in the tombstones set.

Step 3 (BL-DM-001): Node tombstones are applied to the assembled graph. For each tombstoned UUID: - The node is removed from nodes. - Every relation where from or to equals the tombstoned UUID is removed from relations and rels_by_node. - The node is removed from parent_of and children_of. - The node's ref path is removed from ref_index and ref_by_uid via key_derivation.unindex_ref(), which is careful not to drop a colliding ref entry that belongs to another node (BL-DM-010). - The tombstoned UUID is removed from the children list of its parent in children_of.

Step 4: The remaining overlay nodes and relations are merged into the base graph. nodes.update(overlay["nodes"]) applies the overlay-wins rule. Structural indexes (parent_of, children_of, trees[i]["children"], ref_index, ref_by_uid, rels_by_node) are rebuilt for all overlay nodes in two passes so that children of overlay parents resolve correctly regardless of the order files were scanned.

After merge_overlay() returns, base_graph["relations"] has been reassigned to the filtered and extended list. Callers must read from base_graph["relations"] after the call, not from any reference they captured before it.

Invariants

The two invariants named in overlay.yaml map directly to code:

  • overlay-wins-on-conflict: enforced by nodes.update(overlay["nodes"]) in merge_overlay() (line 167 of graph_merge.py). Because dict.update overwrites existing keys, any base entry for a UUID that appears in the overlay is replaced unconditionally.
  • atomic-write: enforced by write_atomic() in DomainStore (lines 246–260 of mutation_store.py). Every overlay file write — nodes, relations, tombstones, relation tombstones — goes through this function. There is no code path that writes overlay JSON without fsync + os.replace.

A developer who bypasses write_atomic — for example, by calling path.write_text() directly — risks leaving a partially written file on disk if the process crashes mid-write. load_overlay() would then fail to parse the malformed JSON, log a load_overlay_*_parse_failed warning and skip the file, and the entity would disappear from the Effective Graph until the file was manually repaired or overwritten. This is the primary blast radius of violating the atomic-write invariant.