Skip to content

Overlay

Verified against the code · 19d408be · 2026-08-25

The overlay.md page's claims about the JSON-LD overlay layer, its on-disk layout, the write_atomic/load_overlay/write_tombstone/write_node_tombstone functions, the _normalize_node field mapping, and the merge_overlay tombstone-ordering logic (including the specific line-134 nodes.update(overlay['nodes']) citation) all match backend/persistence/mutation_store.py and backend/domain/graph_merge.py exactly as they exist on disk. Function signatures, line-number citations (write_atomic 224-238, invalidate mechanics, DomainStore fields), directory names (nodes/, relations/, tombstones/, node_tombstones/), and both stated invariants (overlay-wins-on-conflict, atomic-write) are all borne out by the code. The sources.files directory entry backend/mutations/nodes/ is currently empty on this checkout (no nodes have been created yet) but is a valid runtime path created by DomainStore.ensure_dirs(), not a code-path divergence.

What the Overlay Is

The overlay is the mutable layer of the EIDOS data model. It is a directory of JSON-LD files that records every node and relation change made since the base .eidos file was last published. When the system assembles the Effective Graph — the merged view that API callers and the frontend consume — the overlay always wins over the base on any ID that appears in both.

The mental model is a sparse patch layer on top of an immutable base:

Effective Graph = base .eidos data + overlay mutations

"Sparse" is important. The overlay does not contain a copy of every node; it contains only the nodes and relations that have been created or modified since the base was last published, plus tombstone markers for entities that have been deleted. Everything else is read directly from the base.

The overlay lives at backend/mutations/ and is governed by backend/persistence/mutation_store.py. The merge that combines the base with the overlay is performed by backend/domain/graph_merge.py, called by eidos_loader._load().

What the overlay contains

The overlay directory holds four kinds of data relevant to the merge:

  • New and updated nodes — every node created at runtime, plus any base node that was subsequently modified. These live in mutations/nodes/.
  • New relations — every relation created at runtime. These live in mutations/relations/.
  • Relation tombstones — soft-delete markers for seeded relations. These live in mutations/tombstones/.
  • Node tombstones — soft-delete markers for nodes. These live in mutations/node_tombstones/.

There are additional subdirectories (ownership/, idempotency/, audit/, snapshots/, pending/) that support other subsystems but are not part of the overlay merge path.

On-Disk Layout

mutations/
  nodes/              one .jsonld file per created or updated node
  relations/          one .jsonld file per created relation
  tombstones/         one .jsonld file per soft-deleted seeded relation
  node_tombstones/    one .jsonld file per soft-deleted node
  ownership.jsonld    all ownership assignments (single shared file)
  idempotency/        one .jsonld file per committed idempotency key
  snapshots/          one .jsonld file per snapshot
  audit/              append-only audit log, one file per calendar day
  pending/            short-lived preview commands (one .json per preview)
  _cache_version      monotonic integer counter for cross-process cache invalidation

File naming convention

Every node and relation file is named after its UUID (UUID v5):

mutations/nodes/{uuid}.jsonld
mutations/relations/{uuid}.jsonld
mutations/tombstones/{uuid}.jsonld
mutations/node_tombstones/{uuid}.jsonld

The filename stem must equal the eidos:id field inside the file. load_overlay() detects and logs discrepancies between the filename and the embedded eidos:id (Contract 06-R28), skipping any file where they do not match.

What a node overlay entry looks like

A file at mutations/nodes/abc-123.jsonld has this shape:

{
  "@context": {"eidos": "https://ontoteq.com/ns/eidos#"},
  "@type": "eidos:Node",
  "eidos:id": "abc-123",
  "eidos:label": "My Component",
  "eidos:nodeType": "Component",
  "eidos:treeId": "tree-xyz",
  "eidos:parent": "parent-456",
  "eidos:productType": "hardware",
  "eidos:properties": {"description": "A new component added at runtime"},
  "eidos:created": "2026-06-01T10:00:00+00:00",
  "eidos:updatedAt": "2026-06-15T09:30:00+00:00",
  "eidos:createdBy": "user@example.com",
  "eidos:rev": 3
}

After reading this file, _normalize_node() in mutation_store.py converts the JSON-LD fields into the flat dict format used throughout the codebase:

JSON-LD field Flat dict key Description
eidos:id id UUID of this node
eidos:label label Display label
eidos:nodeType class Node type (compat key with base format)
eidos:treeId treeId Tree this node belongs to
eidos:parent parent UUID of the parent node
eidos:productType productType Product classification
eidos:properties properties Arbitrary key-value pairs; description lives here
eidos:created created ISO timestamp of creation
eidos:updatedAt modified ISO timestamp of last modification
eidos:createdBy createdBy Identity of creator
eidos:rev rev Monotonic version token; absent on legacy/base nodes resolves to 0
eidos:engineering engineering Optional list of engineering data dicts
(always set) _overlay Boolean True — marks this as an overlay-originated node

The _overlay: True marker is important: merge_overlay() uses it in tombstone filtering to distinguish overlay relations (which are never tombstoned by tombstones) from base relations (which are).

What a relation overlay entry looks like

A file at mutations/relations/def-456.jsonld has this shape:

{
  "@context": {"eidos": "https://ontoteq.com/ns/eidos#"},
  "@type": "eidos:Relation",
  "eidos:id": "def-456",
  "eidos:source": "abc-123",
  "eidos:toNodeId": "ghi-789",
  "eidos:relationType": "implements",
  "eidos:inheritSource": false
}

What a tombstone entry looks like

A relation tombstone at mutations/tombstones/def-456.jsonld:

{
  "@context": {"eidos": "https://ontoteq.com/ns/eidos#"},
  "@type": "eidos:RelationTombstone",
  "eidos:id": "def-456",
  "eidos:created": "2026-06-20T14:00:00+00:00",
  "eidos:createdBy": "user@example.com"
}

A node tombstone at mutations/node_tombstones/abc-123.jsonld:

{
  "@context": {"eidos": "https://ontoteq.com/ns/eidos#"},
  "@type": "eidos:NodeTombstone",
  "eidos:id": "abc-123",
  "eidos:deletedAt": "2026-06-20T15:00:00+00:00",
  "eidos:deletedBy": "user@example.com",
  "eidos:auditRef": "urn:eidos:audit:some-audit-id"
}

Overlay vs Base — The Immutability Contract

The base .eidos file is a ZIP archive that is never modified at runtime. It is loaded by eidos_loader and treated as read-only. The reasons for this constraint are practical and safety-related:

  1. ZIP files are not append-friendly. Modifying a ZIP in place requires rewriting the central directory, which is not atomic. A crash mid-rewrite corrupts the entire base.
  2. The base represents a published snapshot. Mutating it would destroy the ability to reconstruct the exact state that was reviewed and released.
  3. The overlay is the crash-safe write path. Every write to the overlay uses the atomic pattern described below. A crash can leave a partial .tmp file but can never corrupt the base or an existing overlay file.

The invariant is:

Effective Graph = immutable base + mutable overlay, where overlay wins on identical IDs.

This means: any node or relation in the base can be superseded at runtime by writing a file with the same ID to mutations/nodes/ or mutations/relations/. The base entry is not deleted; it is simply shadowed. If the overlay entry is later removed or its ID no longer appears in the overlay, the base entry becomes visible again.

A developer who misunderstands this and tries to write directly into the .eidos ZIP, or who removes an overlay file expecting "undo" semantics without considering the merge order, will get incorrect Effective Graph output. The right way to suppress a base node is to write a node tombstone, not to remove or modify the base.

Writing to the Overlay

All writes go through write_atomic(data: dict, path: Path) in DomainStore (also available as the module-level wrapper mutation_store.write_atomic).

write_atomic — the temp/fsync/rename pattern

def write_atomic(self, data: dict, path: Path) -> None:
Parameter Type Description
data dict JSON-serialisable data to write
path Path Absolute destination path inside mutations/

The function implements CCR Rule 5: temp file → fsync → atomic rename. In detail:

  1. A tempfile.mkstemp() call creates a uniquely named .tmp file in the same directory as path. Using the same directory is essential: os.replace() (step 4) requires that the source and destination be on the same filesystem.
  2. The data is written as indented JSON with json.dump(..., ensure_ascii=False, indent=2) and the file handle is flushed with f.flush().
  3. os.fsync(f.fileno()) is called. This forces the kernel to push the data to the storage device before the rename. Without fsync, a crash after the rename could leave a zero-byte or partially written file that appears complete to os.replace.
  4. os.replace(tmp, path) performs an atomic rename. On POSIX systems, this is guaranteed to be atomic by the kernel: any reader of path sees either the complete old content or the complete new content — never a partial write.
  5. If any step fails, the .tmp file is deleted with contextlib.suppress(OSError) and the original exception is re-raised. The destination file is never corrupted.

The method is serialised by self._write_lock (a threading.Lock), which prevents two coroutines in the same process from interleaving writes to the same path. For the single shared files (ownership.jsonld) an additional cross-process advisory fcntl lock is acquired by the callers before calling write_atomic.

What happens if a write is interrupted

If the process crashes between the fsync and the os.replace, the only artefact is an orphaned .tmp file in the same directory as the intended target. The target file (if it already existed) is unchanged. The .tmp file is inert: load_overlay() globs *.jsonld and will never pick up *.tmp files. The next successful write to that path will replace the .tmp or the surviving old file, whichever applies.

Reading from the Overlay

load_overlay() — what it assembles

def load_overlay(self) -> dict:

Returns:

{
    "nodes":           {uuid: normalized_node_dict, ...},
    "relations":       [rel_dict, ...],
    "tombstones":      {uuid, ...},        # set of soft-deleted seeded relation UUIDs
    "node_tombstones": {uuid, ...},        # set of soft-deleted node UUIDs
}

The function scans four directories in order:

  1. mutations/nodes/*.jsonld — each file is parsed, UUID-checked, and passed through _normalize_node(). The result is keyed by UUID in the nodes dict.
  2. mutations/relations/*.jsonld — each file is parsed, UUID-checked, and the relevant fields (eidos:source, eidos:toNodeId, eidos:relationType, etc.) are extracted into a flat relation dict with _overlay: True.
  3. mutations/tombstones/*.jsonld — each file's eidos:id is added to the tombstones set.
  4. mutations/node_tombstones/*.jsonld — each file's eidos:id is added to the node_tombstones set.

Malformed files (JSON parse errors, missing fields) are silently skipped with a bare except Exception: pass. This is intentional: a single corrupted overlay file must not prevent the rest of the overlay from loading. The UUID-mismatch case is the exception — it logs a structured warning at WARNING level with rule="C-01" so it can be found in audit tooling.

Caching and cross-process invalidation

load_overlay() caches its result in self._overlay_cache. The cache is valid as long as the monotonic counter in mutations/_cache_version matches the value recorded when the cache was last built (self._overlay_cache_sentinel_mtime). Every call to invalidate() bumps the counter via bump_version_counter(), which uses os.replace internally (same atomic-rename guarantee). Because the counter is stored on disk, other uvicorn worker processes see the change on their next load_overlay() call and discard their own warm caches. This replaces the old st_mtime comparison, which had whole-second resolution and could miss two invalidations in the same second.

Concrete Example

Suppose the base .eidos file contains this node:

{
  "id": "node-001",
  "label": "Engine Mount",
  "class": "Mechanical",
  "treeId": "tree-A",
  "properties": {"description": "Original description from base"}
}

A user updates the label and description at runtime. The system writes mutations/nodes/node-001.jsonld:

{
  "@context": {"eidos": "https://ontoteq.com/ns/eidos#"},
  "@type": "eidos:Node",
  "eidos:id": "node-001",
  "eidos:label": "Engine Mount (Rev B)",
  "eidos:nodeType": "Mechanical",
  "eidos:treeId": "tree-A",
  "eidos:properties": {"description": "Updated description after engineering review"},
  "eidos:updatedAt": "2026-06-23T08:00:00+00:00",
  "eidos:rev": 1
}

When eidos_loader._load() runs, it:

  1. Loads the base graph. nodes["node-001"] contains the original data.
  2. Calls load_overlay(). The overlay returns nodes = {"node-001": <normalized overlay entry>}.
  3. Passes both to merge_overlay(). Inside merge_overlay(), line 134: nodes.update(overlay["nodes"]) — the overlay entry for "node-001" overwrites the base entry.

The Effective Graph now contains:

{
    "id": "node-001",
    "label": "Engine Mount (Rev B)",
    "class": "Mechanical",
    "treeId": "tree-A",
    "properties": {"description": "Updated description after engineering review"},
    "modified": "2026-06-23T08:00:00+00:00",
    "rev": 1,
    "_overlay": True,
    # engineering and productType are preserved from base (see merge_overlay lines 130-133)
}

The base file is untouched. If mutations/nodes/node-001.jsonld were deleted (do not do this directly — use a node tombstone), the next load would show the original base entry again.

Tombstones

Why tombstones exist instead of deleting files

The overlay follows an additive invariant: files are never deleted from it. Instead, a separate tombstone file is written to signal deletion. This design has two motivations:

  1. Audit trail. The layer stack of mutations/nodes/{uuid}.jsonld + mutations/node_tombstones/{uuid}.jsonld tells the full history: "this entity existed, then was deleted." Deleting the overlay file would erase that evidence.
  2. Atomic semantics. Deleting a file and ensuring that no concurrent reader sees a half-deleted state is harder than writing a new file. write_atomic gives the same crash-safe guarantee for tombstones as for all other overlay writes.

write_node_tombstone — soft-deleting a node

def write_node_tombstone(
    self, node_id: str, deleted_by: str = "", audit_id: str = ""
) -> None:
Parameter Type Description
node_id str UUID of the node to soft-delete
deleted_by str Identity of the user requesting deletion
audit_id str Reference to the audit log entry for this deletion

Writes a eidos:NodeTombstone document to mutations/node_tombstones/{node_id}.jsonld via write_atomic. The node's overlay file at mutations/nodes/{node_id}.jsonld (if any) is intentionally left on disk. The operation is idempotent: re-writing the same tombstone is an overwrite with no duplicate-file error.

write_tombstone — soft-deleting a seeded relation

def write_tombstone(self, rel_uuid: str, created_by: str = "") -> None:
Parameter Type Description
rel_uuid str UUID of the seeded relation to soft-delete
created_by str Identity of the user requesting deletion

Writes a eidos:RelationTombstone document to mutations/tombstones/{rel_uuid}.jsonld via write_atomic. Only base (seeded) relations use this mechanism. Overlay relations (those in mutations/relations/) are deleted by removing their file — but again, do not do this manually; use the appropriate API endpoint which manages invalidation.

How merge_overlay handles tombstones

merge_overlay() in graph_merge.py applies tombstones in a strict order to prevent re-introduction of deleted entities:

Step 1 (BL-FE-037): Before any merge work, node_tombstones are used to filter overlay["nodes"]. If a node has both an overlay file and a node tombstone, the overlay entry is removed from the working copy of the overlay dict. This prevents the subsequent nodes.update() call from silently resurrecting a deleted node. The original _overlay_cache dict is not mutated; a shallow copy is made (overlay = dict(overlay)) so the cache remains valid.

Step 2: Relation tombstones are applied to the base relations list and to rels_by_node. The filter keeps overlay relations (identified by r.get("_overlay")) because overlay relations have a separate deletion path and are never in the tombstones set.

Step 3 (BL-DM-001): Node tombstones are applied to the assembled graph. For each tombstoned UUID: - The node is removed from nodes. - Every relation where from or to equals the tombstoned UUID is removed from relations and rels_by_node. - The node is removed from parent_of and children_of. - The node's ref path is removed from ref_index and ref_by_uid via key_derivation.unindex_ref(), which is careful not to drop a colliding ref entry that belongs to another node (BL-DM-010). - The tombstoned UUID is removed from the children list of its parent in children_of.

Step 4: The remaining overlay nodes and relations are merged into the base graph. nodes.update(overlay["nodes"]) applies the overlay-wins rule. Structural indexes (parent_of, children_of, trees[i]["children"], ref_index, ref_by_uid, rels_by_node) are rebuilt for all overlay nodes in two passes so that children of overlay parents resolve correctly regardless of the order files were scanned.

After merge_overlay() returns, base_graph["relations"] has been reassigned to the filtered and extended list. Callers must read from base_graph["relations"] after the call, not from any reference they captured before it.

Invariants

The two invariants named in overlay.yaml map directly to code:

  • overlay-wins-on-conflict: enforced by nodes.update(overlay["nodes"]) in merge_overlay() (line 134 of graph_merge.py). Because dict.update overwrites existing keys, any base entry for a UUID that appears in the overlay is replaced unconditionally.
  • atomic-write: enforced by write_atomic() in DomainStore (lines 224–238 of mutation_store.py). Every overlay file write — nodes, relations, tombstones, relation tombstones — goes through this function. There is no code path that writes overlay JSON without fsync + os.replace.

A developer who bypasses write_atomic — for example, by calling path.write_text() directly — risks leaving a partially written file on disk if the process crashes mid-write. load_overlay() would then parse malformed JSON, log nothing (the except Exception: pass in the scan loop silently skips it), and the entity would silently disappear from the Effective Graph until the file was manually repaired or overwritten. This is the primary blast radius of violating the atomic-write invariant.