Skip to content

Mutation Store

Verified — minor divergences from the code — 2 finding(s) · 19d408be · 2026-08-25

All documented canonical operations exist with matching signatures and behaviour in backend/persistence/mutation_store.py (write_atomic, load_overlay, invalidate, write_tombstone, write_node_tombstone, load_ownership, write_ownership, cleanup_stale_previews, get_effective_graph all verified against DomainStore methods and module-level wrappers). Directory layout, field-name mapping table, write_lock(), ensure_dirs() subdirectory list, and cross-process cache-version mechanism all match the code exactly. Two minor omissions found.

Divergences from the code — details
Sev Where Doc says Code does Evidence
LOW canonical_operations: write_node_tombstone(node_id, deleted_by) -> None canonical_operations lists write_node_tombstone(node_id, deleted_by) -> None The actual function signature is write_node_tombstone(self, node_id: str, deleted_by: str = "", audit_id: str = "") -> None — it takes a third audit_id parameter (used to populate eidos:auditRef), which the YAML signature omits. The .md doc page itself documents audit_id correctly. backend/persistence/mutation_store.py:428-430
LOW canonical_operations The doc's canonical_operations / Key Functions list mutation_store's public surface (write_atomic, get_effective_graph, load_overlay, invalidate, write_node_tombstone, write_tombstone, load_ownership, write_ownership, cleanup_stale_previews) as the operations this engine owns. The module also exports remove_node_tombstone(node_id) -> bool (the un-delete/rollback counterpart to write_node_tombstone, referenced by its own docstring as 'the sanctioned rollback path'), which appears in neither the YAML canonical_operations list nor the compiled doc's Key Functions section. backend/persistence/mutation_store.py:452-470

Layer: persistence

mutation_store — Overlay File System Engine

Purpose and Responsibilities

mutation_store.py is the overlay file I/O and cache management layer for EIDOS Explorer. It owns:

  • Atomic writes of all overlay documents to the mutations/ directory tree (CCR Rule 5: temp → fsync → rename).
  • Overlay scanning — reading nodes/*.jsonld, relations/*.jsonld, tombstones/*.jsonld, and node_tombstones/*.jsonld back into an in-memory cache dict.
  • Cache invalidation — clearing the per-process cache and advancing a cross-process monotonic counter (_cache_version) so other uvicorn workers rebuild their caches on the next load.
  • Preview persistence — writing and consuming single-use pending command files via the shared filesystem so any worker can commit a preview created by a different worker.
  • Idempotency record persistence — write-once records keyed by an idempotency key string.
  • Tombstone writes — soft-deletion markers for seeded relations (tombstones/) and overlay nodes (node_tombstones/).
  • Ownership persistence — the shared ownership.jsonld file, protected by a cross-process advisory file lock.

mutation_store does not own:

  • Document construction (that is mutation_engine.build_overlay_*).
  • UUID derivation (that is key_derivation.py).
  • The effective-graph merge algorithm (that is graph_merge.merge_overlay, called from eidos_loader._load).
  • Business-level validation or mutation orchestration (that is graph_crud_engine.py).
  • Subtree locking (callers hold subtree locks before calling write functions).

All public functions at module level are backward-compatible wrappers that delegate to a process-wide singleton DomainStore. Tests and multi-tenant code can instantiate DomainStore directly with a custom mutations_dir.


Directory Layout

mutations/
  nodes/{uuid}.jsonld          — one file per created/updated overlay node
  relations/{uuid}.jsonld      — one file per created overlay relation
  tombstones/{uuid}.jsonld     — soft-delete marker for a seeded relation
  node_tombstones/{uuid}.jsonld — soft-delete marker for a node (BL-DM-001)
  ownership.jsonld             — all ownership assignments (@graph array)
  idempotency/{key}.jsonld     — committed idempotency record per key
  pending/{uuid}.json          — single-use preview command (worker-shared)
  snapshots/{uuid}.jsonld      — pre-mutation snapshots (written by snapshot_service)
  audit/{YYYY-MM-DD}.jsonld    — append-only audit log per day
  _cache_version               — monotonic integer; cross-process invalidation sentinel
  _write.lock                  — advisory fcntl lock file (cross-process write serialisation)

The MUTATIONS_DIR constant is resolved at import time through the canonical config register in backend/infra/config.py — the single owner of the MUTATIONS_DIR environment read and its one default, backend/mutations (hardening #518: the bootstrap and drift paths previously carried diverging defaults).

from infra.config import resolve_mutations_dir
MUTATIONS_DIR = resolve_mutations_dir()  # env MUTATIONS_DIR, default backend/mutations

Key Functions

read_version_counter(path) -> int

def read_version_counter(path) -> int:

Parameters

Parameter Type Meaning
path str \| Path Path to the sentinel file whose integer contents are read.

Returns int — the value stored in the file, or 0 if the file is absent, empty, or unparseable.

Errors — none raised; all FileNotFoundError, ValueError, and OSError are caught and silently mapped to 0.

The design intent (quoted from the docstring): "On any doubt we return a value that will differ from a warm cache's recorded version, so the caller rebuilds rather than risk serving stale data (fail toward freshness)."

This is a generalised primitive. Both the overlay cache sentinel (_cache_version) and any rulebook sentinel use this single function, parameterised by path (Contract 02.01 R11).

# Check the overlay cache sentinel
version = read_version_counter(MUTATIONS_DIR / "_cache_version")

The backward-compatible alias _read_cache_version is also exported for callers that imported it under that name.


bump_version_counter(sentinel_path) -> int

def bump_version_counter(sentinel_path) -> int:

Parameters

Parameter Type Meaning
sentinel_path str \| Path Path to the counter file to atomically increment.

Returns int — the new counter value after incrementing.

ErrorsOSError from the underlying filesystem can propagate; callers that cannot tolerate this must wrap it in try/except (the internal _bump_cache_version does exactly this).

Atomicity guarantee: the read-modify-write is serialised by a per-sentinel advisory file lock (<sentinel>.lock) so two concurrent workers cannot lose each other's bumps. The new integer is published with os.replace so concurrent readers always see a complete old-or-new integer — never a torn write.

new_version = bump_version_counter(MUTATIONS_DIR / "_cache_version")

write_atomic(data: dict, path: Path) -> None

def write_atomic(data: dict, path: Path) -> None:

Parameters

Parameter Type Meaning
data dict Any JSON-serialisable dict that will be written as indented UTF-8 JSON.
path Path Absolute destination path. The parent directory is created automatically.

Returns None.

Errors — any Exception from json.dump, os.fsync, or os.replace propagates to the caller; the temp file is deleted before re-raising so no partial write is ever left at path.

Implements CCR Rule 5: write to a temp file in the same directory (tempfile.mkstemp), fsync the file descriptor, then os.replace to the final path. os.replace is atomic on both POSIX and Windows for same-filesystem renames.

from mutation_store import write_atomic
from pathlib import Path

write_atomic(
    {
        "@context": {"eidos": "https://ontoteq.com/ns/eidos#"},
        "@type": "eidos:OverlayNode",
        "eidos:id": "3f2504e0-...",
    },
    MUTATIONS_DIR / "nodes" / "3f2504e0-....jsonld",
)

load_overlay() -> dict

def load_overlay() -> dict:

Parameters — none.

Returns dict with the structure:

{
    "nodes":           dict[str, dict],  # uuid → normalised flat node dict
    "relations":       list[dict],       # normalised flat relation dicts
    "tombstones":      set[str],         # relation UUIDs that are soft-deleted
    "node_tombstones": set[str],         # node UUIDs that are soft-deleted
}

Errors — individual malformed files are silently skipped (the except Exception: pass guard in each scan loop ensures one bad file never prevents the rest from loading). File-level integrity violations (C-01: eidos:id in the JSON does not match the filename stem) are logged as WARNING with load_overlay_uuid_mismatch or load_overlay_rel_uuid_mismatch and the offending file is skipped.

The result is cached in _overlay_cache. Cache validity is checked on every call by comparing the current value of _cache_version against the value stored when the cache was last populated. If they differ, the cache is discarded and rebuilt.

overlay = load_overlay()
node = overlay["nodes"].get(some_uuid)
live_relations = [
    r for r in overlay["relations"]
    if r["source"] not in overlay["tombstones"]
]

Do not mutate the returned dict. The cache holds the same object reference. Mutating it corrupts the cache for all subsequent callers in the same worker process.


invalidate(*, rel_uuid: str | None = None) -> None

def invalidate(*, rel_uuid: str | None = None) -> None:

Parameters

Parameter Type Default Meaning
rel_uuid str \| None None When provided, enables a granular cache patch — only that relation is removed from eidos_loader._cache in place, without clearing the whole effective graph. Pass None (the default) for a full cache eviction.

Returns None.

Errors — post-invalidate hooks that raise are caught and logged as post_invalidate_hook_error; they do not interrupt invalidation.

This is the only correct way to signal that a mutation has been committed. It does three things atomically under _write_lock:

  1. Clears _overlay_cache (so load_overlay rebuilds on next call).
  2. Bumps the cross-process _cache_version counter (so other uvicorn workers rebuild too).
  3. Fires all registered post-invalidate hooks.
# After writing a relation file:
write_atomic(rel_doc, MUTATIONS_DIR / "relations" / f"{rel_uuid}.jsonld")
invalidate()  # full eviction

# After deleting a single overlay relation:
write_atomic(rel_doc, ...)   # not applicable — you deleted the file
invalidate(rel_uuid=rel_uuid)  # granular patch

write_tombstone(rel_uuid: str, created_by: str = "") -> None

def write_tombstone(rel_uuid: str, created_by: str = "") -> None:

Parameters

Parameter Type Meaning
rel_uuid str UUID of the seeded relation being soft-deleted. Becomes the filename and eidos:id.
created_by str Email or identifier of the user performing the deletion. Defaults to empty string.

Returns None. ErrorsOSError / json.JSONEncodeError can propagate from write_atomic.

Writes tombstones/{rel_uuid}.jsonld. After calling this you must call invalidate() so the tombstone is picked up by the next load_overlay scan.

The tombstone UUID is derived by eidos_loader.seeded_rel_id(from_id, to_id, predicate)uuid5(NAMESPACE_SEEDED_RELATION, f"{from_id}::{predicate}::{to_id}"). The namespace is frozen — changing it invalidates all existing tombstone files.

write_tombstone(rel_uuid="6ba7b810-...", created_by="alice@example.com")
invalidate(rel_uuid="6ba7b810-...")

write_node_tombstone(node_id: str, deleted_by: str = "", audit_id: str = "") -> None

def write_node_tombstone(
    node_id: str, deleted_by: str = "", audit_id: str = ""
) -> None:

Parameters

Parameter Type Meaning
node_id str UUID of the node being soft-deleted.
deleted_by str Email/identifier of the actor.
audit_id str Audit event UUID; stored as eidos:auditRef.

Returns None. Errors — same as write_atomic.

Implements BL-DM-001: additive soft-delete. The node's overlay file in nodes/ is intentionally left on disk. The presence of both overlay file and tombstone represents "existed, then deleted". The operation is idempotent — re-writing the same tombstone overwrites the file without error.

write_node_tombstone(
    node_id="3f2504e0-...",
    deleted_by="alice@example.com",
    audit_id="audit-uuid-here",
)
invalidate()

get_idempotency_record(idem_key: str) -> dict | None

def get_idempotency_record(idem_key: str) -> dict | None:

Parameters

Parameter Type Meaning
idem_key str The idempotency key to look up. Maps to idempotency/{idem_key}.jsonld.

Returns dict (the stored record) if the key has been previously committed, None otherwise. Returns None on parse errors (treated as cache miss).

record = get_idempotency_record(idem_key)
if record is not None:
    # Already committed — return stored entity_id
    return record["eidos:entityId"]

write_idempotency(idem_key, entity_id, result, created_at, payload_hash) -> None

def write_idempotency(
    idem_key: str,
    entity_id: str,
    result: str,
    created_at: str,
    payload_hash: str,
) -> None:

Parameters

Parameter Type Meaning
idem_key str The idempotency key (e.g. uuid4 or sha256-based). Becomes the filename.
entity_id str The UUID of the entity created or updated by this mutation.
result str Typically "committed".
created_at str ISO 8601 UTC timestamp.
payload_hash str SHA-256 hex of the canonical command payload. Used to detect payload changes on replay.

Returns None. Errors — propagates from write_atomic.

Always call this inside the subtree lock, after the overlay file has been written and before calling invalidate(). Calling it outside the lock can create a race where two workers both pass the idempotency check and both write the same overlay file.


write_preview(preview_id: str, cmd: dict) -> None

def write_preview(preview_id: str, cmd: dict) -> None:

Parameters

Parameter Type Meaning
preview_id str UUID identifying the preview (typically uuid4). Becomes the filename.
cmd dict The full command dict to be committed later.

Writes pending/{preview_id}.json. The file is shared across all uvicorn workers via the filesystem.


consume_preview(preview_id: str) -> dict | None

def consume_preview(preview_id: str) -> dict | None:

Parameters

Parameter Type Meaning
preview_id str UUID of the preview to consume and delete.

Returns the stored cmd dict, or None if the file does not exist or is corrupted. The file is deleted atomically under _write_lock — this is a single-use operation (consistency contract C-05).

cmd = consume_preview(preview_id)
if cmd is None:
    raise HTTPException(400, "PREVIEW_REQUIRED")

load_ownership() -> list

def load_ownership() -> list:

Returns list — the @graph array from ownership.jsonld, or [] if the file is absent or unreadable.


write_ownership(graph: list) -> None

def write_ownership(graph: list) -> None:

Parameters

Parameter Type Meaning
graph list The complete ownership @graph array to persist. Always overwrites the full file.

Combines write_atomic (CCR Rule 5) with a cross-process advisory file lock on ownership.lock. On Windows the advisory lock is skipped; the threading _write_lock covers the intra-process case.


write_lock() (async context manager)

@contextlib.asynccontextmanager
async def write_lock():

Exclusive write lock covering the full read-modify-write of a mutation. Provides two levels of serialisation:

  1. In-process: a per-event-loop asyncio.Lock keyed by id(loop) so it is correct under both the single production loop and per-call test loops.
  2. Cross-process: an fcntl advisory lock on MUTATIONS_DIR/_write.lock (no-op on Windows).
async with write_lock():
    overlay = load_overlay()           # read
    # ... compute new document ...
    write_atomic(doc, target_path)     # write
    invalidate()

ensure_dirs() -> None

def ensure_dirs() -> None:

Creates all required subdirectories under mutations_dir if they do not exist: nodes, relations, snapshots, audit, idempotency, tombstones, node_tombstones, pending.

Call this at server startup before any reads or writes.


cleanup_stale_previews(ttl_seconds: int = 86400) -> int

def cleanup_stale_previews(ttl_seconds: int = 86400) -> int:

Parameters

Parameter Type Default Meaning
ttl_seconds int 86400 (24 hours) Files older than this many seconds are deleted.

Returns int — count of files deleted.

Call this at startup after ensure_dirs(). Removes pending/*.json files left by abandoned preview dialogs or worker crashes. Per-file errors are logged as warnings and do not abort the sweep.


register_post_invalidate_hook(fn) -> None

def register_post_invalidate_hook(fn) -> None:

Register a callable invoked after every cache invalidation. Hooks run under _write_lock and are called in registration order. A hook that raises is logged and skipped — it does not block subsequent hooks or the invalidation itself.

Hooks cannot be deregistered. Any object captured in a hook closure is pinned for the worker's lifetime. Register hooks only at startup.


Data Structures

Overlay Cache Dict (returned by load_overlay)

{
    "nodes": {
        "3f2504e0-4f89-11d3-9a0c-0305e82c3301": {
            "id":          "3f2504e0-...",
            "label":       "Node Label",
            "class":       "SystemNode",      # maps from eidos:nodeType
            "treeId":      "TREE-001",
            "productType": "Widget",
            "parent":      "parent-uuid-...", # used for parent_of index in merge
            "properties":  {"description": "...", "note": "..."},
            "created":     "2026-05-13T10:00:00+00:00",
            "modified":    "2026-05-13T11:00:00+00:00",  # from eidos:updatedAt
            "createdBy":   "alice@example.com",
            "rev":         3,                 # monotonic version token; 0 on legacy
            "_overlay":    True,              # internal marker; do not persist
            # "engineering" key present only when eidos:engineering was in file
        },
    },
    "relations": [
        {
            "from":             "source-node-uuid",
            "to":               "target-node-uuid",
            "predicate":        "hasComponent",
            "source":           "relation-uuid",   # identity key (confusingly named)
            "inherit_source":   False,
            "_overlay":         True,
            # Optional cross-domain fields:
            "external_domain":        "location",
            "external_uuid":          "target-node-uuid",
            "external_ref":           "+FFL.CTP10.TA37",
            "external_source_domain": "engineering",
            # Optional unresolved internal ref:
            "unresolved_ref":         "+FFL.CTP10",
        },
    ],
    "tombstones":      {"relation-uuid-1", "relation-uuid-2"},  # set of str
    "node_tombstones": {"node-uuid-1"},                         # set of str
}

Important field name mapping — the _normalize_node function renames keys when converting from JSON-LD to the flat dict. The most confusing one:

JSON-LD key Flat dict key Notes
eidos:nodeType class Name change — not type or nodeType
eidos:updatedAt modified Fixed in audit Phase 3 (SL-04)
eidos:id source On relations — the relation UUID is stored as source, not id

Idempotency Record (as stored in idempotency/{key}.jsonld)

{
    "@context":          {"eidos": "https://ontoteq.com/ns/eidos#"},
    "@type":             "eidos:IdempotencyRecord",
    "eidos:key":         "idem-key-string",
    "eidos:entityId":    "entity-uuid",
    "eidos:result":      "committed",
    "eidos:createdAt":   "2026-05-13T10:00:00+00:00",
    "eidos:payloadHash": "sha256-hex",
}

Relation Tombstone (as stored in tombstones/{uuid}.jsonld)

{
    "@context":        {"eidos": "https://ontoteq.com/ns/eidos#"},
    "@type":           "eidos:RelationTombstone",
    "eidos:id":        "relation-uuid",
    "eidos:created":   "2026-05-13T10:00:00+00:00",
    "eidos:createdBy": "alice@example.com",
}

Node Tombstone (as stored in node_tombstones/{uuid}.jsonld)

{
    "@context":        {"eidos": "https://ontoteq.com/ns/eidos#"},
    "@type":           "eidos:NodeTombstone",
    "eidos:id":        "node-uuid",
    "eidos:deletedAt": "2026-05-13T10:00:00+00:00",
    "eidos:deletedBy": "alice@example.com",
    "eidos:auditRef":  "urn:eidos:audit:audit-event-uuid",
}

DomainStore Instance Attributes

Attribute Type Meaning
mutations_dir Path Root of the mutations directory tree for this domain.
_overlay_cache dict \| None Cached result of the last load_overlay scan; None when invalid.
_overlay_cache_sentinel_mtime float Version counter value at the time the cache was populated. Used to detect cross-process invalidation.
_write_lock threading.Lock Intra-process lock protecting all writes and cache mutations.
_version int Monotonically increasing counter bumped on every invalidate() call. Readable via get_version().
_cache_sentinel Path Path to mutations_dir/_cache_version.
_post_invalidate_hooks list Callables fired after each invalidation.
_ownership_file Path Path to mutations_dir/ownership.jsonld.

Design Constraints

1. Always call ensure_dirs() at startup before any read or write. The individual write functions create parent directories on demand, but load_overlay silently skips non-existent directories. If the directory tree is not present, an empty overlay will be returned even though files exist elsewhere.

2. Always call invalidate() after every mutation commit. load_overlay caches its result. If you write an overlay file and do not call invalidate(), the current worker serves stale data until something else clears the cache. Other workers will never see the change.

3. Pass rel_uuid to invalidate() only when a single overlay relation was deleted. Passing rel_uuid triggers a granular patch (eidos_loader.patch_remove_relation) rather than a full graph reload. Passing a node UUID, a non-existent UUID, or any string that is not a just-deleted relation UUID will silently leave the effective graph in an inconsistent state.

4. Never mutate the dict returned by load_overlay(). The cache holds the exact object. Mutating a node dict, the relations list, or either tombstone set corrupts the cache for all subsequent callers in the same process. Always take a copy if you need to modify.

5. Write the idempotency record inside the subtree lock, after the overlay file. The correct order is: acquire lock → write overlay file → write idempotency record → call invalidate(). Writing the idempotency record before the overlay file means a crash between the two steps leaves a committed key with no entity file. Writing it after invalidate() means another request can see the new entity but find no idempotency record and attempt a duplicate write.

6. Call cleanup_stale_previews() at startup, not on every request. The function performs a directory scan. Calling it per-request is expensive. It is safe to call multiple times concurrently, but there is no benefit in doing so.

7. The _write_lock threading lock inside DomainStore does not replace write_lock() for multi-step mutations. _write_lock protects individual atomic writes and invalidate() calls. The async context manager write_lock() is needed when you must hold the lock across the entire read-modify-write sequence of a mutation to prevent interleaving with other concurrent mutations.

8. Do not set MUTATIONS_DIR after module import. MUTATIONS_DIR and _CACHE_SENTINEL are module-level constants resolved at import time. Changing the environment variable after the module loads has no effect. The default DomainStore singleton is created on first use; to use a different directory, instantiate DomainStore directly.


Common Pitfalls

Pitfall 1: Forgetting to call invalidate() after writing. This is the single most common mistake. The overlay cache is not time-based — it only clears when invalidate() is called or when another worker bumps _cache_version. A write without a subsequent invalidate() will be invisible to load_overlay() callers in the same process and to all other workers indefinitely.

Pitfall 2: Assuming load_overlay() returns a fresh scan on every call. It does not. Repeated calls within the same request typically return the same cached object. The cache is invalidated only by explicit invalidate() calls or by detecting a changed _cache_version counter.

Pitfall 3: Confusing the relation source field with the source node UUID. In the normalised relation flat dict, source holds the relation's own UUID (from eidos:id). The source node UUID is in from. This naming is a historical accident documented in the audit; do not rename it without updating all callers.

Pitfall 4: Using the _CACHE_SENTINEL path directly instead of going through invalidate(). eidos_loader accesses mutation_store._CACHE_SENTINEL directly for reading. This is an allowed backward-compat use. Callers must never touch or write this path directly — always use invalidate(), which delegates to _bump_cache_version() and also handles the in-process cache and hook callbacks.

Pitfall 5: Using os.stat().st_mtime to detect changes. The old implementation compared st_mtime, which has whole-second resolution on many filesystems. Two invalidations in the same second produced the same mtime, and workers served stale caches. The current implementation uses the monotonic integer counter — every invalidate() call advances it regardless of timing. Do not revert to mtime-based comparisons.

Pitfall 6: Not handling consume_preview returning None. consume_preview returns None both when the file does not exist and when the file is corrupted. On the happy path a missing preview means the user never called the preview endpoint, or another worker already consumed it (single-use). Always check for None and return an appropriate error (HTTP 400 PREVIEW_REQUIRED).

Pitfall 7: Placing files manually in nodes/ or relations/ with a filename that does not match the eidos:id field inside. Consistency contract C-01 requires that the filename stem equals eidos:id. If they differ, load_overlay logs a WARNING (load_overlay_uuid_mismatch / load_overlay_rel_uuid_mismatch) and skips the file entirely. The node or relation will be invisible to the graph. This is the expected behaviour for corrupted files, but it means a manually-placed file with a typo in either the filename or the eidos:id field will silently disappear from the overlay.

Pitfall 8: Registering post-invalidate hooks after startup. Hooks registered via register_post_invalidate_hook cannot be deregistered and run on every subsequent invalidate() call. If a hook captures a large object in its closure or is registered inside a request handler (so it accumulates per-request), memory will grow without bound for the lifetime of the worker.

Pitfall 9: Assuming tombstones delete the overlay node file. write_node_tombstone (BL-DM-001) is additive. It writes a tombstone file and leaves the node's nodes/{uuid}.jsonld on disk by design. The overlay file and tombstone together represent the full history. Code that checks only overlay["nodes"] without also checking overlay["node_tombstones"] will incorrectly treat a deleted node as live.

Pitfall 10: Calling get_effective_graph() inside a mutation's write sequence. get_effective_graph() delegates to eidos_loader._load(), which acquires no lock and reads the (potentially warm) graph cache. Calling it after write_atomic but before invalidate() may return a graph that does not include the just-written file. Always read the effective graph before performing writes, not after.