Cache Coherence

Verified against the code · 4996d1ceb770fb4545edebc5fd140792bfb23830 · 2026-10-20

956 D1b-4 (2026-10-20): 5 finding(s) fixed — the browser caches (EidosNodeStore._nodes, EidosTree._childrenOf, labelMap) are unbounded plain objects with no LRU/eviction; current_graph() is the sole loader entry (_load removed in #868 Slice 8); mutation_store pins invalidate :293, bump_version_counter :108, write_lock :166 (decorator :165, mkdirs MUTATIONS_DIR before the flock). Re-verified against main 4996d1ce. #935 S1 (2026-10-21): line pin moved (_cache at eidos_loader.py:40). Checked against archon/issue-935. #935 S7 (2026-10-22): a rebuild reads the overlay only (no .eidos file since S5a) and the tree declaration from the runtime domains.json. Checked against eidos_loader.py at a11d2636.

flowchart LR
%% generated_from: cache-coherence
%% verified_at_commit: c6a749dd8c609e43f904cb8e43ab24d867f4b6f4
%% description: Cache Coherence lifecycle
%% legend: owoc=overlay-wins-on-conflict
  subgraph frontend
    browser-cache-lookup["Browser Cache Lookup"]
    populate-browser-cache["Populate Browser Cache"]
  end
  subgraph domain
    effective-graph-lookup["Effective Graph Lookup"]
  end
  subgraph persistence
    effective-graph-rebuild["Effective Graph Rebuild · owoc"]
  end
  browser-cache-lookup --> effective-graph-lookup
  effective-graph-lookup --> effective-graph-rebuild
  effective-graph-rebuild --> populate-browser-cache

Overview — Why Caching EffectiveGraph

Every time a caller needs the current state of the EIDOS knowledge graph it calls get_effective_graph() in backend/persistence/mutation_store.py, which delegates immediately to eidos_loader.current_graph() (the sole loader entry point; the old _load() wrapper was removed entirely in #868 Slice 8). That function performs a substantial amount of work:

1. Declares the domain's tree from the overlay root and the runtime domains.json (_overlay_only_tree). Since #935 S5a no data.eidos archive is opened; a legacy one is only checked for existence.
2. Starts from an empty base: nodes, relations, trees, parent_of, children_of, ref_index and rels_by_node are filled by the merge.
3. Calls mutation_store.load_overlay(), which in turn scans every file in mutations/nodes/*.jsonld, mutations/relations/*.jsonld, mutations/tombstones/*.jsonld, and mutations/node_tombstones/*.jsonld.
4. Passes the (empty) base graph and the overlay result to graph_merge.merge_overlay(), which resolves conflicts (overlay wins on conflict — the overlay-wins-on-conflict invariant from the workflow YAML).
5. Builds the full-text search corpus by traversing every merged node's attributes, properties, and engineering entries — field-filtered through the canonical content-field denylist in domain/search_index.py (_CORPUS_DENIED_FIELDS / _flatten_indexable), so document body, OCR, webpage and attachment content never enters the corpus (BL-FE-080; hardening #511).

Without caching, every HTTP request that touches any node, relation, or tree data would pay the full cost of steps 1–6. On a non-trivial EIDOS model this takes tens to hundreds of milliseconds. Under normal usage — a team of engineers navigating the graph, running searches, inspecting relations — dozens of requests per second would each trigger a full overlay directory scan and merge.

The strategy is cache-then-invalidate: the result of current_graph() is stored in the module-level _cache dict inside eidos_loader.py and returned directly on all subsequent calls until something invalidates it. Invalidation is triggered only when a write operation modifies the overlay on disk. This reduces the common case (a read against an unchanged graph) to a dict truthiness check and a sentinel comparison — essentially free.

The workflow as a whole enforces a single end-to-end invariant: no caller ever receives a graph that is older than the most recent committed write, across all uvicorn worker processes. The four stages described below are how that invariant is maintained.

---

Stage 1: Write Received

What triggers invalidation

Every mutation that changes the on-disk overlay must eventually call mutation_store.invalidate(). In practice this includes:

• Creating an overlay node (writes mutations/nodes/{uuid}.jsonld)
• Updating an overlay node (overwrites the same file)
• Creating an overlay relation (writes mutations/relations/{uuid}.jsonld)
• Deleting an overlay relation (deletes the file, then writes a tombstone or not depending on whether the relation was seeded or overlay-native)
• Soft-deleting a seeded relation (writes mutations/tombstones/{uuid}.jsonld)
• Soft-deleting a node (writes mutations/node_tombstones/{uuid}.jsonld)

All of these go through mutation_store.write_atomic() first, then call invalidate(). The file is on disk before the cache is invalidated — this ordering ensures that when the cache is rebuilt it will find the new file.

Exactly what "invalidate" does

DomainStore.invalidate() (in backend/persistence/mutation_store.py, line 293) acquires self._write_lock (a threading.Lock) and then calls _clear_caches():

def invalidate(self, *, rel_uuid: str | None = None) -> None:
    with self._write_lock:
        self._clear_caches(rel_uuid=rel_uuid)
        for hook in self._post_invalidate_hooks:
            try:
                hook()
            except Exception as _exc:
                _log.warning("post_invalidate_hook_error", hook=repr(hook), exc=str(_exc))

Inside _clear_caches(), three things happen atomically under that lock:

1. self._overlay_cache = None — the in-process overlay cache held by DomainStore is dropped. The next call to load_overlay() will perform a full directory scan.
2. self._version += 1 — the in-process monotonic version counter is advanced. This counter is readable via get_version() and used by callers that want to detect "has anything changed since I last checked?"
3. self._bump_cache_version() is called — this advances the cross-process sentinel file (see Stage 2).

Additionally, _clear_caches directly reaches into eidos_loader._cache and calls either .clear() (full eviction) or eidos_loader.patch_remove_relation(rel_uuid) (granular patch when a single relation was deleted). Both the overlay cache and the effective-graph cache in eidos_loader are cleared in the same lock acquisition. Any exception from _bump_cache_version or from the eidos_loader manipulation is silenced — neither can block the mutation commit.

---

Stage 2: Version Bump

bump_version_counter: the sentinel file, atomic increment, why it's needed

The _cache dict in eidos_loader.py is module-level state. Each uvicorn worker process has its own copy. When worker A commits a mutation and calls invalidate(), that clears worker A's in-process cache. Worker B's eidos_loader._cache is unaffected — it is in a different process with separate memory.

To signal worker B (and any other worker), the system maintains a cross-process monotonic counter stored in the file mutations/_cache_version. The file contains a single ASCII integer.

bump_version_counter(sentinel_path) (line 108 of mutation_store.py) performs an atomic read-modify-write on that file:

def bump_version_counter(sentinel_path) -> int:
    sentinel = Path(sentinel_path)
    sentinel.parent.mkdir(parents=True, exist_ok=True)
    lock_fh = _acquire_file_lock(sentinel.with_name(sentinel.name + ".lock"))
    try:
        new = read_version_counter(sentinel) + 1
        tmp = sentinel.with_name(f"{sentinel.name}.{new}.tmp")
        tmp.write_text(str(new))
        os.replace(tmp, sentinel)
        return new
    finally:
        _release_file_lock(lock_fh)

The sequence is:

1. Acquire an advisory fcntl file lock on _cache_version.lock (on Linux/macOS). This serialises concurrent bumps from multiple workers so no increment is lost.
2. Read the current integer from _cache_version (returns 0 if the file is absent or unparseable).
3. Write current + 1 to a temp file (_cache_version.{new}.tmp).
4. os.replace() the temp file over _cache_version. This is atomic on both POSIX and Windows for same-filesystem renames — a concurrent reader always sees either the old or the new integer, never a partial write.
5. Release the advisory lock.

The reason this uses an integer counter rather than a simple touch (updating mtime) is stated explicitly in the code comments: st_mtime has whole-second resolution on many filesystems. Two invalidations within the same second would produce the same mtime, and other workers would incorrectly treat their warm cache as still valid. The monotonic counter advances on every invalidation regardless of wall-clock timing.

How the version counter enables external cache checks

eidos_loader.current_graph() calls _sentinel_mtime() (the name is a historical artifact — it returns the integer counter, not an mtime):

def _sentinel_mtime() -> float:
    try:
        from persistence import mutation_store
        return mutation_store._read_cache_version(mutation_store._CACHE_SENTINEL)
    except Exception:
        return 0.0

At the end of a successful load, current_graph() stores the sentinel value it observed into _cache_sentinel_mtime. On the next call to current_graph() it compares the live sentinel value against the stored one. If they differ, the cache is stale and is cleared before rebuilding. If they are equal, the warm cache is returned immediately.

mutation_store._CACHE_SENTINEL is a module-level constant: MUTATIONS_DIR / "_cache_version". It is set once at import time and never changes. _read_cache_version is the backward-compatible alias for read_version_counter.

---

Stage 3: Cache Miss

What happens on the next get_effective_graph call after invalidation

DomainStore.get_effective_graph() calls eidos_loader.current_graph() directly:

def get_effective_graph(self) -> dict:
    from persistence import eidos_loader
    return eidos_loader.current_graph()

At the top of current_graph():

def current_graph() -> dict:
    global _cache_sentinel_mtime
    if _cache:
        if _sentinel_mtime() == _cache_sentinel_mtime:
            return _cache
        _cache.clear()  # stale — rebuild below

The _cache attribute: what it stores, how it's checked

_cache is a module-level plain dict defined at line 40 of eidos_loader.py:

_cache: dict = {}
_cache_sentinel_mtime: float = 0.0

The truthiness check if _cache: is True when the dict is non-empty (i.e., a warm cache exists). After invalidation, _clear_caches calls eidos_loader._cache.clear(), which empties the dict. On the next current_graph() call, if _cache: evaluates to False — the sentinel comparison is skipped entirely, and the full rebuild proceeds unconditionally.

This two-level check matters: a cold cache (empty dict) skips even reading the sentinel file from disk. A warm cache with a stale sentinel (_sentinel_mtime() != _cache_sentinel_mtime) calls _cache.clear() before rebuilding. In both paths, the code falls through to the overlay load and merge sequence below.

---

Stage 4: Cache Rebuilt

The full call chain from get_effective_graph to load_overlay to merge_overlay

Starting from a user HTTP request reaching any endpoint that reads graph data:

HTTP request
  → eidos_loader.get_nodes() / get_relations() / etc.
  → eidos_loader.current_graph()
      → _overlay_only_tree() — overlay root + tree declaration (#935 S5a: no .eidos read)
      → mutation_store.load_overlay()
          → DomainStore.load_overlay()
              → read _cache_version sentinel
              → if overlay cache is warm and sentinel unchanged: return cached overlay
              → else: scan mutations/nodes/*.jsonld, mutations/relations/*.jsonld,
                       mutations/tombstones/*.jsonld, mutations/node_tombstones/*.jsonld
              → store result in self._overlay_cache
              → store current sentinel value in self._overlay_cache_sentinel_mtime
              → return overlay dict
      → graph_merge.merge_overlay(_base, overlay)
          → apply overlay nodes (overlay wins on conflict per invariant)
          → filter tombstoned relations
          → extend ref_index with overlay node refs
      → _build_search_corpus(nodes, relations)
      → _cache.update({nodes, relations, trees, parent_of, ...})
      → _cache_sentinel_mtime = _sentinel_mtime()
      → return _cache

What is cached and what is not

Cached (stored in eidos_loader._cache after a successful current_graph()):

• nodes — the fully merged dict of {uuid: node_dict} including both base and overlay nodes, with node_tombstoned entries absent.
• relations — the merged list of relation dicts with tombstoned relations removed.
• trees, parent_of, children_of, ref_index, ref_by_uid, rels_by_node — all derived indexes built from the merged graph.
• search_corpus — the flat lowercased text strings per node, built after overlay merge so overlay-added nodes are included.
• tree_name_by_id, domain_root_uid, domain_root_path, propagation_rules, all_rules, domains — additional derived and raw data.

Not cached / recomputed on every rebuild:

• The search_index (inverted index for Global Search v2) is built lazily on the first get_search_index() call after a cache miss and stored inside the same _cache dict. It is effectively cached alongside the rest until the next invalidation.
• The overlay scan result inside DomainStore._overlay_cache is a separate cache with its own sentinel check. It is populated by load_overlay() during the rebuild and cleared by invalidate() in the same lock acquisition that clears eidos_loader._cache. In practice both caches are warm or cold together under normal operation.

current_graph() logs eidos_loaded (info level) with counts of nodes, relations, and trees every time it performs a full rebuild. The absence of this log line during normal traffic confirms the cache is working.

---

Concurrency

_loop_write_lock: the asyncio.Lock, what it protects

The async context manager write_lock() (line 166 of mutation_store.py; @contextlib.asynccontextmanager decorator at 165) combines two levels of serialisation for the full read-modify-write sequence of a mutation:

@contextlib.asynccontextmanager
async def write_lock():
    async with _loop_write_lock():
        try:
            MUTATIONS_DIR.mkdir(parents=True, exist_ok=True)
        except OSError:
            pass
        fh = _acquire_file_lock(MUTATIONS_DIR / "_write.lock")
        try:
            yield
        finally:
            _release_file_lock(fh)

_loop_write_lock() returns a per-event-loop asyncio.Lock from the _write_loop_locks registry — a WeakKeyDictionary keyed on the loop object (trin 3.3; auto-evicts when a loop is garbage-collected, so it does not leak one entry per loop). An asyncio.Lock serialises concurrent coroutines within the same event loop. Because each uvicorn worker runs one event loop, this lock prevents two requests within the same worker from interleaving their read→write sequences.

The outer fcntl advisory lock on MUTATIONS_DIR/_write.lock extends this serialisation across uvicorn worker processes on Linux/macOS. On Windows, the advisory lock is a no-op (the threading.Lock inside DomainStore covers the single-machine development case).

write_lock() is the correct scope for a mutation: it must be held across the entire sequence of read overlay → compute new document → write file → write idempotency record → call invalidate(). Holding it for shorter spans risks lost updates; holding it after invalidate() is not necessary.

What happens if two invalidations race

If two coroutines both call invalidate() concurrently (e.g., two independent mutation endpoints firing simultaneously), each acquisition of self._write_lock (the threading.Lock inside DomainStore) will queue them sequentially. The second invalidation will find _overlay_cache already None (cleared by the first) and will call _bump_cache_version() again, advancing the sentinel by another increment. This is correct: the sentinel ends up at least 2 higher than before, all workers will detect the change and rebuild, and no invalidation is lost.

If two invalidations race at the bump_version_counter level across two workers, the advisory lock on _cache_version.lock serialises them. Each worker's bump is an independent read-increment-write protected by that lock, so both increments are preserved. A worker that was mid-read when the lock was held will retry (or next call will see the changed counter).

---

Failure Modes

What happens if the sentinel file is corrupted

read_version_counter(path) catches all of FileNotFoundError, ValueError, and OSError and returns 0:

def read_version_counter(path) -> int:
    try:
        return int(Path(path).read_text().strip())
    except (FileNotFoundError, ValueError, OSError):
        return 0

If _cache_version contains non-integer content (e.g., a partial write that somehow bypassed the temp-rename mechanism, or manual corruption), read_version_counter returns 0. The sentinel mtime stored in _cache_sentinel_mtime from the previous successful load will likely be a positive integer. Since 0 != positive_integer, _sentinel_mtime() != _cache_sentinel_mtime evaluates to True, and current_graph() treats the cache as stale and rebuilds. The design explicitly fails toward freshness: when in doubt, rebuild rather than serve stale data.

Similarly, if _cache_version disappears entirely (deleted, filesystem error, etc.), read_version_counter returns 0. All workers will clear their caches on the next load call, rebuild, and store 0 as the new reference value. Subsequent loads will compare 0 == 0 and return the warm cache until the next invalidate() call writes a 1 back to the file.

The one scenario that would silently serve stale data: if the counter file is somehow frozen at its current value (e.g., a filesystem mount made read-only after the last write) at the exact moment a mutation is committed. bump_version_counter will raise an OSError when attempting to write, which _clear_caches catches and silences (the comment reads: "silence: cache sentinel update must not block mutation commit"). In this case the in-process eidos_loader._cache.clear() still fires (clearing the local worker's cache), but other workers will not see a changed sentinel and will continue serving their warm caches. This is a known trade-off: sentinel failures must not abort mutations, so cross-process coherence degrades silently. Monitoring for write errors to mutations/_cache_version is the detection mechanism.

What happens if merge_overlay raises

In eidos_loader.current_graph(), the call to graph_merge.merge_overlay(_base, overlay) is inside a try: ... except ImportError: pass block. This guard catches only ImportError (intended for secondary backends that do not have graph_merge installed). An unexpected exception from within merge_overlay itself — a bug, a data shape violation, an assertion failure — propagates up through current_graph() and back to the HTTP request handler, which should return a 500 error.

Critically, because _cache.update(...) comes after merge_overlay in the code, a failed merge_overlay leaves _cache in its cleared state (it was cleared at the top of current_graph() by the stale-sentinel branch). The next request will attempt another rebuild, calling merge_overlay again with the same data. If the data that caused the failure is still on disk (e.g., a malformed overlay node that load_overlay accepted but merge_overlay cannot process), every request will fail until that file is corrected or removed.

The load_overlay scan is defensive: individual malformed JSON files are silently skipped. But merge_overlay sees only the overlay that load_overlay returned, and if that overlay contains structurally valid JSON that nonetheless produces an unexpected shape (e.g., a node missing required fields), merge_overlay may raise. The failure surface is: a malformed overlay file that passes load_overlay's file-level validation but fails merge_overlay's graph-level processing will cause every get_effective_graph() call to fail until the file is corrected. Identifying such a file requires inspecting the structlog output for eidos_loading log lines without a following eidos_loaded line, then examining recent overlay writes.

Stage Reference

Browser Cache Lookup

Engine: frontend-crud — Zone: frontend

check EidosNodeStore / EidosTree (childrenOf, parentOf) / labelMap (plain session-scoped objects; unbounded, no LRU/eviction)

Effective Graph Lookup

Engine: eidos-loader — Zone: domain

check _cache (EffectiveGraph in-memory dict); return immediately if present

Effective Graph Rebuild

Engine: eidos-loader — Zone: persistence

on cache miss: read the overlay files from disk (#935 S5a: no .eidos base); invoke graph_merge.merge_overlay to produce new EffectiveGraph; store result in _cache

Invariants enforced: overlay-wins-on-conflict

Populate Browser Cache

Engine: frontend-crud — Zone: frontend

store API response in EidosNodeStore (upsert/patch) / EidosTree (childrenOf) (unbounded; entries leave only via explicit invalidate/reset)