Cache Coherence

Verified — minor divergences from the code — 1 finding(s) · 19d408be · 2026-08-25

The cache-coherence.md page's behavioural claims are highly accurate: get_effective_graph()/_load()'s cache-then-invalidate strategy, the four-stage lifecycle, DomainStore.invalidate()/_clear_caches(), bump_version_counter()/read_version_counter(), the cross-process integer sentinel in mutations/_cache_version, the write_lock() asyncio+fcntl double-lock, and the merge_overlay/failure-mode discussion all match backend/persistence/mutation_store.py and backend/persistence/eidos_loader.py line-for-line (including specific cited line numbers such as write_lock at line 149 and bump_version_counter at line 97). The one divergence is a stale file path in the KM entity's sources.files list itself.

Divergences from the code — details

Sev	Where	Doc says	Code does	Evidence
MEDIUM	sources.files	sources.files lists the code file as 'backend/eidos_loader.py'	No such file exists at that path; the module actually lives at backend/persistence/eidos_loader.py (confirmed by Glob and by mutation_store.py's own from persistence import eidos_loader import)	docs/manual/knowledge/workflows/cache-coherence.yaml:37

flowchart LR
%% generated_from: cache-coherence
%% verified_at_commit: d50caf3143612a76bde48787104e90a4ac202ff3
%% description: Cache Coherence lifecycle
%% legend: owoc=overlay-wins-on-conflict
  subgraph frontend
    browser-cache-lookup["Browser Cache Lookup"]
    populate-browser-cache["Populate Browser Cache"]
  end
  subgraph domain
    effective-graph-lookup["Effective Graph Lookup"]
  end
  subgraph persistence
    effective-graph-rebuild["Effective Graph Rebuild · owoc"]
  end
  browser-cache-lookup --> effective-graph-lookup
  effective-graph-lookup --> effective-graph-rebuild
  effective-graph-rebuild --> populate-browser-cache

Overview — Why Caching EffectiveGraph

Every time a caller needs the current state of the EIDOS knowledge graph it calls get_effective_graph() in backend/persistence/mutation_store.py, which delegates immediately to eidos_loader._load(). That function performs a substantial amount of work:

1. Opens and decompresses the data.eidos ZIP archive from disk.
2. Parses manifest.json, the snapshot JSON-LD document, relations.eidr, and optionally make.eidm.
3. Builds flat nodes, relations, trees, parent_of, children_of, ref_index, and rels_by_node structures from the raw data.
4. Calls mutation_store.load_overlay(), which in turn scans every file in mutations/nodes/*.jsonld, mutations/relations/*.jsonld, mutations/tombstones/*.jsonld, and mutations/node_tombstones/*.jsonld.
5. Passes the assembled base graph and the overlay result to graph_merge.merge_overlay(), which resolves conflicts (overlay wins on conflict — the overlay-wins-on-conflict invariant from the workflow YAML).
6. Builds the full-text search corpus by traversing every merged node's attributes, properties, and engineering entries — field-filtered through the canonical content-field denylist in domain/search_index.py (_CORPUS_DENIED_FIELDS / _flatten_indexable), so document body, OCR, webpage and attachment content never enters the corpus (BL-FE-080; hardening #511).

Without caching, every HTTP request that touches any node, relation, or tree data would pay the full cost of steps 1–6. On a non-trivial EIDOS model this takes tens to hundreds of milliseconds. Under normal usage — a team of engineers navigating the graph, running searches, inspecting relations — dozens of requests per second would each trigger a ZIP extraction and a full directory scan.

The strategy is cache-then-invalidate: the result of _load() is stored in the module-level _cache dict inside eidos_loader.py and returned directly on all subsequent calls until something invalidates it. Invalidation is triggered only when a write operation modifies the overlay on disk. This reduces the common case (a read against an unchanged graph) to a dict truthiness check and a sentinel comparison — essentially free.

The workflow as a whole enforces a single end-to-end invariant: no caller ever receives a graph that is older than the most recent committed write, across all uvicorn worker processes. The four stages described below are how that invariant is maintained.

---

Stage 1: Write Received

What triggers invalidation

Every mutation that changes the on-disk overlay must eventually call mutation_store.invalidate(). In practice this includes:

• Creating an overlay node (writes mutations/nodes/{uuid}.jsonld)
• Updating an overlay node (overwrites the same file)
• Creating an overlay relation (writes mutations/relations/{uuid}.jsonld)
• Deleting an overlay relation (deletes the file, then writes a tombstone or not depending on whether the relation was seeded or overlay-native)
• Soft-deleting a seeded relation (writes mutations/tombstones/{uuid}.jsonld)
• Soft-deleting a node (writes mutations/node_tombstones/{uuid}.jsonld)

All of these go through mutation_store.write_atomic() first, then call invalidate(). The file is on disk before the cache is invalidated — this ordering ensures that when the cache is rebuilt it will find the new file.

Exactly what "invalidate" does

DomainStore.invalidate() (in backend/persistence/mutation_store.py, line 271) acquires self._write_lock (a threading.Lock) and then calls _clear_caches():

def invalidate(self, *, rel_uuid: str | None = None) -> None:
    with self._write_lock:
        self._clear_caches(rel_uuid=rel_uuid)
        for hook in self._post_invalidate_hooks:
            try:
                hook()
            except Exception as _exc:
                _log.warning("post_invalidate_hook_error", hook=repr(hook), exc=str(_exc))

Inside _clear_caches(), three things happen atomically under that lock:

1. self._overlay_cache = None — the in-process overlay cache held by DomainStore is dropped. The next call to load_overlay() will perform a full directory scan.
2. self._version += 1 — the in-process monotonic version counter is advanced. This counter is readable via get_version() and used by callers that want to detect "has anything changed since I last checked?"
3. self._bump_cache_version() is called — this advances the cross-process sentinel file (see Stage 2).

Additionally, _clear_caches directly reaches into eidos_loader._cache and calls either .clear() (full eviction) or eidos_loader.patch_remove_relation(rel_uuid) (granular patch when a single relation was deleted). Both the overlay cache and the effective-graph cache in eidos_loader are cleared in the same lock acquisition. Any exception from _bump_cache_version or from the eidos_loader manipulation is silenced — neither can block the mutation commit.

---

Stage 2: Version Bump

bump_version_counter: the sentinel file, atomic increment, why it's needed

The _cache dict in eidos_loader.py is module-level state. Each uvicorn worker process has its own copy. When worker A commits a mutation and calls invalidate(), that clears worker A's in-process cache. Worker B's eidos_loader._cache is unaffected — it is in a different process with separate memory.

To signal worker B (and any other worker), the system maintains a cross-process monotonic counter stored in the file mutations/_cache_version. The file contains a single ASCII integer.

bump_version_counter(sentinel_path) (line 97 of mutation_store.py) performs an atomic read-modify-write on that file:

def bump_version_counter(sentinel_path) -> int:
    sentinel = Path(sentinel_path)
    sentinel.parent.mkdir(parents=True, exist_ok=True)
    lock_fh = _acquire_file_lock(sentinel.with_name(sentinel.name + ".lock"))
    try:
        new = read_version_counter(sentinel) + 1
        tmp = sentinel.with_name(f"{sentinel.name}.{new}.tmp")
        tmp.write_text(str(new))
        os.replace(tmp, sentinel)
        return new
    finally:
        _release_file_lock(lock_fh)

The sequence is:

1. Acquire an advisory fcntl file lock on _cache_version.lock (on Linux/macOS). This serialises concurrent bumps from multiple workers so no increment is lost.
2. Read the current integer from _cache_version (returns 0 if the file is absent or unparseable).
3. Write current + 1 to a temp file (_cache_version.{new}.tmp).
4. os.replace() the temp file over _cache_version. This is atomic on both POSIX and Windows for same-filesystem renames — a concurrent reader always sees either the old or the new integer, never a partial write.
5. Release the advisory lock.

The reason this uses an integer counter rather than a simple touch (updating mtime) is stated explicitly in the code comments: st_mtime has whole-second resolution on many filesystems. Two invalidations within the same second would produce the same mtime, and other workers would incorrectly treat their warm cache as still valid. The monotonic counter advances on every invalidation regardless of wall-clock timing.

How the version counter enables external cache checks

eidos_loader._load() calls _sentinel_mtime() (the name is a historical artifact — it returns the integer counter, not an mtime):

def _sentinel_mtime() -> float:
    try:
        from persistence import mutation_store
        return mutation_store._read_cache_version(mutation_store._CACHE_SENTINEL)
    except Exception:
        return 0.0

At the end of a successful load, _load() stores the sentinel value it observed into _cache_sentinel_mtime. On the next call to _load() it compares the live sentinel value against the stored one. If they differ, the cache is stale and is cleared before rebuilding. If they are equal, the warm cache is returned immediately.

mutation_store._CACHE_SENTINEL is a module-level constant: MUTATIONS_DIR / "_cache_version". It is set once at import time and never changes. _read_cache_version is the backward-compatible alias for read_version_counter.

---

Stage 3: Cache Miss

What happens on the next get_effective_graph call after invalidation

DomainStore.get_effective_graph() calls eidos_loader._load() directly:

def get_effective_graph(self) -> dict:
    from persistence import eidos_loader
    return eidos_loader._load()

At the top of _load():

def _load() -> dict:
    global _cache_sentinel_mtime
    if _cache:
        if _sentinel_mtime() == _cache_sentinel_mtime:
            return _cache
        _cache.clear()  # stale — rebuild below

The _cache attribute: what it stores, how it's checked

_cache is a module-level plain dict defined at line 33 of eidos_loader.py:

_cache: dict = {}
_cache_sentinel_mtime: float = 0.0

The truthiness check if _cache: is True when the dict is non-empty (i.e., a warm cache exists). After invalidation, _clear_caches calls eidos_loader._cache.clear(), which empties the dict. On the next _load() call, if _cache: evaluates to False — the sentinel comparison is skipped entirely, and the full rebuild proceeds unconditionally.

This two-level check matters: a cold cache (empty dict) skips even reading the sentinel file from disk. A warm cache with a stale sentinel (_sentinel_mtime() != _cache_sentinel_mtime) calls _cache.clear() before rebuilding. In both paths, the code falls through to the ZIP extraction and merge sequence below.

---

Stage 4: Cache Rebuilt

The full call chain from get_effective_graph to load_overlay to merge_overlay

Starting from a user HTTP request reaching any endpoint that reads graph data:

HTTP request
  → eidos_loader.get_nodes() / get_relations() / etc.
  → eidos_loader._load()
      → zipfile.ZipFile(EIDOS_FILE) — open and parse data.eidos
      → [build nodes, relations, trees, parent_of, ref_index from snapshot]
      → mutation_store.load_overlay()
          → DomainStore.load_overlay()
              → read _cache_version sentinel
              → if overlay cache is warm and sentinel unchanged: return cached overlay
              → else: scan mutations/nodes/*.jsonld, mutations/relations/*.jsonld,
                       mutations/tombstones/*.jsonld, mutations/node_tombstones/*.jsonld
              → store result in self._overlay_cache
              → store current sentinel value in self._overlay_cache_sentinel_mtime
              → return overlay dict
      → graph_merge.merge_overlay(_base, overlay)
          → apply overlay nodes (overlay wins on conflict per invariant)
          → filter tombstoned relations
          → extend ref_index with overlay node refs
      → _build_search_corpus(nodes, relations)
      → _cache.update({nodes, relations, trees, parent_of, ...})
      → _cache_sentinel_mtime = _sentinel_mtime()
      → return _cache

What is cached and what is not

Cached (stored in eidos_loader._cache after a successful _load()):

• nodes — the fully merged dict of {uuid: node_dict} including both base and overlay nodes, with node_tombstoned entries absent.
• relations — the merged list of relation dicts with tombstoned relations removed.
• trees, parent_of, children_of, ref_index, ref_by_uid, rels_by_node — all derived indexes built from the merged graph.
• search_corpus — the flat lowercased text strings per node, built after overlay merge so overlay-added nodes are included.
• tree_name_by_id, domain_root_uid, domain_root_path, propagation_rules, all_rules, domains — additional derived and raw data.

Not cached / recomputed on every rebuild:

• The search_index (inverted index for Global Search v2) is built lazily on the first get_search_index() call after a cache miss and stored inside the same _cache dict. It is effectively cached alongside the rest until the next invalidation.
• The overlay scan result inside DomainStore._overlay_cache is a separate cache with its own sentinel check. It is populated by load_overlay() during the rebuild and cleared by invalidate() in the same lock acquisition that clears eidos_loader._cache. In practice both caches are warm or cold together under normal operation.

_load() logs eidos_loaded (info level) with counts of nodes, relations, and trees every time it performs a full rebuild. The absence of this log line during normal traffic confirms the cache is working.

---

Concurrency

_loop_write_lock: the asyncio.Lock, what it protects

The async context manager write_lock() (line 149 of mutation_store.py) combines two levels of serialisation for the full read-modify-write sequence of a mutation:

@contextlib.asynccontextmanager
async def write_lock():
    async with _loop_write_lock():
        fh = _acquire_file_lock(MUTATIONS_DIR / "_write.lock")
        try:
            yield
        finally:
            _release_file_lock(fh)

_loop_write_lock() returns a per-event-loop asyncio.Lock keyed by id(loop) from the _write_loop_locks dict. An asyncio.Lock serialises concurrent coroutines within the same event loop. Because each uvicorn worker runs one event loop, this lock prevents two requests within the same worker from interleaving their read→write sequences.

The outer fcntl advisory lock on MUTATIONS_DIR/_write.lock extends this serialisation across uvicorn worker processes on Linux/macOS. On Windows, the advisory lock is a no-op (the threading.Lock inside DomainStore covers the single-machine development case).

write_lock() is the correct scope for a mutation: it must be held across the entire sequence of read overlay → compute new document → write file → write idempotency record → call invalidate(). Holding it for shorter spans risks lost updates; holding it after invalidate() is not necessary.

What happens if two invalidations race

If two coroutines both call invalidate() concurrently (e.g., two independent mutation endpoints firing simultaneously), each acquisition of self._write_lock (the threading.Lock inside DomainStore) will queue them sequentially. The second invalidation will find _overlay_cache already None (cleared by the first) and will call _bump_cache_version() again, advancing the sentinel by another increment. This is correct: the sentinel ends up at least 2 higher than before, all workers will detect the change and rebuild, and no invalidation is lost.

If two invalidations race at the bump_version_counter level across two workers, the advisory lock on _cache_version.lock serialises them. Each worker's bump is an independent read-increment-write protected by that lock, so both increments are preserved. A worker that was mid-read when the lock was held will retry (or next call will see the changed counter).

---

Failure Modes

What happens if the sentinel file is corrupted

read_version_counter(path) catches all of FileNotFoundError, ValueError, and OSError and returns 0:

def read_version_counter(path) -> int:
    try:
        return int(Path(path).read_text().strip())
    except (FileNotFoundError, ValueError, OSError):
        return 0

If _cache_version contains non-integer content (e.g., a partial write that somehow bypassed the temp-rename mechanism, or manual corruption), read_version_counter returns 0. The sentinel mtime stored in _cache_sentinel_mtime from the previous successful load will likely be a positive integer. Since 0 != positive_integer, _sentinel_mtime() != _cache_sentinel_mtime evaluates to True, and _load() treats the cache as stale and rebuilds. The design explicitly fails toward freshness: when in doubt, rebuild rather than serve stale data.

Similarly, if _cache_version disappears entirely (deleted, filesystem error, etc.), read_version_counter returns 0. All workers will clear their caches on the next load call, rebuild, and store 0 as the new reference value. Subsequent loads will compare 0 == 0 and return the warm cache until the next invalidate() call writes a 1 back to the file.

The one scenario that would silently serve stale data: if the counter file is somehow frozen at its current value (e.g., a filesystem mount made read-only after the last write) at the exact moment a mutation is committed. bump_version_counter will raise an OSError when attempting to write, which _clear_caches catches and silences (the comment reads: "silence: cache sentinel update must not block mutation commit"). In this case the in-process eidos_loader._cache.clear() still fires (clearing the local worker's cache), but other workers will not see a changed sentinel and will continue serving their warm caches. This is a known trade-off: sentinel failures must not abort mutations, so cross-process coherence degrades silently. Monitoring for write errors to mutations/_cache_version is the detection mechanism.

What happens if merge_overlay raises

In eidos_loader._load(), the call to graph_merge.merge_overlay(_base, overlay) is inside a try: ... except ImportError: pass block. This guard catches only ImportError (intended for secondary backends that do not have graph_merge installed). An unexpected exception from within merge_overlay itself — a bug, a data shape violation, an assertion failure — propagates up through _load() and back to the HTTP request handler, which should return a 500 error.

Critically, because _cache.update(...) comes after merge_overlay in the code, a failed merge_overlay leaves _cache in its cleared state (it was cleared at the top of _load() by the stale-sentinel branch). The next request will attempt another rebuild, calling merge_overlay again with the same data. If the data that caused the failure is still on disk (e.g., a malformed overlay node that load_overlay accepted but merge_overlay cannot process), every request will fail until that file is corrected or removed.

The load_overlay scan is defensive: individual malformed JSON files are silently skipped. But merge_overlay sees only the overlay that load_overlay returned, and if that overlay contains structurally valid JSON that nonetheless produces an unexpected shape (e.g., a node missing required fields), merge_overlay may raise. The failure surface is: a malformed overlay file that passes load_overlay's file-level validation but fails merge_overlay's graph-level processing will cause every get_effective_graph() call to fail until the file is corrected. Identifying such a file requires inspecting the structlog output for eidos_loading log lines without a following eidos_loaded line, then examining recent overlay writes.

Stage Reference

Browser Cache Lookup

Engine: frontend-crud — Zone: frontend

check nodeCache / childrenCache / labelMap / parentMap (LRU, session-scoped)

Effective Graph Lookup

Engine: eidos-loader — Zone: domain

check _cache (EffectiveGraph in-memory dict); return immediately if present

Effective Graph Rebuild

Engine: eidos-loader — Zone: persistence

on cache miss: read .eidos ZIP base + overlay files from disk; invoke graph_merge.merge_overlay to produce new EffectiveGraph; store result in _cache

Invariants enforced: overlay-wins-on-conflict

Populate Browser Cache

Engine: frontend-crud — Zone: frontend

store API response in nodeCache / childrenCache (LRU eviction enforced)