Atomic Write¶

Verified against the code · 19d408be · 2026-08-25

Every code-checkable claim matches the code exactly. The quoted write_atomic implementation (temp file via tempfile.mkstemp, fsync, os.replace, cleanup-on-exception) is byte-for-byte identical to backend/persistence/mutation_store.py:224-238. _acquire_file_lock/_release_file_lock (lines 73-94), the per-loop _loop_write_lock/_write_loop_locks dict (lines 136-145), write_ownership's lock-then-write_atomic pattern (lines 494-515), the CCR Rule 5 module-header annotation, and the load_overlay exception-swallowing comment (line 335, verbatim 'silence: malformed overlay file; skip entry') are all confirmed present and behaving as described. No divergence found.

The atomic-write invariant is the foundational crash-safety guarantee for all overlay file I/O in EIDOS Explorer. It is encoded as CCR Rule 5 in the module header of backend/persistence/mutation_store.py and is enforced exclusively through DomainStore.write_atomic.

The Guarantee¶

"Atomic" here means exactly one thing: a reader opening an overlay file will always find either the complete previous content or the complete new content — never a partial or truncated intermediate state.

This guarantee holds even if the server process is killed, the machine loses power, or the OS kernel crashes at the exact moment a write is in progress.

The write-to-temp-then-rename pattern¶

The implementation never opens the target file for writing directly. Instead it writes to a freshly created temporary file in the same directory, syncs the data to durable storage, and then replaces the target file with a single rename operation.

The reason this works is that os.replace (which wraps the POSIX rename(2) syscall) is atomic at the filesystem level on POSIX systems. The kernel updates the directory entry in a single operation — no other process can observe the directory in a state where the old entry has been removed but the new entry has not yet appeared. The target path either points to the old inode or the new inode; there is no in-between state.

If the process dies before os.replace is called, the temporary file is left behind as an orphan (its name ends in .tmp). The target path is unchanged. The next read sees the old file as if nothing happened. The orphaned .tmp file is inert: nothing reads .tmp files during normal operation.

If the process dies after os.replace returns, the new file is fully in place. The rename is the commit point.

Where It Is Enforced¶

Every overlay write in the system — nodes, relations, tombstones, node tombstones, ownership, idempotency records, previews, and the cache-version counter — passes through DomainStore.write_atomic. There is no legitimate code path that writes a .jsonld or .json file in mutations/ by opening the target path directly.

`DomainStore.write_atomic`¶

def write_atomic(self, data: dict, path: Path) -> None:
    """temp → fsync → rename, serialised by _write_lock. Never corrupts on failure."""
    with self._write_lock:
        path.parent.mkdir(parents=True, exist_ok=True)
        fd, tmp = tempfile.mkstemp(dir=path.parent, suffix=".tmp")
        try:
            with os.fdopen(fd, "w", encoding="utf-8") as f:
                json.dump(data, f, ensure_ascii=False, indent=2)
                f.flush()
                os.fsync(f.fileno())   # CCR Rule 5
            os.replace(tmp, path)      # atomic rename
        except Exception:
            with contextlib.suppress(OSError):
                os.unlink(tmp)
            raise

Location: backend/persistence/mutation_store.py, class DomainStore, lines 224–238.

Parameters:

Param	Type	Description
`data`	`dict`	The Python dictionary to serialise as JSON-LD. Must be JSON-serialisable.
`path`	`Path`	Absolute path of the target overlay file. The parent directory is created if absent.

Return type: None. Raises on I/O failure after cleaning up the temp file.

Step-by-step execution:

Acquires self._write_lock (a threading.Lock). This serialises all calls from concurrent threads within the same process before any filesystem work begins.
Creates the parent directory tree with path.parent.mkdir(parents=True, exist_ok=True). This is a no-op when the directory already exists.
Calls tempfile.mkstemp(dir=path.parent, suffix=".tmp"). mkstemp creates the file atomically with a unique name, returns a raw OS file descriptor and the temp path as a string. Placing the temp file in the same directory as the target is critical: os.replace is only atomic when both paths are on the same filesystem (same mount point). Cross-device renames fall back to copy-then-delete and lose the atomicity guarantee.
Wraps the file descriptor in a Python file object via os.fdopen and serialises data as indented UTF-8 JSON.
Calls f.flush() to push data from Python's user-space write buffer into the OS kernel page cache.
Calls os.fsync(f.fileno()) to force the kernel to flush dirty pages to durable storage before closing. See the fsync discussion below.
Closes the file object (implicit at the end of the with block).
Calls os.replace(tmp, path) — the atomic rename. From this point on the new content is visible to any subsequent reader.
On any exception in steps 3–8, attempts to delete the orphaned temp file with os.unlink(tmp), then re-raises.

What it mutates: The single file at path. It does not mutate self._overlay_cache or any other in-memory state; cache invalidation is handled separately by the caller.

What it reads: Nothing. It is a pure write operation.

The `fsync` before rename¶

f.flush() alone is not enough. On Linux, flush() moves data from CPython's internal buffer into the kernel's page cache. The data is in RAM, not on disk. If the machine loses power at this point, the data is lost and the temp file may contain zeros or garbage depending on the filesystem's journaling mode.

os.fsync(f.fileno()) issues a fsync(2) syscall, which blocks until the drive's write cache confirms that the data has reached persistent storage. Only after fsync returns is it safe to rename the temp file into place.

This distinction was identified as a real defect. The YAML entity records:

The retired write-engine had exactly this defect (Finding 10 in ARCHITECTURE_REVIEW.md): it used os.replace without a preceding fsync, so a power loss after rename but before the drive flushed its write cache could leave a file containing zeros at the target path.

With fsync in place, a power loss after the rename commits a fully durable file. A power loss before fsync returns leaves the old target file intact (the rename has not happened yet).

The File Lock¶

The single write_atomic method is called from multiple contexts that require different coordination strategies. The system uses two independent locking layers, each targeting a different adversary.

`_acquire_file_lock` and `_release_file_lock`¶

def _acquire_file_lock(lock_path):
    if _sys.platform == "win32":
        return None
    import fcntl
    fh = open(lock_path, "a")
    fcntl.lockf(fh, fcntl.LOCK_EX)
    return fh

def _release_file_lock(fh) -> None:
    if fh is not None:
        import fcntl
        fcntl.lockf(fh, fcntl.LOCK_UN)
        fh.close()

Location: backend/persistence/mutation_store.py, lines 73–94.

These functions implement an advisory cross-process file lock using fcntl.LOCK_EX (POSIX exclusive lock). The lock is held on a dedicated sibling lock file (e.g., ownership.lock, _write.lock), not on the data file itself.

_acquire_file_lock is used in two places:

write_ownership acquires a lock on ownership.lock before calling write_atomic. This is necessary because ownership.jsonld is a single shared file that every uvicorn worker process may need to update. Without a cross-process lock, two workers could both read the current ownership graph, both compute an updated version, and one of them would silently overwrite the other's update (lost-update anomaly).
write_lock() (the async context manager) acquires a lock on _write.lock for the duration of a full mutation read-modify-write cycle.

For UUID-named node and relation files (nodes/{uuid}.jsonld, relations/{uuid}.jsonld), no cross-process advisory lock is needed in write_atomic itself. Each worker writes to a different filename, so os.replace — which is atomic — is sufficient: two workers cannot race on the same target path for distinct UUIDs.

`_loop_write_lock`: the asyncio lock for concurrent async writers¶

_write_loop_locks: "dict[int, _asyncio.Lock]" = {}

def _loop_write_lock() -> "_asyncio.Lock":
    loop = _asyncio.get_running_loop()
    lk = _write_loop_locks.get(id(loop))
    if lk is None:
        lk = _asyncio.Lock()
        _write_loop_locks[id(loop)] = lk
    return lk

Location: backend/persistence/mutation_store.py, lines 136–145.

fcntl locks are per-process, not per-coroutine. Within a single uvicorn worker process, all coroutines share the same fcntl lock state. Two coroutines in the same event loop that both call _acquire_file_lock will not block each other — both will succeed because they are in the same process and fcntl sees them as a single holder.

The asyncio.Lock stored in _write_loop_locks closes this gap. It serialises all coroutines within one event loop, regardless of whether they await at any point inside the critical section.

The lock is keyed on id(loop) rather than stored as a simple module-level variable. This is necessary for test correctness: each call to asyncio.run() in a test creates a fresh event loop with a fresh id. A module-level lock created under one event loop would be unusable from a different loop and would raise a RuntimeError. The per-loop dictionary ensures each test run gets a fresh asyncio.Lock that belongs to its own loop.

Why two layers?¶

Threat	Layer that stops it
Two coroutines in the same process racing on a shared file	`asyncio.Lock` (via `_loop_write_lock`)
Two threads in the same process racing	`threading.Lock` (`self._write_lock` inside `write_atomic`)
Two uvicorn worker processes racing on a shared file	`fcntl.LOCK_EX` (via `_acquire_file_lock`)

The threading.Lock inside write_atomic handles the thread-level case. The asyncio.Lock handles the coroutine-level case. The fcntl lock handles the cross-process case. No single layer covers all three; all three are required.

What Violation Looks Like¶

A violation occurs when any code writes to a target path in mutations/ by opening that path directly:

# VIOLATION — do not do this
with open(path, "w", encoding="utf-8") as f:
    json.dump(data, f)

What a reader can observe¶

If a reader opens path between the moment the kernel truncates the file (which happens at open(..., "w")) and the moment json.dump finishes writing, the reader sees an empty file or a truncated JSON document. json.loads raises json.JSONDecodeError. In load_overlay, the exception is silently swallowed:

except Exception:  # silence: malformed overlay file; skip entry
    pass

The node or relation disappears from the effective graph for the duration of that reader's call. This is a transient ghost deletion — the node reappears on the next load after the write completes. Depending on timing, a user action based on the stale graph may succeed when it should fail (e.g., creating a duplicate relation), or fail when it should succeed (e.g., rejecting a valid parent reference).

Data races: the interleaved-write scenario¶

With direct writes and no locking, two concurrent async mutation handlers can interleave on a single shared file such as ownership.jsonld:

Handler A reads ownership.jsonld → list [owner_1]
Handler B reads ownership.jsonld → list [owner_1]
Handler A appends owner_2, writes [owner_1, owner_2] to ownership.jsonld
Handler B appends owner_3, writes [owner_1, owner_3] to ownership.jsonld

After step 4, owner_2 has been silently lost. Neither handler raised an error. The audit log recorded both mutations as successful. The on-disk state is inconsistent with reality. This is a lost update and it is undetectable at read time.

write_ownership prevents this by acquiring the fcntl lock before the read-modify-write, holding it across both the read and the write, so handlers A and B are serialised and the write in step 3 is visible to the read in step 2 for the second handler.

Windows Caveats¶

On Windows, os.replace uses MoveFileExW with the MOVEFILE_REPLACE_EXISTING flag. On NTFS this is effectively atomic from the perspective of userspace readers: a reader opening the path will see either the old or the new file, not a partially overwritten state. However, the Windows kernel does not guarantee the same metadata-journal durability semantics as Linux ext4/XFS, and os.fsync on Windows flushes the file's data but not necessarily the parent directory's journal entry.

For the EIDOS Explorer deployment target (Linux in production), the full POSIX atomicity guarantee applies. On Windows the implementation remains correct for single-process development use because _write_lock (the threading.Lock) prevents concurrent writes within the same process, and _acquire_file_lock returns None on win32 rather than attempting fcntl. The cross-process advisory lock is explicitly documented as a POSIX/deployment guarantee:

# On Windows, no-op: the per-process threading.Lock (_write_lock) is sufficient
# in the development environment (single machine, single writer at a time).

A developer running multiple uvicorn workers locally on Windows does not have cross-process write serialisation. This is an accepted limitation of the development environment. Production deployment is Linux only.

Testing for Violation¶

Verifying crash-safety mid-write¶

To confirm that a crash before os.replace leaves the old file intact, inject a fault between os.fsync and os.replace:

import os
import json
from pathlib import Path
from unittest.mock import patch

def test_crash_before_rename_leaves_old_file_intact(tmp_path):
    target = tmp_path / "test.jsonld"
    original = {"eidos:id": "original"}
    target.write_text(json.dumps(original), encoding="utf-8")

    def exploding_replace(src, dst):
        raise OSError("simulated crash")

    store = DomainStore(tmp_path)
    with patch("os.replace", side_effect=exploding_replace):
        try:
            store.write_atomic({"eidos:id": "corrupted"}, target)
        except OSError:
            pass

    # Target must still contain original content
    assert json.loads(target.read_text(encoding="utf-8"))["eidos:id"] == "original"
    # No .tmp orphan should survive (write_atomic cleans up on exception)
    assert list(tmp_path.glob("*.tmp")) == []

The test simulates a crash at the rename step. Because write_atomic calls os.unlink(tmp) in its except block before re-raising, the temp file is cleaned up. The target file is untouched.

Verifying that direct writes are absent¶

A static check confirms no code in backend/persistence/ opens a .jsonld or .json overlay file for writing outside of write_atomic:

grep -rn "open(.*['\"]w['\"]" backend/persistence/ | grep -v "write_atomic\|mkstemp\|fdopen\|test_"

Any line returned by this command that writes to a path inside mutations/ is a CCR Rule 5 violation and must be replaced with a call to write_atomic.

Verifying fsync is present¶

Remove the os.fsync call and run the crash-safety test against a filesystem that does not journal data writes (e.g., a RAM-backed tmpfs). On a journaled filesystem the test may still pass because the OS flushes dirty pages quickly; on tmpfs it will reliably expose the gap. In production, omitting fsync passes all tests but silently re-introduces Finding 10 from ARCHITECTURE_REVIEW.md.

Blast Radius if Violated¶

If write_atomic is bypassed for any overlay file type, the consequences depend on the file's role:

Node or relation file (nodes/{uuid}.jsonld, relations/{uuid}.jsonld): A concurrent reader sees a truncated JSON document. The node or relation vanishes from the effective graph for the duration of one or more load_overlay calls. API responses return incomplete graphs. Clients may cache the incomplete result.
ownership.jsonld: A lost update silently removes an ownership assignment. Authorization checks based on ownership return incorrect results until the next write corrects the file. No error is logged.
idempotency/{key}.jsonld: A failed write means the idempotency record is never persisted. A retry of the same request is not recognised as a duplicate and is executed a second time, creating a duplicate entity.
_cache_version: A torn write of the monotonic counter causes read_version_counter to return 0 (the integer parse fails and the fallback 0 is returned). Workers that read 0 when the previous sentinel was, say, 42 will treat their cache as stale and reload — a performance cost, not a correctness failure. This is the one file where the blast radius of a violation is bounded to a spurious cache miss rather than data corruption.