Overlay Wins on Conflict

Verified — minor divergences from the code — 2 finding(s) · 19d408be · 2026-08-25

The invariant, its enforcement point (merge_overlay in backend/domain/graph_merge.py), the Step 4a/4b node-merge logic, the tombstone-ordering fix, and the BL-FE-037 regression test all match the code precisely, including exact line 134 for nodes.update(overlay["nodes"]) and near-verbatim code quotes for the Step-1 tombstone pre-filter. The one divergence is in the KM YAML itself: sources.files cites a nonexistent path for the enforcement file.

Divergences from the code — details

Sev	Where	Doc says	Code does	Evidence
MEDIUM	sources.files / enforced_by: graph-merge	sources.files lists the enforcement file as backend/graph_merge.py	No such file exists; the module is at backend/domain/graph_merge.py (the compiled doc page itself correctly uses this path throughout, so only the KM YAML's own sources.files entry is stale)	docs/manual/knowledge/invariants/overlay-wins-on-conflict.yaml:23
LOW	For relations the mechanism is different	Relations: relations = relations + overlay["relations"]	The actual code first filters overlay relations against node_tombstones into overlay_rels, then does relations = relations + overlay_rels (graph_merge.py:176-181) — the doc's quoted snippet is a simplification that omits the tombstone-gating step (which is otherwise correctly described in prose elsewhere in the same page)	backend/domain/graph_merge.py:176-181

The Guarantee

When a node or relation exists in both the base .eidos ZIP and the overlay (mutations/ directory), the overlay version is the authoritative value. For any given node UUID, once an overlay entry exists the base is never consulted for that ID again — not merged, not preferred, not blended.

In concrete terms: if the base graph contains node faaa4f34-… with label "OldLabel" and the overlay contains the same UUID with label "NewLabel", the effective graph presents "NewLabel". The base entry for that UUID is completely shadowed.

This applies equally to the node entry as a whole. If the overlay entry omits fields that the base contained, those fields disappear from the effective node (with two narrowly-scoped exceptions for engineering and productType — see the Step 4a discussion below). The overlay entry is not patched on top of the base entry field-by-field; it replaces it.

The same guarantee extends to relations. Overlay relations are appended to the effective relation list; they do not require a matching base relation, and a base relation can be hidden by a tombstone entry in the overlay. The rule governing relation ordering is append-only rather than replacement, but the directional authority runs the same way: overlay assertions always win.

Where It Is Enforced

The invariant is enforced in a single function:

# backend/domain/graph_merge.py

def merge_overlay(base_graph: dict, overlay: dict) -> dict:

The function is the only path through which the effective graph is produced. eidos_loader._load() calls it after assembling the base graph and scanning the mutations/ directory. No other call site generates an EffectiveGraph from a base-plus-overlay combination.

The Core Decision — nodes.update()

The overlay-wins rule for nodes is implemented in Step 4b of merge_overlay, at line 134 of backend/domain/graph_merge.py:

nodes.update(overlay["nodes"])

nodes is a reference to base_graph["nodes"] — a dict[str, dict] mapping node UUID → node entry. dict.update() overwrites any existing key with the value from the incoming dict and inserts keys that were absent. Because overlay["nodes"] is passed as the argument, overlay entries land on top of base entries for identical UUIDs. There is no conditional: the update is unconditional for every key in overlay["nodes"].

Immediately before this line, Step 4a runs a narrow preservation pass:

for uid, onode in overlay["nodes"].items():
    if "engineering" not in onode and uid in nodes:
        onode["engineering"] = nodes[uid].get("engineering", [])
    if "productType" not in onode and uid in nodes and nodes[uid].get("productType") is not None:
        onode["productType"] = nodes[uid]["productType"]

This copies engineering and productType from the base into the overlay entry only when the overlay entry does not contain those keys at all. It is not an exception to the invariant — it is a field-level back-fill for two specific metadata fields that the overlay authoring process does not write. The overlay entry still wins for every other field. After this back-fill, nodes.update(overlay["nodes"]) runs and the overlay entry (now enriched with the back-filled fields) replaces the base entry in full.

For relations the mechanism is different — overlay relations are appended rather than merged by UUID — but the authority direction is the same:

relations = relations + overlay["relations"]

Base relations that conflict with overlay intent are removed via tombstones (see the Tombstones section below). An overlay relation is never suppressed by a base relation.

Why It Must Be Unconditional

The User-Edit Use Case

The primary reason the rule has no exceptions is the user-edit lifecycle.

When a user modifies a node that originated in the base .eidos ZIP, the mutation engine writes a new overlay file for that node UUID in mutations/nodes/. The overlay entry holds the user's new values. From that point forward, the only way the user's changes can survive across reloads is if the overlay entry unconditionally wins when the same UUID appears in both layers.

If the rule had an exception — for example, "overlay wins unless the base entry has a higher version number" — then a base ZIP update could silently overwrite user edits with no warning and no recovery path. The user edited "NewLabel", the base ZIP was replaced with a new version that still carries the node but now calls it "UpdatedBaseLabel", and the user's work is gone.

The unconditional rule eliminates this entire class of defect. User-authored overlay data is durable against base replacements.

What Would Break if the Rule Had Exceptions

Conditional merge by field type. If the code attempted to merge base and overlay entries field-by-field rather than replacing wholesale, the result would depend on which fields the overlay happened to write. Overlay authoring does not guarantee completeness — an overlay file for a node may contain only the fields the user explicitly changed. A field-level merge would mix base and overlay data in ways neither layer intended, producing composite node entries that correspond to no real authored state.

Version-gated overlay. If a newer base version were allowed to override an older overlay, the effective graph would silently change on base ZIP updates with no user action and no notification. The user would see a node revert to a previous state with no explanation.

Per-field ownership tracking. Any scheme that routes individual fields to either base or overlay depending on provenance metadata would require that metadata to be consistent across all mutation paths. A missing or incorrect provenance tag would silently mis-route a field. The dict.update() approach requires no such metadata; the overlay file's existence is the only signal needed.

Tombstones as Overlay Assertions

The overlay-wins invariant also governs deletions. A node tombstone — a file at mutations/node_tombstones/{uuid}.jsonld — is an overlay-layer assertion that the node with that UUID must not exist in the effective graph, regardless of whether it exists in the base.

This is the same invariant applied to the delete case: if the base contains a node and the overlay says that node is deleted, the overlay wins and the node is absent from the effective graph.

In merge_overlay, node tombstones are applied during Step 3:

node_tombstones: set = overlay.get("node_tombstones", set())
if node_tombstones:
    for uid in list(nodes):
        if uid in node_tombstones:
            del nodes[uid]
    relations = [r for r in relations
                 if r.get("from") not in node_tombstones
                 and r.get("to") not in node_tombstones]
    # ... plus parent_of, children_of, ref_index, ref_by_uid, rels_by_node cleanup

The tombstone removes the node from every index structure. The base file and overlay file remain on disk — the on-disk layer stack preserves the full lifecycle — but the effective graph projection does not contain the node.

Relation tombstones follow the same pattern via the tombstones set (Step 2):

tombstones: set = overlay.get("tombstones", set())
if tombstones:
    relations = [r for r in relations
                 if r.get("_overlay") or r.get("source") not in tombstones]

A relation tombstone targets a seeded (base-graph) relation by its source UUID. Overlay relations carry "_overlay": True and are never targeted by relation tombstones — they have no base source to reference.

The important design consequence is that tombstones must be applied before the overlay-add merge. If tombstones were applied after nodes.update(), a node that was created in the overlay and later tombstoned would be re-introduced from overlay["nodes"] before the tombstone filter ran. This was the exact mechanism behind BL-FE-037.

Testing for Violation

A test that catches a violation of the overlay-wins invariant must construct a scenario where base and overlay both define the same node, then assert that the effective graph reflects only the overlay value.

Minimal test structure:

def test_overlay_wins_for_conflicting_node():
    base_graph = {
        "nodes": {
            "uid-A": {"label": "BaseLabel", "engineering": [], "productType": None},
        },
        "relations": [],
        "trees": [],
        "parent_of": {},
        "children_of": {},
        "ref_index": {},
        "ref_by_uid": {},
        "rels_by_node": {},
    }
    overlay = {
        "nodes": {
            "uid-A": {"label": "OverlayLabel"},
        },
        "relations": [],
        "tombstones": set(),
        "node_tombstones": set(),
    }
    result = merge_overlay(base_graph, overlay)
    assert result["nodes"]["uid-A"]["label"] == "OverlayLabel"

A test that catches the tombstone variant of the invariant (the BL-FE-037 scenario) must place the same UUID in both overlay["nodes"] and overlay["node_tombstones"] and assert that the node is absent from every output structure:

def test_overlay_tombstone_wins_over_overlay_node():
    uid = "uid-created-then-deleted"
    base_graph = {
        "nodes": {},
        "relations": [],
        "trees": [],
        "parent_of": {},
        "children_of": {},
        "ref_index": {},
        "ref_by_uid": {},
        "rels_by_node": {},
    }
    overlay = {
        "nodes": {uid: {"label": "CreatedViaOverlay"}},
        "relations": [],
        "tombstones": set(),
        "node_tombstones": {uid},
    }
    result = merge_overlay(base_graph, overlay)
    # Node must be absent from every output structure
    assert uid not in result["nodes"]
    assert uid not in result["parent_of"]
    assert uid not in result["children_of"]
    assert uid not in result["ref_by_uid"]
    assert uid not in result["rels_by_node"]

This second test is the pattern used in tests/test_node_tombstone.py::test_merge_removes_tombstoned_overlay_node_from_effective_graph, which was added as the regression test for BL-FE-037.

To test the base-tombstone variant (a base-graph node deleted via overlay), place the UUID in base_graph["nodes"] and in overlay["node_tombstones"] and assert the same absence.

BL-FE-037 — Historical Violation

What Went Wrong

On 2026-05-20, users reported that deleting a node appeared to succeed — the delete dialog closed without error — but the node remained visible in the tree and detail panel. A hard browser refresh did not help; the defect was server-side.

Investigation confirmed (via SSH to the DEV deployment) that the tombstone file mutations/node_tombstones/{uuid}.jsonld was present on disk. The tombstone had been written correctly. The defect was in backend/graph_merge.py::merge_overlay.

At that point the function applied node_tombstones filtering to base structures — removing the tombstoned node from nodes, parent_of, children_of, and so on — but then unconditionally ran nodes.update(overlay["nodes"]) with the raw, unfiltered overlay["nodes"] dict. For any node that existed both as an overlay file (because it was created via overlay) and as a tombstone (because it was subsequently deleted), the tombstone filter's work was immediately undone by the dict.update() call. The overlay-add pass re-inserted the deleted node into every structure that the tombstone filter had just cleaned.

The defect affected any node that went through the create-in-overlay then delete lifecycle, which is the normal lifecycle for any user-created node. Base-graph nodes that had never been written as overlay files were not affected — their tombstones removed them from the base structures, and there was no overlay entry to re-introduce them.

The Fix

The fix was applied in commit e0df3a4 (2026-05-20). It added Step 1 to merge_overlay: filter overlay["nodes"] against overlay["node_tombstones"] before any other processing runs. The filtered dict replaces overlay["nodes"] for the remainder of the function. Because overlay is potentially the cached mutation_store._overlay_cache dict shared across requests, a shallow copy is made before mutating:

_node_tombstones_pre: set = overlay.get("node_tombstones", set())
if _node_tombstones_pre and overlay.get("nodes"):
    _filtered_overlay_nodes = {
        uid: n for uid, n in overlay["nodes"].items()
        if uid not in _node_tombstones_pre
    }
    if len(_filtered_overlay_nodes) != len(overlay["nodes"]):
        overlay = dict(overlay)           # shallow copy — do not mutate cached dict
        overlay["nodes"] = _filtered_overlay_nodes

After Step 1 runs, no subsequent step in the function can see the tombstoned UUID in overlay["nodes"]. The nodes.update(overlay["nodes"]) call in Step 4b never encounters it. The invariant — tombstone wins over overlay node entry — is enforced at the input boundary of the overlay layer, before the first write to any output structure.

Blast Radius of Violation

If the order of operations in merge_overlay ever drifts back to the pre-fix shape — tombstone applied to output structures before the nodes.update() call — the consequences are:

• Every node with the create-via-overlay-then-delete lifecycle will be permanently visible in the effective graph, with no server-side mechanism to hide it. Hard reloads do not help.
• Users see deleted nodes in tree responses, detail panels, and search results. Subsequent deletes of the same node are idempotent (the tombstone already exists), so the delete appears to succeed while the node persists.
• The defect accumulates silently. Each overlay-created-then-deleted node adds a permanent phantom entry. There is no self-correcting mechanism.

The invariant is also checked by GCF Contract 06 §3.11 R37–R38 (added after BL-FE-037), which requires a regression test covering the composite-view tombstone ordering and a canonical flow diagram at docs/architecture/flow-merge-overlay.md. Any change to the order of operations in merge_overlay must be accompanied by a regression test update before merge.

Related Files

• backend/domain/graph_merge.py — the sole enforcement point for this invariant.
• docs/manual/knowledge/invariants/overlay-wins-on-conflict.yaml — the KM entity for this invariant.
• docs/manual/knowledge/workflows/composite-view-assembly.explanation.md — full explanation of the four-step merge, data structures, and all pitfalls.
• docs/architecture/flow-merge-overlay.md — canonical Mermaid flow diagram showing the strict order of operations.
• tests/test_node_tombstone.py — regression suite including the create-then-delete test added for BL-FE-037.
• backlog/items/frontend/BL-FE-037.md — full triage, root-cause analysis, and resolution record for the historical violation.
• backlog/items/data-model/BL-DM-001.md — node tombstone contract.