Documents Engine¶

Diverges from the code — 6 finding(s) · 19d408be · 2026-08-25

All six first-pass findings survived adversarial re-verification against the code on disk: the blocklist section describes read-time filtering that GET /node/{id}/documents and the admin browse endpoint genuinely do not perform (confirmed no is_blocked call in either read path), and five canonical_operations signatures in the KM YAML are confirmed stale/wrong against the actual documents_engine.py/documents_import.py functions — status remains diverges on the blocklist behavioural gap alone (kind=behaviour, severity=high), independently reinforced by two medium-severity api-signature contradictions.

Divergences from the code — details

Sev	Where	Doc says	Code does	Evidence
HIGH	Moderation / Blocklist	"Blocked documents never appear in GET /node/{id}/documents or the Documents admin browse; the entry is stored, not the resolved output" and blocking is by "id or URL".	Confirmed: documents_blocklist.is_blocked(url, blocklist) only matches by hostname (no id-based blocking exists at all). Confirmed call sites: only the create-time gate in routes/documents.py:1444 (rejects new external links) and admin_external_links at routes/documents.py:826-844 (sets a 'blocked' display flag, does not drop the row). Read `get_node_documents` (routes/documents.py:246-288) and `admin_browse_documents` (routes/documents.py:731-752) in full: neither calls is_blocked or filters on a blocked flag anywhere in their bodies — a document already attached before its domain was blocked keeps appearing in both reads exactly as the finding states.	`backend/routes/documents.py:246-288, 731-752, 826-844, 1444-1447; backend/domain/documents_blocklist.py:57-71`
MEDIUM	canonical_operations: build_document_node	canonical_operations: build_document_node(doc_uuid, source, title, classification_id, description, parent_id) -> dict	Confirmed at backend/domain/documents_engine.py:223-224: actual def is build_document_node(doc_uuid, source, title, classification, description) -> dict — five parameters, no parent_id param, and the field is named classification not classification_id.	`backend/domain/documents_engine.py:223-224`
MEDIUM	canonical_operations: document_node_to_payload / document_node_to_browse_payload	canonical_operations: document_node_to_payload(node, merged_entry, lib_index) -> dict and document_node_to_browse_payload(node, lib_index) -> dict	Confirmed at backend/domain/documents_engine.py:340-342 and :408: actual signatures are document_node_to_payload(node, merged_entry, *, inherit=False, role=None, relId=None) and document_node_to_browse_payload(node) -> dict — neither takes a lib_index parameter, and document_node_to_payload's own docstring states 'There is no library join — the library abstraction is removed (documents-fresh-rebuild)'.	`backend/domain/documents_engine.py:340-342, 408`
MEDIUM	DocuGraph FME Import: preview()	DocuGraph FME Import table: preview(imported, eff) -> dict returns {fingerprint, summary:{added,updated,unchanged}, nodes:[...], relations:[...]}	Confirmed at backend/domain/documents_import.py:263-265: the actual return statement includes a fourth top-level key, 'risks': _compute_risks(imported, eff), which the doc's return-shape description omits entirely.	`backend/domain/documents_import.py:231-265`
LOW	canonical_operations: finalize_persist_metadata	canonical_operations: finalize_persist_metadata(node, parent_id, tree_id, created_by, audit_ref, origin) -> dict	Confirmed at backend/domain/documents_engine.py:257-260: actual keyword-only params are (node, *, parent_id, tree_id, username, audit_event_id="", origin="manual", idempotency_key=None) — created_by/audit_ref don't exist (they are username/audit_event_id), and idempotency_key is omitted from the doc entirely.	`backend/domain/documents_engine.py:257-260`
LOW	The `documents` Domain — First-Class, Visible by Default	"its own container running the same server binary with EIDOS_DOMAIN=documents"	Confirmed: config/domains.yaml:90 declares id: document (singular); deploy.py:1031 sets EIDOS_DOMAIN from that same catalog key; every runtime gate in routes/documents.py (16+ call sites, e.g. lines 189, 414, 459, 858, 881, 1018) checks `_THIS_DOMAIN != "document"` / `== "document"`, singular — never 'documents'.	`config/domains.yaml:90; backend/routes/documents.py:189,414,459,858,881,1018; deploy.py:1031`

Layer: domain

Overview — What the Documents Module Is¶

The DOCUMENTS module makes a document a first-class object instead of a URL-valued property. A document is a node in the first-class, visible-by-default documents domain, attached to items via hasDocument relations, resolved for display through three channels (direct / from-type / inherited) with locked dedup priority and a strict no-leak rule.

A plain URL string cannot carry classification, description, sharing (one document referenced by many items), propagation (documents on a %Type shown on instances), or mass-assignment. All of these require identity — hence a node.

Repo content (whole trees of DocumentContainer + Document nodes) enters the domain via DocuGraph FME import (see §DocuGraph FME Import below). The legacy DocumentLibrary + StorageBackend + generations upload machinery was retired in PR #431 — the new spec (docs/superpowers/specs/2026-08-18-documents-domain-docugraph-import-design.md) is authoritative.

Code layout (single-owner, GCF 02.01):

Concern	Owner
Pure domain logic: identity, normalization, node shapes, merge/dedup, URL resolution, classification catalog	`backend/domain/documents_engine.py`
DocuGraph FME import engine: parse → preview → commit	`backend/domain/documents_import.py`
Moderation / blocklist	`backend/domain/documents_blocklist.py`
Orchestration: all `/documents`, `/node/{id}/documents`, `/admin/documents` endpoints	`backend/routes/documents.py` (router factory; `server.py` contact = one `include_router` line)
Classification catalog data	`backend/document_classification_catalog.json` (admin-extensible, atomic writes)
Frontend user view	`DocumentsZone` (read/list) in `frontend/plugins/detail-panel.js`; attach runs through the generic Add Information → Document dialog (`openAddInformationDialog` in `frontend/app.js`) — the retired bespoke `AttachDocumentFlow` modal is kept only as a design reference
Frontend admin view: Data Exchange import panel	`frontend/app.js`

The `documents` Domain — First-Class, Visible by Default¶

documents is a normal, first-class secondary domain like any other: its own container running the same server binary with EIDOS_DOMAIN=documents, internalPort: 8097, prefix @, structural predicate contains (registered in config/domains.yaml, the domain SSOT).

Two predicates are in play and must not be confused:

contains — the domain's own structural (Hierarchy) predicate, used within the documents tree (container → document), exactly like every other domain's structural predicate.
hasDocument — the cross-domain attachment edge from an item in any domain to a document node in the documents domain.

Visibility is controlled by the generic per-domain hidden option (config/domains.yaml). Backend access is never gated by hidden — it is a UI-surface flag only; per-node IAM (read/write gates) governs actual access regardless of visibility.

Node classes inside the domain: Document (leaf, carries the source), DocumentContainer (holds imported repo trees; also the single "External links" container for user-attached external documents).

Data Model — Two Source Types¶

Every document node carries source_type plus source-specific engineering entries. Identity is deterministic uuid5 — external documents via derive_document_uuid("external", normalized_url); imported repo nodes keep DocuGraph's own uuid5 verbatim (never reminted, so the same FME export always maps to the same node).

Source type	Origin	URL resolution	Engineering entries
`external`	A URL reference the user adds through the generic "Add Information → Document" dialog — either attach an existing document, or paste a new hyperlink (`POST /documents` with `type=external`, external-only since PR #459)	stored URL as-is (a path-less URL like `https://www.dr.dk/` is labelled by its host, `www.dr.dk`)	`url`
`repo` (DocumentContainer / Document under one)	DocuGraph FME import (see §DocuGraph FME Import below)	read-model fallback: `docLink` → `webURI` → legacy `url`	`docLink`, `webURI` (reserved), `source_type=repo`, plus `import_fingerprint` / `import_source` on the container

Imported Document nodes render with readOnly: true in the read-model payload (documents_engine.document_node_to_payload), so the detail panel disables edit affordances — DocuGraph is the sole producer for repo content. Since the fresh rebuild (PR #459) external nodes are read-only references too: document-content editing was removed (PATCH /documents/{id} deleted), along with the entire legacy DocumentLibrary route/upload surface (GET /documents/libraries, the library source branch of POST /documents, and the DocumentLibrary node class). A document is created once — a repo import, or an external URL reference — and thereafter only attached / detached, never edited in place. hasRelatedDocument (doc-to-doc, 1-hop) is a registered relation-catalog predicate surfaced read-only in the subpanel.

Shared metadata entries: classification (default unspecified, catalog-validated ids), description, filename (derived by derive_filename), fileType (derived by derive_file_type, fallback LINK). Title defaults to filename without extension. derive_file_type maps a known extension to an uppercase token via _KNOWN_FILE_TYPES — PDF, the Office family (doc/docx, xls/xlsx/xlsm, csv, pptx), CAD (dwg/dxf/step/ifc), images, txt, zip — and returns LINK for anything unrecognised (a bare hyperlink with no file extension).

Attaching a Document (UX)¶

Since the fresh rebuild, attaching is done through the one generic relation dialog every other relation uses — there is no separate document modal. The user opens Add Information, picks Document, and lands in the standard (source node) —hasDocument→ (target) picker:

Attach an existing document — pick a target document node in the tree. The commit routes to POST /documents with a documentId (the cross-domain attach path; a generic CreateRelation to a document node is not used, since that fails for cross-domain document targets).
Add a hyperlink — paste a URL into the in-dialog "＋ Add hyperlink" field. POST /documents with an external source creates the reference node and the hasDocument relation in one step. A path-less URL falls back to its host for the label.

The attach request is sent to the source node's own domain backend (resolved from the node's domain, not the panel's current base), so a document attaches correctly to items in any domain (product, location, signal, type, discipline). hasDocument is only valid from those model domains — you do not attach a document to another documents node.

File-Type Colour Families (frontend — one classification)¶

Both surfaces that show a file-type badge — the attach-dialog chips (.adl-type-chip--*) and the document-list / panel / header-pin pills (.ndp-doc-fmt--*) — colour by family, never by raw extension. The extension / fileType-token → family mapping is a single canonical registry (GCF 02.01 §3.6): _fileTypeFamily() in frontend/plugins/detail-panel.js. _fileTypeChipClass (chips) and _docFmtPillClass (pills) both derive from it, so the classification lives in exactly one place.

Families: pdf; docx (doc/docx/rtf/odt/txt/md); xlsx (xls/xlsx/xlsm/csv/ods); dwg (dwg/dxf/step/stp/ifc); png (png/jpg/jpeg/gif/webp/svg); and other (everything else — neutral base).

The family tones are defined once as --ft-* CSS custom properties in frontend/index.html (--ft-pdf, --ft-docx, --ft-xlsx, --ft-dwg); both class families reference those variables, so a tone is never duplicated. There are no per-extension colour rules — the former .ndp-doc-fmt.<ext> store was a parallel implementation that drifted from the chip classes (e.g. xlsm coloured in one surface but grey in the other) and was removed (#464/#465).

Adding a new file type is therefore at most two edits: the backend token (_KNOWN_FILE_TYPES in derive_file_type, so the badge text is right) and the frontend family map (_fileTypeFamily, so the colour is right). No CSS change is needed unless a genuinely new family/tone is introduced.

DocuGraph FME Import¶

backend/domain/documents_import.py is the sole write path for repo content. It is a three-stage boundary — parse_and_validate → preview → commit — matching the Excel importer's shape (docs/manual/dev/import-pipeline.md).

Function	Purpose
`parse_and_validate(raw) -> ImportedRepo`	Parse a single DocuGraph EIDOS-FME 1.3.0 view-mode `.jsonld` payload and enforce shape (root count, allowed classes, container `source_type=repo`). Raises `ImportValidationError(code, message)` — Contract 05 structured error — on any shape or version drift.
`preview(imported, eff) -> dict`	Diff the parsed repo against the effective graph. Node status is keyed on the DocuGraph uuid5 id vs `eff["nodes"]`; relation status is keyed on `seeded_rel_id()` membership in `eff["relations"]`. Read-only. Returns `{fingerprint, summary:{added,updated,unchanged}, nodes:[...], relations:[...]}`.
`commit(imported, eff, username) -> dict`	Write overlay nodes + relations via `mutation_store.write_atomic`. Fingerprint-idempotent: if every container already carries `import_fingerprint == imported.fingerprint`, returns `{unchanged: True, skipped: N}` without writing. Re-import of a previously-tombstoned repo raises the nodes from the dead (overlay-wins).
`ImportedRepo` (dataclass)	Parsed payload: `containers`, `documents`, `nodes`, `relations`, `fingerprint`, `source`, `root_id`.
`ImportValidationError`	`ValueError` subclass carrying stable `.code` + `.message`; endpoint boundary maps into the mixed-outcome per-file response shape.

Endpoints (backend/routes/documents.py, all admin-gated, all forwarded to the documents instance when this backend is not it):

Method + Path	Purpose
`POST /admin/documents/import/preview`	Batch-preview one or many `.jsonld` payloads. Always returns 200 with a per-file mixed-outcome table (`{name, summary, nodes, relations}` or `{name, error:{code, message}}`).
`POST /admin/documents/import/commit`	Commit one or many pre-previewed payloads. Blocks the whole batch with 400 `IMPORT_CONFLICTS` on any validation failure; writes overlay nodes+relations via `commit()`, invalidates the cache ONCE after the batch (Contract 13 bounded cost), audits `ImportDocumentRepo` per file.
`GET /admin/documents/imported-repos`	List every imported `DocumentContainer + source_type=repo` with its descendant Document count, source, fingerprint.
`DELETE /admin/documents/import/{repo_id}`	Cascade-tombstone the container and every descendant Document + their relations via `delete_node(cascade=True)`. Idempotent — repeat delete returns `{removed: false}`, not 404.
`POST /admin/documents/purge`	DEV-only clean slate (guarded by `confirm=='PURGE'`): snapshot the domain, then cascade-tombstone every direct child of the domain root; per-child failures are collected into `failed:[{id,code,message}]` in the response.

Root binding. commit() re-binds any node whose parent is the exported root to the live eff["domain_root_uid"] — the exported root is never written; the live #Documents root is authoritative.

Cascade semantics. merge_overlay (backend/domain/graph_merge.py) filters both base and overlay relations whose endpoint is in node_tombstones (Contract 06 §3.11 R37-R39). This keeps the composite view free of dangling edges after a cascade delete.

Document Resolution — Where It Runs, and Four Channels¶

GET /node/{node_id}/documents returns the union of up to four channels, deduplicated with a locked priority order. The first three are provenance channels for documents attached to the viewed item; the fourth surfaces the viewed node's own doc-to-doc links.

Direct — the item's effective outgoing hasDocument relations targeting the documents domain.
From-type — the item's hasType targets (resolved via _resolve_cross_domain_target); each type node's own hasDocument external relations contribute refs labeled with the type. Cross-domain walk.
Inherited — ancestor hasDocument relations only when they carry inherit_source: true on the relation (locked: absent/false = not inherited), labeled with the ancestor.
Related — when the viewed node itself carries hasRelatedDocument relations (a Document node's References / Referenced-by), its 1-hop neighbours, both directions, surface as a top-level related key: {"references": […], "referencedBy": […], "count": <int>}, projected via de.related_doc_ref — the identical projection each direct document's own per-row inline expander already uses (_attach_related, #445/#444). Omitted entirely when there are no neighbours; this channel is local-only (the viewed node's own relations), so it never appears on the cross-domain secondary-item path below.

merge_document_sources(direct, from_type, inherited) dedupes channels 1-3 by document with locked priority direct > from-type > inherited. primarySource is the strongest channel and sources preserves all provenance labels. Channel 4 is independent of that merge — it augments the response, it does not compete with a document's provenance.

By design, not a bug: channel 4 only ever appears on a Document node's own panel. Channel 4 is sourced from the viewed node's own hasRelatedDocument relations — it has nothing to do with which domain the viewed node lives in, only with whether the viewed node itself has doc-to-doc neighbours, which is only ever true for a Document node. A Products/Type/Location item's Documents panel therefore never shows a "Related documents" section, while a Document node's own panel does — this is a real, visible structural difference between the Documents domain's own subpanel and every other domain's, and is intentional (see _related_node threading above), not something the union-read fix or any other panel-parity work should try to equalise.

Rule registry mapping. The two propagation channels (type→instance, parent→child-with-inherit) are registered as read-only builtin descriptors in the canonical RuleCatalog (domain/rule_catalog.py _BUILTIN_DOCUMENT_PROPAGATION, category documentPropagation, provenance builtin) so the All-Rules audit (GET /admin/rule-catalog) enumerates them. The descriptors are DESCRIPTIVE ONLY.

Where the resolver runs — primary vs. secondary vs. document domain¶

The canonical rule: assemble where the relations and the documents domain are both visible.

Primary-domain items (e.g. product) and document-domain nodes (viewing a doc) resolve locally: get_node_documents (backend/routes/documents.py) reads the item's own effective outgoing relations via _resolve_effective_outgoing (canonical accessor), classifies them into the three channels (classify_document_refs), and — since this instance holds the viewed node itself — also wires channel 4 from the node's own hasRelatedDocument relations.
Non-document secondary-domain items (location, type, signal, discipline, …) resolve on the primary, not on the secondary instance. A secondary-domain item's hasDocument/hasType relations can live in either of two stores: the primary's overlay (external_source_domain == <domain>, the exact convention _apply_secondary_overlay uses to enrich the detail panel) when the relation was written through the primary, or the secondary instance's own local store, because the existing attach route (POST /api/proxy/{server}/documents) forwards the write verbatim to the secondary instance itself, which persists it there. Reading only the primary's overlay (as the read path did before this fix) found zero relations for anything attached through that route, so the Documents panel showed nothing for a Location/Type/etc. item even when a real hasDocument attachment existed (bug #1).

GET /api/proxy/{server}/node/{id}/documents (backend/api/proxy_router.py, proxy_node_documents) now validates server against the registered secondary domains first (an unknown/invalid server 404s "Unknown domain", the same shape every other proxy route gives — it no longer silently returns an empty documents list), then branches: for server == "document" it still forwards verbatim (a doc node's own channels are local to the documents instance, per above). For any other secondary, it does not forward — the primary assembles the response itself: - collect_cross_domain_document_refs(server_mod, server, node_id, auth) rebuilds the same effective-relations shape from the primary's overlay (same external_source_domain == server scoping and ancestor-set construction _apply_secondary_overlay already implements for inherited pills), hands them to the same classify_document_refs used by the local path (Contract 02.01, single owner) — and then unions in the secondary node's own hasDocument relations, read directly off its raw node fetch (GET /node/{id} on the secondary, identity-forwarded) via _secondary_own_hasDocument_refs. The two lists are deduped by document id with the primary-overlay entry winning on conflict, so a relation visible in both stores is never shown twice. A 403/404 from that raw-node fetch (no access, or the node genuinely has no local record) falls back to overlay-only results, never an error; any other non-200 is a domain-level infra failure and raises (Contract 05), matching _fetch_document_nodes's own policy. - _assemble_documents_response(...) then does the identical fetch (_fetch_document_nodes, identity-forwarded) + merge (merge_document_sources) + payload (document_node_to_payload) + per-doc related-attach (_attach_related) as the local path. The cross-domain call passes no related_node, so channel 4 never appears here — it is only wired for the node the resolving instance actually holds.

Net effect: a Location or Type node's Documents panel is populated from the primary — reading BOTH the store the write path historically used and the store the read path originally expected — through the same merge/priority machinery as a primary-domain item, with no duplicated channel logic between the local and cross-domain paths. This union-read is a compatibility fix, not the end state: the write and read paths still disagree about which store is authoritative, and collapsing to one store (with a migration for existing secondary-store data) is tracked separately (BL-DM-020).

Error handling (Contract 05) — infra failure vs. per-document no-leak¶

_fetch_document_nodes fetches each resolved document id from the documents domain with the caller's real identity forwarded (on_behalf=auth.user_id), so the no-leak filter always evaluates the actual principal, never a service identity:

A per-document 404/403 (missing or restricted) is dropped silently — the id is simply omitted from the result, so no filename/path/url/title of a withheld document ever appears (Contract 09, unchanged).
Any other non-200 from the documents domain (5xx, or a transport/timeout failure surfaced by the proxy layer) raises (ExternalServiceError) instead of being swallowed. Previously any non-200 was treated as "not found," so an unreachable or broken documents domain silently read as an empty document list. Now a genuine infra failure surfaces as an error to the caller, while a normal missing/restricted document still degrades gracefully to "not shown."

URL Normalization and Scheme Guard¶

normalize_external_url strips whitespace, lowercases scheme+host, drops #fragment, keeps ?query, and preserves path case. The scheme guard accepts only http://, https://, file:// — anything else (javascript:, data:, vbscript:, scheme-relative //…, relative paths) raises ValueError so it can never become a document node or a stored, clickable URL. This is the stored-XSS guard.

normalize_relative_path rejects any traversal (../, ..\\, embedded /../).

Moderation / Blocklist¶

backend/domain/documents_blocklist.py implements the admin blocklist. The admin surface (/admin/documents/blocklist/*) accepts blocklist entries, and the resolver silently drops any document whose id or URL matches at read time. Blocked documents never appear in GET /node/{id}/documents or the Documents admin browse; the entry is stored, not the resolved output — unblocking restores visibility without re-attaching.

Documentation 360°¶

Documentation 360° (issue #490) is a separate, downstream consumer of this module's three direct document channels — it does not add a fourth channel here, and it is not owned by this engine. For a selected node (optionally its whole subtree) it produces an A4 PDF document list explaining, per node, which documents belong there and why.

The three direct channels above (direct, from-type, inherited) are its starting point for every analysed node. On top of them, a new v2 rulebook category, documentContext, adds two further, rulebook-governed context kinds: doc-to-doc (one hop over a document's own hasRelatedDocument neighbours) and via-node (the direct-channel documents of another node reached over an associative relation). Both are opt-in, rulebook-driven, and fully editable/removable — there is no hardcoded fallback.

See the Documentation 360 Engine page for the occurrence model, the documentContext category, deduplication, ordering, the IAM invariants, and the API; see the Rulebook Engine page for the documentContext schema entry.

Route Layer (orchestration, not owned by this engine)¶

backend/routes/documents.py owns orchestration for the /documents*, /node/{id}/documents and /admin/documents* endpoints. It composes the engine's pure functions (documents_engine, documents_import, documents_blocklist) with IAM gating, cross-domain proxying, and the internal POST /documents/nodes/upsert handler (service token + X-Eidos-Username; refuses to run unless EIDOS_DOMAIN=documents). The engine layer is route-agnostic and holds no HTTP or FastAPI dependencies — Contract 02.01 layer discipline.

The /node/{id}/documents resolver itself is split into small, independently reusable pieces so the local and cross-domain placements above share one implementation: classify_document_refs (pure channel classification over an already-resolved effective-outgoing-relations list), collect_cross_domain_document_refs (the primary-side overlay sourcing that feeds it for a secondary item), _fetch_document_nodes (identity-forwarded, no-leak, infra-failure-raising document fetch), and _assemble_documents_response (fetch + merge + payload + related-attach, shared verbatim by get_node_documents and proxy_router.proxy_node_documents). backend/api/proxy_router.py owns the cross-domain entry point (proxy_node_documents) that decides, per server, whether to forward to the documents instance or assemble on the primary.