Export Engine¶
Verified against the code · 59cdd26ce9c6fd83e4c19969b3f9cffd1a189edb · 2026-10-20
956 D1b-2 (2026-10-20): 7 finding(s) fixed — admin helpers are _buildFullModelDownloadPath/_buildProjectExportPath, export_service imports infra.iri (id_from_envelope) and infra.target_ref, build_export_doc lists all four extra kwargs, the share link is origin+pathname+'#'+EidosIRI.toIri(uuid), 15 export formats, exports filter via IamScope, plugin tests use .test.mjs (no export-boq test). Re-verified against main 4d5c9b57. #946 M1 (2026-10-21): FME 1.7.0 — relation @id in every mode, mode=migration (strict provenance/tombstones/auditByDomain, derived markers), GET /admin/export/full-model/audit; written against export_service.py and routes/domain_io.py on the #946 branch, pin moves to the merge commit after merge. #946 M2 (2026-10-21): Row shape corrected — archive mode, not only migration, drops created/createdBy (every _RELATION_PROVENANCE_KEYS key) on inherited rows, M1 behaviour the page omitted; migration node/relation fan-out now strict (ExternalServiceError, no reachable:false); tools/migration package builder documented. Checked against export_service.py _stamp_relation_ids at 79cc74e7 and routes/domain_io.py on the #946 branch. #946 M3 (2026-10-21): migration package sidecars documented — per-domain access_rules.db/interfaces.db (online backup + VACUUM), notifications.db (notifications + user_changelog_seen only), instance.json, relation_catalog.json; --shared users.json (tokens dropped, class secret), rulebook, property catalogs; access_rules reference check and secret scan refuse the build. Checked against tools/migration on the #946 branch (base 59cdd26c). #1049 RDF WP0-c (2026-10-21): migration mode now carries secondary provenance and full secondary tombstones via the new strict GET /admin/export/full-model/provenance fan-out; the "not included yet" sentence removed. Checked against routes/domain_io.py and export_service.py on the #1049 branch, KM pin re-verified at main 59cdd26c. #946 M4b (2026-10-21): migration import now rewrites legacy urn:eidos:* @id values in mutations/{nodes,relations,node_tombstones,tombstones} to the canonical IRI of the same uuid, and urn:eidos:audit: values of eidos:createdAuditRef/updatedAuditRef/auditRef to audit_iri of the same uuid (eidos:source stays literal: the external placeholder uuid is derived from it), refuses an unparseable or file-name-mismatched id or an unparseable audit reference before any write, writes id-mapping.json (rewritten + auditRefsRewritten) per domain target, fsyncs staged files and refuses equal or nested --domain/--shared targets; M4a import paragraph corrected. Checked against tools/migration on the #946 branch (base e29f1810). #935 S2 (2026-10-22): tools.migration convert subcommand noted with a pointer to the EIDOS Loader page. Checked against tools/migration/cli.py on the #935 branch (base 3fd2448f). #1071 RDF WP0-c (2026-10-22): an audit reference that is exactly urn:eidos:audit: (empty id) is kept byte-identical and listed under emptyLegacyRefs in id-mapping.json instead of refusing; an audit reference must name a uuid, and an empty urn in @id still refuses. Checked against tools/migration/identity_rewrite.py on the #1071 branch (base 19059cc0). #935 S7 (2026-10-22): the raw data endpoints read the overlay-only effective graph, no .eidos seed. Checked against mutation_store.py at a11d2636. #1102 S8a (2026-10-22): migration import writes every relation in the current eidos:target form with the placeholder uuid explicit as eidos:node, drops the four pre-WP4b keys, makes an IRI-form eidos:source the bare uuid, refuses a ref-without-domain or uuid-less in-model target, and lists every changed field under relationFieldsRewritten in id-mapping.json; the M4b "eidos:source stays literal" sentence removed. Checked against tools/migration/identity_rewrite.py on the #1102 branch (base 538bb015). #989 WP9 S4a (2026-10-23): FME 2.0.0 — every node/relation identity value (world id, hasDomain from/to, node id/parent, relation from/to/source/id/inherited_onto/inherited_from/external_uuid, engineeringInherited source_uid, tombstone id) is its IRI in every mode via export_service._identity_iris on copies the export owns (a placeholder to kept as its IRI, via target_ref.relation_target_iris), after every bare-uuid join and before the fingerprint; audit sections and treeId keep the bare uuid; /internal/reconstruct passes identity_iris=False; tools.migration builds, verifies and imports 2.0.0, the DocuGraph reader is unchanged (S4b). Checked against export_service.py on the #989 branch (base 8e8c8dd3). #1112 WP9 S4b (2026-10-23): the readers accept 2.0.0 — the DocuGraph importer reads it to the same bare uuids as 1.x, tools.migration build refuses an export whose formatVersion is not in SUPPORTED_FME_VERSIONS or that carries a legacy urn node id, an unknown major is a typed refusal naming the version, import accepts 1.7.0 and 2.0.0; checked against documents_import.py, tools/migration/cli.py, builder.py and refcheck.py on archon/issue-1112 at 86bac62b. #1125 #607 S1 (2026-10-25): every FME node row carries the additive eidos:propertyOrigin (own / inherited with the source node IRI), from domain.property_origin via routes.domain_io._flat_export_node, also on a secondary's GET /nodes?full=true rows; eidos:engineering unchanged, fingerprint moves with the content. Checked against domain_io.py on archon/issue-1125 (base b4bead2f). #1125 send-back r1 (2026-10-25): FME 2.1.0 — FORMAT_VERSION bumped for eidos:propertyOrigin and the #1126 inherited-row occurrence_id/occurrence_ref/derivation; documents_import and tools.migration accept 2.1.0, 2.0.0 stays readable. Checked against export_service.py, documents_import.py and tools/migration/importer.py on archon/issue-1125 (base 535c9321).¶
Layer: api
Overview — Export Architecture¶
The Export Engine is responsible for all file downloads that EIDOS Explorer produces: structured data tables (CSV/XLSX) and rendered diagrams (PDF). It is built around a plugin-per-format model: every export format lives in a dedicated JavaScript file under frontend/plugins/. Each plugin registers itself with a central in-process registry (EidosExportRegistry) at page load time. The application core (app.js) does not know about any specific format; it only knows how to open the registry, present the user with a choice, and dispatch to the chosen plugin.
The split between "data exports" and "PDF reports" is fundamental to the architecture:
- Data exports run entirely in the browser for data assembly (calling backend REST endpoints to fetch raw graph data), then POST the rendered rows to a backend staging endpoint (
POST /export/stage) which returns a short-lived token. The browser immediately follows up withGET /export/d/{token}to download the file. - PDF report exports (marked
kind: 'pdf-report') follow a three-stage pipeline mandated by ADR-007 Approach C: the browser GETs a Mermaid diagram source from the backend, renders the SVG client-side using the Mermaid library, then POSTs the SVG back to the server for PDF assembly using ReportLab/svglib.
All 15 Export Formats¶
| Registry ID | Plugin File | Category | Description |
|---|---|---|---|
boq |
export-boq.js |
data | Bill of Quantities: one row per hasType-grouped component type with count and paths |
component_list |
export-electrical.js |
data | All Component-typed nodes with type, description, location |
signal_list |
export-electrical.js |
data | Terminals carrying hasSignal relations, grouped by signal |
cable_list |
export-electrical.js |
data | IEC-prefixed cable nodes with from/to system and component detail |
relation_list |
export-relations.js |
model | All effective outgoing relations for the primary domain, import-compatible |
property_list |
export-properties.js |
model | Engineering properties (direct and inherited) for every node |
product_nodes |
export-products.js |
model | Complete product-node tree snapshot with root, import-compatible |
product_nodes_template |
export-products.js |
model | Product nodes without the root row, for re-import under any anchor |
system_list |
export-products.js |
data | Functional and/or Technical systems filtered by node type |
location_nodes |
export-locations.js |
model | Complete location-domain node tree with root |
location_nodes_template |
export-locations.js |
model | Location nodes without root, for re-import under any anchor |
reference-whitelist |
export-reference-whitelist.js |
model | Styled XLSX per DS/ISO 81346-12:2018 Annex A, whole-project only |
fsb-depth |
export-system-breakdown.js |
model | Functional System Breakdown PDF, depth-gradient palette |
fsb-category |
export-system-breakdown.js |
model | Functional System Breakdown PDF, category-hue palette |
system-context |
export-system-context.js |
model | System Context & Interface Diagram PDF anchored at selected node |
Note: report-ref-arch.js also loads as a plugin but does not register with EidosExportRegistry; it
publishes EidosCore.module('RefArch').triggerRefArchDownload(lang, setMsg) on the reverse registry
(4.5c-3 #774, previously window.triggerRefArchDownload) and is invoked from the Insights → Export
submenu independently.
Export Plugin Interface¶
Registry Contract¶
Every export plugin must obtain references to the two global singletons and call registry.register() before returning:
const registry = window.EidosExportRegistry;
const core = window.EidosCore;
if (!registry || !core) {
console.warn('[my-plugin] EidosExportRegistry or EidosCore not found — plugin skipped');
return;
}
registry.register('my-export-id', { /* definition */ });
Data Export Definition¶
For formats that produce tabular rows (CSV/XLSX), the definition object must include:
registry.register('my-export-id', {
label: 'Human-readable name shown in the dropdown',
headers: ['Column A', 'Column B'], // header row for the file
keys: ['col_a', 'col_b'], // property names on each row object
build: buildMyRows, // async function (scope, setMsg, options?) → row[]
summarize: summarizeMyRows, // optional: (rows) → [{label, count}]
category: 'model' | 'data', // 'model' = schema exports; 'data' = derived listings
options: [ /* radio option defs */ ], // optional: triggers configure card before preview
});
build(scope, setMsg, options) is the core contract:
async function buildMyRows(scope = 'global', setMsg = () => {}, options = {}) {
// scope: 'global' | 'subtree'
// setMsg: (message: string) => void — call to update the loading indicator
// options: object with values from any declared options (radio choices)
// Return: array of plain objects whose keys match the keys array
}
| Param | Type | Description |
|---|---|---|
scope |
'global' \| 'subtree' |
Whether to include all nodes or only the selected subtree |
setMsg |
(msg: string) => void |
Callback to update progress text in the export preview panel |
options |
object |
Key/value pairs from any declared option radios (e.g. {systemType: 'functional'}) |
PDF Report Definition¶
For PDF report exports, the definition uses kind: 'pdf-report' and replaces build/headers/keys with a trigger function:
registry.register('my-pdf-id', {
kind: 'pdf-report',
category: 'model',
label: 'My PDF Report',
description: 'Description shown on the configure card.',
wholeProjectOnly: false, // true → disables subtree scope radio
actionLabel: 'Generate PDF', // button text on the card
options: [ /* radio option defs */ ],
trigger: myTriggerFn,
});
trigger(opts, setMsg) is the PDF plugin contract:
async function trigger(opts, setMsg) {
// opts: merged object of {scope: 'global'|'subtree', ...optionValues}
// setMsg: (message: string) => void
// Must throw on error — the caller surfaces err.message to the user
}
| Param | Type | Description |
|---|---|---|
opts |
object |
Includes scope from the dialog plus values of any declared option radios |
setMsg |
(msg: string) => void |
Progress indicator callback |
How the Application Activates Plugins¶
Plugins are not statically imported. At boot, loadConfig() fetches /config from the backend, which returns the list of plugin IDs enabled for this instance. _activatePlugins(ids) then injects a <script> tag into the document head for each enabled plugin:
function _activatePlugins(ids) {
return Promise.all((ids || []).map(id => {
const path = _KNOWN_PLUGINS[id];
if (!path) return Promise.resolve();
return new Promise(resolve => {
const s = document.createElement('script');
s.src = path + '?v=' + _DEPLOY_VER; // cache-busted per deploy
s.onload = resolve;
s.onerror = () => {
console.warn(`[EIDOS] Plugin failed to load: ${id}`);
resolve(); // non-fatal: other plugins still load
};
document.head.appendChild(s);
});
}));
}
Each plugin script executes as it loads and calls registry.register() during that execution. After all plugin promises settle, _pluginsLoaded is set to true. openExportModal() is a no-op if called before this flag is set (BL-IE-015).
The mapping of plugin ID to file path lives in _KNOWN_PLUGINS in app.js. A plugin ID absent from this map is silently ignored, which means new plugins require a corresponding entry in _KNOWN_PLUGINS in addition to their file.
Backend Export Endpoints¶
Data Staging Endpoints¶
POST /export/stage — accepts a pre-built data payload and returns a one-time download token.
Request body:
{
"headers": ["Col A", "Col B"],
"rows": [["val1", "val2"]],
"format": "xlsx",
"filename": "my-export"
}
| Field | Type | Description |
|---|---|---|
headers |
list[str] |
Column header row |
rows |
list[list] |
Data rows; each inner list must align with headers |
format |
"xlsx" \| "csv" |
Output format; CSV is UTF-8 BOM-encoded for Excel compat |
filename |
str |
Base filename (extension is appended by the server) |
Response:
{ "url": "/export/d/<token>" }
The token is a 16-byte URL-safe random string with a 5-minute TTL. Maximum 100,000 rows. The staged file is written to the EXPORT_STAGE_DIR directory (default /tmp/eidos-export-stage).
GET /export/d/{token} — streams the staged file with Content-Disposition: attachment. Bearer auth required, and the caller must be the user who staged the token (meta.owner, recorded by /export/stage) — an unguessable token is not sufficient on its own; a leaked URL is not replayable by anyone else (hardening 0.4, #514). The ownership check runs before the file is consumed, so a foreign caller (403) cannot burn the single-use token. The file is deleted from disk immediately after a successful owner download.
_render_export_bytes(headers, rows, fmt, column_meta=None) — internal function called by the staging endpoint. /export/stage forwards payload.get("column_meta"); the xlsx branch uses it for per-column styling, alignment and numeric formatting, the csv branch ignores it:
def _render_export_bytes(headers: list, rows: list, fmt: str, column_meta: list = None) -> tuple[bytes, str]:
# Returns (file_bytes, content_type)
# fmt == 'xlsx': uses openpyxl Workbook
# fmt == 'csv': uses csv.writer with ';' delimiter, UTF-8 BOM prefix
Raw Data Endpoints (called by plugins)¶
GET /mutations/products/export
Query parameters:
include_root: bool (default false) — include the tree root row
Returns one row per product node in depth-first order. Row shape: {node_path, reference, description, nodeType, note}. Powered by build_product_export_rows() in backend/api/excel_export.py. IAM-filtered: nodes the caller cannot read never appear.
GET /mutations/relations/export
Returns all effective outgoing relations for the primary domain. Row shape: {source_path, target_path, relation_type, inherit_source, is_inherited}. Includes inherited relations computed by _resolve_effective_outgoing(). IAM-filtered: targets outside the caller's read scope are replaced with [restricted].
GET /mutations/properties/export
Returns all engineering properties (direct and inherited) for every node in the primary domain. Row shape: {node_path, property, value, unit, group, inherited, source_path}. IAM-filtered at both the node and the inheritance source ancestor.
GET /api/proxy/{server}/mutations/products/export
Proxy endpoint that surfaces a secondary domain's node export (e.g. the Location domain) through the primary server. Used by export-locations.js so the location plugin can call apiFetch('/mutations/products/export', locationBase()) without CORS issues.
PDF Report Endpoints¶
These are owned by the Reports Engine but are called directly by PDF export plugins:
| Endpoint | Method | Description |
|---|---|---|
GET /api/reports/system-breakdown |
GET | Returns {mermaid_src, anchor_description} for the breakdown diagram |
POST /api/reports/system-breakdown/pdf |
POST | Accepts {svg, palette, scope, ...}, returns application/pdf |
GET /api/reports/system-context |
GET | Returns {mermaid_src, anchor_label, anchor_description} for context diagram |
POST /api/reports/system-context/pdf |
POST | Accepts {svg, lang, anchor_label, anchor_description}, returns application/pdf |
GET /api/reports/reference-whitelist.xlsx |
GET | Returns a styled multi-sheet XLSX directly |
The PDF POST endpoints set response headers X-Render-Scale, X-Render-Pt, and X-Render-Legible so the frontend can warn the user when body text falls below 9pt.
A Complete Export Flow¶
Example: Bill of Quantities (BOQ) Export¶
1. User opens the Export dialog
The user clicks the Export button. openExportModal() calls EidosExportRegistry.getAll(), groups the registered types by category, and populates the <select> element. The user selects "Bill of Quantities" and chooses scope "All nodes", then clicks "Preview".
2. openExportPreview() dispatches to _runDataExportPreview()
async function openExportPreview() {
const def = EidosExportRegistry.get('boq');
// def.kind is undefined (not 'pdf-report'), def.options is undefined
// → goes directly to _runDataExportPreview
await _runDataExportPreview(def, 'boq', 'global');
}
3. buildBoqRows(scope, setMsg) runs in the browser
async function buildBoqRows(scope = 'global', setMsg = () => {}) {
// Step A: fetch the full node list
const probe = await apiFetch('/nodes?limit=1&offset=0');
const total = probe.total || 0;
const data = await apiFetch(`/nodes?limit=${total}&offset=0`);
// Step B: in batches of 20, fetch /node/{id} for each candidate
// and resolve hasType relations to type nodes
const BATCH = 20;
for (let i = 0; i < candidates.length; i += BATCH) {
setMsg(`Resolving types… ${i} / ${candidates.length} nodes`);
await Promise.allSettled(batch.map(async c => {
const nd = await apiFetch(`/node/${c.id}`);
const rel = outgoing.find(r => r.predicate === 'hasType');
// group by type target URI → {typeName, count, paths[]}
}));
}
// Step C: sort, number rows, return
return rows; // [{linjeId, typeName, typeDesc, docLink, quantity, components}]
}
4. Preview table renders in the browser
_runDataExportPreview stores the rows in _exportRows, calls _repaintExportWrap() to render the table, and enables the Download buttons.
5. User clicks "Download XLSX"
The download handler collects _exportRows, maps each row to an array using the plugin's keys definition, and POSTs to /export/stage:
const body = {
headers: def.headers, // ['No.', 'Type', 'Description', ...]
rows: _exportRows.map(r => def.keys.map(k => r[k] ?? '')),
format: 'xlsx',
filename: 'bill-of-quantities-2026-06-23',
};
const { url } = await EidosApi.postJson('/export/stage', body); // canonical layer (4.1)
// The download is owner-gated bearer-auth (hardening 0.4, #514) — a plain
// navigation cannot carry the Authorization header, so we fetch through the layer
// (which injects the Bearer header) and save the returned blob under the real name:
const dl = await EidosApi.raw(url); // as:'response', auth:true, no timeout
_downloadBlob(await dl.blob(), filename); // deferred-revoke helper in app.js
6. Server stages and streams the file
export_stage renders the XLSX using openpyxl, writes it to the staging directory with a random token, records the issuing user as meta.owner, and returns {"url": "/export/d/<token>"}. The browser fetches that URL with its bearer header; export_download verifies the session and that the caller equals meta.owner (before consuming — a foreign 403 does not burn the token), then reads the file, deletes it from disk, and returns it as Content-Disposition: attachment.
Example: System Context PDF Export¶
1. User selects a node in the tree, then opens the Export dialog and selects "System Context & Interface Diagram".
2. openExportPreview() routes to _renderPdfReportCard(def, scope)
The configure card appears showing a language radio (English/Dansk) and an action button.
3. User clicks "Generate PDF"
The card's click handler calls def.trigger({lang: 'en', scope: 'global'}, setMsg).
4. trigger() in export-system-context.js runs the three-stage pipeline
async function trigger(opts, setMsg) {
const anchorId = core.selectedNodeId;
if (!anchorId) throw new Error('No node selected...');
// Stage 1: Fetch Mermaid source from backend (IAM-filtered). Since 4.1c (#676)
// the raw fetch + Bearer auth + primary routing live in the canonical EidosApi
// layer; EidosApi.raw() returns the Response unthrown so the plugin keeps its own
// !resp.ok / _friendlyExportError handling and response-header reads unchanged.
setMsg('Building diagram…');
const srcResp = await EidosApi.raw(
'/api/reports/system-context?anchor_node_id=' + anchorId + '&lang=en'
);
const srcBody = await srcResp.json();
// srcBody: { mermaid_src: '...', anchor_label: '...', anchor_description: '...' }
// Stage 2: Render SVG client-side (ADR-007 Approach C)
setMsg('Rendering diagram…');
const mermaid = await loadMermaid(); // lazy CDN load, singleton
const { svg } = await mermaid.render('eidosContext_' + Date.now(), srcBody.mermaid_src);
// Stage 3: POST SVG to backend for PDF assembly (through the canonical layer)
setMsg('Building PDF…');
const pdfResp = await EidosApi.raw(
'/api/reports/system-context/pdf',
{
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ svg, lang: 'en', anchor_label: srcBody.anchor_label, anchor_description: srcBody.anchor_description }),
}
);
// Stage 4: Trigger download
const blob = await pdfResp.blob();
const a = document.createElement('a');
a.href = URL.createObjectURL(blob);
a.download = 'system-context-2026-06-23.pdf';
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(a.href);
}
The backend /api/reports/system-context applies IAM filtering before building the Mermaid source. The backend /api/reports/system-context/pdf trusts the SVG from the browser and assembles the PDF using svglib + ReportLab. Legibility is checked post-render and surfaced via X-Render-Legible / X-Render-Pt headers.
Data Structures¶
Plugin Definition Object¶
{
// Required for all plugins:
label: string, // shown in the export type dropdown
category: 'model' | 'data',
// For data exports (kind absent or not 'pdf-report'):
headers: string[], // column header row
keys: string[], // property names matching headers, same order
build: async (scope, setMsg, options?) => object[],
summarize: (rows) => {label: string, count: number}[], // optional
options: OptionDef[], // optional, triggers configure card
cellRender: { [key]: (val) => string }, // optional HTML renderer per column
// For PDF reports (kind === 'pdf-report'):
kind: 'pdf-report',
description: string,
wholeProjectOnly: boolean,
actionLabel: string,
trigger: async (opts, setMsg) => void,
options: OptionDef[],
}
OptionDef (Radio Option)¶
{
id: string, // key in the options bag passed to build()/trigger()
label: string, // displayed label in the configure card
type: 'radio',
default: string, // value of the initially selected choice
choices: { value: string, label: string }[],
}
Export Row (data exports)¶
Each row is a plain object. Keys are defined by the plugin's keys array. Values are always strings (the staging endpoint serialises each cell as a string). For example, the BOQ row:
{
linjeId: '1',
typeDesc: 'Centrifugal process pump',
typeName: 'Pump',
quantity: '4',
}
Staged Export Token¶
// POST /export/stage response:
{ "url": "/export/d/<token>" }
// Filesystem layout on server:
// <EXPORT_STAGE_DIR>/<token>.bin — rendered file bytes
// <EXPORT_STAGE_DIR>/<token>.meta.json — { filename, format, content_type, owner }
// (owner = user_id of the staging caller; the download requires owner match — #514)
Error Handling¶
Plugin-Level Guards¶
Every plugin begins with a guard that bails out silently when global singletons are unavailable. This prevents a failed plugin from blocking the page load:
if (!registry || !core) {
console.warn('[my-plugin] EidosExportRegistry or EidosCore not found — plugin skipped');
return;
}
Load errors are caught by _activatePlugins's s.onerror handler, which resolves the plugin's promise without rejecting, so a network error fetching one plugin file does not prevent others from loading.
PDF Export Error Mapping¶
Both export-system-breakdown.js and export-system-context.js implement a _friendlyExportError(rawText, status, stage) helper that parses the server's {detail: "..."} JSON envelope and maps known error cases to user-actionable copy:
| Condition | User-facing message |
|---|---|
| "components cannot anchor" in detail | Direct user to select a non-Component node |
| HTTP 403 / "forbidden" in detail | Permission denied, contact admin |
| HTTP 404 / "not found" in detail | Node deleted or moved, refresh tree |
| HTTP 5xx | Server error, try again |
| Unknown | Raw detail string from server |
Staging Endpoint Validation¶
POST /export/stage enforces:
- format must be "xlsx" or "csv" — returns HTTP 400 otherwise
- headers and rows must be lists — returns HTTP 400 otherwise
- rows length must not exceed 100,000 — returns HTTP 413 otherwise
GET /export/d/{token} requires a valid bearer session (401 without) and validates the token against a strict [A-Za-z0-9_\-]{8,64} pattern before any filesystem access, returning HTTP 400 on mismatch. Expired or missing tokens return HTTP 404. A caller who is not the staging owner (meta.owner) receives HTTP 403 — without consuming the single-use token (#514).
Data Export Batch Errors¶
Data plugins fetch node detail in batches using Promise.allSettled(), so a single failed /node/{id} request does not abort the whole export — the failing node is simply skipped. The export-electrical.js cable list explicitly handles missing from/to sides by returning an EMPTY_SIDE sentinel, so a cable with no connections still appears in the output with blank endpoint columns.
Data Freshness Invariants¶
Per-run cache (cross-domain nodes): Every build function that resolves cross-domain targets (location, type) creates a fresh const runCache = new Map() at the top of the function. This Map deduplicates identical target URLs within a single export run while guaranteeing that a new export always fetches live data — no stale cross-domain node data can bleed in from a previous run. A module-level Map is forbidden for node data; only _domainsJsonCache (domain config, which does not change during a session) is permitted at module scope.
From/To component nodes in cable list: buildCableListRows must fetch the component nodes at each cable end via apiFetch('/node/{id}'), not via fetchNode() / core._fetchNodeByTarget(). _fetchNodeByTarget reads the module-level nodeCache, which holds a snapshot from the last time the user navigated to that node. After an in-session location edit the cached copy still carries the old isLocatedIn relation, making the export show stale location data. Direct apiFetch bypasses the cache and always returns the current backend state. (Regression: BL-IE-036)
Integration with Other Engines¶
Mutation Store / Effective Graph: The three raw data endpoints (/mutations/products/export, /mutations/relations/export, /mutations/properties/export) all call mutation_store.get_effective_graph() to obtain the live merged graph (the overlay-only effective graph + rule-derived edges). This means exports always reflect the current state including unsynchronised overlay changes.
IAM Engine: All export endpoints depend on the canonical auth surface (api/deps.py — CurrentUser for read exports, AdminOrService for admin/service exports) and filter rows through the IamScope predicate (iam.scope.build_scope → filter_readable / can_read, hærdning 3.2) in the /mutations/*/export handlers of backend/api/mutations_io_router.py. Nodes or relation targets outside the caller's read scope are excluded (products/properties) or masked as [restricted] (relations) before rows are returned.
Reports Engine: PDF export plugins directly call the Reports Engine's endpoints (/api/reports/system-breakdown, /api/reports/system-context). The Export Engine owns only the trigger-and-download half of the pipeline; diagram source generation and PDF assembly are fully owned by the Reports Engine (backend/reports/).
Excel Importer/Exporter: build_product_export_rows() in backend/api/excel_export.py is the single canonical owner of the product node row shape. It is imported by /mutations/products/export and mirrors excel_importer.parse_product_node_rows() column for column, guaranteeing that exported files round-trip through the importer without data loss.
Domain Registry: export-locations.js reads core.DOMAIN_REGISTRY.domains.location.baseURI at call time to resolve the Location domain's base URL. If no Location domain is registered, the plugin throws a descriptive error rather than silently producing empty output.
Adding a New Export Format¶
Follow these exact steps to add a new export type to EIDOS Explorer.
Step 1: Create the plugin file¶
Create frontend/plugins/export-myformat.js. Use an IIFE to avoid polluting the global scope:
(function () {
'use strict';
const registry = window.EidosExportRegistry;
const core = window.EidosCore;
if (!registry || !core) {
console.warn('[export-myformat] EidosExportRegistry or EidosCore not found — plugin skipped');
return;
}
const { apiFetch } = core;
async function buildMyRows(scope = 'global', setMsg = () => {}) {
setMsg('Fetching data…');
const rows = await apiFetch('/mutations/myformat/export');
return rows;
}
registry.register('myformat', {
label: 'My Format',
headers: ['Column A', 'Column B'],
keys: ['col_a', 'col_b'],
build: buildMyRows,
category: 'data', // or 'model'
});
}());
Step 2: Register the plugin ID in app.js¶
Add an entry to _KNOWN_PLUGINS in frontend/app.js:
const _KNOWN_PLUGINS = {
// ... existing entries ...
'export-myformat': 'plugins/export-myformat.js',
};
Without this entry, the backend config can list the plugin ID but _activatePlugins will silently skip it because _KNOWN_PLUGINS[id] returns undefined.
Step 3: Add a backend data endpoint (if needed)¶
If the format requires new data that no existing endpoint provides, add a FastAPI route to backend/server.py:
from api.deps import CurrentUser
@app.get("/mutations/myformat/export")
async def mutations_myformat_export(auth: CurrentUser):
"""Return rows for My Format export."""
user_id = auth.user_id
_require_mutations()
eff = mutation_store.get_effective_graph()
# Build and return rows as a list of dicts
return rows
Apply IAM filtering before returning rows (see /mutations/products/export as the reference implementation).
Step 4: Enable the plugin in the instance configuration¶
The backend /config endpoint returns the list of plugin IDs for the current instance. Add 'export-myformat' to the plugins list in the instance configuration file (domains.json or the equivalent deployment config). The exact mechanism is instance-specific; consult backend/routes/setup_router.py and the deployment documentation for the target instance.
Step 5: Write tests¶
Add a test file tests/test_myformat_export.py (backend endpoint) and frontend/plugins/__tests__/export-myformat.test.mjs (plugin build function; plugin tests in __tests__/ use the .test.mjs suffix). The existing test files tests/test_bl_ie_011_product_export.py and frontend/plugins/__tests__/export-electrical-freshness.test.mjs are the canonical references (there is no export-boq test).
Full-Model Export (FME)¶
export_service.build_export_doc() (in backend/persistence/export_service.py) assembles the full-model JSON-LD dump from pre-fetched domain data. The full-model export passes through the @id values already written by the façade-backed writers upstream (infra.iri.node_iri / relation_iri, G3 #930); export_service.py imports from infra.iri import at_id_entry, id_from_envelope, node_iri, relation_iri (id_from_envelope reads each tombstone/overlay envelope's id; at_id_entry + relation_iri stamp the relation @id, FME 1.7.0; node_iri + relation_iri write the identity values, FME 2.0.0) plus the infra.target_ref helpers external_domain_of / external_ref_of. The document contains:
| Field | Value |
|---|---|
@type |
eidos:FullModelExport |
eidos:formatVersion |
"2.1.0" (additive-only within the major version — a major bump requires a migration procedure) |
eidos:mode |
"view" (default), "archive" or "migration" (see Migration Mode) |
eidos:delivery |
{"source": "live"} — always present; signals the export was assembled from the live server |
eidos:fingerprint |
SHA-256 of canonical JSON of domains + auditLog + relationCatalog |
eidos:auditLog |
Included only when audit_log is not None |
eidos:metrics |
Archive-mode only — a model-metrics snapshot (see eidos:metrics below) |
eidos:metricsDefinitionVersion |
Archive-mode only — the definition-meaning version of eidos:metrics (e.g. "m1") |
eidos:auditByDomain |
Migration-mode only — [{domain, reachable, seq, events}] for the primary and every secondary; not covered by the fingerprint |
build_export_doc(name, domains, audit_log, mode="view", interface_lookup=None, web_uris=None, metrics=None, metrics_definition_version=None, metrics_failure_marker=None, inheritance_resolver=None, inheritance_eff=None, primary_domain_name=None, audit_by_domain=None, identity_iris=True)
# SOURCE: backend/persistence/export_service.py
doc = {
"@context": EIDOS_CONTEXT,
"@type": "eidos:FullModelExport",
"eidos:formatVersion": FORMAT_VERSION, # "2.1.0"
"eidos:mode": mode, # "view" | "archive" | "migration"
"eidos:delivery": {"source": "live"},
"eidos:fingerprint": fingerprint,
...
}
FORMAT_VERSION = "2.1.0" is a module-level constant. Evolution is additive-only within the major version; consumers parse eidos:formatVersion and reject exports whose major does not match. The 1.4.0 bump (from 1.3.0) is additive and archive-only — it adds eidos:metrics; view-mode bytes are unchanged, so a view-mode reader (e.g. the DocuGraph document importer) accepts both 1.3.0 and 1.4.0.
The 1.5.0 bump (from 1.4.0, #954 WP5 S6) is additive in both modes: every node in the primary domain's eidos:domains[].nodes[] gains @id — the canonical identity IRI infra.iri.node_iri(id) (https://id.ontoteq.com/{uuid}) — and notation, the node's WP2 slug segment (skos:notation, null on nodes created before WP2). Both keys are added by routes/domain_io.py::_flat_export_node (the @id through infra.iri.at_id_entry). The bare id and every other 1.4.0 node field are unchanged, so a reader written against 1.4.0 keeps working; the fingerprint stays stable across two exports of an unchanged graph. In 1.5.0, secondary-domain nodes (fetched from the secondary's own /nodes?full=true) did not carry the two keys; 1.6.0 closes that gap. The DocuGraph document importer (domain/documents_import.py) is a separate reader; its accepted-version set _SUPPORTED_FORMAT_VERSIONS is {1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 2.0.0, 2.1.0}.
The 1.6.0 bump (from 1.5.0, #988) is additive in both modes: every secondary-domain node in eidos:domains[].nodes[] now has the same @id and notation as a primary node, so all nodes in one export share one shape. routes/domain_io.py::_secondary_export_node stamps @id from the façade (infra.iri.node_iri(id)) and never takes it from the peer's payload, so a secondary still running pre-1.6.0 code exports the same shape. notation is only what the secondary supplies (GET /nodes?full=true now returns it), and null when it has none — never a guessed slug. No field is removed or renamed, so a 1.5.0 reader still reads the file. The fingerprint changes once, because the new keys enter the hashed content, and then stays stable across two exports of an unchanged model. .eidx and the other export formats are unchanged.
The 1.7.0 bump (from 1.6.0, #946 M1) is additive in every mode: each relation row in eidos:domains[].relations[] gains its own @id = infra.iri.relation_iri(source) (source is the relation's own uuid), stamped by export_service._stamp_relation_ids after inherited-row materialisation. Only stored relations carry an @id: a materialised inherited row (inherited_onto set) is derived, so it has no @id — sharing the stored relation's would make JSON-LD consumers merge the N+1 rows into one — and no provenance; it points to its origin with the non-identity eidos:derivedFrom = relation_iri(source). A row without source gets no @id, and eidos:hasDomain edges never carry one (they are not stored relations). It also adds the third mode, migration (below). View and archive output is otherwise unchanged, so a 1.6.0 reader still reads the file; the fingerprint changes once because the relation @id enters the hashed content. The DocuGraph importer accepts 1.7.0 (it reads view mode only, and only each relation's from/to/predicate).
The 2.0.0 bump (from 1.7.0, #989 WP9 S4a) is a major bump: the keys stay the same, but every node and relation identity value is now its IRI (https://id.ontoteq.com/{uuid}), in every mode — view, archive, migration and an as_of reconstruction. export_service._identity_iris writes them, through infra.iri.node_iri / relation_iri, as the last step before the fingerprint: eidos:world.id, eidos:hasDomain[].from/.to, every node's id and parent, every relation's from, inherited_onto and inherited_from (node IRIs), its target channels to and external_uuid (through infra.target_ref.relation_target_iris, node IRIs whatever the target's kind — in-model, external with a known remote uuid, external placeholder or unresolved; a placeholder to is kept, as its IRI, like the overlay's eidos:target since #1102 S8, and the kind stays readable from the external and unresolved fields) and source and, on a reconstruction-fold row, id (its relation IRI, so a stored row's @id equals its source), a migration node's eidos:engineeringInherited[].source_uid, and the id of every node and relation tombstone, the reduced archive shape and the full migration shape alike. An empty value stays empty — an unresolved cross-domain to is "" and a root's parent is null, never a guessed IRI — and a value that is already an IRI is not wrapped twice. Every step that joins on the bare uuid (WORLD extraction, hasDomain, the IIS lookup, inheritance materialisation, relation ids, external-ref resolution) runs before it, unchanged. It rewrites copies the export owns — every rewritten row, nested eidos:engineeringInherited[] entries included, is a new dict — so the effective graph, the tombstone and provenance loaders' results and the world row keep their bare uuids. The fingerprint is computed over the IRI content and stays stable across two exports of an unchanged model. Two things keep the bare uuid: the audit sections (eidos:auditLog, eidos:auditByDomain) are the recorded command payloads, and the uuid stays the internal storage key for audit; and treeId, which names a tree, not a node or a relation. A secondary's /internal/reconstruct builds its domain with identity_iris=False, because the primary joins that domain on the bare uuid and writes the IRIs once in its own document. The primary reads a secondary's /nodes, /relations and /tombstones rows through their IRI companions (domain_io._peer_ids, #1119 WP9 S5b), since S5c (#1120) dropped the bare fields from those responses, and then drops the response-only companions (parentIri, fromIri, toIri, external_iri, a tombstone's iri; _without_peer_iris, #1101 WP9 S2), so a secondary on either side of the cut gives the same export. Because the value shape changed, a 1.x reader must not read a 2.0.0 file; the export ships only after GraphAnalyzer and eidosdash accept major version 2. The readers accept 2.0.0 beside the 1.x files they read before (#1112 WP9 S4b), and read every identity through infra.iri. The DocuGraph importer accepts 2.0.0 in _SUPPORTED_FORMAT_VERSIONS and reads a node id/parent and a relation from/to through infra.iri.canonical_node_uuid, which takes the bare uuid of 1.x and the IRI of 2.0.0 alike, so a 2.0.0 file and the same model as 1.7.0 import to the same overlay; DocuGraph's own 1.3.0 files, with bare ids, import as before. tools.migration reads the 2.0.0 package this backend builds: import accepts SUPPORTED_FME_VERSIONS = {"1.7.0", "2.0.0", "2.1.0"} (2.1.0 since #1125, below), build refuses an export of any other version, and the build's access-rule reference check reads each node id through infra.iri.from_iri (a 1.x export's bare uuid is kept as it is). An unknown major, such as 3.0.0, is refused with a typed error that names the version — UNSUPPORTED_VERSION in the importer, a refused build or import in tools.migration — and never read on a guess. A legacy urn:eidos: node or relation id in an FME file is refused as the overlay readers refuse it since #1102 S8b: LEGACY_ID_REFUSED in the importer, and LegacyFormRefusedError from the reference check, which refuses the build.
Property origin (#1125, #607 D4). Every FME node row carries eidos:propertyOrigin, in every mode: one row per key of the node's effective property view, {key, property, group, origin, source, rule}, from domain.property_origin.property_origins (routes.domain_io._flat_export_node). origin is own or inherited, and an inherited row names its ancestor in source as the node IRI, so the S4a scanner finds no bare uuid. An own value that overrides an inherited one is own. The FME carries no propagated values, so it has no propagated rows. A secondary's GET /nodes?full=true rows carry the same field, and the export passes it through. The field is additive: eidos:engineering and eidos:engineeringInherited are byte-equal to before, and the fingerprint changes only because the content does.
The 2.1.0 bump (from 2.0.0, #1125 #607 S1) is additive and names two additions: the node eidos:propertyOrigin above, and the occurrence_id, occurrence_ref and derivation fields on materialised inherited relation rows (#1126 #607 S2, PR #1128), which shipped under 2.0.0. No field is removed or renamed and no value shape changes, so a 2.0.0 reader still reads the file. The DocuGraph importer accepts 2.1.0 in _SUPPORTED_FORMAT_VERSIONS and ignores both additions, so a 2.1.0 file imports to the same overlay as the same model at 2.0.0; tools.migration builds and imports a 2.1.0 package, and 2.0.0 stays readable in both.
Migration Mode (FME 1.7.0)¶
GET /admin/export/full-model/download?mode=migration (#946 M1) is a lossless, strict export meant for moving a project to a new backend. Identity is never re-derived: every id and @id is the stored one. On top of the view-mode content it carries:
- Provenance (every domain).
export_service.load_node_provenance/load_relation_provenanceread the overlay files straight off disk — not throughmutation_store.load_overlay, whose_normalize_nodeand relation dict drop these fields. Node rows gaincreatedBy,origin,createdAuditRef,idempotencyKey,rev,updatedAt,updatedBy,updatedAuditRef; relation rows gaincreated,createdBy,createdById,createdAuditRef,origin,inheritSource(next to the existinginherit_source). Every stored primary node and relation row carries every key (merge_node_provenance/merge_relation_provenance): the overlay file's value when the file has the key, else the loaded graph's value (updatedAtfrom the graph'smodified,inheritSourcefrominherit_source), elsenull— a key missing from the file never erases a known value. E.g. an updated node's file has noidempotencyKey/createdAuditRef, because the update writer does not carry them forward; a base (non-overlay) node takes every key from the graph ornull. Secondary rows get the same keys by the same rule (see Secondary provenance and tombstones below). - Full tombstones (every domain).
load_tombstones_fullreturns every raw field of eachnode_tombstones/andtombstones/file, plus the bare uuid asid, astombstoned_nodes/tombstoned_relations. The provenance endpoint below sendstombstoned_nodesthroughinfra.iri.without_bare_ids(#1120 WP9 S5c), so a node tombstone goes out named by its@idonly, and the primary restoresidfrom it (_fetch_secondary_provenance). A secondary's come from its provenance endpoint (below), not from the reduced archive-modeGET /tombstones. - Strict integrity. The three readers run before any secondary is contacted. A file that is unreadable, is not a JSON object, has no envelope id, or whose id differs from its file name raises
DomainExportError→ HTTP 422domain_export_error, and nothing is delivered. The error names the file only. View and archive keep the lenient skip-with-warning behaviour. - Strict secondary fan-out (#946 M2). A secondary whose
GET /nodesorGET /relationsfails or answers non-200 fails the whole export withExternalServiceError(the client's typed error, e.g. timeout 504; otherwise 502secondary_bad_status) instead of being listedreachable: falsewith no rows — a migration artifact never silently lacks a domain. View and archive keep listing itreachable: false. - Secondary provenance and tombstones (#1049). Every backend serves the internal
GET /admin/export/full-model/provenance(samedomain_export_authgate as the audit endpoint: admin on the primary, a peer's service token on a secondary;Cache-Control: no-store). It returns{nodes, relations, tombstoned_nodes, tombstoned_relations}: the backend's ownload_node_provenance/load_relation_provenancemaps (uuid → fields on disk) andload_tombstones_fulllists, so a bad file answers 422 like the primary's own export. In migration mode only, after a secondary'sGET /nodesandGET /relationssucceed, the primary fetches this endpoint from it with the longTier.EXPORTtimeout, merges it onto that secondary's node rows (keyed byid, which the primary reads off each row'siricompanion since #1119 WP9 S5b) and relation rows (keyed bysource) withmerge_node_provenance/merge_relation_provenance— file, then the secondary's graph value, thennull— and uses its full tombstones instead ofGET /tombstones. The fetch is strict: a failure or non-200 (or a body without the four fields) fails the whole export withExternalServiceError(timeout 504, otherwise 502secondary_bad_status). View and archive never call it, so their output is unchanged. - Derived content marked. Inherited relation rows are materialised exactly as in archive mode and keep
inherited_onto; like in every mode they carryeidos:derivedFrominstead of an@id, and no provenance. A node's owneidos:engineeringis never merged with inherited values; those go toeidos:engineeringInherited(each entry withsource_uid/source_label).eidos:worldand everyeidos:hasDomainedge carryderived: true. - Audit (every domain).
eidos:auditByDomainlists{domain, reachable, seq, events}for the primary (localaudit/log and the globalaudit/_seqwatermark, never below the highesteidos:seqamong the returned events) and for each secondary, fetched from the new internalGET /admin/export/full-model/audit(samedomain_export_authgate: admin, or a peer's service token;Cache-Control: no-store) with the longTier.EXPORTtimeout. The fetch is strict: a secondary whose audit fetch fails or answers non-200 fails the whole export withExternalServiceError(timeout 504, otherwise 502secondary_bad_status) — view and archive are unaffected.collect_audit_by_domainlists a domain passed in as unreachable withreachable: false, seq: null, events: []without fetching it; since #946 M2 the strict node/relation fan-out means a migration export never reaches it with one. Independent ofinclude_audit, which still controls only the flat primaryeidos:auditLog.
Migration mode has no IIS stamping and no eidos:metrics, and as_of is rejected (it requires mode=archive).
Migration package (#946 M2). The host CLI python -m tools.migration build packages a quiesced instance for a move: it refuses unless every backend's public GET /config reports capabilities.canEdit: false (the EDIT_ENABLED kill-switch; unreachable or malformed answers also refuse), and checks again before writing the manifest. Per domain it copies data.eidos and the mutations/ tree to domains/<key>/ — never mutations/pending/ (AddNode previews), mutations/_cache_version or lock files (*.lock, e.g. _cache_version.lock, _write.lock) — fetches this mode=migration export once from the primary — logged in as an admin, never with the service token (M5, below) — as fme-migration.jsonld (an export whose eidos:formatVersion is not in SUPPORTED_FME_VERSIONS, or that carries a legacy urn node id, is refused before anything is packaged, #1112), and writes manifest.json: every file's POSIX path, sha256 and size, appVersion (the primary's /config version), fmeVersion, per-domain nodeCount/relationCount (keyed by domain_key; the export's domains must equal the packaged ones), exportedAt, and export — the export file's path, its eidos:fingerprint (the content fingerprint, stable across unchanged exports; an export without one is refused) and its own eidos:exportedAt. python -m tools.migration verify --package <dir> reports every changed (C), missing (M) and unlisted (X) file and exits non-zero on any. python -m tools.migration compare <dir-a> <dir-b> compares two manifests by content: it ignores the timestamp fields (exportedAt, export.exportedAt) and judges the export file by export.fingerprint rather than its sha256/size (the live export's own timestamp changes its bytes every build); every other field and file entry must match. Two builds of an unchanged instance print SAME (exit 0); otherwise each difference is printed as D <field> (exit 1). Importing a package into an empty target is import (M4a, below).
Migration package sidecars (#946 M3). Each domains/<key>/ also carries the domain's sidecar stores, each copied only if that domain has it: access_rules.db and interfaces.db (copied with SQLite's online backup, safe beside a -wal file, then vacuumed; row values, including the IIS manual fields owner_user_id, notes, status, criticality_class and history, are unchanged), instance.json, relation_catalog.json, and notifications.db — the notifications and user_changelog_seen tables of sessions.db and nothing else: sessions, invite tokens, reset tokens and API keys are dropped and the file vacuumed, so their bytes are not in the package. With --shared DIR (the instance's /shared directory) every domain also gets its own copy of users.json — password hashes kept, any field whose name contains token dropped —, the live project.rulebook.json and property_catalog.json, property_groups.json, property_library.json; without --shared the build prints a note and packages none of them. The rulebook history is already inside mutations/rulebook_history/. Every file is listed in manifest.json with a SHA-256 computed in 1 MiB chunks (a large file is never read whole); the users.json entries carry "class": "secret". Before the manifest is written the build refuses, leaving the partial package and no manifest (exit 1), when an access_rules row's subject or granted_by is not a user in the packaged users.json (__super__ is exempt, and granted_by may also be the documented system grantor bootstrap that instance setup records, #1061) or its non-null subtree_root is not a node of its own domain in the migration export, and when a secret scan finds, in any file other than the users.json copies, the build's service token, a JWT-shaped string, or a non-empty value under a credential-named key (service token, JWT/session secret, SMTP or SSH password, API key, private key). Findings name the file (and table/column), never the value.
Migration package import (#946 M4a). python -m tools.migration import --package <dir> --domain KEY=PATH … [--backend-url KEY=URL …] [--shared DIR] lands a package on a freshly provisioned target byte for byte — except legacy-form ids in node, relation and tombstone files, which are rewritten (M4b, below) — and the audit files under mutations/audit/ arrive verbatim. Every package domain needs exactly one --domain target directory, which must already exist; no two targets (--domain or --shared) may be the same directory or lie one inside the other (usage error, exit 2); --shared is required when the package carries users.json, the rulebook or the property catalogs, which land there once (every domain's copy must have the same SHA-256, otherwise the import refuses). fme-migration.jsonld and manifest.json stay in the package, and domains.json is never written — it remains the target's own config. A domain's packaged notifications.db (M3) never lands under that name (#1056): its notifications and user_changelog_seen rows are merged into the target's own sessions.db — the only file the backend reads them from — built on a copy in the staging directory and promoted and rolled back like every other file. A target without sessions.db gets one holding only those two tables, in the backend's own schema; the backend's init_db adds the session, token and API-key tables at its first start. No session, invite-token, reset-token or API-key row is ever carried. Preflight, before any write, refuses (exit 1) a package whose schemaVersion is newer than the tool's or whose fmeVersion is not a supported one (1.7.0, 2.0.0, 2.1.0), any manifest entry whose path is absolute, contains .. or an empty segment, names an invalid domain key or would land outside the staging directory (the manifest is not signed), any file that does not match its manifest.json hash (re-verified in 1 MiB chunks), a target that already holds data.eidos, any file under mutations/, users.json or a migration.json — so a second import into the same target is refused —, a target sessions.db that already has notifications or user_changelog_seen rows or still has a non-empty -wal/-journal file beside it (the backend did not close it cleanly), and any given --backend-url whose backend is not proven stopped: only a refused connection (ConnectionRefusedError in the httpx.ConnectError cause chain) passes, while any answer to GET /health, any other connect error (a DNS failure, a TLS handshake failure, an unreachable network), a connect or read timeout, a protocol error or an invalid URL refuses. A --backend-url without an http:// or https:// scheme is a usage error (exit 2). Other target files — an existing access_rules.db, rulebook or property catalog — are not checked and are overwritten by the package's copy; the backups taken while promoting exist only for a rollback. Files are then copied into a staging directory beside each target, re-hashed while they stream, fsynced, and promoted only once every target staged cleanly; a failure while staging or promoting — of any kind, not only an I/O error — rolls every target back to exactly its earlier state. If that rollback itself fails, every staging directory that holds a backup of a replaced file is kept, and the import exits 1 without a traceback, printing the original error, the rollback error and the kept directories on stderr. Every imported file is 0600, like the backend's own overlay files (the manifest records no other mode; a merged sessions.db keeps the target file's mode), and the imported users.json directory is 0700. Every file and directory the import creates in a target is owned (#1068) by that target directory's uid/gid, or by the numeric --owner UID:GID for every target; a non-numeric --owner is a usage error (exit 2). The backend runs as appuser (uid 1000) and cannot read a root-owned 0600 file, so a process that may not chown to that owner — not root, and not that uid with that gid among its groups — is refused (exit 1) before any write. The import is recorded in a migration.json in every target — packageFingerprint, manifestSha256, importedAt, toolVersion — never in the audit log.
Migration package import, identity rewrite (#946 M4b). The import normalises identity while it stages: every mutations/nodes, mutations/relations, mutations/node_tombstones and mutations/tombstones file whose @id is still a legacy form (urn:eidos:node:…, urn:eidos:relation:…, …/node/…) lands with the canonical IRI https://id.ontoteq.com/{uuid} of the same uuid, under the same file name. The audit references eidos:createdAuditRef, eidos:updatedAuditRef and eidos:auditRef of those files are normalised the same way: a urn:eidos:audit:{uuid} value becomes infra.iri.audit_iri of the same uuid, while an already-canonical or empty value stays as it is. A value that is exactly the legacy prefix with an empty id, urn:eidos:audit:, is an empty legacy reference (#1071) — an older writer's way of saying "no audit reference": it stays as it is, the file arrives byte for byte if nothing else in it changes, and the field is reported in id-mapping.json. This rule covers only those three audit-reference fields; an empty urn in @id still refuses. The audit log files under mutations/audit/ themselves are never touched. Only those values, and a relation's target and source (#1102 S8a, below), change — key order and every other field stay as they are — and the file is serialised the way the backend writes one (2-space indent, UTF-8, no trailing newline). A file already in canonical form, or carrying no @id (an eidos:id-only legacy tombstone) and no legacy audit reference, arrives byte for byte; audit files, sidecars and every other file are never opened by this step. Before any target is touched every such file of every domain is checked: an @id or audit reference that does not parse (an audit reference must name a uuid, so urn:eidos:audit:not-a-uuid refuses too), an envelope id that disagrees with the file name, or a file that is not a JSON object refuses the whole import (exit 1) with nothing written. Each domain target gets an id-mapping.json next to migration.json — {"rewritten": [{"path", "uuid", "oldId", "newId"}, …], "auditRefsRewritten": [{"path", "field", "oldValue", "newValue"}, …], "emptyLegacyRefs": [{"path", "field"}, …]}, all sorted by path and empty when nothing needed rewriting, plus relationFieldsRewritten (#1102 S8a, below); --shared gets none. Every relation target reads the same through identity_rewrite.read_target before the import as through the backend's read_target_ref_jsonld after it. The backend loader reads the same node, relation and tombstone ids from the imported target as from the source.
Migration package import, relation target and source (#1102 S8a). Every relation file also lands in the current form. Its target is first read off the untouched file with identity_rewrite.read_target (since #1102 S8b the one sanctioned reader of the legacy forms the backend refuses: the four pre-WP4b target keys and a legacy urn id, read as the backend read them before; read_id does the same for an @id) and written back as eidos:target through infra.target_ref.target_ref_jsonld: an in-model node IRI, or an eidos:ExternalRef / eidos:UnresolvedRef that carries a non-empty placeholder uuid explicitly as eidos:node. The four pre-WP4b keys (eidos:toNodeId, eidos:externalDomain, eidos:externalTargetRef, eidos:unresolvedTargetRef) are dropped; eidos:target takes the place of the first of them. A legacy external or unresolved placeholder is the file's eidos:toNodeId; a WP4b eidos:ExternalRef without eidos:node gets the placeholder the reader derives from its literal eidos:source. Only then does an eidos:source in a node-IRI form (urn:eidos:node:…, https://id.ontoteq.com/…) become the bare uuid the backend writer emits. The reader prefers an explicit eidos:node and derives an eidos:ExternalRef placeholder only when it is absent, so no relation id or placeholder uuid changes, and the loader reads an IRI-form source as its uuid (infra.iri.relation_source_id), so the effective graph and the FME view export (fingerprint included) stay the same. A legacy target ref without a domain, an in-model target with no uuid, an unconvertible target or a source that does not parse to a uuid refuses the import (exit 1) before anything is written. Each changed field is listed in id-mapping.json under relationFieldsRewritten as {"path", "field", "oldValue", "newValue"} (null for an absent key — a dropped legacy key has newValue: null, an added eidos:target has oldValue: null). A package that is already in this form imports byte for byte, so a re-import is idempotent. The legacy readers stay in the backend until S8b.
Migration package build, admin login (#946 M5). build fetches the mode=migration export through the normal login, so nothing goes around IAM. --admin-user EMAIL names an admin account. Its password is read only from $EIDOS_ADMIN_PASSWORD or, when that is unset, an interactive getpass prompt — never a command-line argument, and never printed or logged. The tool calls POST /auth/login on the primary and fetches GET /admin/export/full-model/download?mode=migration with the returned Bearer token. The request carries no X-Service-Token and no X-Eidos-Username, so it passes the same admin_or_service gate as any admin (backend unchanged). Once a session exists, POST /auth/logout follows on every outcome, success or refusal; a failed logout only prints a warning. A wrong password refuses at the login (HTTP 401 Invalid credentials, exit 1). A non-admin account logs in but the export refuses (HTTP 403 Admin access required, exit 1). In both cases --out is never created, because the export is fetched and checked before the package directory is made. --service-token / $SERVICE_TOKEN is optional and authenticates nothing; an empty value prints a stderr warning, since the secret scan can then not check the package for that literal. The secret scan (M3) still refuses a package in which that token, or the admin password, appears outside the users.json copies. The golden round-trip suite (tools/migration/tests/test_golden_roundtrip.py) runs build, verify and import against two real backend processes, logged in as the real admin, with no mocked auth; it also runs build as a real non-admin user and with a real wrong password, against the same running backend, and checks the HTTP 403/401 refusals, that --out is never created and that a session is logged out only where one was issued.
Domain base conversion (#935 S2). python -m tools.migration convert --domain PATH --key KEY [--dry-run | --rollback] moves one domain's base root and WORLD node from data.eidos into overlay files and archives data.eidos; it runs on one domain directory, not on a package. See EIDOS Loader, Converting a domain to this mode.
Migration runbook (instance A → instance B, #946)¶
- Quiesce A. Restart every backend of A (primary and secondaries) with
EDIT_ENABLED=false. Check that eachGET /configanswerscapabilities.canEdit: false.buildchecks this again itself, before and after assembling the package. -
Build with
--admin-user. On a host that reaches every backend of A and its data directories:export EIDOS_ADMIN_PASSWORD='…' # or leave unset and answer the prompt export SERVICE_TOKEN='…' # A's real service token, used only by the secret scan python -m tools.migration build \ --domain product=/opt/eidos/domains/product \ --domain location=/opt/eidos/domains/location \ --backend-url product=http://127.0.0.1:8090 \ --backend-url location=http://127.0.0.1:8091 \ --primary product --admin-user admin@example.com \ --shared /opt/eidos/shared --out /opt/eidos/migrate-2026-10-21 unset EIDOS_ADMIN_PASSWORD SERVICE_TOKENSet
SERVICE_TOKENto A's actual service token even though the build does not authenticate with it. The secret scan can only refuse a package that leaks this token if it knows the literal, so leaving it unset drops that check — the build prints a stderr warning when--service-tokenis empty, rather than dropping it silently. The build exits 1 (refused) for a non-admin account, a wrong password, a backend that is not quiesced, an unresolved IAM reference or a secret outsideusers.json. It exits 2 for a usage error. 3. Verify.python -m tools.migration verify --package /opt/eidos/migrate-2026-10-21must printOK. 4. Copy. Copy the package directory to B's host unchanged; it holds password hashes (users.json, 0600 in a 0700 directory), so keep it owner-only. Runverifyagain on B's host. 5. Import into an empty B. B's domain and shared directories must exist and hold no model data, and B's backends must be stopped. Runimportas root, or as the backend's uid; every created file and directory gets the owner of its target directory, or the numeric--owner UID:GID(e.g.--owner 1000:1000, the containers'appuser) when the directories are not owned by the backend yet:python -m tools.migration import --package /opt/eidos/migrate-2026-10-21 \ --domain product=/srv/eidos/domains/product \ --domain location=/srv/eidos/domains/location \ --backend-url product=http://127.0.0.1:8090 \ --backend-url location=http://127.0.0.1:8091 \ --shared /srv/eidos/shared --owner 1000:1000The import refuses (exit 1, B unchanged) on a hash mismatch, an unsupported version, a non-empty target, a backend that answers
GET /healthor a process that cannot chown to the target's owner. 6. Start B with the samedomains.jsonand ports as A, so every public URI stays the same, and wait forGET /healthon each domain. 7. Compare. Log in to B as the admin. B'smode=viewfull-model export must equal A's, except foreidos:exportedAt. B'smode=migrationexport must equal the package'sfme-migration.jsonldonce each domain'sid-mapping.jsonis applied, ignoringeidos:exportedAtandeidos:fingerprint(the fingerprint follows the rewritten ids). The node, relation and tombstone ids, the audit log,users.jsonand the IAM access rules must be the same on both sides. The golden round-trip suite checks exactly these properties (letters A–D). To confirm that A stayed quiesced while the package was built, runpython -m tools.migration compareon two builds of A; it printsSAME(letter F).
eidos:metrics (archive mode) — model-metrics snapshot¶
In mode: archive, the export carries a frozen snapshot of the model-scope
metrics at eidos:metrics, so a downstream consumer (the eidosdash
dashboard) reads the numbers straight from the archive instead of recomputing
them. It is archive-only: mode: view never carries it.
One shape, one code path. eidos:metrics is exactly the response of the live
GET /metrics/model endpoint — metrics.contract.model_response:
"eidos:metrics": {
"scope": "model",
"graph_version": 42,
"computed_at": "2026-09-17T09:00:00+00:00",
"days": 30, // the archive always freezes the 30-day window
"partial": false,
"metrics": { "nodes_total": { "value": 1972, "unit": "count", "domains": {…} }, … },
"errors": {}
}
The per-metric map is under .metrics[<id>] (each entry {value, unit, …}),
not .values. Dash reads exactly one format from both the endpoint and the
archive.
How it is produced (the injection pattern). build_export_doc is a pure,
synchronous assembler in the persistence layer — it has no access to the metrics
engine, the IIS DB, or the secondary-domain fan-out, and it gains none. Instead the
caller injects the snapshot via the metrics= parameter (the same
dependency-injection pattern as interface_lookup). The single canonical producer
is metrics.collection.collect_model(server_mod, wanted, days=30) — the same
cross-domain collector the /metrics/model endpoint delegates to (02.01 §3.6: one
implementation, no copy). The live full-model download (routes/domain_io.py) and
the daily archive job (server.py) both call it and pass the result as metrics=.
Two independent fan-outs. An archive assembles from two separate
cross-domain fan-outs that do not share reachability state: the domain-data
fan-out (assemble_full_model_domains → each domain's nodes/relations, with a
per-domain reachable flag) and the metrics fan-out (collect_model → each
domain's /metrics/model?domain=self, folded into eidos:metrics.partial and the
per-domain null entries). They are queried independently, so their signals are
independent: a domain can be reachable: true in eidos:domains yet contribute
null to eidos:metrics (its metrics call failed), or vice-versa. A consumer must
read eidos:metrics.partial for metric completeness and the domains' reachable
flags for data completeness — one does not imply the other.
Collector hard-failure marker. The per-secondary case above degrades within
eidos:metrics (partial + null). If the whole snapshot cannot be produced (the
primary's own compute raised), the archive is still written — a durable record is
never lost — but with no eidos:metrics at all. To keep that state
distinguishable from a genuine pre-W2 archive (which also had no such field),
eidos:delivery["metrics"] is stamped "collector_failed". Both archive
callers obtain the snapshot through the single metrics.collection.collect_archive_snapshot
helper, which owns this fault-isolation policy (02.01 §3.6).
Never a silent zero. The whole envelope is frozen, including partial and
errors: an unreachable secondary freezes partial: true + a null per-domain
entry; a failed audit source freezes the affected ids in errors. A consumer can
always tell an incomplete archive from a complete one — the same signal names the
live endpoint uses.
Definition-meaning version. eidos:metricsDefinitionVersion (e.g. "m1",
sourced from metrics.catalog.DEFINITION_VERSION) versions the meaning of the
frozen definitions. It is bumped whenever an archived metric's definition changes
meaning, so Dash can detect that an older archive was computed under a different
definition. A guard test (tests/metrics/dash_contract.py) pins a SHA-256 of the
archived ids' definition strings to this label, so a silent meaning-drift fails CI.
What each version means — the change log describing how a version's numbers
differ from earlier ones (and from Dash's collector-fallback) — lives on the
metrics-engine page: What the definition version means.
Reconstruction carries none. A reconstruction / as_of export rebuilds the
graph as of a past date T; metrics are a live snapshot and cannot be recomputed
as of T, so a reconstruction omits eidos:metrics and its eidos:delivery block
carries an explicit marker "metrics": "not_available_for_reconstruction" — the
absence is deliberate, not a dropped field.
Not in the fingerprint. eidos:metrics carries computed_at (and a live
graph_version), so it is deliberately excluded from eidos:fingerprint
(_compute_fingerprint covers domains + auditLog + relationCatalog only) — the
same model sample keeps the same fingerprint whether or not metrics are attached.
Not a metric: changesOnRemovedNodes is a field on the /activity/heatmap
response, not a catalog metric — it is deliberately absent from eidos:metrics.
The WORLD node is extracted from every domain's node list by _extract_world() and placed once at eidos:world. hasDomain edges from WORLD to each domain root are materialized by _has_domain_edges() — they are implicit in per-domain files but explicit in the merged export. _resolve_external_refs() runs after archive-mode stamping so that lookup keys match the stored uuids the IIS detector indexed.
webURI (view mode) — canonical node weblinks¶
In mode: view exports, each eidos:hasDomain edge carries a webURI — the canonical deep-link origin: the primary public origin (BASE_URL), the same value for every domain. A node deep link is domain-agnostic — {origin}/node/{uuid}, where {uuid} is a locator — because the consumer resolves the owning domain itself (see the canonical-deep-link invariant; Contract 02.01: route by identity, never transport). So webURI must not be a per-domain routing publicURI — for a secondary that is the /api/proxy/{domain} path, which would leak transport into the link. Consumers build a direct link to a node as:
webURI + "/node/" + <node uuid>
Since FME 2.0.0 (#989 WP9 S4a) a node's id is its IRI https://id.ontoteq.com/{uuid}, so the <node uuid> is that IRI's local part, never the IRI itself.
The Explorer share link itself (since #1000/#1006, canonicalNodeShareUrl in frontend/app.js) puts the unescaped identity IRI in the hash instead of a url-encoded /node/ URI:
<explorer origin><pathname>#https://id.ontoteq.com/<uuid> (origin + pathname + '#' + EidosIRI.toIri(uuid))
The value is built by export_service.canonical_web_uris() from the primary's BASE_URL and is identical on every hasDomain edge; the per-domain publicURI is intentionally not used (it stays the internal routing key). If BASE_URL is empty the map is empty and no webURI is stamped. webURI is omitted in mode: archive and in reconstruction / as_of exports (host-neutral). It lives only on hasDomain edges — never on the domains payload or per node — so it does not affect eidos:fingerprint. Added in formatVersion 1.3.0 (additive, non-breaking).
filename_for(name) produces a sanitised <name>-<YYYY-MM-DD>.jsonld filename for the export payload.
Tombstones and Deletion History¶
load_tombstones(mutations_dir) reads deletion history for the archive export (FMDE D3). It scans two directories:
MUTATIONS_DIR/node_tombstones/*.jsonld— node tombstones withdeletedAt,deletedBy,auditRefMUTATIONS_DIR/tombstones/*.jsonld— relation tombstones (creation time = deletion moment; exposed asdeleted_at)
Malformed files are skipped with a warning log, never fatal — same tolerance as mutation_store.load_overlay. Migration mode uses the strict, full-field load_tombstones_full instead (see Migration Mode).
GET /tombstones — served from backend/server.py. Returns the full tombstone payload ({"nodes": [...], "relations": [...]}) for admin and archive consumers. No pagination; the tombstone set grows with each deletion event. IAM-protected: requires a valid session.
Admin Data Exchange Tab¶
The admin panel exposes a single Data Exchange tab (#admin-content-dataexchange in frontend/index.html, activated by switchAdminTab('dataexchange') in frontend/app.js). It consolidates every "get data in or out of the project" affordance the admin surface offers, replacing the historical Snapshots + Export split that arrived in three separate feature waves (April 2026 FME, May 2026 domain snapshot, June 2026 project exchange, July 2026 FMDE). The consolidation is pure UI-IA — every endpoint, request/response shape, server filename, auth model, capability model, and file format is unchanged (see issue #425 §5 for the full compatibility statement).
Section order and vocabulary¶
The tab is grouped by scope first, purpose second, in this fixed order:
- Project (round-trip) —
GET /admin/project/exportandPOST /admin/project/import/{preview,commit}. Portable.eidx-projectZIP bundling every domain plus a manifest. Promoted to the top because it is the newest and highest-fanout surface (it internally callsserialize_domain()per reachable domain and zips the result). - Domain (round-trip) —
GET /domain-exportandPOST /domain-import/{preview,commit}. Single-domain.eidx.json. A convenience surface over the sameserialize_domain()primitive that Project Export bundles. - Full Model — analysis snapshot —
GET /admin/export/full-model/{info,download}. Merged, IRI-addressed.jsonld(FME2.1.0, every identity value is its IRI since2.0.0) for downstream consumers. No import counterpart — it is a frozen analysis snapshot, not a round-trip file. Both facets live behind one endpoint (mode=viewandmode=archive&as_of=YYYY-MM-DD).
#### GET /admin/export/full-model/info — response shape
Each domain entry in domains[]:
{
"name": "primary",
"nodes": 1234,
"relations": 800,
"relations_total": 950,
"reachable": true
}
"partial_reason": "unreachable" (secondary fan-out returned non-200 or raised), "pre_w1" (reachable secondary that exposes relations without relations_total, so its inherited count is unknown), or "metric_errors" (the primary's own compute_model returned errors, so its counts are unreliable).
- relations = explicit-only (base + overlay − tombstoned; matches what the view-mode download exports).
- relations_total = explicit + inherited (matches what the archive-mode download materialises).
Top-level aggregates: partial (bool), nodes_total, relations (Σ per-domain explicit), relations_total (Σ per-domain explicit + inherited), domains[], history (earliest/latest audit dates). The export dialog renders both totals as "N relations (M incl. inherited)" so the view/archive distinction is visible to the operator.
Partial totals (#647). The top-level sums (nodes_total, relations, relations_total) cover only fully-known domains — a degraded domain (partial_reason set) is excluded from all three, so relations and relations_total always range over the same domain set and a partial figure is never presented as complete. When any domain is degraded, partial: true (the same signal name /metrics/model uses). The per-domain counts stay in domains[] so a client can still see what was left out; the export dialog prefixes "⚠ partial totals" and lists the excluded domains with their reason. Before #647 a pre-W1 secondary's explicit count was summed into relations_total (understating the model-wide inherited total) and an unreachable domain contributed a silent 0 with no top-level flag.
4. Audit Log (provenance) — GET /admin/audit/export. Append-only .jsonl provenance stream, one record per mutation. No import counterpart by design.
5. Mutation checkpoints — GET /snapshots and GET /snapshots/{id}/download. Subtree eidos:Snapshot files written before every mutation commit; tier-gated by snapshot_service.compute_and_write (silent no-op below the domain_snapshot capability). Kept as the last section because it is a read-only recovery aid, not a user-triggered exchange. GET /snapshots accepts optional limit (1–500, newest first) and before (opaque cursor, newest-first) query parameters so the admin panel can page large histories — a deployment with thousands of files would otherwise exceed the frontend's 8 s apiFetch timeout because every file was opened for its nodeCount/relationCount. Omitting limit preserves the pre-#688 all-files-opened behaviour. Checkpoint files carry eidos:schemaVersion "1.2" (#954 WP5 S6): every node in eidos:nodes gains @id (infra.iri.node_iri), added to a copy so the live graph is not touched; the bare id and all 1.1 fields are unchanged. The paging contract is the opaque _cursor field each row carries: the client echoes the last visible row's _cursor back as before to request the next page. The cursor is pinned to the same server-side sort key as the merged list (the file st_mtime plus the file stem as a tiebreaker, consistent across a primary and its secondaries), so pagination is lossless even when many files are written within the same filesystem tick — the PR #920 review's bulk-import regression. An ISO-8601 timestamp is still accepted as a legacy fallback for pre-#920 clients that passed last.createdAt as the cursor; this fallback is lossy under bursty writes (createdAt is captured a few milliseconds before the file lands on disk) and must not be relied on by new clients.
The word "snapshot" is reserved for one meaning across the tab: the FMDE archive facet (mode=archive&as_of=…). The former "Export/Import snapshot" microcopy on the domain surface is now "Export/Import domain" (matches serialize_domain() and the .eidx.json file convention). "Auto-snapshots created before each mutation commit" is now "Mutation checkpoints" (describes what the file is — a pre-mutation safety net — rather than how it was implemented).
Surfaced capabilities¶
Two capabilities that were delivered on the server side but never reached the admin UI are now visible on the tab:
- FMDE historical
as_of— the Full Model section renders a date picker whosemin/maxare read from/admin/export/full-model/info.historyat load time. Picking a date and clicking "Download archive snapshot (.jsonld)" issuesGET /admin/export/full-model/download?mode=archive&as_of=YYYY-MM-DD— the same endpoint the eidosdash collector already consumes via service token. The server's existing error envelope (before_history_floor,invalid_as_of,reconstruction_failed,rulebook_before_history_floor) is surfaced verbatim.rulebook_before_history_floorfires when T precedes the earliest snapshot-bearing rulebook event (distinct frombefore_history_floor, which fires when T precedes the earliest audit event — issue #604, builds on #592). - Project Export options — an "Options" disclosure below the Export/Import buttons exposes two checkboxes that map to server-side query params the endpoint has always accepted:
include_configandinclude_provenance. Both default to unchecked, which produces a request byte-for-byte identical to the pre-#425 call (server defaults hold).
Filename handling¶
Every admin download prefers the server's Content-Disposition filename over any frontend-side guess. The three shared helpers in frontend/app.js are (the first two return EidosPaths-relative paths and take no baseUrl; the old _build*Url(baseUrl, …) forms were removed in #687):
_buildFullModelDownloadPath({name, audit, mode, asOf})— backs both the "Download current" button (mode=view, the default) and the "Download archive snapshot" button (mode=archive); emits only the query params that were actually set, so the URL stays minimal._buildProjectExportPath({includeConfig, includeProvenance})— same shape; unchecked options are omitted, preserving the endpoint's default behaviour._filenameFromContentDisposition(cd, fallback)— pullsfilename="…"out of the header, or returns the caller's fallback. Used by every admin download path (Full Model current, Full Model archive, Domain export, Project export, Audit Log) so the server's date-stamped filenames survive the browser round-trip. This closes a long-standing inconsistency (G7 in the pre-implementation report) where the domain export handler used to override the server's{safe_domain}-{YYYY-MM-DD}.eidx.jsonwith${domain}-snapshot.eidx.json, both discarding the date-stamp and re-injecting the overloaded word "snapshot" into filenames.
Deep linking¶
Admin tabs are switched entirely inside switchAdminTab() — no location.hash, no location.search, no external URL points into a specific admin tab. The rename from ?snapshots|export to ?dataexchange therefore has no deep-link impact. (The canonical deep-link contract at docs/superpowers/specs/2026-07-27-canonical-deep-link-contract.md governs /node/{uuid} and is unrelated to admin surfaces.)
Archive Mode and Inherited-Relation Materialisation (FMDE)¶
In mode: archive, build_export_doc materialises descendant inherited rows through the canonical Effective Relations Engine via the 5.1b façade (domain.relation_reads.read_relations(mode=EFFECTIVE)) — see the engine's own doc: Archive materialisation (FMDE). The caller (server._write_today_archive_once + routes/domain_io.py:/admin/export/full-model/download) injects three DI arguments:
inheritance_resolver—(node_id, parent_of) -> list[dict], bound toserver._resolve_effective_outgoing(same adapter the detail panel reads).inheritance_eff— the primary's effective graph ({nodes, parent_of}).primary_domain_name— thed["name"]whose nodes the resolver answers for.
Persistence does NOT import the engine (01.01 R4/R6 — only the façade domain.relation_reads). An archive mode build without these three kwargs logs archive_inheritance_resolver_missing and skips materialisation (same ergonomics as interface_lookup=None).
Row shape (invariant — read by consumers)¶
Inherited rows post-materialisation:
from= descendant (viewing node) — overrides engine's ancestorfrom.inherited_onto= descendant — the base-vs-inherited split (reconstruction_servicefilters oninherited_onto is None; graphanalyzertests/graphanalyzer/check_fme.mtsconsumes the shape).inherited_from= ancestor source id (additive provenance from the engine).inherit_source= preserved as on the stored source row (a rulebook-only inherited row keepsfalse).- No
@idand no provenance (FME 1.7.0, #946 M1):_stamp_relation_idsdrops every_RELATION_PROVENANCE_KEYSkey (created,createdBy,createdById,createdAuditRef,origin,inheritSource) from any row withinherited_onto, in archive mode as well as migration mode — an inherited row no longer carries thecreated/createdBythe resolver copied from the stored row; only that stored row, linked byeidos:derivedFrom, has them.
Base rows (not inherited) are unchanged. No FORMAT_VERSION bump (shape preserved — spec §10.7-2). Fingerprints for the same model change post-5.1c-2 because the content is corrected — this is expected and documented in the slice.
Archive Mode and IIS Criticality¶
When mode="archive" and interface_lookup is provided, build_export_doc stamps interface relations with IIS criticality before external-ref resolution:
# SOURCE: backend/persistence/export_service.py build_export_doc()
iface_preds = {p["id"] for p in catalog if p.get("creates_interface")}
for d in domains:
for rel in d.get("relations", []):
if rel.get("predicate") not in iface_preds:
continue
hit = interface_lookup(rel.get("from"), rel.get("to"), rel.get("predicate"))
if hit and hit.get("criticality_class") is not None and hit.get("status") is not None:
rel["eidos:iis"] = {
"criticality_class": hit["criticality_class"],
"status": hit["status"],
}
Only predicates with creates_interface: true in the relation catalog qualify. The IIS stamp (criticality_class, status) is injected directly onto the relation object, so downstream consumers compute IIS criticality without calling back to the live server. D1 invariant: engine results are materialized in the file; consumers apply no logic after export.