Skip to content

Operation Record Engine

Verified against the code · 4d5c9b5781eaaa62048db19cd10dc7039e402c98 · 2026-10-20

956 D1b-2 (2026-10-20): 1 finding(s) fixed — OperationRecord carries no event-id field; id_from_envelope is used only by record()/record_rejected() to return the appended event's id. Re-verified against main 4d5c9b57.

Layer: domain

Overview — one meaning for every audit event

Every mutation the system commits (and every one it rejects) is written to an append-only provenance log — one eidos:AuditEvent per line in mutations/audit/{date}.jsonl (older days may still be legacy .jsonld @graph files). Five different features read that log back: the point-in-time reconstruction engine, a node's history timeline, the changelog, the activity metrics, and the activity heatmap.

Before trin 2.4 each of those five re-derived the meaning of a record on its own — the "only count committed events" gate, the rule that a CreateRelation / DeleteRelation is charged to its source node (not the relation id), and the dispatch on intentType. Worse, the heatmap hand-parsed only {date}.jsonld files and never read the .jsonl format that every real mutation now writes — so on any modern deployment the heatmap silently counted zero activity (a live bug).

The Operation Record Engine (backend/domain/operation_record.py) is now the single interpretation of an audit event. It reads through the format-agnostic accessor audit_service.read_events (both .jsonld and .jsonl) and projects each raw event into a frozen OperationRecord with the shared accessors:

Accessor Meaning
is_committed the record's commit result is "committed"
attributed_entity_id the entity a change is charged to — the source node for a relation create/delete, otherwise the commit's entityId
day, actor, intent, payload, seq, snapshot_ref the record's fields, defaulted so nothing raises

read_committed(...) returns the committed records sorted by (timestamp, seq) — the canonical replay/aggregation order. All five readers consume the engine: reconstruction folds read_committed; changelog, metrics and the heatmap read through it; node history projects the route's events with from_event. The heatmap now reads both formats, so .jsonl activity is counted.

Audit event envelope — @id / audit_iri (WP1 #950)

Every eidos:AuditEvent record audit_service.append writes carries the WP1 envelope shape: the event's identity is an @id IRI emitted via infra.iri.audit_iri(event_uuid) → https://id.ontoteq.com/{event_uuid} (flat IRI space, shared with nodes and relations; @type: "eidos:AuditEvent" carries the type distinction). eidos:id is no longer emitted — the pre-WP1 duplicate was removed. Both audit_service.make_committed and audit_service.make_rejected build the envelope via at_id_entry(audit_iri(event_uuid)), so there is one owner for the shape (R4-writer baseline in tests/test_rdf_g1_r4_ratchet.py; HTTPS prefix pinned by tests/test_rdf_wp1_r5_ratchet.py).

OperationRecord (and from_event) expose no event id at all — its fields are intent, payload, entity_id, result, actor, ip, timestamp, seq, snapshot_ref, domain_key, raw. infra.iri.id_from_envelope, which keys on @id and falls back to eidos:id only for pre-WP1 legacy .jsonld day files, is called only inside record() / record_rejected() on the freshly built envelope, to return the appended event's id. Legacy urn:eidos:audit:{uuid} @id values written before WP1 continue to load unchanged via the from_iri fallback (debug log from_iri_legacy_form); rewrite is deferred to #946. The route layer (and record() / record_rejected() facades) never hand-build the envelope, so no caller re-emits eidos:id — the forward-only policy on WP1 (docs/specs/identity-facade.md §4a; governance exception .archon/exceptions/06-TEST-ROLLBACK-wp1-forward-only.yaml) depends on this.

Because the engine depends only on persistence.audit_service and never on server, it lifts into eidos-core unchanged — it is the P4 provenance engine.

The single provenance sink

record(...) and record_rejected(...) are the only way a non-engine surface records provenance. Graph node/relation mutations are audited inside the graph CRUD engine itself; everything else — the Documents admin actions, the Documentation 360° report, the cross-domain relation heal, and (new in 2.4) IAM AccessRule and Rulebook / propagation-rule mutations — records through this facade. audit_service.append is the single writer underneath.

This closes a real provenance gap: before 2.4 the IAM and Rulebook admin routers never wrote to the audit log at all, so a rule change left no trace (project_full_provenance requires the state to be reconstructible from disk + the audit log). A CI guard (tests/test_mutation_transaction_guard.py) now forbids any route from hand-building an audit event — all provenance must flow through the facade.

Config events are not model changes — excluded from every change metric

The IAM / Rulebook provenance intents (CONFIG_INTENTS — CreateIAMRule, UpdateIAMRule, DeleteIAMRule, UpsertPropagationRule, DeletePropagationRule, UpsertMovementRule, DeleteMovementRule, ReplaceRulebook) are admin plumbing, not model changes. They stay in the audit log (and are still reconstructable — node history, read_committed, admin export), but they never count as changes to the model. The changelog and the activity heatmap have always filtered them out (record.is_config), so a non-admin never sees that IAM rules are changing (a mild info-disclosure the owner ruled out). Owner decision 2026-09-17 extended this to every changes*/active_users* metric — the exclusion now lives in the one canonical metrics projection (metrics._audit.committed_events), so /metrics/model, /activity/heatmap and the node chips (/metrics/batch) count model-data changes only. "How much did the model change" must not be inflated by configuration churn. Reconstruction and node history are unaffected (they read the full audit record, config events included).

Unknown intent types never break reconstruction

The new IAM / Rulebook provenance events (UpdateIAMRule, UpsertPropagationRule, …) land in the same append-only log as graph events, but they have no graph effect. The engine never dispatches on intent — it only exposes it — and reconstruction's fold treats an unrecognised intentType as a graceful no-op (skipped, logged at debug, never raised). This is a binding invariant: a future event type must never be able to break a point-in-time reconstruction. It is covered by an explicit test.

Rulebook events are reconstructable (#592)

Config events are hidden from users, but they must still satisfy the full-provenance invariant — canonical state = disk + audit log, reconstructable at any T. Before #592 the rulebook/propagation events (UpsertMovementRule, DeleteMovementRule, UpsertPropagationRule, DeletePropagationRule, ReplaceRulebook — the RULEBOOK_INTENTS subset) wrote thin payloads (category + keys), so the rulebook could not be rebuilt from history. #592 closes that gap without an envelope change:

  • Content-addressed snapshot per mutation. Inside the same locked read-modify-write that writes project.rulebook.json, the full post-state book is written to <mutations_dir>/rulebook_history/<sha256>.json (persistence.rulebook_history, idempotent — same content ⇒ same file). The snapshot bytes are the canonical rulebook serialization (rulebook_store.serialize_rulebook, written with newline="\n" so the file bytes equal the serialization on every OS), so a snapshot is byte-identical to the live rulebook at that state. The snapshot is durable before the audit event is appended, so a committed snapshotRef can never dangle. The only possible failure ordering is the harmless one — if the snapshot write itself fails, the mutation is applied but no event is recorded (a snapshot without an event, never an event without a snapshot).
  • The pointer rides the existing field. record(..., snapshot_ref="sha256:…") puts it in eidos:snapshot.snapshotRef (already in the envelope; OperationRecord exposes it as snapshot_ref). Granular events also carry a human-readable before/after diff; ReplaceRulebook carries {categories, before_hash, after_hash} — never the inline book.
  • Reconstruction is a lookup, not a fold. rulebook_history.rulebook_as_of(T) returns the snapshot referenced by the latest committed rulebook event with timestamp <= T. A T before the first snapshot-bearing event raises RulebookHistoryFloorError (404, canonical envelope) — mirroring the graph's HistoryFloorError; there is no reconstructable rulebook state before provenance existed.
  • Actor parity (#601 review). These events record actor.username = the display name (resolved the same way engine mutations resolve it, via server._resolve_actor_display_name), with the stable uuid kept in payload.actorUserId — so an audit event's actor is a name, uniformly, whether it came from a graph mutation or a rulebook edit.

IAM mutations through /admin/iam/rules* are reconstructable (#605). Create/Update/DeleteIAMRule events carry the same model: a content-addressed snapshot of the full post-state rule set in <mutations_dir>/iam_history/<sha256>.json, referenced via snapshotRef. persistence.iam_history.iam_rules_as_of(T) is the lookup function; IAMHistoryFloorError (error_code="iam_before_history_floor", HTTP 404) is the floor error. The IAM_INTENTS frozenset in operation_record marks the three intents. Cascade coverage (#912): rules dropped by the user-delete cascade (admin_users_router.admin_delete_user) now emit one DeleteIAMRule event per rule — same shape as a direct /admin/iam/rules* delete, with an additive payload.cascade='user_deleted' + payload.deleted_user_id marker and a non-empty snapshot_ref. iam_rules_as_of(T) replays the cascade through the same pointer-lookup path; the pre-#912 orphan gap is closed. Failure behaviour (#912 round 2): the cascade loop runs BEFORE the user record is removed, so a mid-cascade exception propagates out of the handler with the user still on disk; events for rules already committed stay durable and a second DELETE /admin/users/{user_id} finishes the remaining rules and finally removes the user — the audit log stays reconstructable across the retry.

Both snapshot stores share one primitive, persistence.snapshot_store.ContentAddressedSnapshotStore (#605 r1), parametrised by (directory, serializer, intent set, floor-error class). rulebook_history and iam_history are thin shims over it.

Derived quantities use the historical rulebook (#604). The reconstruct_as_of graph fold now feeds rulebook_as_of(T) into the inherited-relation expansion, so an as_of export is historically faithful in its derived quantities too — the eidos:delivery.reconstruction_semantic marker on an as_of response is raw_graph_and_derived_quantities_as_of_T. Pre-#604, reconstruction did NOT wire an inheritance resolver at all (so a reconstructed as_of archive carried no inherited rows); fingerprints over the reconstruction delivery path therefore change for every T whose model has an inheritable relation. IIS criticality is the one derived quantity still computed by today's engine (no historical IIS store yet — a scoped simplification tracked for a sibling issue). See effective-relations-engine for the inheritance expansion itself; the rulebook_as_of(T) pointer-lookup mechanics (snapshot refs, history floor, no-history sentinel) are described in the "Rulebook events are reconstructable (#592)" section ABOVE on this page — the sole owner of the rebuild mechanics, with persistence.rulebook_history as the implementation module.