Record durable extraction runs and proposition run lineage

[jimador]

Goal

Record durable, tenant-scoped extraction runs and link propositions to every run that
produced or confirmed them without changing source-grounding cardinality.

Context

Projection and collector runs establish the DICE storage pattern. Extraction needs its
own lifecycle, metadata envelope, indexes, parent/superseded links, and a dedicated
proposition-to-run relation. ProvenanceEntry and DerivedFrom remain source evidence.

A run may contain zero, one, or many model invocations because of chunking or retries.
Requested configuration, observed execution facts, optional provider-effective facts,
and protected replay evidence must remain distinct.

The design borrows the separation of run/job/input/output from OpenLineage, the
parameters/metrics/context/artifact split from MLflow, requested-versus-response model
facts and opt-in content from OpenTelemetry GenAI, and #64's S3/W3C-PROV-style stable
source plus opaque revision. It uses typed fields rather than generic facet/tag maps.

The resulting evidence supports troubleshooting failures, comparing and evaluating runs,
approximate reproduction, auditing proposition output, and investigating user-context
effects. Experiment/user-context comparisons use tenant-scoped run pages and host-side
grouping; they do not imply new push-down filters or REST endpoints.

Acceptance criteria

• ExtractionRun records tenant, profile, ordered source revisions, pass, parent/
  superseded lineage, experiment/cohort, prompt/template and schema fingerprints,
  metamodel, extractor/host/runtime, status, timing, counts, and bounded failures.
• Requested model configuration records portable model/role, temperature, penalties,
  max tokens, top-k/top-p, thinking/selection fingerprints, and timeout without
  serializing provider extension objects.
• Repeatable invocation records capture only genuinely observed configured service,
  usage, timing, attempt, outcome, and optional provider-response facts.
• Opaque actor, request, session, personalization, and deployment references support
  audit without retaining PII or arbitrary user-context maps.
• Raw prompt/source/user-context payloads are absent by default; optional replay
  material is represented only by classified, expiring protected-content references.
• Replay fidelity is explicit and never promises exact deterministic replay.
• DICE ships scoped in-memory and Drivine stores, indexes, schema registration,
  lifecycle coordination, invocation correlation, and autoconfiguration.
• Empty, completed, failed, and cancelled headers persist; retries are idempotent and
  incompatible terminal rewrites are rejected.
• Proposition-to-run is dedicated many-to-many attribution using canonical saved ids.
• Parent and superseded chains are axis-explicit, bounded, cycle-safe, and queryable.
• Repository, authorized REST, and bundle lineage expose run refs separately from sources.
• Two identical-content runs produce one proposition/source grounding and two run links.
• Every lookup, join, inverse query, chain, page, and aggregate fails closed across tenants.
• Legacy callers, Java descriptors, existing /why JSON, and hosts without configured
  run storage retain current behavior.

Constraints

• Do not add run identity to source-provenance equality.
• Depends on Carry opaque source revisions in proposition provenance #64's final source-revision contract and Carry versioned extraction profiles through compatible DICE entry points #66's profile/current-run refs.
• Do not guess provider-effective values from requested configuration.
• Do not persist credentials, raw prompts/source text, full user/session objects, provider
  SDK responses, native usage objects, or arbitrary extension maps.
• Follow current collector/projection store conventions.
• Land through a reviewed PR to main. PR Embabel Agent 1.5.0 #71 completed the Spring/Embabel upgrade;
  the operator confirmed main is the sole delivery target and no backport is required.

Out of scope

• Atomic product commit.
• Product batching and later-pass orchestration.
• Host replay policy or exact replay guarantees.
• Encrypted blob storage, KMS, retention scheduling, authentication, or actor-directory ownership.

Original request

Record which model, prompt, schema, profile, source revisions, runtime, user/request
context, and observed execution conditions produced knowledge so extraction incidents,
evaluation, cost, drift, testing, and re-analysis are auditable.

Activity

• 2026-07-24T22:42:28Z [groom] Rewritten against the validated proposition extraction roadmap. Claims were checked against current Me and DICE contracts; dependencies and verification boundaries are now explicit.
• 2026-07-24T22:45:05Z [groom] Mirrored to GitHub: Record durable extraction runs and proposition run lineage #67
• 2026-07-26T11:39:01Z [codex] Brigade claimed the ticket for expanded run-metadata and reproducibility design. New input: consider model hyperparameters, user/tenant context, replay, troubleshooting, testing, evaluation, cost, and privacy use cases.
• 2026-07-26T12:50:00Z [codex] Expanded metadata/lineage plan complete: ten packets, shared wire contract, zero Brigade blockers, and Claude Fable ready with no unresolved blocking/high findings. Corrected dependencies to Carry opaque source revisions in proposition provenance #64 and Carry versioned extraction profiles through compatible DICE entry points #66; awaiting implementation approval and upstream landings.
• 2026-07-26T14:06:00Z [codex] Added the cross-domain API analogy and five-use-case matrix. Claude Fable verified every use maps to owned typed fields/tests, provider extensions remain host-supplied fingerprint-or-null, and no unsupported filter/endpoint or exact-replay promise was introduced.
• 2026-07-26T14:06:00Z [codex] Ticket blocked on Carry opaque source revisions in proposition provenance #64/Carry versioned extraction profiles through compatible DICE entry points #66 landings; no implementation branch created.
• 2026-07-26T14:23:00Z [codex] Verified PR Embabel Agent 1.5.0 #71 completed the Spring/Embabel upgrade with successful build/DCO gates. Operator confirmed main-only delivery; removed the backport decision.