ModelProvider: declare the kernel port for autolabeling — local and cloud adapters both ordinary

[JArmandoAnaya]

Audited 2026-08-16 at f31653d: description updated to match HEAD. Every acceptance
criterion this issue was opened on is met in the shipped code; what keeps it open is the
scope appended on 2026-08-08, which the title and the criteria never described. The
decisions recorded in the comments below are untouched and remain the arbiter.

The decision this encodes

Recorded by Armando in the 2026-08-07 design session on the AI-assist substrate issue (#81),
which covered ModelProvider pre-annotation, the review loop in the UI, and importers:

ModelProvider is designed and declared now as a kernel port with no implementations,
shaped so that a local runner and a cloud service are both ordinary adapters later: MCP stays
a native first-class path; autolabeling will run locally and in the cloud eventually; nothing
ships yet, and nothing may be designed to preclude the cloud half. Local-first,
cloud-compatible after.

Status at HEAD: the port is designed, and the original deliverable is done

src/visionset/kernel/ports/model_provider.py is no longer the placeholder this issue was
written to replace. It answers all three of the design questions it posed, and argues each
against the dual test:

• Invocation unit — per batch. PredictionRequest carries a tuple of targets; a local
  provider loops, a hosted one does not pay a round trip per image.
• Result shape — Iterator[AssetPrediction], not a materialised Sequence of
  Annotation. A provider answers in its own terms and stamps model_ref; the write gate
  turns claims into labels, so no provider needs to know which schema version a batch pinned.
• Progress without a transport — nothing in the port names a ProgressReporter, a queue, a
  callback or a future. Yielding is the whole mechanism.
• Discovery — settled as no entry-point group, and argued in the docstring: adapters are
  instantiated from user-created InferenceConnection rows, so a provider found by entry point
  would have nothing to be instantiated from, and a workspace could acquire the ability to
  predict by an unrelated pip install.

The local adapter shipped alongside it as slice 2 (cf. #434), so the "zero implementations"
criterion is superseded rather than unmet. src/visionset/inference/ now holds
sam_provider.py and transformers_provider.py behind that port.

What actually keeps this open

The hosted adapter and the HTTP endpoint contract — the cloud half of the dual test, which
the port was shaped for and nothing has exercised. That is the whole of the remaining scope, and
it is neither in this issue's title nor in the criteria below.

Two things to settle at pickup rather than in advance:

• Overlap with the provider contract and plugin architecture (cf. Open providers over closed capabilities: the provider contract and plugin architecture #498). That issue designs
  a provider toolkit above this port, with its own contract version, its own entry-point group
  and its own refusal rules. Whether a hosted adapter is the first consumer of that layer, or
  ships beneath it as an ordinary adapter of this port, decides whether this issue is work or a
  pointer.
• The constraint inherited from exercise the AI-assist substrate — ModelProvider pre-annotation, review loop in the UI, importers #81 and preserved here: predictions are written into
  review_pending, never silently annotated, and schema validation is not relaxed for
  them.

Out of scope

• Any UI.
• Any job-queue infrastructure. If async turns out to need more than a port can express, flag
  it rather than build it — that is a separate decision.

Acceptance criteria

• ModelProvider is a designed port rather than a placeholder: every element of its shape
  justified against both a local runner and a hosted service
• Discovery is settled — an entry-point group is named, or its absence is argued in the
  docstring
• Zero implementations ship — superseded 2026-08-08 by the local adapter (cf. feat(inference): ModelProvider port, local adapter, weight download (#418 slice 2) #434)
• The docstring states the contract and its non-goals, in importer.py's and
  job_queue.py's register
• A hosted adapter and its endpoint contract, or a written decision that they belong to
  Open providers over closed capabilities: the provider contract and plugin architecture #498 instead

Original description, as filed 2026-08-08 (including the 2026-08-08 scope update)

The decision this encodes

Recorded by Armando in the 2026-08-07 design session on the AI-assist substrate issue (#81),
which covered ModelProvider pre-annotation, the review loop in the UI, and importers:

ModelProvider is designed and declared now as a kernel port with no implementations,
shaped so that a local runner and a cloud service are both ordinary adapters later: MCP stays
a native first-class path; autolabeling will run locally and in the cloud eventually; nothing
ships yet, and nothing may be designed to preclude the cloud half. Local-first,
cloud-compatible after.

Nothing in this issue ships an implementation. The deliverable is a port whose shape
survives both futures.

Correction to the premise: the port already exists, as a placeholder

src/visionset/kernel/ports/model_provider.py is at HEAD (904b37a) and is exported from
kernel/ports/__init__.py:24. It is not a designed port and its own docstring says so —
"Phase 3 — declared now to fix the port surface". Its entire surface is:

def predict(self, asset: Asset, schema: AnnotationSchema) -> Sequence[Annotation]: ...

That shape fails the dual test in three ways, which is precisely the work:

• Per-asset, so a provider that batches — which every hosted inference service does — pays
  one round trip per image.
• Synchronous, returning a materialised Sequence, so a remote provider that takes minutes
  has nowhere to be except blocking its caller.
• Discovered by nothing. Importer and Exporter name the visionset.formats entry-point
  group; this port names no group at all, so there is no way to install a provider even in
  principle.

So this issue re-shapes a declared placeholder. It does not add a new file to empty space,
and a PR that only creates one has not done the work.

The pattern to follow

kernel/ports/importer.py — a @runtime_checkable Protocol, discovered through an entry-point
group, zero implementations, documented as awaiting its producers. formats/registry.py
already filters one group by port (isinstance on an instance, because issubclass against a
protocol with data members raises), so reusing that group or declaring a new one both have
precedent — but the choice has to be made and argued rather than inherited.

Design questions the implementer answers in the PR, not in advance here

Every answer must hold for a hypothetical local ONNX runner and a hypothetical HTTP service
equally. That dual test is the acceptance criterion.

• The invocation unit — asset, or batch? A per-asset call is simplest locally and worst
  remotely, which is exactly the tension the port has to resolve rather than pick a side of by
  accident.
• The result shape — annotations carrying provenance='model', confidence, and model
  identity/version. The domain already requires model_ref whenever provenance is model
  (domain/annotation.py:69-71); the open question is whether that one string is the whole of
  model identity or whether the port owes more. These fields align with the work that surfaces
  provenance and confidence in the review loop (Model-produced annotations: provenance and confidence surfaced in the review loop #417), which is the rendering half of the same
  split.
• How a slow or asynchronous provider reports progress without the port assuming a
  transport. The JobQueue port in kernel/ports/job_queue.py, which came out of the embedded
  job system with its SQLite adapter and in-process dispatcher (Embedded job system: a JobQueue port, a SQLite adapter, and an in-process spawn dispatcher #328), already exists and is
  explicitly not an executor — its separation of "where rows live" from "how the product asks
  for work" is the shape to lean on, not to duplicate.

Out of scope

Scope updated 2026-08-08: the port and the local adapter shipped as slice 2 (#434),
including cross-box NMS, the fp16 shims, and the weight-download job. This issue remains open
for the hosted adapter and the HTTP endpoint contract. The original scope text is preserved in
this issue's edit history.

• Any adapter, local or hosted. When one is eventually built it inherits a constraint
  recorded on exercise the AI-assist substrate — ModelProvider pre-annotation, review loop in the UI, importers #81 and preserved here so it does not die with that thread: predictions are
  written into review_pending, never silently annotated, and schema validation is
  not relaxed for them.
• Any UI.
• Any job-queue infrastructure. If async turns out to need more than a port can express, flag
  it rather than build it — that is a separate decision.

Acceptance criteria

• ModelProvider is a designed port rather than a placeholder: every element of its shape
  justified against both a local runner and a hosted service
• Discovery is settled — an entry-point group is named, or its absence is argued in the
  docstring
• Zero implementations ship
• The docstring states the contract and its non-goals, in importer.py's and
  job_queue.py's register

[JArmandoAnaya]

These decisions were first recorded on the AI-assist substrate issue (#81), which closed on
2026-08-07 and was split into the work surfacing model provenance and confidence in the review
loop (#417) and the port work tracked here (#418). They are re-anchored on this issue so that it
is the arbiter for the port:

Decision (Armando, 2026-08-06): This project does not take runtime dependencies on
third-party inference ecosystems. Auto-labeling is built on neutral foundations
(transformers, weights from original sources). Third-party inference stacks are
design reference only, never dependencies.

Decision (Armando, 2026-08-07): The ModelProvider port is designed to run models both
locally and remotely. Adapters are instantiated from user-created model connections
(cf. #421), never from a bundled default. Nothing may be designed in a way that
precludes remote execution later — local-first, remote-compatible after.

Decision (Armando, 2026-08-07): VisionSet never downloads models or weights at install
time. Weight download is an explicit user action when setting up a local connection,
surfaced with the download size beforehand. The local runtime ships as the
visionset[local-inference] extra (canonical spelling: local-inference) — it is the
runtime for local-type connections, not the auto-labeling feature, which is always
present. Supersedes the earlier [autolabel] framing.

— supersedes any prior direction; cf. #417, #421.

[JArmandoAnaya]

Durable findings from the Phase 0 measurement spike, run against Grounding DINO base in fp16 on a 16 GB GPU. The worktree has been deleted and the full report extracted, so these are the parts worth keeping.

1. fp16 requires shims on transformers 5.14.1. Several Grounding DINO internals — the deformable-attention reference points that go through grid_sample, and the decoder query positional embeddings — are built in float32 regardless of the weight dtype. The local adapter therefore needs torch.autocast("cuda", float16) around the forward pass plus a grid_sample dtype cast, or inference fails with dtype mismatches. Weights stay fp16-resident, so VRAM is unaffected.
2. Raw output needs cross-box NMS. At box_threshold=0.35 and text_threshold=0.25, duplicate detections of the same instance are a primary failure mode. The adapter, or the pipeline above it, must apply NMS before pseudo-labels reach the write gate.
3. Memory does not force sequencing on the 16 GB profile. The model sits at 0.44 GiB resident and around 1.4 GiB while inferring at batch 1, so a co-resident SAM-class model at roughly 4.5 GiB worst case still fits with about 8 GiB spare.
4. Batching does not help. Throughput decreases as batch size grows on mixed-size images, because the padding cost exceeds the batching gain. Batch 1 is the operating point, at roughly 115 ms per image. Uniform-size batches were not measured.
5. The reference hardware changed. Measurements were taken on an RTX 5070 Ti rather than the 4070 Ti previously recorded as the reference profile — the same 16 GB, on newer silicon. Throughput is therefore an upper bound for the stated OSS profile, while the VRAM numbers transfer as-is.

One caveat on quality: zero-shot performance was measured off-domain, on 11 cat and dog images. Classification was reliable and localization approximate, with head-only boxes and duplicates. On-domain quality for traffic and robotics remains unmeasured, which is relevant to the open ordering question.

[JArmandoAnaya]

Decision (Armando, 2026-08-07): Auto-labeling ships both modes — interactive
click-to-suggest in the editor canvas, and batch pre-labeling as a background job. This
decision sequences them; it does not choose between them: interactive first, batch
second.

Rationale: the two modes form one loop at maturity — batch pre-labels at scale, humans
review, and the interactive tool is what makes correcting batch output cheap. The
dependency is asymmetric: interactive is self-sufficient and delivers value with no
batch stage in front of it, while batch reaches full value only once the interactive
correction tool exists. Interactive-first also generates the on-domain quality signal
(suggestion acceptance rate) that the Phase 0 spike could not measure, informing batch
threshold/NMS tuning before batch ships. Measured facts on record in the spike-findings
comment above: neither VRAM nor throughput forces an ordering on the 16 GB profile.

Scope note for the interactive mode: it operates against a user-configured model
connection (cf. #421) and must respect the editor self-sufficiency principle — no flow
may force navigation out of the editor, including the no-connection-configured empty
state. — supersedes any prior direction; cf. #417, #421.