Skip to content

MPT-24908 Attach a Meta to models yielded by stream() - #409

Closed
jentyk wants to merge 1 commit into
mainfrom
bugfix/MPT-24908/streamed-models-carry-meta-none-while-iterated-carry-meta
Closed

jentyk wants to merge 1 commit into
mainfrom
bugfix/MPT-24908/streamed-models-carry-meta-none-while-iterated-carry-meta

Conversation

@jentyk

@jentyk jentyk commented Sep 4, 2026 •

Copy link
Copy Markdown
Member

🤖 AI-generated PR — Please review carefully.

What

Fixes MPT-24908: models yielded by stream() carried meta=None, while models from the paged path carry a Meta.

  • Paged reads (iterate(), fetch_page(), fetch_one()): ServiceBase.make_collection builds each model as model_class(resource, meta), so every model carries the page's Meta.
  • Streaming reads (stream(), both wire formats, sync and async): deserialize_stream_record built models as model_class(record), leaving meta=None.

Code migrated from for m in svc.iterate(): use(m.meta.response.headers) — or any .meta access — to svc.stream() therefore crashed with AttributeError: 'NoneType' object has no attribute ... on the first record. Neither stream() docstring mentioned the gap, and in envelope format the $meta member that actually arrives on the wire was parsed into a StreamedTotal and then discarded rather than surfaced to the models.

How

  • New streaming_meta() in mpt_api_client/http/mixins/streaming_mixin.py builds one Meta per streaming read, shared by every model of that stream — mirroring how one page's models share their page's Meta. deserialize_stream_record now takes it and calls model_class(record, meta).
  • meta.response is a headers-and-status snapshot with an empty body. The body is the stream and is never buffered; buffering it to populate the snapshot would give up the one-record memory bound that makes streaming worth using. content/text are empty and json() raises; headers and status_code are real. Header names come out lowercased by dict(response.headers) — the paged path does the identical conversion, so the same lookup works on both.
  • meta.pagination.total is the declared MPT-Item-Count, identical in both wire formats. limit/offset stay 0, so has_next() is False and total_pages() is 0: a stream is one response, not a cursor into a sequence of them.
  • Deletion stubs stay meta-less. A DeletionStub is not a model and guarantees nothing but its id, so .meta on a streamed object still requires the existing stub branch.
  • The envelope's own $meta.pagination.total is still parsed and dropped rather than sourced — the header remains the single authoritative total, so a faulty response whose two carriers disagree cannot overwrite it.
  • Reading the declared count once for both the progress receiver and the Meta moved into a _negotiated_meta helper, keeping stream() within its WPS210 local-variable budget. The events generator is passed unstarted, so set_total_items still fires before the first record leaves the wire; the existing progress-ordering tests cover that.

deserialize_stream_record gained a required third parameter. That is not a public break: it is not exported from http.mixins or http, has no callers outside the mixin and its tests, and stream() has not shipped in a release.

Testing

  • 10 tests added or updated in test_streaming_mixin.py: the declared count reaching meta.pagination (sync, async, and envelope format), the response snapshot's headers/status/empty body, one Meta shared across a stream's models, meta staying out of to_dict(), and deserialize_stream_record attaching the meta — plus the three existing deserialize_stream_record call sites updated for the new parameter.
  • Every claim in the new documentation section was verified against the interpreter rather than reasoned out: has_next(), total_pages(), ignored, text, json() raising, and the stub's AttributeError.
  • Documented in docs/streaming.md: a new What model.meta Carries On A Streamed Model section with a paged-vs-streamed comparison table, plus a model.meta row in the stream()-versus-iterate() table.
  • make check passes (ruff format, ruff check, flake8, mypy, uv lock).
  • make test passes: 2635 tests, 99% overall coverage, streaming_mixin.py at 100%.

🤖 Generated with Claude Code

Closes MPT-24908

  • Attach shared Meta data to models from synchronous and asynchronous stream() calls.
  • Include response headers, status code, item count, and zero limit and offset values.
  • Preserve meta-less DeletionStub objects.
  • Ignore envelope pagination totals in favor of the MPT-Item-Count response header.
  • Update streaming documentation and tests.

deserialize_stream_record built models as model_class(record), leaving
meta=None, while the paged path (ServiceBase.make_collection) builds
model_class(resource, meta). Code moved from iterate() to stream() then
crashed with AttributeError on the first .meta access, and neither
stream() docstring mentioned the gap.

Build one Meta per streaming read and share it across every model of
that stream, as one page's models share their page's Meta. Its response
is a headers-and-status snapshot with an empty body: the body is the
stream itself and buffering it to fill the snapshot would give up the
one-record memory bound that makes streaming worth using. Its
pagination.total is the declared MPT-Item-Count, identical in both wire
formats, with limit and offset left at zero because a stream has no
pages. Deletion stubs stay meta-less — a stub is not a model and
guarantees nothing but its id.

The envelope's own $meta.pagination.total is still parsed and dropped
rather than sourced: the header stays the single authoritative total,
so a faulty response cannot overwrite it. Reading the count once up
front for both the progress receiver and the Meta moved into a
_negotiated_meta helper, keeping stream() within its local-variable
budget; the events generator is passed unstarted, so set_total_items
still fires before the first record leaves the wire.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jentyk
jentyk requested a review from a team as a code owner September 4, 2026 08:46
@jentyk
jentyk requested review from alephsur and robcsegal September 4, 2026 08:46
@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown

✅ Found Jira issue key in the title: MPT-24908

Generated by 🚫 dangerJS against ce70254

@sonarqubecloud

sonarqubecloud Bot commented Sep 4, 2026

Copy link
Copy Markdown

@coderabbitai

coderabbitai Bot commented Sep 4, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Essentials

Run ID: a4e6c141-761c-4933-b37f-e7054b8d82cc

📥 Commits

Reviewing files that changed from the base of the PR and between 4c714dc and ce70254.

📒 Files selected for processing (3)
  • docs/streaming.md
  • mpt_api_client/http/mixins/streaming_mixin.py
  • tests/unit/http/mixins/test_streaming_mixin.py
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • softwareone-platform/mpt-extension-skills (manual)

Included review availability: 2 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 3 reviews per hour.

📜 Recent review details
🧰 Additional context used
📓 Path-based instructions (5)
Review documentation changes against `docs/documentation.md` and the linked repository's `standards/documentation.md`.

⚙️ CodeRabbit configuration file

Files:

  • docs/streaming.md
For each subsequent commit in this PR, explicitly verify if previous review comments have been resolved

⚙️ CodeRabbit configuration file

Files:

  • docs/streaming.md
  • mpt_api_client/http/mixins/streaming_mixin.py
  • tests/unit/http/mixins/test_streaming_mixin.py
When repository behavior changes, update the narrowest relevant document under `docs/`

📄 CodeRabbit inference engine (docs/contributing.md)

Files:

  • docs/streaming.md
Topic-specific documentation must live in the matching file under `docs/` directory Shared engineering rules must be linked from `mpt-extension-skills` instead of copied into this repository

📄 CodeRabbit inference engine (docs/documentation.md)

Files:

  • docs/streaming.md
Put topic-specific documentation under `docs/` directory instead of expanding `README.md` Link shared engineering rules from `mpt-extension-skills` instead of duplicating them locally in documentation

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • docs/streaming.md
🧠 Learnings (1)
📚 Learning: 2026-02-02T13:05:41.144Z
Learnt from: albertsola
Repo: softwareone-platform/mpt-api-python-client PR: 201
File: tests/unit/resources/accounts/mixins/test_activatable_mixin.py:132-139
Timestamp: 2026-02-02T13:05:41.144Z
Learning: In the mpt-api-python-client repository, tests are configured to use pytest asyncio mode auto, which auto-detects async test functions and runs them without requiring pytest.mark.asyncio. Reviewers should rely on this behavior for all Python test files under tests/, and avoid adding unnecessary asyncio markers in async tests. Ensure test files in tests/ adhere to this convention unless a specific test requires an explicit marker.

Applied to files:

  • tests/unit/http/mixins/test_streaming_mixin.py
🔇 Additional comments (3)
mpt_api_client/http/mixins/streaming_mixin.py (1)

36-44: LGTM!

Also applies to: 169-194, 465-465, 474-481, 491-491, 557-562, 596-598, 637-650, 653-674, 683-695, 765-770, 804-806, 846-862, 863-884, 893-905

tests/unit/http/mixins/test_streaming_mixin.py (1)

27-28: LGTM!

Also applies to: 155-166, 234-279, 396-405, 948-962, 980-981, 1027-1038

docs/streaming.md (1)

26-26: LGTM!

Also applies to: 83-115


📝 Walkthrough

Walkthrough

Streaming models now share response and pagination metadata for the complete stream. Synchronous and asynchronous paths create this metadata before record consumption. Deletion stubs remain metadata-free. Tests and documentation cover the new behavior.

Changes

Streaming model metadata

Layer / File(s) Summary
Metadata construction and record deserialization
mpt_api_client/http/mixins/streaming_mixin.py
Response headers, status, and declared item count create shared Meta data. Deserialization attaches it to models and excludes it from deletion stubs.
Synchronous and asynchronous stream wiring
mpt_api_client/http/mixins/streaming_mixin.py
Both stream paths negotiate metadata and progress totals before consuming records. Envelope totals remain ignored.
Metadata validation and documentation
tests/unit/http/mixins/test_streaming_mixin.py, docs/streaming.md
Tests cover shared metadata, response snapshots, declared counts, JSON envelopes, serialized records, and deletion stubs. Documentation describes stream-scoped metadata.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to ce702

Streaming models now expose shared response and pagination metadata in synchronous and asynchronous formats while preserving meta-less deletion stubs. The covered behavior is ready to merge with no active current-head risk identified.

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Documentation Up To Date ✅ Passed PASS: The PR changes user-facing stream() metadata behavior and updates the matching source-of-truth document, docs/streaming.md, in the same commit. The added section documents shared Meta, res…

Comment @coderabbitai help to get the list of available commands.

@jentyk jentyk closed this Sep 4, 2026
@jentyk
jentyk deleted the bugfix/MPT-24908/streamed-models-carry-meta-none-while-iterated-carry-meta branch September 4, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant