Skip to content

feat(query): ask for a sample export with buildQuery(sample=True) - #63

Draft
Gcolon021 wants to merge 7 commits into
mainfrom
ALS-13677-sample-dataset-export
Draft

Gcolon021 wants to merge 7 commits into
mainfrom
ALS-13677-sample-dataset-export

Conversation

@Gcolon021

@Gcolon021 Gcolon021 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

buildQuery(sample=True) asks the server for a small sample of an export, so a researcher can write analysis code before pulling the full dataset. This is the adapter side of ALS-13677 and needs the pic-sure change for that ticket, hms-dbmi/pic-sure#487. The request body is unchanged when sample is off, so the adapter still works against a server that predates the option.

  • Query carries sample (default False). runQuery(..., "participant"), runQuery(..., "timestamp") and exportAsPFB send it as "sample": true.
  • loadQueryByID and runQueryByID read sample from the stored query, so a sample dataset created in the PIC-SURE UI runs as a sample again. Older adapter versions drop the flag on reload and export the whole dataset.
  • When a result download answers 409 result_not_ready, the adapter polls status again and downloads once the result is available. The whole wait stays inside the connect(timeout=...) budget.
  • saveQueryByName takes type (any runQuery type or "pfb", default "count"), so the caller chooses what the saved query is. A sample query saves as participant, timestamp or pfb. The adapter sends every combination and lets the server decide. Saving as one of those three starts that export on the server, and the save neither waits for it nor downloads it.
  • When the server refuses a request with a 4xx and a JSON body, the raised error quotes the server's message as plain text with its error type, for example "The server rejected the saveQueryByName query submit (HTTP 400, sample_disabled). The server said: Sample datasets are not available at this time". It used to show the raw JSON body cut at 200 characters. This changes the text of every 4xx error the adapter raises. A 5xx or a body that is not JSON reads as before.

buildQuery takes sample, Query carries it, and the submit body writes
"sample": true only when it is set, so a body without it is unchanged
for servers that predate the field. loadQueryByID reads it from the
stored query and always returns a Query for a sample, so the flag
survives a reload of a filter-only query.
The result route answers HTTP 409 with errorType result_not_ready when a
finished result was lost to a server restart, when the export is being
run again after an access change, and when the job is still running. The
job flow now returns to the status poll and downloads again, inside the
budget that started at the submit. A result still not ready when the
budget passes raises PicSureConnectionError.

A status poll answered with the same 409 counts as still running. A 409
without that error type and a 400 on the download raise at once, as
before.
A result_not_ready error type is read as a result to wait for only on a 409. Any other status carrying it keeps its status-based error.

Query refuses a sample that is not a bool when constructed directly, and buildQuery relies on that check. The guide, changelog and saveQueryByName docstring now say how large a sample is, what a 400 can mean, and that a sample query cannot be saved by name.
A sample of a cohort at or below the server's minimum size comes back whole, so the docstrings no longer say a sample replaces the whole dataset. The budget error after a download that was not ready now says the result did not become ready again. The guide lists open access as a fourth cause of a refused sample, and the sample type error names the module of a type that is not builtin.
@Gcolon021 Gcolon021 added the enhancement New feature or request label Nov 12, 2026
@coderabbitai

coderabbitai Bot commented Nov 12, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

saveQueryByName takes a type argument: any type runQuery accepts, or
"pfb" for a PFB export. The default stays "count", so existing calls
send the same request.

The adapter sends every combination of type and sample and leaves the
decision to the server. A refusal, such as a sample query saved as a
count or a sample query while sample datasets are switched off, raises
PicSureValidationError quoting the server's message, and no named
record is written.

Saving as participant, timestamp or pfb submits that export on the
server. The save does not poll for it or download it.
A 4xx answer whose body is a JSON object with a string message used to
be quoted as the first 200 characters of the raw JSON, request id
included, which could cut the message short. The error now quotes the
message as plain text, up to 1000 characters, and names the errorType
beside the status:

  The server rejected the saveQueryByName query submit (HTTP 400,
  bad_request). The server said: Sample datasets are only available
  for the result types DATAFRAME, DATAFRAME_PFB and
  DATAFRAME_TIMESERIES

This changes the text of the errors raised for 400 and other 4xx, 401,
403 and 404 from every call that goes through the shared translator.
The exception classes are unchanged. A body that is not JSON or has no
string message, and every 5xx, read as before.
The text quoted from a JSON 4xx body is now cleaned before it reaches
an error message. Control characters become spaces and whitespace runs
collapse, so a decoded newline or escape sequence cannot reach a
terminal or a notebook. Credentials are redacted before the text is cut
to its cap, so a token lying across the cut is replaced whole. A body
nested too deeply to decode falls back to the truncated raw body
instead of raising RecursionError out of the translator.

The changelog no longer says a sample query cannot be saved by name, or
that saveQueryByName always saves a count.
@gitguardian

gitguardian Bot commented Nov 13, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
40042353 Triggered JSON Web Token 83a147b tests/unit/test_service_errors.py View secret
40042353 Triggered JSON Web Token 6fbcc40 tests/unit/test_service_errors.py View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices. AWS Best Practice
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant