Skip to content

Handle async query result-polling (getResultUrl ping-pong) - #77

Open
muratdursun wants to merge 1 commit into
andrusha:mainfrom
peakacom:upstream-async-result-polling
Open

muratdursun wants to merge 1 commit into
andrusha:mainfrom
peakacom:upstream-async-result-polling

Conversation

@muratdursun

Copy link
Copy Markdown

Problem

When a query doesn't complete within the initial queries/v1/query-request window — most commonly while a warehouse resumes from auto-suspend, or for any slow statement — Snowflake returns an async "query in progress" envelope whose data contains only getResultUrl, queryId, progressDesc, queryAbortsAfterSecs (no result rows). The client is expected to GET that getResultUrl until the real result is returned.

This crate never followed that ping-pong, so the untagged ExecResponse rejected the in-progress envelope (every variant's required result fields were absent) and it surfaced as reqwest's opaque error decoding response body. In practice this makes the very first query against a cold warehouse fail.

Reproduced against a live account: the raw 200 response body was

{"data":{"getResultUrl":"/queries/<id>/result","queryId":"<id>","progressDesc":"","queryAbortsAfterSecs":300},"code":"333334","success":true}

Fix

  • Poll getResultUrl until a completed result comes back (await_query_result): 500ms interval, ~60s ceiling. Synchronous results (no getResultUrl) pass straight through unchanged.
  • Connection::get_result performs the polling GET with a fresh request id + the session auth header, matching the original request's Accept MIME.
  • Soft-default the result-bearing fields of QueryExecResponseData so the in-progress envelope deserializes; the caller detects it via get_result_url.
  • Decode responses via serde_json::from_str so a schema mismatch surfaces serde's precise error instead of reqwest's opaque error decoding response body.
  • Adds SnowflakeApiError::QueryInProgressTimeout and a regression test for the in-progress envelope.

Testing

cargo test + cargo clippy clean. Verified end-to-end against a live Snowflake account (PAT auth): queries against a cold COMPUTE_WH now succeed via the polling path where they previously failed to decode.

When a query doesn't complete within the initial request window (a suspended
warehouse resuming, or a slow statement), Snowflake returns an "in progress"
envelope carrying only `getResultUrl`/`queryId` and no result rows. The client
must GET that URL until the real result arrives. This crate never did, so the
untagged `ExecResponse` rejected the response (every variant's required fields
were absent) and surfaced as reqwest's opaque "error decoding response body".

- Soft-default the result-bearing fields of `QueryExecResponseData` so the
  in-progress envelope deserializes (detected via `get_result_url`).
- `Connection::get_result` GETs a result URL with a fresh request id + the
  session auth header.
- `await_query_result` polls `getResultUrl` (500ms × up to 120 = ~60s ceiling)
  until a completed result comes back; synchronous results pass straight through.
- Decode responses via `serde_json::from_str` so a schema mismatch surfaces
  serde's precise error instead of reqwest's opaque one.
- Regression test for the in-progress envelope.

Fixes the Dataglot Snowflake connector failing at catalog build (peakacom
OSS-241).

(cherry picked from commit ec8d581)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant