Repository navigation
Conversation
xlai20
force-pushed
the
integration-tests-bidi-read
branch
2 times, most recently
from
September 30, 2026 03:16
08556ce to
985388e
Compare
…l_rapid, and regional_rapid submodules
…s with full RCU lifecycle
…ional rapid tests
…s in conformance.rs
…n DEFAULT_TEST_ENDPOINT
…id non-colocated cases
… and is_hns flags
…handle b/565175323
…e, clarify conformance output
… conformance tests
…_ENDPOINT and HTTP_ENDPOINT
…_GRPC_ENDPOINT and GOOGLE_CLOUD_TEST_HTTP_ENDPOINT
…Rapid and Regional Rapid
… legacy bidi read tests verbatim Use rcu_crud.rs-style banners and SUCCESS/Warning messages in the bidi read conformance suite, without test numbering. Restore bidi_read/features.rs as an exact copy of the original bidi_read.rs from main (comments, private helpers, and run(bucket_name) signature), and drop internal suite numbering from the module docs.
- Clean up buckets even when a test panics, then re-raise the panic. - Use a valid bucket id for the non-existent bucket read. - Fail the out-of-range test if the read returns neither data nor an error. - Explain why Regional Standard (HNS) only runs the multiple ranged read test. - Add a module doc to bidi_read/features.rs.
…escriptor Dispatch a valid range and an out-of-bounds range concurrently via tokio::join! on the same ObjectDescriptor session (matching Python's asyncio.gather pattern), verifying that the valid range succeeds while the out-of-bounds range returns OutOfRange.
…s unset Guard Zonal Rapid conformance testing in run() with #[cfg(google_cloud_unstable_storage_bidi)], cleanly skipping Zonal Rapid bucket creation and execution when compiled without the unstable flag.
…7021) Resolve idempotency per request for `google.storage.v2.Storage` RPCs, following the [GCS retry strategy](https://cloud.google.com/storage/docs/retry-strategy#idempotency-operations) table. Idempotent mutations carry an `x-goog-gcs-idempotency-token` header, minted once per call so all retry attempts share the same token. `librarian.yaml` sets `idempotency_hook: resolve_idempotency` for `google/storage/v2`, so the generated transport calls `req.resolve_idempotency(options)`. The rules live in `src/idempotency.rs`: - Reads and lists are always idempotent and carry no token. - `CreateBucket`, `DeleteBucket`, and `LockBucketRetentionPolicy` are always idempotent. - `UpdateBucket` and `UpdateObject` require `if_metageneration_match`. - `ComposeObject`, `RestoreObject`, `RewriteObject`, and `MoveObject` require the destination `if_generation_match`. - `DeleteObject` requires a `generation` or `if_generation_match`. - Negative preconditions (`*_not_match`) never make a request idempotent. An explicit `with_idempotency()` always takes precedence, and `with_idempotency(false)` also suppresses the token. Behavior changes: `create_bucket()` and `delete_bucket()` are now retried. Part of the split of googleapis#7000 (towards googleapis#5974): - [x] PR 1: Request-level idempotency rules and `resolve_idempotency` hook for `google.storage.v2.Storage` RPCs - [ ] PR 2: gRPC mock integration tests (`grpc_mock_idempotency.rs`) for `StorageControl` - [ ] PR 3: Idempotency token stamping and precondition check for single-shot uploads - [ ] PR 4: Idempotency token stamping for resumable upload session creation - [ ] PR 5: Retry code samples and `src/storage/GEMINI.md` documentation --------- Co-authored-by: Chen Chen <chensg@google.com>
Refactor `Value::as_*` accessors (`as_str`, `as_bool`, `as_f64`,
`as_struct`, `as_list`) to return `Option<...>` instead of panicking,
aligning the API with standard Rust conventions (such as
`serde_json::Value`).
Key changes:
- `as_str(&self) -> Option<&str>`: Added as the canonical borrowed
string slice accessor.
- `as_bool`, `as_f64`, `as_struct`, `as_list`: Updated return types from
scalar/reference types to `Option<...>`.
- `is_null(&self) -> bool`: Added helper to test for protobuf null or
unspecified value states.
- `as_string` & `try_as_*`: Marked as deprecated with migration guidance
pointing to the corresponding `as_*` accessors.
- Documentation: Added three-part rustdoc comments with runnable
doctests for all public accessors, and documented Spanner's wire format
behavior for non-finite floats (`NaN`, `±Infinity`) on `as_f64`.
- Convenience methods: Replaced bare `.unwrap()` calls in `Row::is_null`
and `Row::get` with descriptive `.expect(...)` panic messages.
BREAKING CHANGE: `Value::as_bool`, `Value::as_f64`, `Value::as_struct`,
and `Value::as_list` now return `Option<...>` instead of panicking.
`Value::as_string` is deprecated in favor of `Value::as_str` and also
returns `Option<&str>`.
Migration Guide:
- If you were handling missing/mismatched values:
- Before: `value.try_as_bool()`
- After: `value.as_bool()`
- If you were expecting panics on type mismatch:
- Before: `value.as_bool()`
- After: `value.as_bool().expect("expected bool")`
- For string slices:
- Before: `value.as_string()`
- After: `value.as_str().expect("expected string")`
Make TimestampBound constructors infallible to prevent panics in public client APIs, while adding fallible `try_*` variants for types that require conversion. Previously, `TimestampBound::exact_staleness`, `max_staleness`, `read_timestamp`, and `min_read_timestamp` called `.expect()` internally, which could panic if an input value was out of range. Changes: - Infallible staleness constructors: `exact_staleness` and `max_staleness` accept `std::time::Duration` and clamp out-of-range values safely. - Infallible timestamp constructors: `read_timestamp` and `min_read_timestamp` accept `wkt::Timestamp`. - Fallible `try_*` constructors: `try_exact_staleness`, `try_max_staleness`, `try_read_timestamp`, and `try_min_read_timestamp` accept generic `TryInto` inputs (supporting `&str`, `SystemTime`, `OffsetDateTime`, etc.) and return `Result<Self, T::Error>`. - Implement `Default` (returning `Self::strong()`) and derive `PartialEq` on `TimestampBound`. - Add cross-references in doc comments and fix the doc link anchor on `read_timestamp`. BREAKING CHANGE: `TimestampBound::read_timestamp` and `min_read_timestamp` now accept `wkt::Timestamp` directly instead of generic `T: TryInto<Timestamp>`. Callers passing ecosystem date/time types (e.g., `time::OffsetDateTime`, `std::time::SystemTime`, or string slices) must use `TimestampBound::try_read_timestamp` or `try_min_read_timestamp` and handle the `Result`. Callers passing `std::time::Duration` to `exact_staleness` or `max_staleness` are unaffected and continue to compile unchanged.
Document the safety invariant on `From<wkt::Timestamp> for Value`, clarifying that this conversion cannot panic at runtime because `wkt::Timestamp` is strictly bounded to years 0001 through 9999, which is well within the range supported by `time::OffsetDateTime`. Also clean up the tests in `to_value.rs` to conform to the workspace styleguide: - Remove the `test_` prefix from all test function names. - Replace `.unwrap()` calls with descriptive `.expect(...)` messages. - Add explicit error messages to boolean assertions. - Replace fully-qualified paths (`time::...`, `prost_types::...`) with top-level imports in the test module. - Expand abbreviated variable names to full words. - Add boundary tests verifying `Timestamp::MIN_SECONDS` and `Timestamp::MAX_SECONDS` convert cleanly.
Several public API methods in the Spanner client used types that lived in internal modules without public re-exports. In practice, this made it impossible for applications to name these types in their own function signatures, test fixtures, or error-handling code. This change exposes those types in their respective modules and makes them straightforward to use: - Transaction results: Callers can now explicitly type-annotate TransactionResult, inspect commit timestamps and stats directly, or construct mock results in unit tests. - Custom transaction retries: Developers can now implement TransactionRetryPolicy to supply custom retry logic for read/write transactions and Partitioned DML. - Generic row helpers: Callers can now use the ColumnIndex trait to write generic helper functions that accept either column names or zero-based indices. - Error inspection: RowError, ConvertError, and TlsError are now exported under the error module, with extract() helpers on RowError and ConvertError to easily inspect deserialization failures without having to manually walk the source error chain. Missing Debug implementations on public builder and response stream types have also been added for consistent logging.
In preparation for the 1.0 (GA) release of google-cloud-spanner, remove the pre-1.0 `opentelemetry` (0.33) dependency from the crate's default public API: - Introduce an opt-in `unstable-metrics` feature flag that gates `pub use opentelemetry;` and the custom `MeterProvider` builder methods (`with_meter_provider` and `with_export_builtin_metrics_to_custom_provider`). This isolates pre-1.0 OpenTelemetry SemVer breaking changes behind an explicit opt-in, following the `unstable-*` convention used in other crates (such as `unstable-stream` in google-cloud-storage). - Remove the deprecated `metrics` feature. - Introduce an internal `_internal-metrics` feature to decouple `builtin-metrics` from the public API. Background export of latency and request metrics to Google Cloud Monitoring remains enabled by default, but OpenTelemetry is treated strictly as an internal implementation detail without leaking into user code. - Add docs.rs configuration with feature badges so APIs requiring `unstable-metrics` are clearly marked in documentation.
xlai20
force-pushed
the
integration-tests-bidi-read
branch
from
October 1, 2026 09:23
985388e to
9ae202b
Compare
…error checks State explicitly in conformance.rs that out-of-range and post-stream-close checks on Regional Standard (HNS) are duplicate because the bidi read transport logic is identical between flat and HNS buckets.
…tracking (googleapis#7007) Connect streaming RPCs (execute_streaming_sql and streaming_read) to location-aware routing feedback and active connection lifecycle management. Key changes: - Active Request Tracking: Hold an ActiveRequestGuard on direct ServerConnection instances throughout the lifetime of the stream, ensuring in-flight request counters accurately reflect active streaming load for tablet selection. - TTFB Latency Measurement: Measure time-to-first-byte (TTFB) upon arrival of initial gRPC response headers/handshake, feeding accurate network round-trip and server startup latency into Paxos group candidate selection without being inflated by slow client consumers or long streams. - Stream Lifecycle & Cooldown Feedback: - Record endpoint success upon initial handshake and clean stream completion (EOF), allowing consecutive successes to heal expired cooldown tiers. - Intercept initial handshake and mid-stream errors (UNAVAILABLE, RESOURCE_EXHAUSTED), extract server retry delay hints (RetryInfo), and apply endpoint cooldowns and group penalties. - StreamGuard Extension: Add record_first_response, record_error, and record_success to StreamGuard with default no-op implementations, keeping standard Spanner channel pooling unaffected. - Test Coverage: Add unit and mock integration tests covering initial TTFB latency, mid-stream failures with cooldown floors, early client stream drops, and end-to-end ResultSet lifecycle.
Register `google_cloud_unstable_bigquery_arrow` and `google_cloud_unstable_bigquery_storage_read` compiler cfg flags to support upcoming Arrow + jobs.query work and Storage Read acceleration work. Towards googleapis#7034 googleapis#7037
Increase the size limit of responses we can receive over gRPC. Note that tonic does not pre-allocate a buffer. So we don't pay for this. The only draw back is if the server reports a bad size in the headers. I don't think that happens. Fixes googleapis#7078
Tombstone the library by emptying all symbols, removing dependencies, bumping to 2.0.0, and keeping it configured in librarian.yaml so librarian workflows continue to work.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a cross-SDK conformance suite for bidirectional reads (
open_object/BidiReadObject) and runs it against four bucket configurations: Regional Standard (flat), Regional Standard (HNS), Zonal Rapid, and Regional Rapid (HNS with a Rapid Cache Ultra cache attached).Changes
New conformance suite (
tests/storage/src/bidi_read/conformance.rs)test_non_existent_bucket_readNotFound(orPermissionDenied)test_read_post_stream_closeNone, and dropping an unfinished reader does not break the descriptortest_out_of_rangeOutOfRange/InvalidArgument, while valid reads on the same descriptor still succeedtest_multiple_ranged_readopen_appendable_object. This requires--cfg google_cloud_unstable_storage_bidi. All other buckets use regular JSON uploads.GOOGLE_CLOUD_TEST_GRPC_ENDPOINTandGOOGLE_CLOUD_TEST_HTTP_ENDPOINT. If either is unset, the test prints a message and exits successfully, so it is a no-op in CI today.Reorganized existing bidi read tests
tests/storage/src/bidi_read.rsnow only declares two submodules:conformanceandfeatures.tests/storage/src/bidi_read/features.rs. The file is byte-for-byte identical to the oldbidi_read.rs;git diff main -Cshows this.tests/storage/tests/driver.rs,run_storage_bidiis split intostorage::bidi_read::features(same behavior as before) andstorage::bidi_read::conformance.Endpoint variable rename
rcu_crud.rsandrun_storage_control_rapid_cachenow readGOOGLE_CLOUD_TEST_GRPC_ENDPOINTinstead ofGOOGLE_CLOUD_TEST_STORAGE_CONTROL_ENDPOINT, so every test that needs a custom endpoint uses the same variable. No CI configuration references the old name.Testing
Existing tests, with the endpoint variables unset (matches CI):
Result:
featurespasses against production andconformanceis skipped.Conformance suite, against a custom endpoint:
Result: all scenarios passed on all four bucket types (121s).
cargo fmt --checkandcargo clippy --tests -- -D warningsare clean.