The format is based on Keep a Changelog, and this project adheres to Semantic Versioning. See MAINTAINERS.md for instructions to keep up to date.
-
Every store reads the
compression_configquery parameter of its URL to tune how it compresses the files it writes: for zstd stores, the level and window as<level>or<level>/<window MiB>(best,better/32,best/64); for gzip stores, an integer level from-2to9. The setting applies to that store only, and is kept bySubStoreandClone. A value not valid for the store's compression, or any value on a store without compression, makes the constructor fail. Readers need no change, but must have memory for the zstd window, and thezstdcommand line tool refuses windows above 128 MiB by default. On BNB Chain merged blocks,best/64writes files 26% smaller than the defaults, encoding at 64 MB/s instead of 879 MB/s and decoding at 892 MB/s instead of 2362 MB/s. See the README for the other configurations. -
zstd stores accept the decoder settings
lowmem=<bool>andpool=<name>incompression_config, after the level or alone, separated by commas (best/32,lowmem=false,pool=blocks,pool=cache).lowmem=falsemakes objects larger than their window decode with far less memory copying, at twice the window in memory per decoder.pool=<name>reuses decoders across objects and across every store using that name. Reading 100 MiBbest/32objects 10 at a time,lowmem=false,pool=blocksused 72% less CPU and 26% less peak heap than the defaults. -
The
Compressionoption is now honoured by stores created directly withNewGSStore,NewS3Store,NewAzureStore,NewLocalStoreandNewMemoryStore, and byClone, not only throughNewStore. -
StoregainedListFoldersFromTo(ctx, prefix, inclusiveFrom, exclusiveTo, max),ListFoldersrestricted to a slice of the key space. A single folder listing is paged one round trip at a time however few folders come back, so a caller holding tens of thousands of them can now split the key space and list the slices concurrently: on a Google Cloud Storage folder holding 39k sub-folders that takes the listing from 8.2s to 1.1s.GSStorepushes both bounds down as the listing's start and end offsets andS3Storepushes the lower one down asStartAfter; the others filter what they listed. The bounds must start withprefix, the same contractWalkFromToenforces. -
StoregainedListFolders(ctx, prefix, max), returning the immediate sub-folders ofprefix(each relative to the store and ending with/) without reporting anything nested deeper. Object stores answer it with a single delimited listing that never looks at the objects below;LocalStorereads the one directory. A negativemaxmeans unlimited, andprefixis accepted with or without its trailing/. -
StoregainedWalkAttributes(ctx, prefix, f), walking objects and yielding anObjectEntry— name, size and modification time — instead of just the name. Every backend reads all three straight out of the listing it was already making, so this costs whatWalkcosts whereObjectAttributesper object used to be the only way to get them. It follows the same rules asWalk,StopIterationincluded.Both are implemented natively by
GSStore,S3Store,AzureStore,LocalStoreandMemoryStore.MockStoreis served by the genericWalk-based implementations, which return the same results at the usual cost. -
Azure store:
AZURE_STORAGE_ENDPOINToverrides the service URL, pointing the client at an emulator or a private endpoint instead of<account>.blob.core.windows.net. Unset, nothing changes. -
Azurite service in
docker-compose.yml, so the Azure store can be exercised locally without an Azure account, plus the matchingstoretestsentry points.
-
Azure store:
WalkFromandWalkFromTohave Azure start the listing at the starting point, with thestartFromparameter of List Blobs, instead of listing every file under the prefix and skipping those before it. A walk from near the end of a store holding millions of files now takes a few calls instead of one per 5000 files before the starting point. The names are still checked against both bounds, so an endpoint that ignoresstartFromreturns the same files, at the old cost. -
zstd stores read with decoders kept in a process-wide
defaultpool unlesscompression_confignames another pool, and those decoders work on the reading goroutine instead of 4 background ones. Reading 100 MiBbest/32objects 10 at a time, that used 6% less CPU and 53% less peak heap; on 2 MiB objects, 42% less CPU.compression_config=pool=nonebrings back a decoder per object with 4 background goroutines. Reading from a zstd reader after closing it now returns an error. -
Bumped
github.com/klauspost/compressfrom v1.10.2 to v1.20.1, which requires Go 1.25. In v1.10.2 thebetterandbestlevels were aliases ofdefault. -
S3 store:
CopyObjectis now done by the service itself, with a singleCopyObjectcall up to the 5 GiB S3 allows and a multipart copy of 1 GiB parts above that. A backend answeringNotImplementedorMethodNotAllowedfalls back to the previous behaviour, which downloaded the object and uploaded it back, moving every byte through the client and, on a compressed store, decompressing and recompressing it on the way.
-
A local store whose base path is
.now finds its files.Walk,WalkFromand the listings returned nothing, becausefilepath.Walk(".")reportsfileand not./file, which never matched the./prefix the walk filtered on. The base path is now made absolute when the store is created. -
S3Store.WriteObjectno longer commits a partial object when its reader fails partway. The reader's error was turned into a clean end of stream before the upload was cancelled, so an upload that got there first completed with the content read so far andWriteObjectreturnednil. The upload now fails with the reader's error whatever the timing. -
zstd stores now close the object body, a file or an HTTP response, when the reader returned by
OpenObjectis closed, as gzip stores already did. Closing only stopped the decoder, so a reader closed before the end of the object kept its HTTP stream or connection open until its context was canceled, never when opened withcontext.Background(), and local stores kept the file descriptor until garbage collection. The body is also closed when the zstd reader cannot be created. -
NewStorereturns a nilStoreon error, instead of a nil store pointer wrapped in a non-nil interface. -
ObjectURLkeeps the query of the store URL after the object path (gs://bucket/path/file.dbin.zst?project=p) instead of appending the path to the query. -
S3 and Azure stores:
ListFoldersFromTonow stops paging as soon as the exclusive upper bound is reached, instead of listing the rest of the prefix to discard it.S3Store.WalkFromTodoes the same, and compares the bound against the full name rather than a prefix-stripped one, which made it yield keys past the bound wheneverprefixwas non-empty. -
S3 store:
WalkFromTostarts the server-side scan at the key right before the starting point (helloworld.htmlis walked fromhelloworld.htmk) instead of the starting point with its last character cut off (helloworld.htm), which had the service send every key in between for the walk to filter out. On fixed-width keys such as block numbers that window held every key sharing the truncated stem; it now holds none. A one-character bound used to cut off to the prefix itself and was skipped altogether, so it pushed nothing down at all. -
S3 store:
ListFoldersFromTopushes its lower bound down whole, minus the trailing/that makes it exclusive, where it used to drop the last character of the bound and skip it altogether when a single character was left. Listingchain/fromchain/b/now starts the server-side scan atchain/binstead of walkingchain/a*and filtering it out. -
GS store: the exclusive end point of
WalkFromTokeeps its trailing/in the listing's end offset.filepath.Joinate it, so walking up tosome/folder/skipped the object namedsome/folder, which sorts before the bound. -
S3 store:
WriteObjectnow drains the input reader when skipping a write because the destination already exists andoverwriteis disabled. Previously the reader was left untouched, which would deadlock pipe-based producers (e.g. a goroutine writing to anio.Pipe) and leak both the goroutine and any memory it had captured. -
S3 store: suppress checksum validation warnings from the SDK by setting
DisableLogOutputChecksumValidationSkippedtotrue. -
S3 store: only compute request checksums when required (
RequestChecksumCalculation = WhenRequired). Sinceservice/s3v1.73.0 the SDK computes CRC32 checksums on PUT/multipart uploads by default, which breaks non-AWS S3-compatible backends that reject the parts. This removes the need for theAWS_REQUEST_CHECKSUM_CALCULATION=when_requiredenv var workaround. See aws/aws-sdk-go-v2#2960.
- GCS store: disable gRPC DirectPath when both a project is set and
client_protocol=grpcis used, preventing connection issues in that configuration. - S3 store: share a single HTTP transport across all store clones for proper HTTP/2 multiplexing, replacing the previous per-clone transport that broke connection sharing.
- GCS store: opt-in gRPC transport via
client_protocol=grpcquery parameter (e.g.gs://bucket/path?client_protocol=grpc). Defaults to the existing HTTP client; the gRPC client is selected only when this parameter is explicitly set. - S3 store:
storage_classquery parameter as the canonical snake_case name forstorageClass.
- S3 store: each store clone now gets its own transport for failure isolation; adds
ResponseHeaderTimeoutto prevent hung requests and configures HTTP/2 health checks viax/net/http2; default connection pool sizes are reduced.
- S3 store:
storageClassquery parameter is deprecated in favour ofstorage_class; a warning is logged when the old form is used.
- S3 store: fixed connection pool leak when closing an object after a partial read; the raw HTTP body must be drained before closing the outer reader chain — closing outer first (when there is no compression layer) already closes the body, making the subsequent drain a no-op and preventing connection reuse
- Added
docker-compose.ymlwith MinIO, Ceph RGW, and fake-gcs-server for local integration testing - Added S3 integration tests for Ceph RGW (
TestS3Store_Ceph,TestS3Store_Ceph_EmptyBucket_FilePrefix,TestS3Store_Ceph_CompressionAndMetering) - Added GCS integration tests for fake-gcs-server emulator (
TestGSStore_Emulator,TestGSStore_Emulator_Overwrite,TestGSStore_Emulator_CompressionAndMetering) - S3 store: configurable HTTP connection pool via
DSTORE_S3_MAX_IDLE_CONNS,DSTORE_S3_MAX_IDLE_CONNS_PER_HOST,DSTORE_S3_IDLE_CONN_TIMEOUTenv vars
- S3 store: fixed goroutine leak caused by connection pool exhaustion on single-host S3 stores (e.g. MinIO); HTTP body is now explicitly drained and closed, and the transport is configured with
MaxIdleConnsPerHost=100by default - GCS store now uses the JSON API for object reads when
STORAGE_EMULATOR_HOSTis set, fixing compatibility with fake-gcs-server (which does not handle the XML API with percent-encoded path slashes)
- Migrated from AWS SDK for Go v1 to v2 (
github.com/aws/aws-sdk-go→github.com/aws/aws-sdk-go-v2)- Public API remains backward compatible for dstore users
- Added support for "workload identity credentials" in Azure. Order of preference is:
- If
AZURE_STORAGE_KEYis set, use shared key credential (previous behavior)- Otherwise, use DefaultAzureCredential which supports:
- Managed Identity (for Azure resources)
- Service Principal (via AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID)
- Azure CLI credentials
- Visual Studio Code credentials
- Otherwise, use DefaultAzureCredential which supports:
- If
- Legacy azure library changed from
github.com/Azure/azure-storage-blob-gotogithub.com/Azure/azure-sdk-for-go/sdk
- Fixed
WalkFromonS3andGCSwhen bothPrefixandStartingPointwas provided.
-
Added
SetMetadatato interface, which you can now get back from ObjectAttributes. -
Added
storageClassquery parameter for the s3 store to define a storage class on upload. -
Added Clonable interface so you can call
dstore.Clone(ctx)on a remote store, instantiate a new network client and context. -
Added
dstore.ReadObjectto easily read a single file from afileURL. -
Added
dstore.NewStoreFromFileURLto replacedstore.NewStoreFromURLwhich was not clear that it's usage is meant to create a store from a file directly. -
Added
dstore.OpenObjectthat is able to open a single store element without having to create a separate store, this is a shortcut for splitting the path & filename, creating a new store from the path and then callingstore.OpenObject. -
Added
Store::BaseURL()to retrieve the underlying URL of the store.
-
Improved 'Walk' speed on gstore by 25% by only fetching 'Name'
-
Store
dstore.MockStorenow opened up public access toFilesto easily get all written content. -
BREAKING:
MockStore'sSubStoremethod has changed behavior. It now removes the prefix from files already present, to conform to the behavior of other stores. This might affect your use ofMockStore::SetFile()in tests. -
The
Walk()andListFiles()methods does not have anignoreSuffixparameter anymore. This is managed internally by the LocalStore which was the only one that needed it, when writing temporary files (and renaming afterwards). Simplifies it for everyone else. -
The
dstore.NewLocalStore(local store implementation) sanitize the input if it does not start withfile://. -
BREAKING: The
NewLocalStorenow takes a*url.URLobject instead of astring. Just pass a&url.URL{Scheme: "file", Path: originalString}to fix your code, if you're usingNewLocalStoredirectly and not the recommendedNewStore.
- Deprecated The method
dstore.NewStoreFromURLis deprecated, usedstore.NewStoreFromFileURLwhich is clearer in semantics.