Skip to content

Recover from R2 lock errors when mirroring releases - #1439

Draft
zsol wants to merge 1 commit into
mainfrom
zsol/mirror-create-or-verify
Draft

zsol wants to merge 1 commit into
mainfrom
zsol/mirror-create-or-verify

Conversation

@zsol

@zsol zsol commented Nov 13, 2026

Copy link
Copy Markdown
Member

The PBS mirror skips keys it finds in its initial listing. If an upload commits after that listing but its response is lost, boto3 can retry the write and hit the bucket lock. PBS can then report a failure even though R2 already holds the expected bytes. #1433 · R2 bucket locks

The ty 0.0.102 mirror hit bucket-lock errors on several artifacts when it reran after a multipart completion failure. That prompted a check of how the PBS uploader handles the same lock response. ty mirror rerun

PBS keeps boto3’s upload_file path. If PutObject or CompleteMultipartUpload returns ObjectLockedByBucketPolicy, PBS reads the exact key and accepts it only if its SHA-256 matches the staged SHA256SUMS. A missing object, failed read, or mismatch fails the upload. If the final SHA256SUMS upload hits the same lock error, PBS hashes the local manifest itself for comparison.

In a local HTTP test, the server stored the first PUT and closed the connection before responding. Botocore retried, received the lock error, and PBS fetched the same key and verified its digest.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant