libs/filer: tolerate already-vanished directory during recursive delete#5958
Merged
Conversation
On GCS-backed UC Volumes (and potentially S3), a directory created implicitly
via CreateParentDirectories is only a key prefix with no standalone object, so
it disappears when its last child file is deleted. recursiveDelete deletes all
files first and then the directory objects, so on GCS the directory-delete
returns HTTP 404 and fails the whole delete even though its goal (directory and
contents gone) is already satisfied.
Map that 404 to a not-found error in deleteDirectory via apierr.IsMissing (the
SDK maps any 404 to apierr.ErrNotFound; a 404 on the directory-delete endpoint
can only mean the directory is already gone) and skip not-found directories in
recursiveDelete's reverse-order loop. The tolerance is correct on every
backend: on AWS/Azure the directory-delete still returns 204, so the tolerant
branch is never taken.
Observed on the cli-gcp env — the directory delete that fails during recursive
delete returns a 404 with a specific reason:
DELETE /api/2.0/fs/directories/Volumes/main/<schema>/my-volume/dir/subdir2
< HTTP/2.0 404 Not Found
< {
< "details": [{ "@type": ".../ErrorInfo",
< "domain": "filesystem.databricks.com",
< "reason": "FILES_API_DIRECTORY_IS_NOT_FOUND" }],
< "error_code": "NOT_FOUND",
< "message": "The directory being accessed is not found."
< }
A direct probe (write two files into a dir, delete both, then HEAD/DELETE the
dir) isolates the backend difference:
Step GCP (cli-gcp) AWS (cli-aws)
HEAD dir after writing its files 200 EXISTS 200 EXISTS
HEAD dir after deleting its files 404 NOT_FOUND 200 EXISTS
DELETE dir 404 NOT_FOUND 204 Success
On GCS the implicitly-created directory is a key prefix with no standalone
object, so it vanishes once its last child is deleted; on AWS the directory
persists as an independently-deletable object. All file deletes succeed (204)
before the failing directory delete, so data removal is complete — only the
trailing directory-object delete errored.
Cross-verified against live cli-gcp and cli-aws envs: TestFilerReadWrite/files
and TestFilerRecursiveDelete/files now pass on GCP (previously failed) and stay
green on AWS.
Co-authored-by: Isaac
Collaborator
Integration test reportCommit: 0976ab2
12 interesting tests: 5 flaky, 4 SKIP, 3 RECOVERED
Top 10 slowest tests (at least 2 minutes):
|
janniklasrose
approved these changes
Jul 17, 2026
Co-authored-by: Jan N Rose <janniklas.rose@gmail.com>
pietern
enabled auto-merge
July 17, 2026 13:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On GCS-backed UC Volumes, a directory created implicitly via CreateParentDirectories has no standalone object, so it disappears when its last child file is deleted.
recursiveDeletedeletes files first and directories after, so on GCS the directory-delete returns 404 and fails the whole operation even though its goal (directory and contents gone) is already met. This surfaced asdatabricks fs rm -rfailing on GCS volumes.Map that 404 to a not-found error via
apierr.IsMissingand skip not-found directories inrecursiveDelete. Correct on every backend: on AWS/Azure the directory-delete still returns 204, so the tolerant branch is never taken.Cross-verified on live test environments:
TestFilerReadWrite/filesandTestFilerRecursiveDelete/filesnow pass on GCP (previously failed) and stay green on AWS.This pull request and its description were written by Isaac.