fix CI log capture and races - #462
Merged
mattjala merged 2 commits intoSep 10, 2026
Merged
Conversation
hsds_app.py - the sub-process reader thread used a bytes sentinel on a text-mode pipe, so it never stopped at EOF. The drain loop then never broke and never closed the buffered logfile, so hs.log lost everything written after the first node exit. This PR fixes the sentinel, has the log close from a finally block, and drops the p.communicate() that can race the reader thread for p.stdout. fileClient.py - put_object stat'd the file it had just written to, so a concurrent delete could turn a successful write into a 500 spuriously. This demotes that stat to a warning and falls back to synthesizing the response, the same approach s3Client already takes. It also stops the bare `except Exception` from relabelling HTTP statuses as internal errors, including the path-outside-root 400 raised via _mkdir. datanode_lib.py, async_lib.py - a deleted root stayed queued in root_scan_ids, so bucketScan could write .info.json and .summary.json back into a prefix which bucketGC had just swept. This purges the root on delete, and has scanRoot bail out if it has been deleted since the scan was queued.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The digest is an HTTP ETag, and md5 is the S3 convention - a single-part S3 upload's ETag is the md5 of the object bytes, and s3Client returns that value straight from AWS - so fileClient computes the same thing to keep the backends consistent. Those ETags also feed the per-object checksums that scanRoot folds into a domain's md5_sum, so the algorithm cannot change without the POSIX backend diverging from S3 for identical data. usedforsecurity=False records that intent in the code. It leaves the digest unchanged, and keeps this working under FIPS, where an unqualified hashlib.md5() raises. The domain checksum in async_lib.py and the key prefix in nodeUtil.py are the same kind of use, and are left alone here.
mattjala
force-pushed
the
fix/ci-flake-log-capture-and-gc-race
branch
from
September 9, 2026 21:04
59ea41a to
acf0efe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hsds_app.py - the sub-process reader thread used a bytes sentinel on a
text-mode pipe, so it never stopped at EOF. The drain loop then never
broke and never closed the buffered logfile, so hs.log lost everything
written after the first node exit. This PR fixes the sentinel, has the
log close from a finally block, and drops the p.communicate() that can
race the reader thread for p.stdout.
fileClient.py - put_object stat'd the file it had just written to, so a
concurrent delete could turn a successful write into a 500 spuriously.
This demotes that stat to a warning and falls back to synthesizing the
response, the same approach s3Client already takes. It also stops the
bare
except Exceptionfrom relabelling HTTP statuses as internalerrors, including the path-outside-root 400 raised via _mkdir.
datanode_lib.py, async_lib.py - a deleted root stayed queued in
root_scan_ids, so bucketScan could write .info.json and .summary.json
back into a prefix which bucketGC had just swept. This purges the root
on delete, and has scanRoot bail out if it has been deleted since the
scan was queued.