atelet: add filecache singleflight retrieval (GetFileTo) - #1513
Closed
Dmitry Berkovich (dberkov) wants to merge 2 commits into
Closed
atelet: add filecache singleflight retrieval (GetFileTo)#1513Dmitry Berkovich (dberkov) wants to merge 2 commits into
Dmitry Berkovich (dberkov) wants to merge 2 commits into
Conversation
Introduce cmd/atelet/internal/filecache, the foundation of a node-local artifact cache: opaque entry keys (content-addressed sha256 and immutable-URI forms), the entries/tmp on-disk layout, a startup sweep for crash debris (unfinished fetches, interrupted evictions), and byte accounting for a GC budget. Golden snapshot restores download their files per actor with no reuse, and sandbox-asset fetches race concurrent downloads of the same asset; this package is the shared cache that will back both paths. Retrieval (singleflight fetch, atomic publication, hardlink-out) and eviction build on this skeleton in follow-up changes.
GetFileTo materializes a cached artifact at a destination path via hard link, fetching it on a miss. Concurrent callers for one key share a single fetch (singleflight), and the fetch runs detached from the callers' contexts bounded by the store's fetch timeout, so one canceled caller never aborts a download other callers are waiting on. There is no negative caching: a failed fetch reaches every waiting caller and the next call starts fresh. A fetch lands in tmp/, must produce a regular file, is made read-only (0444) so a consumer's in-place write fails loudly instead of corrupting the shared copy, and is published with one atomic rename. A hit links out and touches the entry's last-use clock under a shared lock that eviction will hold exclusively, closing the hit-vs-evict window. Destinations must not exist and must be absolute paths on the cache's mount; cross-filesystem destinations fail with a dedicated error rather than a silent copy.
Collaborator
Author
|
Closing in favor of a single consolidated PR carrying the whole M1 library as reviewable commits — the fork-based stacking made per-PR diffs and CI awkward. Replacement PR link to follow. |
Collaborator
Author
|
Consolidated into #1517. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second slice of the node-local artifact cache (#690), stacked on #1512 — fork branches can't serve as PR bases, so this targets
mainand includes the skeleton commit underneath; review only the top commit (atelet: add filecache singleflight retrieval). I'll rebase after #1512 merges, which collapses the diff to just this change.Adds the retrieval path the golden-snapshot Restore integration will call:
FileFetchertakes a destination path (not anio.Writer) soategcs.FetchLocalFileFromGCSWithZstd's sparse-aware output plugs in unchanged; fetcher errors are wrapped with%wsoateerrors.CrashIfReasonclassification sees through the store.GetFileTo(ctx, key, dst, fetch)— atomic get-and-link. Miss: singleflight per key, flight oncontext.WithoutCancel+ fetch timeout (waiters select on their ownctx.Done()), fetch intotmp/, validate, chmod0444, publish via one atomic rename. Hit: hard link todst+ last-use mtime touch underhitMu.RLock(eviction will take the exclusive side). No negative caching.dston the cache's mount (same mount, not just same disk — bind mounts faillink(2)withEXDEV), non-nil fetcher.EXDEVandEEXISTget dedicated errors.BasePathtoday; if a future layout splits them (e.g. a cache-only local-SSD mount), a per-store copy mode is a backward-compatibleOption.Tests: 16 concurrent callers → one fetch; canceled waiter returns while the detached flight completes and warms the cache; fetch failure reaches all waiters with zero debris and no published entry, next call refetches; one-inode verification across destinations and cache copy;
0444mutation tripwire; argument rejection; hit refreshes the LRU clock.🤖 Generated with Claude Code