fix(studio-server): remove the history of a project that no longer exists - #4634
Conversation
jrusso1020
left a comment
There was a problem hiding this comment.
Review at 126700c7a3d9addd70f2a8c1eb972f8a7234b018.
Verdict: REQUEST_CHANGES
One blocker: a project on a drive that is simply not plugged in counts as gone, so its history is deleted once it has been idle 14 days. I reproduced it. Everything else about what gets deleted holds up well, and the fix is small.
What it does
openProjectHistorynow startspruneGoneProjectHistoriesDailyin the background (setImmediate) once the open succeeds. A.last-prunestamp in the history root skips runs for 24 h, and it is written only after a run finishes.- The CLI's direct owner passes
pruneGoneProjectsBudgetMs: 1000. The Studio and Vite hosts pass nothing, so they have no budget. - A history
<root>/<uuid>is removed when all of these hold:- its
project.jsonhas a stringdir; - it has been idle 14 days (the newer mtime of
project.jsonorlog.jsonl), ordiris underos.tmpdir(); dir/.hyperframes/history-idis missing, or holds a different valid id;- the owner lock can be taken with no wait.
- its
- The whole check runs again under the lock. Then the folder is renamed to
.pruned-<id>-<uuid>and removed withrm -rf. Leftover.pruned-*folders are finished on the next run.
Safety of what gets deleted
How "gone" is decided (blocker)
abandonedProject(pruneHistories.ts:139) treats a missing id file as gone. It never asks whether the path is missing because the project was deleted or because the disk holding it is not attached./Volumes/EXT/MyFilmon an unplugged drive,E:\MyFilmwith no E:, or an offline share all give ENOENT, so all of them read as gone.- The only guard is the 14-day idle window, and that counts from the last open, not from when the drive went away. A project last opened three weeks ago on a drive that is unplugged today loses its history the next time any other project is opened.
- Repro (throwaway root, probe test run against this head): I opened a project under a "volume" folder, closed it, and renamed the volume away to simulate unplugging it. The day-0 prune removed nothing. The prune at day 14 + 1 ms removed it (
history exists: false). After I renamed the volume back, the project and its id file were intact, but the history was gone for good. - Video projects with their media on an external SSD are a common setup, so this is more than a corner case. The PR's "what a user can lose" section covers moved projects but not this one.
- Suggested fix, smallest first:
- Count a project as gone only when the recorded folder's parent still exists. That covers
/Volumes/X, Windows drive letters, and the/media/<user>/Xpaths udisks removes on unmount. - Sturdier: record
devnext toino/borninproject.json(whereFolderalready comparesdev). Then count the project as gone only when the nearest existing ancestor ofdiris on that same device. An empty Linux mount point would then read as "can't tell" and be kept. - Either way, add a test that removes the project's parent folder and checks the history is kept.
- Count a project as gone only when the recorded folder's parent still exists. That covers
- Error handling is right.
statSync(..., { throwIfNoEntry: false })only suppresses ENOENT (and ENOTDIR on Node). EACCES, EIO and ESTALE throw, go toonError, and that history is skipped. Treating EACCES as gone fails a test (see below). One difference: Bun throws on ENOTDIR where Node returns undefined, which only matters in the safe direction. - An id file that exists but cannot be read is kept (
readId→ null → keep). Tested. - Case-insensitive disks:
staton the recorded path still resolves after a rename that only changes letter case, so the project is kept.
Temp folder
- The check is
isSafePath(os.tmpdir(), dir), which realpaths the base and the nearest existing ancestor. So/var/foldersvs/private/var/folderson macOS resolves correctly, and so doesTMPDIR. - The temp rule only skips the 14-day window. A temp project is still removed only when its id file is gone or replaced, so a live agent project under temp that exists on disk is never touched.
- Edge, not blocking:
diris stored withresolve(), not the real path. A project opened through a symlink that sits under temp, with that link later removed, loses its history immediately even though the real folder is intact. Reproduced (P2 pruned at day 0: true | real project still has id: true). This is unlikely in practice. Storing the real path would close it. - A project scaffolded under temp and then moved out loses its history at the next run unless it is reopened first. The body says so.
Owner dead
- This reuses
takeHistoryOwnership(home, 0):process.kill(pid, 0), with EPERM counted as alive. A reused pid reads as alive, so the history is kept (safe side). - A corrupt
owner.pidreads as dead. That was already how the lock behaved before this PR. - Note: a pid from another pid namespace (a container sharing the cache folder) reads as dead. Paths inside the container also do not exist on the host. If a container keeps its projects under
/tmpand shares the host's cache, the host could remove a history the container has open right now. That setup is unusual, but a short line in the docs, or keying the lock on hostname plus boot id, would cover it.
What is removed (scope)
- Only direct children of the history root are touched: names that match the UUID shape, plus
.pruned-*.record.diris only stat'ed and read, never deleted, so a craftedproject.jsoncannot sendrmoutside the store. - A symlinked entry is renamed and removed as a link. Its target is never followed.
- Dropping the name filter fails a test.
Races
- The owner lock plus the recheck blocks the common race. But
projectHistoryIdwrites the newdirintoproject.jsonbeforetakeHistoryOwnership(projectHistory.ts:1081-1085). If a moved project is reopened after the prune's recheck and before its rename, the open writes into a new empty home and the old history goes to trash. The window is narrow, and it only applies to histories already past 14 days. Not blocking. - Two pruners racing: the loser's
takeHistoryOwnershiprunsmkdirSync(home, { recursive: true })after the winner has renamedhome. That leaves an empty UUID folder with noproject.json, which is never cleaned up. It is a tiny leak, not a data risk. - The in-process
pruningset only stops the same process from pruning twice. Other processes are guarded by the lock and by the rename happening in one step, and that holds.
The 1 s CLI budget
- The budget is checked only before each history starts. A history that is already being removed runs to the end, and the CLI's normal exit sets
process.exitCodeand waits for pending work, so a large history can keep a command alive past 1 s. - The prune also runs next to the command rather than after it, so a fast command can end up taking about 1 s on days a backlog is draining.
- If a hard exit cuts a removal short, the
.pruned-*leftover is finished on the next run. Tested. - All filesystem calls on the scan path are synchronous (
readdirSync,statSync,readFileSync). A hung network mount in some olddirwould block the Studio server's event loop during the daily run. Worth a note in the docs, or a move to asyncstat.
Timestamps
project.jsonis rewritten on every open, so "idle" means "not opened". A future mtime keeps the history (safe side). A clock that has jumped back failslast <= nowon the stamp, so the prune simply runs again.
Tests and mutations
pruneHistories.test.ts: 9/9 pass at this head. The wholesrc/historyfolder: 123 passed, 2 skipped.- I verified the claim "writes no stamp unless it finished" against the source (line 182). Writing the stamp regardless fails a test.
- I mutated one line at a time and ran the prune tests after each change:
| Mutation | Result |
|---|---|
| Ignore a live owner (busy lock treated as free) | caught |
| Drop the 14-day check | caught |
| 14 days → 1 day | survives |
| Treat EACCES / any stat error as gone | caught |
| Never treat anything as temp | caught (5 tests) |
| Treat everything as temp | caught |
| Unreadable id file counts as gone | caught |
| Drop the recheck under the lock | survives |
| Drop the id-shape name filter | caught |
| Write the stamp even when the budget ran out | caught |
Skip finishing .pruned-* leftovers |
caught |
| Dry run actually deletes | caught |
| Never delete a replaced folder's history | caught |
- Two gaps:
- The window length is not pinned. The test only checks day 0 and day 14. Add a day-13 case that keeps the history.
- Nothing tests the recheck, even though the body names it as the race guard. A test could change
project.jsonto a live folder between the first check and taking the lock (for example by holding the lock, rewriting the record, then releasing it) and assert the history is kept.
- I reran the probes for the two findings above (unplugged volume, symlink under temp) against this head. They are not in the PR.
Checks
- Check runs at this head: 43 success, 11 skipped, none failing or pending. The rollup shows 48 success and 12 skipped.
- No reviews yet.
reviewDecisionisREVIEW_REQUIREDand the merge state isBLOCKED. The repo requires an approval on the latest push.
Notes
- The design is careful. It confines deletion to UUID-named history folders, rechecks under the lock, renames before removing, counts a history it cannot check as kept, and writes the stamp only after a full run.
- The unplugged-drive case is the only one I would hold on. With the parent-folder (or device) check and a test for it, plus the two missing tests above, I would approve.
— Rames
f96d898 to
5f4cae0
Compare
5f4cae0 to
f00f8d9
Compare
jrusso1020
left a comment
There was a problem hiding this comment.
Re-review at f00f8d9361cb5e5ab672c73e079309a8f29bf167. This replaces my CHANGES_REQUESTED 5334441272 at 126700c7.
Verdict: REQUEST_CHANGES
The unplugged-drive fix is right for every history this version writes. Two things still hold it. Every history that exists today was written without dev, and the fallback rule for those still deletes a history whose disk is unmounted but whose mount point stays behind. That is the same data loss as my first blocker, on the histories people already have. And the two tests I asked for (window length, recheck) are still missing.
Prior blockers
1. A project on a disconnected drive counted as gone: fixed.
recordProject(historyId.ts:86-94) now storesdevnext toinoandborn. Every open rewritesproject.jsonfrom a freshstatSync, so a history picks updevthe first time its project is opened on this version.projectGone(pruneHistories.ts:143) callsdiskStillHere(pruneHistories.ts:152) when the id file is missing. That walks up to the nearest folder that still exists. Withdevrecorded, the project counts as gone only if that folder is on the same device (line 160).- How this plays out on each platform:
- macOS,
/Volumes/EXT/MyFilmunplugged:/Volumes/EXTis gone, the walk stops at/Volumeson the system disk, the device differs, and the history is kept. The same holds when the project is the volume root itself. - Linux, udisks removes
/media/<user>/X: the walk stops at/media/<user>on the root disk, so the history is kept. An fstab mount point that stays behind as an empty folder is on the root disk too, so the history is kept. - Windows, no
E::stat("E:\\")fails anddirname("E:\\") === "E:\\", so line 156 returns false and the history is kept. An offline UNC share walks up to\\server\share\and stops the same way. - Deleted on a disk that is still attached: the walk reaches a folder on the same device, so the history is removed. This includes the case where the user trashed the enclosing folder. With
devrecorded, a deleted parent no longer leaves the history behind forever.
- macOS,
- Errors are handled well.
throwIfNoEntry: falseonly covers ENOENT (and ENOTDIR on Node), so the walk steps over a missing path. On ENOTDIR (a parent that is now a file) the walk keeps going to a real ancestor, which is correct. EACCES, EIO and ESTALE throw from inside the walk, go toonError, and that history is skipped. Bun throws on ENOTDIR, which only errs toward keeping. - Tests: "keeps a gone project's history while the disk that held it is not here" covers the unplugged drive (parent gone, other device), the empty mount point (folder there, id gone, other device) and a tree deleted from the same disk. It is a good shape.
2. No test pins the 14-day window length or the recheck under the lock: not addressed.
- The delta adds no day-13 case and no recheck test. Both mutations still survive (table below).
- The code for both is right, but these are the two constants that decide when history is deleted, and nothing fails if either changes. Still a blocker. A day-13 case that keeps the history is one line. The recheck can be tested by holding the lock, making the project reappear, then releasing it and asserting the history is kept.
3. New: the fallback for histories written before this version. Details are in the first note below. Released versions write project.json without dev, so today every history is on the fallback until its project is reopened. On Linux, a data disk mounted by fstab at /mnt/data, with a project at /mnt/data/MyFilm last opened on an older version, is deleted after 14 idle days while the disk is unmounted. A project at a macOS volume root on an old record is hit the same way. Only pruning a fallback history when the project folder itself is there without its id (at === dir) closes it, at the cost of keeping deleted-project histories until they are reopened.
What else changed
- The delta is
historyId.ts,pruneHistories.tsand two tests. Nothing else moved. - The PR body now describes the disk check and the fallback for older histories. I checked it against the code and it matches.
Non-blocking notes
- Histories written before this version use a weaker rule (blocker 3). Released versions write
project.jsonwithoutdev, so every history that exists today is on the fallback until its project is opened again. The fallback (line 161) counts the project as gone when the nearest existing folder is the project folder itself or its parent. That keeps the common cases: macOS/Volumes/EXT/...with the project one or more levels down, udisks mounts, Windows drive letters. It misses a mount point that stays behind:- Scenario: on Linux, a data disk mounted by fstab at
/mnt/data, project at/mnt/data/MyFilm, last opened on an older version. The disk is unmounted and the history has been idle 14 days./mnt/datais still there as an empty folder, it is the parent, and the history is removed. - The same applies when a project sits directly at a volume root on macOS (
/Volumes/MyFilm, where/Volumesis the parent). - The test "keeps a history recorded without its disk once the project's parent is gone too" expects
parentKeptto be pruned. That is the same on-disk state as an unmounted fstab disk. - A
devcomparison cannot catch this, because an empty mount point sits on the root disk like any other folder. The simplest fix is to only prune a fallback history when the project folder itself is there without its id (at === dir), and keep it otherwise until the project is reopened and gets adev. It still misses a project that is itself the mount point, and it leaks on the safe side for deleted projects. - The pool of fallback histories only shrinks, but today it is every history, and a wrong prune deletes history that cannot be recovered. That is why this is now blocker 3.
- Scenario: on Linux, a data disk mounted by fstab at
- Fallback histories whose parent was trashed are never cleaned unless they sit under temp. That leaks on the safe side and the second new test pins it. That is fine by me.
- Device numbers are not always stable. On Linux, the same USB disk can come back as
sdc1instead ofsdb1. If a project on it is then deleted, the ancestor'sdevno longer matches and the history is kept until it is reopened, which may never happen. That leaks on the safe side too. The unsafe direction needs a different disk that gets the same device number and the same mount point, with the project missing from it. That is a swapped drive with the same label in the same port, which is rare. - Branches with no test. These mutations survive: dropping the temp rule inside the fallback (line 161), dropping
at === dirin the fallback, and returning true when the walk reaches the root (line 156). The root case is the Windows missing-drive path, and a test would need to walk up to/. A record whosediris under a path that does not exist from the root down would do it on POSIX. - Unchanged from last time and still not blocking: the reopen race around
projectHistoryIdwritingproject.jsonbefore the lock, the symlink-under-temp edge, synchronousstaton a hung network mount, and the 1 s budget not bounding a removal in progress.
Tests run
Worktree at f00f8d93, throwaway HOME, core, parsers, lint and studio-server built first.
studio-server:tsc --noEmitclean.vitest run: 864 passed, 2 skipped.src/history: 125 passed, 2 skipped.pruneHistories.test.ts: 11/11.cli:src/commands/history.test.ts32/32.- Mutations, one at a time, on
pruneHistories.test.ts:
| Mutation | Result |
|---|---|
| Treat a missing id file as gone again (the old behavior) | caught (2 tests) |
diskStillHere always true |
caught (2 tests) |
Ignore dev, fallback rule only |
caught |
| Fallback always says gone | caught |
| Fallback drops the parent rule | caught |
| Fallback drops the temp rule | survives |
Fallback drops at === dir |
survives |
| Reaching the root counts as gone | survives |
| 14 days → 1 day | survives |
| Drop the recheck under the lock | survives |
Checks
- Check runs at this head: 83 success, 22 skipped, none failing or pending.
- Reviews: only my CHANGES_REQUESTED at
126700c7. This review replaces it.
— Rames
834a1a7 to
947d93d
Compare
947d93d to
276ac45
Compare
|
Addressed at 276ac45. Window length and the check under the lock are now pinned.
Histories written before One exception is kept on purpose: an older history under the system temp folder is still removed once its project is gone. Removable disks are not mounted under the temp folder, and those scratch histories are the bulk of the leak this change is for; without the exception none of them would ever be cleaned. If the temp folder itself is missing (for example Mutations now caught (each fails a test): window shortened to 1 day, no re-check under the lock, fallback without the temp rule, fallback without the emptied-folder rule, fallback with the parent rule restored. Not tested, with reason: "reaching the filesystem root counts as gone" can only happen on Windows with a missing drive letter; on macOS and Linux The body's "What a user can lose" section now says that an older history of a project deleted outside the temp folder stays until the project is opened again. The other notes (reopen race around recording before the lock, symlink under temp, synchronous stat on a hung mount, the 1 s budget not bounding a removal in progress) are unchanged and remain non-blocking as you rated them. |
jrusso1020
left a comment
There was a problem hiding this comment.
Re-review at 276ac45b888cfe86b1bb47f42f8edd3aaf9e75b4. This replaces my CHANGES_REQUESTED 5334763191 at f00f8d93.
Verdict: APPROVE
All three blockers are closed. The fallback for older histories now only prunes an emptied project folder or a project under temp, which keeps the unmounted-disk cases. The window length and the recheck under the lock each have a test now, and both mutations that survived last time are caught.
Prior blockers
1. A project on a disconnected drive counted as gone: fixed (unchanged since last round).
- The
devpath indiskStillHere(pruneHistories.ts:152-160) did not change. Mutations that ignoredevor always say the disk is here still fail tests.
2. No test pins the 14-day window or the recheck under the lock: fixed.
- Window: the 14-day test now checks day 13 keeps the history and day 14 removes it (pruneHistories.test.ts:101-106). It passes
tempDir: historyRoot, so the temp rule does not skip the window. "14 days → 1 day", "→ 13 days" and "→ 15 days" all fail that test. - Recheck: "keeps a history whose project comes back while the prune takes its lock" wraps
takeHistoryOwnershiponce. The wrapper puts the id file back, then takes the real lock. The first check has already said gone, so only the recheck at line 125 can keep it. Replacing the recheck with a plain record read fails this test. - The mock is set after
projectWithHistoryreturns, and that helper waits out the open's own daily prune, so nothing else uses up the one-time mock. I ran the file 15 times and it passed every time.
3. The fallback for histories recorded without dev: fixed.
- Line 162 is now
return at === dir || isSafePath(tempDir, at);. The parent rule is gone, which is the fix I suggested. - Linux fstab disk unmounted, empty
/mnt/dataleft, project at/mnt/data/MyFilm: the walk stops at/mnt/data. That is not the project folder and not under temp, so the history is kept. - macOS project at a volume root,
/Volumes/MyFilmunplugged: macOS removes the mount point, so the walk stops at/Volumes. Kept. The same holds with the project deeper on the volume. - Windows, no
E::stat("E:\\")fails anddirname("E:\\") === "E:\\", so line 156 returns false before the fallback runs. Kept. An offline UNC share stops at the share root the same way. - Deleted project under temp: the walk reaches a folder under
tmpdir()(ortmpdir()itself),isSafePathpasses, and the history is removed. The new test "prunes a history recorded without its disk under the temp folder once its project is gone" pins this, with the whole parent removed so the walk ends attmpdir(). - The reworked fallback test covers three records without
dev: an emptied project folder (pruned), a missing project folder under a folder that is still there (kept, the mount-point shape), and a missing parent (kept).
What else changed
- The delta is one line in
pruneHistories.tsplus a comment, andpruneHistories.test.ts: the day-13 check, the reworked fallback test, a temp fallback test, and the recheck test with avi.mockofownerLock.jsthat wraps the real function. - The PR body now describes the older-history rule, and says an older history of a project deleted outside temp stays until the project is opened again. That matches the code.
Non-blocking notes
- Older histories of projects deleted outside temp now stay. A project deleted on an older version is never opened again, so its history is never cleaned. This is the tradeoff I asked for. The body says so, and it only affects histories written before this version.
- A project that is itself the mount point, on an older record, can still be pruned. If
/mnt/datais both the mount point and the project, an unmount leaves/mnt/datawith no id in it, soat === dirholds. This is rare and was already noted last round. Opening the project once on this version recordsdevand closes it for that project. - Reaching the root still has no test. Changing line 156 to
return truestill passes all 13 tests. This is the Windows missing-drive path. A record whosediris under a path that does not exist from the root down would cover it on POSIX. - The PR body says the studio-server suite has 864 tests. It is 866 at this head.
- Unchanged from last time and still not blocking: the reopen race around
projectHistoryIdwritingproject.jsonbefore the lock, the symlink-under-temp edge, synchronousstaton a hung network mount, the 1 s budget not bounding a removal in progress, and device numbers that change when a disk is replugged (leaks on the safe side).
Tests run
Worktree at 276ac45b, throwaway HOME, core, parsers, lint and studio-server built first.
studio-server:tsc --noEmitclean.vitest run: 866 passed, 2 skipped.src/history: 127 passed, 2 skipped.pruneHistories.test.ts: 13/13, and 15/15 on repeat runs.cli:src/commands/history.test.ts32/32.- Mutations, one at a time, on
pruneHistories.test.ts:
| Mutation | Result |
|---|---|
| 14 days → 1 day | caught |
| 14 days → 13 days | caught |
| 14 days → 15 days | caught |
| Drop the recheck under the lock | caught |
| Fallback drops the temp rule | caught |
Fallback drops at === dir |
caught |
| Fallback puts the parent rule back | caught |
| Fallback always says gone | caught |
Ignore dev, fallback rule only |
caught (2 tests) |
| Reaching the root counts as gone | survives |
Checks
- Required checks at this head: Semantic PR title, Test: runtime contract, Typecheck, Build, regression, Test, Render on windows-latest, Tests on windows-latest, Studio and player captures and Test reachability all have a successful run.
- A second Windows run on the same head (Render on windows-latest and the four Tests on windows-latest shards) was still in progress when I checked. The first run of each passed.
- Overall: 77 success, 22 skipped, 5 in progress, none failing.
- Reviews: only my two CHANGES_REQUESTED, at
126700c7andf00f8d93. This review replaces them.
— Rames
What this fixes
Each project's undo history lives in its own folder under the history cache (
~/.cache/hyperframes/historyby default), and nothing ever removed one. Deleting a project left its history behind, blobs and all, so the cache only grew, by gigabytes a day on a machine that creates many short-lived projects.Now opening a history also clears out the histories of projects that are gone, in the background, so an open never waits on it. Once a run finishes, a stamp in the history cache holds off the next one for a day, across all processes. A one-off CLI command (such as an agent's
hyperframes historycall) spends at most about one second on it and writes no stamp unless it finished, so a large backlog drains over a few commands and then settles into once a day. A long-running host (the Studio server, the Vite dev server) runs it to the end. A history is removed when all of these hold:The check runs again under the history's own lock before anything is deleted, so an open that races the cleanup keeps its history. The history is first renamed out of the way in one step, so a cleanup interrupted mid-delete leaves a marked folder the next run finishes, never a half-deleted history. A history that cannot be checked (for example a permission error) is reported and skipped; the rest are still cleaned. Only folders named like a history are ever considered.
pruneGoneProjectHistoriesis exported with a dry run that reports what would go, with each project's folder and size, for acleancommand to build on.What a user can lose
History follows a project when it is renamed or moved, but the new location is only learned when the project is opened again. So a project that is moved and then not opened for 14 days loses its undo history; one under the temp folder loses it at the next cleanup. A history whose project id file cannot be read is always kept.
A project on a drive or share that is not connected is not counted as gone. History notes which disk each project is on; when the project is missing, it looks at the nearest folder that still exists (the project folder itself, or one above it), and only if that is on the same disk does the project count as deleted. An unplugged external drive, a missing drive letter, an offline network share, or an empty mount point left where a drive was mounted leaves the history alone until the drive is back. Histories recorded before this change carry no disk note, and an empty mount point looks just like a deleted project's parent. For those, the project counts as deleted only when its own folder is still there without the project in it, or when it was under the temp folder; otherwise the history is kept until the project is opened again (which records its disk). So an older history of a project deleted outside the temp folder stays until then, erring on the side of keeping it.
Tests
history/pruneHistories.test.ts:Results: studio-server and CLI
tsc, whole-treeoxfmt --checkandbun run lint, and fallow are clean. The whole studio-server suite passes (864 tests). On main the new test file fails (the cleanup does not exist). Each safeguard removed on its own fails a test: the daily trigger, the temp-folder rule, finishing interrupted removals, reporting one history's error without stopping, releasing a history whose move failed, writing the daily stamp only after a full run, the disk check (including the mount-point case), the 14-day window, the re-check under the lock, and the rule for histories recorded without a disk.