Repository navigation
Conversation
ad970b5 to
9351b5c
Compare
KnifeLemon
left a comment
There was a problem hiding this comment.
Thanks, @ltsammy, this is a big piece of work. The shared SourceBytes reader, the frame, track and ledge-tree caps and the non-recursive tree walk are what we want. I read the new commit on top of #1609. CI's one red job (WebDropJobTests on Linux x64) is a Core test for code this PR doesn't touch; I re-ran it.
Every count has its own cap, but as in #1609, many records can point at the same bytes, so the caps multiply. A .mdl of a few KB can still ask for terabytes of output or all the memory. A decompile needs one budget for the whole model. Could you add one, counted as things are read or written, that refuses with ConversionRefusedException("error.document_too_large", "…") (as OdsReader line 345 and Ebooks/Book.cs line 98 do) when it's exceeded? These are the places:
1. Animations (the biggest one)
src/Filee.Engines/Models3D/SourceAnimation.cs, Decode (lines 51–89) keeps every frame of every bone in a List<Pose[]> before anything is written.
- An animation needs no data at all to be long: with block 0 and index 0,
Locatereturns −1 and every frame is the bind pose. - So one 100-byte animation record with 100,000 frames on a 1,024-bone skeleton is about 2.4 GB of poses in memory and about 6 GB of SMD text.
ReadAnimations(SourceModelDetails.cs) allows 16,384 such records.
Please cap the bone-frames (frames × bones) across all animations; tens of millions is far above any real model. If you can, also write each frame as it is decoded instead of holding them all.
2. Triangles over every body group model and LOD
The point from #1609 applies here across the whole decompile. SourceModelDetails.ReadBodyParts (lines 164–192) calls SourceModel.ReadLodMeshes (SourceModel.cs, from line 240) for up to 512 parts × 512 models × 8 LODs. Strip groups and strips inside each LOD can share one index table. One triangle budget for all of them covers it.
3. Physics
src/Filee.Engines/Models3D/SourcePhysics.cs, ReadSolid (lines 69–96): every leaf of the ledge tree can point at the same ledge (node + I32(node + 4)). So each of the 1,024 solids can hold about 32,000 copies of a piece with up to 32,767 triangles. A cap on the triangles (or points) of the whole .phy covers it.
4. QC entries
These end up in memory and in the QC's StringBuilder:
ReadSequences(SourceModelDetails.cs:267): 16,384 sequences × 4,096 events.ReadHitboxes(SourceModelDetails.cs:212): 4,096 sets × 16,384 hitboxes.
Each is up to 67 million entries, and their tables can overlap too. Caps on the totals (all events, all hitboxes) would close this.
5. Small: Windows device names
SourceDecompiler.SafeName (SourceDecompiler.cs:472) keeps a name such as con, nul, aux or com1. Windows treats those as devices, and on Windows 10 even with an extension, so an animation or body model with that name can't be written as nul.smd and the like. Prefixing them with _, as ArchiveSafety.SanitizeSegment does, avoids it.
6. Reference credits in comments
As in #1609, please keep comments to the format and to compatibility ("as studiomdl reads it", "the layout Crowbar writes" is fine). Drop where the layout was learned, for example SourcePhysics.cs:3 ("and SourceIO's reader").
Main has moved
#1585 took schema step 95 on main, so I merged main into #1591 and #1608 and moved their steps to 96 and 97. The VTF preset (#1611) gets 98. SMD and "QC + SMD" take the next two free numbers when #1609 and this PR land. If the renumbering gets in your way, push your fixes as they are and we'll do it before merging.
A Source engine model (versions 44 to 49, with the .vvd and .dx90.vtx beside it, all three of the same build) becomes a 3D model in its bind pose with its default body groups, first skin and LOD 0, turned to Filee's Y up, millimetres, counter-clockwise and bottom-left UVs, so it converts to OBJ, GLB, STL and the other 3D formats and to a picture. Its textures are found as Garry's Mod finds them: the .vmt in the materials folder beside the models folder, its $basetexture decoded from VTF. Every triangle is counted against one budget for the whole model (10 million), in strips and lists alike, so strip groups that point at the same table can't multiply it; a larger model is refused as too large. A material file over 1 MB is not read (the material stays plain). MDL is in the Games field; .mdl files learn the models donut (schema 100). Part of KnifeLemon#1526.
… skeleton An MDL becomes a reference SMD studiomdl can compile again: its bones with their bind pose, its triangles counter-clockwise with up to three bone weights per vertex, UVs from the bottom left, in inches, Z up. Any other 3D model becomes a static prop with one root bone, ready for studiomdl; a reference SMD's triangles become a model again, so SMD converts to GLB, OBJ, STL and the rest. SMD is in the Games field; .smd files learn the models donut and the SMD preset joins the 3D models in the palette (schema 100, with MDL). Part of KnifeLemon#1526.
The "QC + SMD" preset turns a .mdl with its .vvd, .vtx, .phy and .ani into the folder studiomdl builds it from: the QC (body groups, LODs, materials and skins, attachments, hitboxes, pose parameters, sequences with their events, collision model or joints, key values), a reference SMD per body group model, an SMD per LOD that differs, the physics SMD and an animation SMD per animation. Animations are decoded from the .mdl or its .ani (raw and run-length encoded rotations and positions, sections, root motion) and streamed into their SMDs a frame at a time; the .phy's convex pieces are walked through their ledge trees. Every count is capped and every offset checked before it is used, and the model as a whole has budgets: bone poses over all animations (50 million, checked before anything is written), triangles over all body groups and LODs, triangles and points over all collision pieces, and hitboxes, events and blended animations in all. A damaged file is refused and the folder removed; an oversized one is refused as too large. File names never name a Windows device (con, nul, com1 ... get a "_" in front). The preset is in the palette after "SMD", not on a donut (schema 101). Part of KnifeLemon#1526.
9351b5c to
30d9cba
Compare
|
Thanks! Everything from the review is in, rebased on the updated #1609:
All of them refuse with Schema: "QC + SMD" is step 101, after #1609's 100. |
What does this change?
Follow-up to #1609 for #1526: a Source engine model can now be decompiled into the folder studiomdl builds it from, the way Crowbar does. The "QC + SMD" preset turns a
.mdl(with its.vvd,.vtx,.phyand.ani) into a folder named after the model:<model>.qc$modelname, model flags,$surfaceprop,$illumposition,$bbox/$cbox,$bodygroups,$lod/$shadowlod,$cdmaterials,$texturegroup,$attachments,$hboxset/$hboxes,$includemodels,$poseparameters, a$sequenceper sequence (animations, blends, activity, fps, options, motion flags, fades, events),$collisionmodelor$collisionjoints(with$jointconstrain),$keyvalues<body model>.smdblank)<body model>_lod<n>.smdremovemodel<model>_physics.smd.phy<model>_anims/<animation>.smdHow it reads them
LODs use the LOD 0 vertices with each LOD's own triangles from the
.vtx. This was checked against a six-LOD Garry's Mod vehicle.Animations come from the
.mdl, or from the.anibeside it when they're in animation blocks, section by section. Each animated bone can have:Frames are decoded one at a time and written straight into their SMD, so an animation is never held in memory whole. Track values are scaled by the bone's scales and added to its bind pose (except in delta animations). The root motion studiomdl took out (movements) is added back.
Physics walks the
.phy's IVP ledge trees and turns metres in IVP's axes into inches. Solids named after bones (a ragdoll) are moved from bone space into model space.Not written: flexes (
.vta), eyeballs, IK rules and chains, bone controllers and procedural bones. A model with them still compiles from the QC, just without them. Compiling back to.mdlstill needs Valve's studiomdl.Hostile files
SourceBytes)..physolids (1,024), ledge-tree nodes (65,536) and points per piece (65,536).error.document_too_large:.phypieces: 1,000,000 each;con,prn,aux,nul,com1–com9andlpt1–lpt9(with any extension) get a_in front.Settings
.mdlcan take it. The name is untranslated like "SMD" and "GMA", so there are no new UI strings.Tests
SourceDecompileTests): the test model builder now also writes an attachment, a second LOD, a three-frame animation (Quaternion64, Vector48 and an RLE rotation track) with a sequence and an event, and a.phywith one convex piece. The tests cover:.phytriangles and points) refusing just past the test model's size, with nothing written;_.SourceFormatTests): the v100 → v101 migration (preset after "SMD", not on a donut).Type of change
How was it tested?
dotnet format --verify-no-changesdotnet testChecklist
src/Filee.App/Assets/i18n/*.jsonfiles (none added)THIRD-PARTY-NOTICES.md(none added)