WIP WIP WIP vulkan schema generator and decode action implementation - #3250
Open
jzulauf-lunarg wants to merge 12 commits into
Open
jzulauf-lunarg wants to merge 12 commits into
jzulauf-lunarg wants to merge 12 commits into
Conversation
jzulauf-lunarg
requested review from
antonio-lunarg,
bradgrantham-lunarg and
fabian-lunarg
as code owners
September 6, 2026 02:19
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…ery structure decoder Pilot of the Generic Field Schema and Action Model (Design-documents, consumer-graph/decode_generic_visitor_action_model.md, v7.0) for one operation family: decoding a Vulkan structure from a capture stream. What is added framework/format/format.h gains a vocabulary of logical kinds, one tag per encode-type typedef, each carrying the wire type it is recorded as. A kind is not a type: VkDeviceSize and uint64_t are the same bytes and the kind is what tells them apart. framework/schema/ is a new header-only, API-agnostic layer: field shapes, the MemberPointer trait and its Addressable/HasMember concepts, GetRef/Get/Set, the kind-and-shape concepts an Action selects on, Schema<ApiElement>::Fields, WalkFields, the command return partition and InvokeFromFields. framework/util/type_list.h supplies the TypeList facility it composes. A new generator, vulkan_schema_generator.py, emits six files from vk.xml: API type descriptors, command tags, Field descriptors and Schema specializations; the decoded-representation traits in both directions (ApiElementTraits and its inverse ApiElementFor); three member-trait partitions for native structures, decoded wrappers and decoded command args; and a compile-time checks file. Curated knowledge the registry lacks is small and named: four refined kinds (size_t, VkDeviceSize, VkDeviceAddress, VkSampleMask) and three shared descriptors (GenericHandle with its selector field, ExternalObject, OpaqueBytes). framework/decode/vulkan_decode_action.h is the one hand-written Action: twelve constrained Apply overloads covering every field shape the registry produces. Every write to the decoded value goes through Set; GetRef appears only where a decoder needs a member's address. vulkan_decode_struct_impl.h defines DecodeStruct as one constrained function template around WalkFields. ValueDecoder and PointerDecoder gain a Decode<Kind> template so the Action names kinds, not widths. What is replaced All 1,343 non-union, non-blacklisted Vulkan structures decode through the field walk. generated_vulkan_struct_decoders.cpp drops 26,000 lines of procedural bodies for one explicit instantiation per structure; one hand-written body (VkBaseOutStructure) remains and is the single entry of the NonSchemaDrivenStructs exclusion list. The forward header declares one constrained template in place of one prototype per structure. Callers are unchanged: every call still names a Decoded_T*, and a hand-written non-template overload still wins. Only that translation unit compiles the walk; the member-trait partitions reach no other target. Verification MSVC Debug and GCC 13.3 Debug build clean; 281 assertions in 19 test cases pass on both. test_vulkan_schema.cpp round-trips one structure per Apply overload against hand-encoded bytes. The checks file holds 7,309 static_asserts: every structure has a schema and no return field, every command has one return field, both trait directions round-trip, every descriptor's element type equals its wrapper's struct_type, every scalar element is as wide as its kind's wire type, and every refined kind records the same bytes as the primitive the encoder resolves it to. Regeneration reproduces the tree byte for byte. Before the procedural bodies were removed, the two decoders were compared field for field on the same buffers. Not in this change No semantic metadata is emitted. Commands have schemas, tags, traits and member traits but no decode Action; two pointer-shaped identifier overloads will be needed when they do. No capture_wrapper_type, no formal API-signature storage, no dispatch or lifecycle traits. Unions keep their hand-written decoders and are reached by legacy descent. No CI check yet enforces the private-include boundary. The capture-file smoke run and encoded-byte comparison the design's proof gates call for have not been recorded. Known latent generator limits, none reachable from the current registry: a two-dimensional static array with a Count sibling would emit an invalid extent; a pointer-returning command would emit an invalid descriptor name; base-header hierarchies (parentstruct) are excluded by is_schema_driven rather than modelled. The 26-commit history this collapses is preserved at tag jzulauf-vulkan-schema-generator-presquash. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One 40k-line header becomes three, in a strict include order, so an
includer pays for what it names and no more.
generated_vulkan_schema_types.h API type descriptors and command
tags, 2,682 lines. The public face:
a descriptor is the key every trait
and Schema is looked up by.
generated_vulkan_schema_fields.h Field descriptors, 17,173 lines.
What an Action's member traits need.
generated_vulkan_schema.h Schema specializations, 20,583
lines. What the field walk needs.
The three carry exactly the old file's content, 36,045 non-comment
lines, verified line for line. Types includes format.h and the API
headers; fields includes types and schema/field.h; schema includes
fields and schema/schema.h. Every declaration stays in the namespace it
was in.
Consumers move down to what they use. The decoded-representation traits
include types rather than the schema, so api_element_traits.h no longer
reaches a Field or a Schema and its comment stops calling itself heavy.
The three member-pointer partitions include fields. The impl header,
the checks file and the test still reach everything through schema.
In the generator, the one identity class becomes three, each with its
own option set, and write_identity becomes write_schema_part, which
opens and closes gfxrecon::schema around whichever writers a part
names.
No product translation unit includes any schema file today, so
preprocessed sizes should be flat for consumers and for the one
translation unit that compiles the walk. The step that follows, giving
each decoded wrapper its api_element through the types file, is the one
that moves them.
MSVC Debug builds; 281 assertions in 19 test cases pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every generated decoded wrapper and args structure carries
using api_element = schema::api_type::vulkan::VkX;
beside the struct_type it already had. A wrapper is its own key into
the schema and the representation traits, so the inverse trait goes:
ApiElementFor and its 2,096 generated specializations are deleted, the
traits header drops from 4,267 lines to 2,166, HasApiElement becomes a
member check, and the entry point reads Wrapper::api_element.
The wrappers header and both args headers include the types file the
previous commit split out, 2,682 lines of descriptors and tags. That
is the first schema content a product translation unit sees, and it is
the measured cost of this change: every TU that includes the wrappers
now parses it.
The shared Khronos header generator gains a make_api_element_alias hook
that emits nothing; the Vulkan subclass overrides it and skips
blacklisted structures, so Decoded_VkBaseOutStructure, which keeps a
hand-written body and has no descriptor, carries no alias. OpenXR
generation is unchanged.
One args structure is hand-written, DeferredOperationJoinKHR, and
carries the alias by hand. Its neighbours in that header do not,
because their commands are on functions-all and have no schema element
to name. STATUS.md PIL-2 records why that one exists and how it goes.
The checks file's round-trip assertions now cross two generators: the
wrapper generator's alias against the schema generator's trait.
MSVC Debug builds; 281 assertions in 19 test cases pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… family for its callers Encode is the second operation family over the schema. EncodeStructAction is one hand-written Action with one overload so far, the value-shaped scalar, writing through ParameterEncoder::Encode<Kind>, which mirrors the decoders' Decode<Kind> so the Action names kinds and never widths. It reads through schema::Get, the read primitive kept for exactly this. VkExtent2D is the first structure the schema encodes. Its procedural body is gone from the generated encoders and one explicit instantiation stands where it stood; VkExtent3D keeps its body as the comparison partner. is_schema_driven_encode in the schema generator is the one predicate both encoder generators read, shaped like the decode predicate and taking the same arguments; the list behind it is private to it and flips to an exclusion at inversion, as decode's did. Two things are pre-inversion by design and only these: DescriptorFor, the native structure to its descriptor, is one hand-written specialization until the generator emits one per structure; and SchemaDrivenStructs is a positive list. Each family now has one header that declares every entry point. decode/vulkan_decode_struct.h includes the generated forward header -- the wrapper declarations, the remaining prototypes, the exclusion list -- and adds the concept and the constrained DecodeStruct beside them. encode/vulkan_encode_struct.h does the same over the generated encoders header. Callers include the umbrella; only the two umbrellas include the generated headers. The concept and template declaration that the generators used to emit as Python strings are C++ in those headers, and the generated headers end with their namespaces and guard like every other. Ten decode includers and seven encode includers moved, hand-written and generated alike; the shared pnext encoder generator keeps its per-API include and the Vulkan options swap it for the umbrella. The test compares schema-encoded bytes against hand-encoded primitives for both structures of the pair. All 18 affected generated files regenerate to the tree. MSVC Debug builds decode, encode, graphics, the framework test and gfxrecon-replay; 285 assertions in 20 test cases pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…two dimensions The second encode idiom, taken in the order the census set. One overload for a scalar-kind field of StaticArray shape: the extents come from the member's declared type through std::extent_v, the count is their product, and the first element's address is taken by one if constexpr on rank. Every named fixed-array entry point, the 2DMatrix family included, is one EncodeArray over a flat pointer and a length, so a matrix is a run of the extent product and the two ranks share a body. ParameterEncoder gains EncodeArray<Kind>(arr, len), forwarding to EncodeArrayConverted, which already converts elements only when the element and wire widths differ. It writes exactly what the named entry points write, the enum and size_t conversions included. Two structures ported, one per rank: VkPipelineCacheHeaderVersionOne, four scalars then uint8_t[VK_UUID_SIZE]; VkTransformMatrixKHR, float[3][4]. Two retained partners exercise the idiom through their procedural bodies: VkPipelineCacheHeaderVersionDataGraphQCOM and StdVideoH264ScalingLists. The test compares all four against primitive oracles, the matrix against EncodeFloat2DMatrix, which pins that a matrix is one flat run. The two ported structures carry hand-written DescriptorFor specializations, the pre-inversion trait embedding; the static_assert in the impl header named the missing ones on the first build. No scalar fixed array in the registry carries a count sibling; the three static arrays that do, memoryTypes, memoryHeaps and physicalDevices, are struct and handle runs and belong to the overloads that read count_field. MSVC Debug builds; 289 assertions in 21 test cases pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every StaticArray Field records extents[] in declaration order, replacing extent and array_dimension, so a fixed-extent array is fully described without a storage type. The generated checks file asserts each recorded list against the declared member type, and the static-array Apply overloads assert the same at every instantiation through DeclaredExtentsMatch. field.h documents the Field descriptor vocabulary. WIP, encode pilot. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…registry fact Every pNext Field records has_extensions, whether the registry declares the owning structure on either side of structextends. The overload selects the procedural bodies' entry point on it: EncodePNextStruct when true, EncodePNextStructIfValid when false. VkSubpassEndInfo and VkPipelineCreateInfoKHR port; VkAttachmentReference2 and VkPerTileBeginInfoQCOM keep their bodies as comparison partners, and a byte-comparison test covers all four with null and one-node chains. field.h documents has_extensions as the registry's fact, not the spec's. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ParameterEncoder gains EncodePointer<Kind>, the kind-keyed twin of EncodeArray<Kind>, forwarding to the converting body as the named pointer entry points do. The overload constrains on ScalarKindField and PointerField and reads the pointer through Get, as the scalar-value and extension-chain overloads now do; GetRef remains only where an address is needed. VkBindMemoryStatus ports; VkRenderingInputAttachmentIndexInfo keeps its body as the comparison partner. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The overload constrains on ScalarKindField and PointerArrayField and reads the count from the sibling Field the descriptor names, the first cross-field read in either Action; the decoder never needed one because the count is on the wire. field.h gains FieldCountField and HasCountField. The element pointer is cast to the Field's element type, which turns the void run of OpaqueBytes into a byte run and leaves a typed run as it is. VkPipelineCacheCreateInfo and VkBufferCreateInfo port; VkValidationCacheCreateInfoEXT and VkRenderPassMultiviewCreateInfo keep their bodies as comparison partners. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
jzulauf-lunarg
force-pushed
the
jzulauf-vulkan-schema-generator
branch
from
September 15, 2026 02:27
a49eebf to
d4e771f
Compare
Encode<Kind> converts with TypeCast, so a pointer or a function pointer reaches the Address kind's wire type by reinterpret_cast, as the named EncodeVoidPtr and EncodeFunctionPtr entry points do. field.h gains GeneralScalarKindField, the scalar family plus the address kind, and the scalar-value overload constrains on it: the kind decides the write, not the C++ type. DescriptorFor rows are one macro line each while the list is hand-written. VkWin32SurfaceCreateInfoKHR, VkSurfaceFullScreenExclusiveWin32InfoEXT, VkImportMemoryWin32HandleInfoNV and VkDebugUtilsMessengerCreateInfoEXT port; VkImportMemoryHostPointerInfoEXT and VkCheckpointDataNV keep their bodies as comparison partners. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The handle's capture wrapper is an encode-private trait keyed on the API type descriptor, CaptureWrapperFor, in encode/vulkan_encode_capture_wrappers.h, with HasCaptureWrapper as the design names it; the public types header never names a wrapper. Two hand-written rows, BufferWrapper and DeviceMemoryWrapper, each asserting the wrapper wraps the descriptor's element type. The overload constrains on HandleKindField, ValueShapedField, HasMember and HasCaptureWrapper and calls EncodeVulkanHandleValue with the trait's type. VkBufferMemoryBarrier and VkMappedMemoryRange port; VkBindBufferMemoryInfo and VkBufferViewCreateInfo keep their bodies as comparison partners. The test registers real wrappers in the state handle table so one case carries genuine ids. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… pointers Three overloads. A handle run combines the counted-run and wrapped-handle overloads: count from the sibling Field, wrapper from CaptureWrapperFor. A generic handle is a HandleField with a selector_field; the overload reads the handle and its object-type sibling and resolves the wrapper id through the two-argument GetWrappedId, recording it through the Handle kind as the decode Action already reads it. A text pointer goes through a kind-keyed EncodeString that reads the wire type and a new text_attribute off the Char and WChar kinds, so the UTF-8 and UTF-16 attributes come from the kind and not the call site; the kind vocabulary moves below the pre-existing content in format.h so the text kinds can name PointerAttributes. Fixed-extent text and the six LPCWSTR fields remain for the next step. VkSubmitInfo and VkDebugUtilsObjectNameInfoEXT port; VkSemaphoreWaitInfo, VkPresentInfoKHR and VkDebugMarkerObjectNameInfoEXT keep their bodies as comparison partners. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implemented all decode idioms with one structure replaced for each. Next step, suppress all structure decode generation.
Compiles MSVC and gcc