Parser redesign doc by ashtum · Pull Request #286 · cppalliance/http

ashtum · 2026-08-22T12:10:26Z

No description provided.

cppalliance-bot · 2026-08-22T12:14:50Z

An automated preview of the documentation is available at https://286.http.prtest3.cppalliance.org/index.html

If more commits are pushed to the pull request, the docs will rebuild at the same URL.

2026-08-25 14:04:56 UTC

cppalliance-bot · 2026-08-22T12:27:55Z

GCOVR code coverage report https://286.http.prtest3.cppalliance.org/gcovr/index.html
LCOV code coverage report https://286.http.prtest3.cppalliance.org/genhtml/index.html
Coverage Diff Report https://286.http.prtest3.cppalliance.org/diff-report/index.html

Build time: 2026-08-25 14:20:12 UTC

vinniefalco · 2026-08-22T20:48:29Z

+    /// decoder's output, into caller memory.
+    std::size_t
+    read_some(
+        std::span<capy::mutable_buffer const> buffers,


The reason why we pass mutable_buffers instead of returning an array of const_buffers is that, when using a decoder, we want to decode directly into the user's buffer and avoid extra copies.

The reason why we are taking a span and not a single mutable_buffer is to simplify the I/O layer's job when implementing the read_some operation.

With the span, we can implement the I/O layer read_some like:

template<capy::ReadStream S> template<capy::MutableBufferSequence MB> capy::io_task<std::size_t> read_some_( S& stream, parser& pr, MB buffers) { capy::buffer_param bp(buffers); for(;;) { std::error_code ec; std::size_t n = pr.read_some(bp.data(), ec); if(ec != http::condition::need_more_input) co_return { ec, n }; ec = co_await refill(stream, pr); if(ec) co_return { ec, 0 }; } }

With a single mutable_buffer, the complexity moves to the caller's site:

template<capy::ReadStream S> template<capy::MutableBufferSequence MB> capy::io_task<std::size_t> read_some_( S& stream, parser& pr, MB buffers) { capy::buffer_param bp(buffers); std::size_t total = 0; for(;;) { std::error_code ec; for(;;) { auto const dest = bp.data(); if(dest.empty()) co_return { std::error_code(), total }; std::size_t const n = pr.read_some(dest[0], ec); bp.consume(n); total += n; if(ec) break; } if(ec != http::condition::need_more_input) co_return { ec, total }; if(total != 0) co_return { std::error_code(), total }; ec = co_await refill(stream, pr); if(ec) co_return { ec, 0 }; } }

why isn't this a buffer sequence

Yes, it should be a buffer sequence. I already did that for the serializer, but totally missed it here. And it’s essentially free since the parser doesn’t need to store the sequence.

Fixed in the implementation and updated the redesign document.

vinniefalco · 2026-08-22T20:51:11Z

+
+This approach also allows support for arbitrary decoding formats without limiting the parser to built-in encodings or adding unnecessary complexity to its implementation.
+
+It is worth noting that this design might also eliminate the need for dedicated encoder/decoder services altogether. The information required to use a particular encoder or decoder is already available at the integration layer. For example, the following shows how Burl adds support for `gzip`, `deflate`, `br`, and `zstd` without relying on a separate encoder/decoder service:


This is an interesting direction. Some questions:

How will Boost.Http take advantage of Burl's support for gzip, deflate, br?

What is the mechanism for run-time binding of codecs?

The idea is to add concrete decoder/encoder implementations as headers to the HTTP library and leave it to the library implementer to include the appropriate headers. For example, for decoders:

#include <boost/http/brotli_decoder.hpp> #include <boost/http/zlib_decoder.hpp> #include <boost/http/zstd_decoder.hpp>

With that, the Burl implementation file will be reduced to the following (note that this is still an implementation detail of Burl, and a user won't need to deal with it):

// This header includes decoders based on their availability at build time. #include <boost/http/decoder.hpp> std::unique_ptr<parser::decoder> make_decoder(http::content_coding coding) { switch(coding) { #ifdef BOOST_HTTP_HAS_ZLIB case http::content_coding::deflate: return std::make_unique<zlib_decoder>(15); case http::content_coding::gzip: return std::make_unique<zlib_decoder>(15 + 16); #endif #ifdef BOOST_HTTP_HAS_BROTLI case http::content_coding::br: return std::make_unique<brotli_decoder>(); #endif #ifdef BOOST_HTTP_HAS_ZSTD case http::content_coding::zstd: return std::make_unique<zstd_decoder>(); #endif default: break; } return nullptr; }

This approach makes three assumptions:

Regular users would be using Beast2 and Burl and won't have to deal with these headers or decoder/encoder installation at all.

Low-level users of lib HTTP who want to manage decoders/encoders in their own layer would need to have knowledge of these services anyway. For example, on the client side, one has to set the appropriate Accept-Encoding in the request based on the availability of the decoders. Having to include a header and construct a decoder instance won't make a meaningful difference in terms of ease of use.

Compression libraries are already built libraries. By keeping the implementation of the decoder/encoder header-only, we don't lose any build speed.

The decoder/encoder management would be an implementation detail of the library anyway, so including the headers wouldn't expose the C API to end-user code.

That would eliminate the need for decoder/encoder services and also the installation sites in end-user applications:

// No need to do these anymore: #ifdef BOOST_HTTP_HAS_BROTLI http::brotli::install_brotli_service(); #endif #ifdef BOOST_HTTP_HAS_ZLIB http::zlib::install_zlib_service(); #endif

The way libraries expose their compression capabilities would be specific to each library. For example, in Burl, it would normally be out of the user's way. A request like the following:

auto json = co_await client.get(url).as<json::value>();

would send the appropriate Accept-Encoding based on the compression libraries available at build time and would install the appropriate decoder based on the server response. All of this would be invisible to the user.

However, it would still give the user control through configuration:

burl::client::config cfg; cfg.gzip = true; cfg.deflate = true; cfg.brotli = false; // Do not advertise or decode br. cfg.zstd = false; // Do not advertise or decode zstd. burl::client client(co_await capy::this_coro::executor, tls_ctx, cfg);

This is already implemented and working:
https://develop.burl.cpp.al/burl/2.guide/2m.compression.html

How does boost.http know if ZLib is available?

Boost.HTTP itself never needs to know about the availability of the compression libraries, since the encoder/decoder implementations can be header-only. It is the downstream libraries that need that knowledge. However, this could be integrated into targets like boost_http_zlib, which would publicly link the compression library and define the appropriate macro, like BOOST_HTTP_HAS_ZLIB. The user would then only need to link against the desired targets. Although, if we go this route, there would be no reason to keep the encoder/decoder implementations header-only.

@vinniefalco, this section of the document has been updated. The new parser uses the same approach as before to access decoder services when they are available.

vinniefalco · 2026-08-22T20:52:12Z

+
+```cpp
+template<capy::ReadStream S>
+class message_reader


This is very interesting. It is essentially, a message read strream?

This provides an interface that satisfies both the capy::ReadSource and http::BufferSource requirements and can be used interchangeably depending on the user's needs.

When the user already owns the buffer—for example, when reading a body into an instance of std::string they can use the capy::ReadSource interface to avoid extra copies:

message_reader reader(&stream, &pr); std::string body; body.resize( pr.get().content_length().value()); auto [ec, n] = co_await reader.read( capy::mutable_buffer(body.data(), body.size()));

Note: even with an installed decoder, the parser will decode directly into the user-provided buffer.

When the user wants to perform an operation that requires an external buffer, they can benefit from the parser's internal buffer. For example, streaming the body into a file:

message_reader reader(&stream, &pr); corosio::stream_file f(co_await capy::this_coro::executor); f.open(dest, fb::write_only | fb::create | fb::exclusive); capy::const_buffer bufs[8]; std::uint64_t total = 0; for (;;) { auto [ec, data] = co_await reader.pull(bufs); if (ec == capy::cond::eof) break; if (ec) co_return { ec }; auto [wec, n] = co_await f.write_some(data); total += n; reader.consume(n); if (wec) co_return { wec }; }

The mixed mode allows implementations such as multipart_form to be both simple and efficient. Depending on the contents of each field whether it is read from a file or is a binary or text blob in memory the implementer can chose the appropriate interface.

A similar facility exists in the serializer side (message_writer).

vinniefalco · 2026-08-22T20:53:27Z

+class basic_message_reader;
+
+using message_reader = basic_message_reader<capy::any_read_stream>;
+```


If we follow the design principles of Corosio then we would give the user the building blocks and allow them to opt-in to any of the three levels of abstraction, by uttering the correct type.

sgerbino · 2026-08-25T14:32:31Z

+        }
+
+        // Refill the parser and try again
+        ec = co_await refill(stream, pr);


Is this refill migrated from Burl's message_reader::refill_? If so it returns io_task<> so it should be std::tie(ec) = co_await refill(...). It might be worth showing refill in the doc.

It serves as a filler to shorten the snippet here. Yes, std::tie(ec) would make it correct.

sgerbino · 2026-08-25T14:47:54Z

+};
+```
+
+The rationale for the existence of `message_reader` is to satisfy the `capy::ReadStream`, `http::ReadSource` and `http::BufferSource` concepts, so it composes with generic stream algorithms.


We should explicitly state the end of body contract for message_reader/parser in the design doc, e.g. the differences between eof and incomplete.

The main reference document is more detailed and explains the eof condition in each interface: https://develop.burl.cpp.al/burl/reference/boost/burl/parser.html
But I think the Errors section in the doc should also explain the end of the body conditon.

sgerbino · 2026-08-25T14:53:44Z

+    get_parser();
+
+    /// Parse and return the header.
+    capy::io_task<request_head_base const&>


I think a reference here is problematic and a footgun. co_return { ec, {} } would compile but bind to an object that is already destroyed by the time the caller sees it.

sgerbino · 2026-08-25T15:00:55Z

+template<typename Stream>
+class basic_message_reader;
+
+using message_reader = basic_message_reader<capy::any_read_stream>;


any_read_stream is a handle, and then we have a constructor that takes Stream* s creating two levels of indirection and two pointers. There is something awkward about it, is there a better way to structure this? Possibly this alias needs to change?

The doc doesn’t mean that basic_message_reader should be exactly like the current message_reader implementation. It’s more a question of whether we should offer such a concrete type at all.

sgerbino · 2026-08-25T15:02:50Z

+    {
+        // Try to read the internally buffered data first
+        std::error_code ec;
+        std::size_t n = pr.read_some(buffers, ec);


I'm not a fan of inconsistent error handling, read_some here is taking an outparam std::error_code, where most of the API returns it through io_result. I understand this function is synchronous. Corosio has taken the approach of using io_result to handle this situation as it is more consistent.

Yes, io_result would be more consistence with capy.

github-project-automation Bot added this to Beast2 Aug 22, 2026

github-project-automation Bot moved this to Backlog in Beast2 Aug 22, 2026

ashtum linked an issue Aug 22, 2026 that may be closed by this pull request

Design document for the new parser #287

Open

ashtum moved this from Backlog to In progress in Beast2 Aug 22, 2026

ashtum moved this from In progress to Backlog in Beast2 Aug 22, 2026

ashtum force-pushed the new-parser-design-doc branch 2 times, most recently from 131f3e0 to 18c45bf Compare August 22, 2026 18:00

vinniefalco reviewed Aug 22, 2026

View reviewed changes

ashtum force-pushed the new-parser-design-doc branch 4 times, most recently from 570be1a to 477b4c9 Compare August 25, 2026 11:04

Parser redesign doc

f7ea467

ashtum force-pushed the new-parser-design-doc branch from 477b4c9 to f7ea467 Compare August 25, 2026 14:00

sgerbino reviewed Aug 25, 2026

View reviewed changes


		This approach also allows support for arbitrary decoding formats without limiting the parser to built-in encodings or adding unnecessary complexity to its implementation.

		It is worth noting that this design might also eliminate the need for dedicated encoder/decoder services altogether. The information required to use a particular encoder or decoder is already available at the integration layer. For example, the following shows how Burl adds support for `gzip`, `deflate`, `br`, and `zstd` without relying on a separate encoder/decoder service:

Uh oh!

Conversation

ashtum commented Aug 22, 2026

Uh oh!

cppalliance-bot commented Aug 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cppalliance-bot commented Aug 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ashtum Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ashtum Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cppalliance-bot commented Aug 22, 2026 •

edited

Loading

cppalliance-bot commented Aug 22, 2026 •

edited

Loading

ashtum Aug 23, 2026 •

edited

Loading

ashtum Aug 25, 2026 •

edited

Loading