Conversation
Adds .deps/ to gitignore so locally-built Apache ORC artifacts at .deps/orc/ do not end up in commits.
Initial reader for the Apache ORC file format (refs daphne-project#985). Includes: - src/runtime/local/io/ReadOrc.h — header-only reader matching PR daphne-project#993's reader signature. Dispatched from Read.h's existing if/else chain until daphne-project#993 lands. - Read.h: .orc dispatch for Read<DenseMatrix<VT>> and Read<Frame>. - CMake: liborc wired in as a static dep at .deps/orc/, built from apache/orc v2.0.0 against the container's system protobuf. - 12 Catch2 tests plus pyarrow-generated fixtures. Build notes for reviewers: - Apache ORC is built locally at .deps/orc/ (gitignored): git clone --branch v2.0.0 https://github.com/apache/orc.git .deps/orc-src cd .deps/orc-src && mkdir build && cd build cmake -DCMAKE_INSTALL_PREFIX=$(pwd)/../../orc \ -DBUILD_JAVA=OFF -DBUILD_TOOLS=OFF -DBUILD_CPP_TESTS=OFF \ -DBUILD_POSITION_INDEPENDENT_LIB=ON \ -DPROTOBUF_HOME=/usr/local .. make -j$(nproc) install - The daphne-dev container should include this step for reproducible builds upstream. Refs: daphne-project#985 Refs: daphne-project#993
Frame columns of ValueTypeCode::STR now decode from ORC STRING into std::string. VARCHAR/CHAR still throw via the type-kind mismatch path. DenseMatrix<string> stays rejected at Read.h.
Reader honours options["columns"], a comma-separated list of ORC file column names. Only listed columns are decoded via liborc's include(). User-requested order is preserved as output column order. Duplicates and missing names throw. Whitespace is trimmed. Absence of the key preserves whole-file behaviour.
Reader honours options["predicate"], a single-clause predicate of the form <col> <op> <literal>. Builds an orc::SearchArgument for stripe-level skipping and does a per-row filter for correctness. Ops: =, !=, <, <=, >, >=. Literals: integer, float, single-quoted string. When the predicate is set the caller must pass *res == nullptr; the reader allocates the final DenseMatrix/Frame at the true matching-row count.
ShreyasGS
force-pushed
the
feat/orc-reader-liborc
branch
from
August 1, 2026 21:46
a758a06 to
2d9d208
Compare
Style cleanup for consistency with the rest of the codebase. - Replace U+2014 em dashes with ASCII hyphens across ReadOrc.h, ReadOrcTest.cpp, gen_orc_test_data.py, and OrcIO.md. No sibling file in src/ or doc/ uses em dashes. - Retitle OrcIO.md headings to Title Case for consistency with HDFSUsage.md and BinaryFormat.md. - Move .deps/ under the '# dependencies' block in .gitignore next to thirdparty/*. - Add Apache 2.0 header to gen_orc_test_data.py to match the convention of the other Python files in the repo.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ORC reader for DAPHNE, using Apache ORC's C++ library directly. Supports
DenseMatrix<double>,DenseMatrix<int64_t>, andFramewith mixed value types. Refs #985.What this does
DenseMatrixandFrame, matching the file-I/O extension signature from An Infrastructure for Extendable Input/Output File Format Support in DAPHNE #993.Framecolumns typedValueTypeCode::STRdecode from ORCSTRING.options["columns"]— a comma-separated list of ORC file column names. Only listed columns are decoded (via liborc'sinclude()), in user-specified order. Duplicate names, missing names, and whitespace edge cases are handled with clear errors.options["predicate"]— a single-clause predicate<col> <op> <literal>(ops:=,!=,<,<=,>,>=; literals: integer, float, single-quoted string). Builds anorc::SearchArgumentfor stripe-level skipping and applies a per-row filter for correctness. When set, the caller passes*res == nullptrand the reader allocates the result at the true matching-row count.VARCHAR/CHAR, and unsupported value types are errors.Tests
44 Catch2 test cases in `test/runtime/local/io/ReadOrcTestpaths, string reads, projection, predicates. All passinside the DAPHNE dev container.
Build
Apache ORC is built from source with position-independent code and installed to
.deps/orc/(git-ignored).