A high-performance, multimodal-native engine for AI workloads

Vane unifies multimodal data, intelligence, and continuous learning with Python and SQL interfaces, seamlessly scaling from local environments to Ray clusters.

Note

Project status

• Vane Data — Supports most of the capabilities described below and is under active development. Its interfaces and internals may continue to evolve as the codebase is reviewed and hardened.
• Vane RL and Vane Agent — In the early stages of design and implementation. Their source code will be released in future updates.

• Vibe Coding and Agentic Engineering — Some parts of our system were initially built through Vibe Coding. We are now continuously analyzing, understanding, and improving the codebase, applying an Agentic Engineering approach to drive iterative optimization and enhance the quality, maintainability, and efficiency of the system.

---

Vane Data

Vane Data is a high-performance, multimodal-native data engine for AI workloads. Built on a fork of DuckDB, it extends the core execution engine with native multimodal processing and a unified framework for local and distributed execution.

Key Features

• Multimodal-native processing — Process images, video, audio, text, documents, events, sensor data, and tables through a unified type system. Dynamic batching and backpressure control handle variations in data size and computational cost.
• Python and SQL interfaces — Build data and AI pipelines with DuckDB SQL or the Python Relation API.
• Built-in AI operations — Invoke LLMs, generate embeddings, and run batch inference through OpenAI and Anthropic APIs or native vLLM integration. Prefix-aware bucketing improves vLLM prefix-cache hit rates and inference throughput.
• Heterogeneous execution — Overlap CPU, GPU, I/O, and model inference workloads through asynchronous scheduling.
• Local-to-cloud execution — Run the same pipeline locally or across distributed Ray clusters, with a foundation for future edge-cloud coordination.
• Designed for production AI workloads — Build multimodal training-data preprocessing pipelines and enterprise-scale batch inference workflows.

---

Getting Started

Installation

Vane supports Python 3.10 through 3.14. Python 3.12 is recommended and is the primary development version.

Install the vane-ai package from PyPI:

pip install vane-ai

Optional DuckDB extensions are separate platform packages. Install a provider with pip from the package index configured for your deployment, then load it by name on the connection that will plan the query:

python -m pip install vane-extension-iceberg

import vane

connection = vane.connect()
vane.load_installed_extension("iceberg", connection=connection)
vane.vane_extensions(connection=connection).show()

vane.extension_catalog() reads the live, independent vane-extensions registry, so new provider packages do not require a Vane release. See the distributed extension architecture for discovery, installation, verification, and Ray worker requirements.

For more details, see the Installation Guide.

Apache Doris Arrow Stream Load

Install the HTTP transport and write Arrow batches directly to a Doris FE or BE Stream Load endpoint:

pip install 'vane-ai[doris]'

import pyarrow as pa
import vane

relation = vane.sql(
    """
    SELECT
        i AS id,
        (CASE WHEN i = 1 THEN [0.1, 0.2, 0.3] ELSE [0.4, 0.5, 0.6] END)::FLOAT[] AS embedding,
        CASE WHEN i = 1 THEN 'one' ELSE 'two' END AS title
    FROM range(1, 3) AS t(i)
    """
)
summary = relation.write_datasink(
    vane.DorisStreamLoadSink(
        "analytics",
        "items",
        endpoint="http://doris-fe.example:8030",
        destination_schema=pa.schema(
            [
                pa.field("id", pa.int32(), nullable=False),
                pa.field("embedding", pa.list_(pa.float32()), nullable=False),
                pa.field("title", pa.string(), nullable=False),
            ]
        ),
        vector_dimensions={"embedding": 3},
        worker_count=4,
    )
)

The ray runner also accepts in-memory from_df() relations and from_arrow() relations built from a PyArrow Table or RecordBatch, for reads and sink writes. It snapshots the referenced columns into the Ray object store during planning; subsequent changes to the source do not change that query's snapshot. Materialize Arrow datasets, scanners, readers, and C Stream capsules into an eager table before using them with Ray. The local runner requires a distributable SQL or file relation for distributed sink writes. write_datasink() is synchronous. In an async caller, offload the complete connection/relation/write operation with asyncio.to_thread(); the Ray runner explicitly rejects blocking execution on the caller's event-loop thread.

The required destination_schema lists the selected Doris columns in upload order and declares their exact Arrow physical types: for example, Doris INT is pa.int32(), FLOAT is pa.float32(), and ARRAY<FLOAT> is pa.list_(pa.float32()). The sink safely casts every input column to this schema before opening an HTTP request, so inferred Python integers (int64 in Arrow) cannot be misread as Doris INT; overflow, incompatible nested values, and nulls for non-nullable fields fail locally. Floating-point narrowing rejects finite values that become infinity while allowing normal rounding. The currently supported destination types are booleans, signed integers, float32/float64, UTF-8 strings, and recursive regular lists of those types. Execution workers may use 64-bit Arrow offsets (large_string, large_list); the sink accepts equivalent input representations and safely normalizes them to the destination schema, including nested lists. Only visible list children are converted; hidden payloads beneath null list slots cannot cause false overflow errors. Supported inputs are null, boolean, integer, floating-point, string, binary, and standard list arrays recursively composed from them. Dictionary, run-end encoded, view, and other unsupported representations must be converted and rebatched before writing. String destinations require string or binary input; explicitly stringify other types in the input relation. Temporal Arrow types are rejected, including nested values, because Doris 4.1.3 does not preserve their timezone semantics; explicitly convert them to a supported non-temporal type before writing.

The sink uses Arrow IPC throughout and does not materialize Python rows. max_batch_bytes limits input Arrow batches to 128 MiB by default, while max_request_bytes independently caps encoded HTTP bodies at 160 MiB. Destination buffer sizes are conservatively checked against the request budget before casting. The full IPC stream, including schema and chunk metadata, is sized before allocating one fixed-size request buffer. Peak worker memory includes at least the input and encoded buffers plus any safe-cast buffers and vector offsets; the HTTP transport drains each request through bounded 256 KiB views instead of enqueueing the complete body again. Each worker performs one synchronous request at a time; increase worker_count for concurrent Stream Loads and tune send_batch_parallelism for Doris-side fan-out. When an FE endpoint redirects to a different BE host, list that host in trusted_redirect_hosts before Vane will send the configured Basic Auth credentials. Entries contain only a hostname or IP address, without a port; IPv6 literals can be bare (2001:db8::42) or bracketed ([2001:db8::42]). Passwords must be supplied through EnvironmentSecret. Vane does not retry an Arrow Stream Load request: if a connection fails after upload, the batch outcome is unknown and its reported Doris label must be inspected before submitting new data. timeout sets the Doris import deadline; the HTTP transport adds 30 seconds to receive the terminal response without racing that server-side deadline. For a large initial vector load, create and build the Doris ANN index after ingestion so index construction does not slow every incoming batch.

Quick Start

Follow the Quickstart guide to build and run your first Vane pipeline.

Connection.execute() routes SELECT queries through the configured runner, including queries with positional or named parameters:

import vane

vane.set_runner_ray()
with vane.connect() as conn:
    rows = conn.execute(
        "SELECT i FROM range(?) AS t(i) WHERE i >= ? ORDER BY i",
        [10, 7],
    ).fetchall()

Set VANE_RUNNER=local-fast before connecting to use native DuckDB execution. Ray is the default when that variable is unset or empty. Each connection fixes its runner at creation; cursors and derived relations inherit that policy. Later environment changes and runner-selection calls affect new connections only. Module-level helpers such as vane.sql() share the default connection and its fixed policy. Set the variable before importing Vane to choose the default connection's policy, or create an explicit connection to choose a new one. Ray and local FTE runner instances are initialized separately and retain their explicit configuration. get_runner() and get_or_create_runner() select by the current environment; teardown_runner() closes both initialized runners. Ray initializes when a data query or write first needs it. Only direct client reads in the allowlist below execute on their owning connection without initializing Ray. Derived state queries and other metadata sources are unsupported. Data queries require auto-commit mode, including when binding a lazy Relation's schema. Distributed queries and writes reject explicit transactions before binding can evaluate table-function arguments; runner-bound table-function arguments also reject unsupported client-context and database-modifying expressions before bind-time evaluation in auto-commit mode. Explicit PyLogicalPlan factories apply the same runner admission regardless of the source connection's runner. Planning and execution errors propagate without local fallback. execute() returns the connection and shares one cursor across row, DataFrame, and Arrow consumers. Multiple statements execute in order and retain only the last result. SQL and Relation terminals share one execution entry after client-side binding, before native optimization. local-fast continues through DuckDB; Ray receives that same bound logical plan and builds its physical plan on the driver. Runners do not bind the SQL or Relation again. Lazy data-query relations remain composable. SQL COPY TO, INSERT, UPDATE, DELETE, MERGE, and CTAS use the same write protocol as their Relation counterparts. execute() returns the runner's Count row; sql() completes the write and returns None. A distributed table write requires a target catalog with a distributed write provider; routing does not make ordinary client DuckDB tables distributed. Unsupported targets fail explicitly before backend mutation.

Ray and local FTE COPY use the Relation writer's dataset layout: a new target such as output.parquet is a directory containing worker output files. Both reject COPY FROM, RETURN_FILES, RETURN_STATS, non-file destinations such as STDOUT/devices/pipes, and explicit transactions. Ray table writes also reject RETURNING, INSERT conflict handling, and CTAS TEMPORARY, OR REPLACE, and IF NOT EXISTS. Read-only targets are checked before runner initialization. ATTACH, DETACH, settings, transaction control, catalog-only DDL, and PRAGMA commands remain client connection operations. Direct SHOW/DESCRIBE/PRAGMA statements and their completed results use native DuckDB, including benchmark query pragmas such as PRAGMA tpch(1). DESCRIBE SELECT ... and DESCRIBE table bind and return schema information on the client without initializing Ray or transporting a plan. SUMMARIZE still uses the configured query runner because it scans data. Ray rejects derived relations over these command results. Runner writes and explicit PyLogicalPlan exports cannot include client command queries. Database-modifying expressions such as nextval() are unsupported in distributed plans, including write defaults and CHECK constraints. Ray INSERT/UPDATE/MERGE reject generated target columns because their runtime expressions are outside the bound write plan.

Ray connections also support a finite allowlist of direct client reads through execute() and sql(). These use DuckDB's native binding and execution on the owning client connection without initializing Ray:

• With no FROM, direct calls to current_setting(), getvariable(), current_query(), current_schema(), current_database(), current_connection_id(), current_query_id(), current_transaction_id(), txid_current(), now() and transaction_timestamp(). Arguments must be string/numeric/NULL literals or bound parameters. These literals and parameters are also allowed as result columns. Negative numeric literals such as -1 are supported. Expressions such as +1 or -(1 + 1) are unsupported; pass their values as bound parameters instead.
• Direct columns or bare * from one of duckdb_tables(), duckdb_views(), duckdb_schemas(), duckdb_databases(), duckdb_settings(), duckdb_variables(), duckdb_extensions() and duckdb_sequences(), without arguments. Direct connection.table_function(name) results have the same native path.

Filtering, aggregates, sorting, limits, casts, nested expressions, macros, CTEs, views, subqueries and relation composition do not extend this allowlist. Other state functions and metadata sources, including duckdb_columns(), pragma_table_info() and pragma_show(), are unsupported as SELECT sources. Aliases implemented as macros (such as current_catalog()) and SQL value keywords (such as CURRENT_TIMESTAMP) are also outside this initial allowlist. Use the listed direct function spellings. Native catalog qualification and column aliases are supported; the classifier does not autoload extensions or invoke bind callbacks. Extension overloads do not inherit native-read eligibility from a built-in name. Overloads with a different argument count leave existing native reads available. If an ineligible overload can accept the same argument count, the call is rejected conservatively without resolving argument types.

For example, SELECT current_setting('threads') and SELECT table_name FROM duckdb_tables() stay on the client. SELECT count(*) FROM duckdb_tables() and SELECT current_setting('threads'), value FROM read_parquet(...) report unsupported operations. Filter or combine returned metadata in Python, or pass a value as an explicit SQL parameter to a distributed query. There is no execution fallback. local-fast retains native DuckDB support for these query shapes.

Allowlisted reads can inspect the client's explicit transaction. Data queries still require autocommit and pass the existing Ray capability checks. Native query verification for ordinary queries requires a local-fast connection; Ray client reads report this restriction when verification is enabled. The existing connection snapshot still carries execution settings, including time zone, to the driver and workers for ordinary data queries.

currval(), setseed(), logging functions (write_log() and parse_duckdb_log_message()) and unary age(timestamp) remain unsupported in runner-bound expressions. Binary age(a, b) remains portable because both timestamps are explicit. This restriction also applies inside defaults, CHECK constraints and CTAS WITH, PARTITIONED BY and SORTED BY metadata. Metadata SQL expressions are checked on the client; macro expansions and values already bound as constants are preserved before transport; client-variable reads are rejected. Extension partition/sort transforms validate their SQL arguments against the created table's columns. Metadata subqueries and lambda expressions are unsupported. System table functions that inspect or change client state, such as duckdb_settings(), duckdb_tables() and logging controls, are also rejected in distributed reads and writes. Native queries and client PRAGMA queries keep access to those functions; static lists such as duckdb_keywords() remain portable. Native query verification requires local-fast; connection controls can still disable verification on Ray/local FTE connections. VACUUM and ANALYZE run on the client connection for every runner. Catalog commands such as SHOW TABLES, SHOW DATABASES and SHOW VARIABLES also use the client connection when issued directly. Ray rejects derived relations and catalog queries revealed by query() expansion before binding their contents. Writes and explicit plan transports reject this query origin for every runner. Distributed plans cannot read or write client temporary tables. Temporary views whose definitions expand into transportable data sources remain supported. SQL CALL has no distributed side-effect contract and requires local-fast, as do SQL PREPARE, EXECUTE, and EXPLAIN ANALYZE. These unsupported wrappers are rejected before binding can evaluate their arguments, including inside plain EXPLAIN. Plain EXPLAIN remains available for supported client-side planning. Runner admission rejection preserves an existing client transaction and its prior work. Pass parameters directly to execute() or sql() for runner execution.

connection.interrupt() cancels active runner writes and waits for their outcome; a commit that wins the race retains its successful result. A committed write whose result cannot be delivered raises CopyResultUnavailableError with safe_to_retry=False; an uncertain outcome remains CopyOutcomeUnknownError. executemany() uses the shared entry for every parameter set and retains the final result. Local-fast reuses one native prepared statement across the batch; runner execution exports a bound plan for each parameter set. The local FTE runner supports COPY and DataSink terminals; its SELECT result consumption continues to use native DuckDB. Other table writes require ray or local-fast. Execution errors never trigger local fallback.

Session configuration, ATTACH, transaction control, and catalog-only DDL continue executing on the client coordinator connection. SQL is bound there; Ray receives serialized bound logical plans for both SQL and Relation queries and writes. Moving catalog and session operations to the driver is outside this routing change. Ray uses the same source support as the Relation runner: scans of ordinary in-memory tables and temporary tables are rejected. Select local-fast for those queries when creating the connection, or use a distributed source such as Parquet.

conn.sql() (also query() and from_query()) returns a lazy relation for SELECT, including when params supplies positional or named values. Values are captured when the relation is created; modifying the original parameter container does not change the query. Filtering, joining, exporting SQL, or creating a view preserves those values. Reading the result uses the configured runner and does not first materialize the SELECT on the coordinator:

with vane.connect() as conn:
    relation = conn.sql(
        "SELECT i + $offset AS value FROM range($rows) AS t(i)",
        params={"offset": 10, "rows": 5},
    )
    rows = relation.filter("value >= 12").order("value").fetchall()

The final SELECT remains lazy; preceding statements in the same call execute in order, with SELECTs using the runner. Parameters belong only to the final statement. Ray result consumption rejects an explicit transaction, including one started after the relation was created.

Call conn.interrupt() from another thread to cancel an active Ray result wait. Row consumers raise InterruptException after query cleanup; exported Arrow readers report interruption through the Arrow stream error. The connection can execute another query afterward, and other connections keep their own queries.

More Resources

• Examples
• Production deployment

---

Multimodal Inference Benchmarks

Hardware configuration: 1 node, 36 CPU cores, 64 GB memory, and 1× NVIDIA GeForce RTX 2080 Ti (22 GB VRAM).

We use the Ray Data benchmark suite to compare Vane with Ray Data and Daft. The benchmark source code is included in this repository.

The Ray runner targets distributed workloads. The current results are single-node only; validation on the multi-node environments used in the Ray Data benchmarks is still pending.

See the benchmarking page for detailed results.

---

Contributing

Contributions and collaborations are welcome. Contribution guidelines and community channels will be published as the project opens further.

---

License

Vane is distributed under the Apache License 2.0. See LICENSE and NOTICE for details and third-party attributions.

---

Acknowledgements

Vane Data is built on top of DuckDB and inspired by infrastructure systems such as Ray Data, Daft, and Trino.

• DuckDB: The core modular architecture and inspiration. A high-performance analytical database system. It is designed to be fast, reliable, portable, and easy to use.
• DuckDB-Python: The core modular architecture and inspiration. The DuckDB Python package.
• Ray Data: A scalable data processing library for AI workloads built on Ray
• Daft: High-Performance Data Engine for AI and Multimodal Workloads
• Trino: A fast distributed SQL query engine for big data analytics.

Special thanks to these projects.

---

Give Vane a ⭐️ if it helps you!