Skip to content

[feat](csv) Distribute CSV reads and writes - #543

Draft
kaka11chen wants to merge 1 commit into
AstroVela:mainfrom
kaka11chen:codex/ray-source-sink-audit
Draft

kaka11chen wants to merge 1 commit into
AstroVela:mainfrom
kaka11chen:codex/ray-source-sink-audit

Conversation

@kaka11chen

Copy link
Copy Markdown
Contributor

Summary

  • Enable Ray CSV sources to distribute both multiple files and one seekable, uncompressed UTF-8 file as aligned byte-range scan tasks, with exact row ownership across quoted multiline fields, CR/LF/CRLF boundaries, BOMs, and UTF-8 code points.
  • Serialize and rebuild CSV read/write bind state, union-by-name metadata, scan identities, casts, and COPY writer statistics so Ray workers execute CSV scans and writes without coordinator-only state.
  • Preserve stable source-task identities for fault-tolerant execution and emit one distributed CSV output file per scan task.
  • Reject unsupported distributed CSV configurations explicitly (compression, non-UTF-8 encodings, parallel = false, skip_rows, and store_rejects); this change intentionally adds no compatibility path or local fallback.

Related issue

N/A

Documentation impact

  • No user-facing documentation update is required.
  • Documentation is updated in this PR or in a linked website PR.
  • A follow-up issue is required in
    AstroVela/vane-website.

The change extends the existing Ray runner implementation without adding a new public API. Its strict unsupported-option behavior is covered by tests and documented above.

Validation

  • Incremental native install in Release mode with VANE_NATIVE_BUILD_JOBS=8 and SKBUILD_BUILD_DIR=$PWD/build/python-release.
  • scripts/format workspace --from-ref upstream/main --check
  • git diff --check upstream/main
  • Regenerated serialize_nodes.cpp into a temporary directory and verified an exact diff against the checked-in generated source.
  • scripts/run_installed_pytest.sh tests/fast/test_ray_relation_to_distributed_plan.py -q: 12 passed.
  • scripts/run_installed_pytest.sh tests/fast/test_ray_e2e_serialization.py -q: 10 passed.
  • scripts/run_installed_pytest.sh tests/fast/test_ray_result_contract.py -q: 224 passed.
  • scripts/run_native_tests.sh "test/sql/copy/return_stats.test" -s: 332 assertions in 1 test case passed.
  • scripts/run_release_tests.sh: passed; Ray shard 14 passed, 525 deselected.
  • Randomized byte-range stress coverage: 12 seeds × 8 tasks over LF/CRLF/CR files, quoted delimiters/quotes, embedded newlines, Unicode, and missing final newlines.
  • scripts/run_fast_tests.sh: non-Ray shard passed (5,972 passed, 622 skipped, 132 deselected, 7 xfailed, 1 xpassed); all new CSV Ray tests passed. The shared-Ray shard retained 15 unrelated generic Ray failures, and representative failures reproduced unchanged in a pristine upstream/main worktree on the same host.

Checklist

  • The change is focused and includes tests or a reason tests are unnecessary.
  • Public behavior and compatibility impact are documented.
  • New dependencies, copied code, model assets, and datasets have compatible
    licenses and are recorded where required.
  • No credentials, private endpoints, personal paths, generated data, model
    weights, or build artifacts are included.
  • Security implications of UDFs, serialization, remote code, network access,
    and untrusted input have been considered.
  • Native changes were compiled; Python-only, documentation, and workflow
    changes passed relevant checks.

No new dependencies, copied code, model assets, or datasets are introduced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant