Skip to content

convert_aloha_data_to_lerobot.py is outdated and extremely memory-heavy #986

Description

@AaronDsouza-2006

The current convert_aloha_data_to_lerobot.py script has two major issues that makes it difficult to train openpi:

  1. Outdated LeRobot format (v2 instead of v3.0)

    • The script produces the old LeRobot v2 dataset format.
  2. Extremely high memory usage

    • The script loads too much data into RAM at once
    • It does not support proper batching

Steps to Reproduce

  1. Prepare a folder with ALOHA-style episode_*.hdf5 files
  2. Run convert_aloha_data_to_lerobot.py.
  3. After conversion, the resulting dataset is in the old v2 format.

Expected Behavior

  • The script should output a LeRobot v3.0 compatible dataset by default.
  • It should support batching / low-memory mode (process episodes in small groups, compute running statistics, write Parquet chunks incrementally).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions