This is the PyTorch implementation for the paper SinCro: Single-View 3D-Aware Representations for Reinforcement Learning by Cross-View Neural Radiance Fields. SinCro is a novel RL framework that leverages 3D-aware representations from single-view RGB inputs, without requiring camera calibration information or synchronized multi-view images for downstream RL. We provide the SinCro code for the data generation and encoder pre-training. For more details about the implementation, please refer to our paper.
Download our code. Then,
cd SinCro
conda create -n SinCro python=3.8
conda activate SinCro
pip install -r requirement.txt
We provide a dataset for the peg-insert environment. You can download the dataset from the drive, and unzip it to your dataset path. After that, you should create a file "configs/peg_dataset_path.txt" (refer to "configs/peg_dataset_path_template.txt").
For pre-training of the 3D Scene Encoder in the peg-insert environment,
conda activate SinCro
python MV_run_nerf.py --config configs/peg.txt
To visualize the reconstruction results of the trained model, you can use 'MV_visualize.py'. This file will provide rendered videos and images from all six viewpoints of an episode, as well as quantitative results. Please modify the variables 'ckpt_folder_dir', 'single_view_input', 'input_view_index', and 'ref_view_index' in this file. Then,
conda activate SinCro
python MV_visualize.py
If you want to use the default camera setting used in this work, skip this procedure.
If you want to use your custom setup camera setting, uncomment add_cam_test() and comment sawyer_scripted_policy_test(). Then,run python nerf_data_gen_custom_coordinate.py
Then, for each environment's xml file [e.g. sawyer_window_horizontal.xml in case of window-open-v2], you have to replace the line with your own camera_tree xml file generated by running add_cam_test().
python nerf_data_gen_custom_coordinate.py
Our code is based on and modified from the official implementations of NeRF and CroCo.
If you found our work useful, please consider citing us.
@ARTICLE{11180891,
author={Cho, Daesol and Yoo, Seungyeon and Shim, Dongseok and Kim, H. Jin},
journal={IEEE Robotics and Automation Letters},
title={Single-View 3D-Aware Representations for Reinforcement Learning by Cross-View Neural Radiance Fields},
year={2025},
volume={10},
number={11},
pages={12039-12046},
keywords={Three-dimensional displays;Neural radiance field;Cameras;Representation learning;Image reconstruction;Visualization;Robots;Robot vision systems;Training;Solid modeling;Reinforcement learning;representation learning;visual learning},
doi={10.1109/LRA.2025.3615035}}