This is the official implementation for LiveScene.

LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control
Delin Qu*, Qizhi Chen*, Pingrui Zhang Xianqiang Gao, Binzhao, Zhigang Wang, Dong Wang†, Xuelong Li†

• Project Pages for LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control [2024-5-18]
• DATASET for LiveScene [2024-10-21]
• Video and slides [2024-11-04]
• Code for LiveScene [2025-01-09]

Follow the official instructions to install the latest version of nerfstudio.

Clone the repository and install the package:

git clone https://github.com/Tavish9/livescene.git
pip install -e .

The InterReal and OmniSim datasets are available on Huggingface.

To download the entire dataset using huggingface_cli:

huggingface-cli download --local-dir livescene_dataset --repo-type dataset IPEC-COMMUNITY/LiveScene

To view available options for training:

ns-train livescene --help

use livescene-real-data for InterReal dataset and livescene-sim-data for OmniSim dataset.

You can use it like any other third-party nerfstudio project.

ns-train livescene --data /path/to/your/livescene_dataset/scene_name livescene-real/sim-data

Slider Interact

Use slider to control specific objects within scene.

Language Interact

Use natural language to interact with the 3D scenes.

Relevancy Map

Visualize relevancy map in the scene.

We adapt codes from some awesome repositories, including Nerfstudio, Omnigibson, Kplanes_nerfstudio, Lerf, and Conerf. Thanks for making the code available! 🤗

If you use this work or find it helpful, please consider citing: (bibtex)

@misc{livescene2024,
  title={LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control}, 
  author={Delin Qu, Qizhi Chen, Pingrui Zhang, Xianqiang Gao, Bin Zhao, Zhigang Wang, Dong Wang†, Xuelong Li†},
  year={2024},
  eprint={2406.16038},
  archivePrefix={arXiv},
}