Code for ICLR 2024 spotlight paper Grounding Language Plans in Demonstrations through Counterfactual Perturbations

conda env create -f environment.yml
conda activate glide

# (Optional) Add kernel for jupyter notebook 
python -m ipykernel install --user --name glide

The navigation domain serves as a 2D abstraction of the modal structure for multi-step manipulation tasks, where each polygon represents a different mode with its boundary showing the constraint. Task success depends on starting in the free space (white) and traversing through the polygon sequence consecutively until reaching the final polygon. Invalid mode transitions, such as direct crossings from the white into colored polygons other than the first polygon entered by demonstrations, will cause execution failures.

To generate a new task of a sequence of randomly sampled connected convex polygons, run

python draw_demo.py --out_dir data_gen/tmp --seed 5 --n_poly 5 

n_poly specifies how many polygons there are. Draw a couple demonstrations by clicking and dragging your mouse.

Demonstration by drawing	A small dataset of successful trajectories

Augment the demonstrations with counterfactual perturbations

python augment_demo.py --dir data_gen/tmp --num 500

num specifies how many trajectories (both successful and unsuccessful after perturbations) to generate by perturbing the demonstration replays.

To train with the generated data, run

python train_classifier.py --dir data_gen/polygon_4 --n_mds 5

and visualize the learned modes on Wandb or locally at weights/{wandb_run_name}. Wandb_run_name is printed out upon running the training script. n_mds specifies the number of modes (number of polygons + the white region).

Please refer to tutorial.ipynb