LIBERO benchmark

Lifelong robot learning benchmark with multiple task suites

LIBERO (Lifelong robot learning benchmark) provides multiple task suites for studying knowledge transfer in robot learning, including spatial reasoning, object generalization, goal-conditioned learning, and long-horizon multi-step tasks.

For more information, see the official website.

Benchmark results

GR00T N1.6 achieves state-of-the-art performance across all LIBERO suites:

Suite	Success rate	Training steps	Batch size
Spatial	195/200 (97.65%)	20K	640
Goal	195/200 (97.5%)	20K	640
Object	197/200 (98.45%)	20K	640
10 (Long)	189/200 (94.35%)	20K	640

Fine-tuning

Remember to set WANDB_API_KEY since --use-wandb is enabled by default. If you don’t have a Weights & Biases account, remove this argument from the training scripts.

LIBERO spatial

1. Download dataset

   huggingface-cli download \
       --repo-type dataset IPEC-COMMUNITY/libero_spatial_no_noops_1.0.0_lerobot \
       --local-dir examples/LIBERO/libero_spatial_no_noops_1.0.0_lerobot/
   
2. Copy modality configuration

   cp -r examples/LIBERO/modality.json examples/LIBERO/libero_spatial_no_noops_1.0.0_lerobot/meta/
   
3. Run fine-tuning

   uv run bash examples/LIBERO/finetune_libero_spatial.sh
   

LIBERO goal

1. Download dataset

   huggingface-cli download \
       --repo-type dataset IPEC-COMMUNITY/libero_goal_no_noops_1.0.0_lerobot \
       --local-dir examples/LIBERO/libero_goal_no_noops_1.0.0_lerobot/
   
2. Apply patches

   # Copy modality configuration
   cp -r examples/LIBERO/modality.json examples/LIBERO/libero_goal_no_noops_1.0.0_lerobot/meta/
   
   # Patch corrupted episode
   cp examples/LIBERO/patches/episode_000082.mp4 \
      examples/LIBERO/libero_goal_no_noops_1.0.0_lerobot/videos/chunk-000/observation.images.wrist_image/
   
3. Run fine-tuning

   uv run bash examples/LIBERO/finetune_libero_goal.sh
   

LIBERO object

1. Download dataset

   huggingface-cli download \
       --repo-type dataset IPEC-COMMUNITY/libero_object_no_noops_1.0.0_lerobot \
       --local-dir examples/LIBERO/libero_object_no_noops_1.0.0_lerobot/
   
2. Copy modality configuration

   cp -r examples/LIBERO/modality.json examples/LIBERO/libero_object_no_noops_1.0.0_lerobot/meta/
   
3. Run fine-tuning

   uv run bash examples/LIBERO/finetune_libero_object.sh
   

LIBERO 10 (long)

1. Download dataset

   huggingface-cli download \
       --repo-type dataset IPEC-COMMUNITY/libero_10_no_noops_1.0.0_lerobot \
       --local-dir examples/LIBERO/libero_10_no_noops_1.0.0_lerobot/
   
2. Copy modality configuration

   cp -r examples/LIBERO/modality.json examples/LIBERO/libero_10_no_noops_1.0.0_lerobot/meta/
   
3. Run fine-tuning

   uv run bash examples/LIBERO/finetune_libero_10.sh
   

Evaluation

Setup environment

Install the required dependencies (only needs to be done once):

sudo apt update
sudo apt install libegl1-mesa-dev libglu1-mesa
bash gr00t/eval/sim/LIBERO/setup_libero.sh

Run evaluation

1. Start policy server

   In Terminal 1:

   uv run python gr00t/eval/run_gr00t_server.py \
       --model-path /tmp/libero_spatial/checkpoint-20000/ \
       --embodiment-tag LIBERO_PANDA \
       --use-sim-policy-wrapper
   
2. Start evaluation client

   In Terminal 2:

   gr00t/eval/sim/LIBERO/libero_uv/.venv/bin/python gr00t/eval/rollout_policy.py \
       --n_episodes 10 \
       --policy_client_host 127.0.0.1 \
       --policy_client_port 5555 \
       --max_episode_steps=720 \
       --env_name libero_sim/KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it \
       --n_action_steps 8 \
       --n_envs 5
   

Task lists

Spatial suite (10 tasks)

Spatial reasoning tasks with varying object locations:

• libero_sim/pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate
• libero_sim/pick_up_the_black_bowl_next_to_the_ramekin_and_place_it_on_the_plate
• libero_sim/pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate
• libero_sim/pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate
• libero_sim/pick_up_the_black_bowl_in_the_top_drawer_of_the_wooden_cabinet_and_place_it_on_the_plate
• libero_sim/pick_up_the_black_bowl_on_the_ramekin_and_place_it_on_the_plate
• libero_sim/pick_up_the_black_bowl_next_to_the_cookie_box_and_place_it_on_the_plate
• libero_sim/pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate
• libero_sim/pick_up_the_black_bowl_next_to_the_plate_and_place_it_on_the_plate
• libero_sim/pick_up_the_black_bowl_on_the_wooden_cabinet_and_place_it_on_the_plate

Goal suite (10 tasks)

Goal-conditioned manipulation tasks:

• libero_sim/open_the_middle_drawer_of_the_cabinet
• libero_sim/put_the_bowl_on_the_stove
• libero_sim/put_the_wine_bottle_on_top_of_the_cabinet
• libero_sim/open_the_top_drawer_and_put_the_bowl_inside
• libero_sim/put_the_bowl_on_top_of_the_cabinet
• libero_sim/push_the_plate_to_the_front_of_the_stove
• libero_sim/put_the_cream_cheese_in_the_bowl
• libero_sim/turn_on_the_stove
• libero_sim/put_the_bowl_on_the_plate
• libero_sim/put_the_wine_bottle_on_the_rack

Object suite (10 tasks)

Object generalization with different items:

• libero_sim/pick_up_the_alphabet_soup_and_place_it_in_the_basket
• libero_sim/pick_up_the_cream_cheese_and_place_it_in_the_basket
• libero_sim/pick_up_the_salad_dressing_and_place_it_in_the_basket
• libero_sim/pick_up_the_bbq_sauce_and_place_it_in_the_basket
• libero_sim/pick_up_the_ketchup_and_place_it_in_the_basket
• libero_sim/pick_up_the_tomato_sauce_and_place_it_in_the_basket
• libero_sim/pick_up_the_butter_and_place_it_in_the_basket
• libero_sim/pick_up_the_milk_and_place_it_in_the_basket
• libero_sim/pick_up_the_chocolate_pudding_and_place_it_in_the_basket
• libero_sim/pick_up_the_orange_juice_and_place_it_in_the_basket

Long suite (10 tasks)

Long-horizon multi-step tasks:

• libero_sim/LIVING_ROOM_SCENE2_put_both_the_alphabet_soup_and_the_tomato_sauce_in_the_basket
• libero_sim/LIVING_ROOM_SCENE2_put_both_the_cream_cheese_box_and_the_butter_in_the_basket
• libero_sim/KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it
• libero_sim/KITCHEN_SCENE4_put_the_black_bowl_in_the_bottom_drawer_of_the_cabinet_and_close_it
• libero_sim/LIVING_ROOM_SCENE5_put_the_white_mug_on_the_left_plate_and_put_the_yellow_and_white_mug_on_the_right_plate
• libero_sim/STUDY_SCENE1_pick_up_the_book_and_place_it_in_the_back_compartment_of_the_caddy
• libero_sim/LIVING_ROOM_SCENE6_put_the_white_mug_on_the_plate_and_put_the_chocolate_pudding_to_the_right_of_the_plate
• libero_sim/LIVING_ROOM_SCENE1_put_both_the_alphabet_soup_and_the_cream_cheese_box_in_the_basket
• libero_sim/KITCHEN_SCENE8_put_both_moka_pots_on_the_stove
• libero_sim/KITCHEN_SCENE6_put_the_yellow_and_white_mug_in_the_microwave_and_close_it