> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/NVIDIA/Isaac-GR00T/llms.txt
> Use this file to discover all available pages before exploring further.

# LIBERO benchmark

> Lifelong robot learning benchmark with multiple task suites

LIBERO (Lifelong robot learning benchmark) provides multiple task suites for studying knowledge transfer in robot learning, including spatial reasoning, object generalization, goal-conditioned learning, and long-horizon multi-step tasks.

For more information, see the [official website](https://libero-project.github.io/main.html).

## Benchmark results

GR00T N1.6 achieves state-of-the-art performance across all LIBERO suites:

| Suite     | Success rate     | Training steps | Batch size |
| --------- | ---------------- | -------------- | ---------- |
| Spatial   | 195/200 (97.65%) | 20K            | 640        |
| Goal      | 195/200 (97.5%)  | 20K            | 640        |
| Object    | 197/200 (98.45%) | 20K            | 640        |
| 10 (Long) | 189/200 (94.35%) | 20K            | 640        |

## Fine-tuning

<Note>
  Remember to set `WANDB_API_KEY` since `--use-wandb` is enabled by default. If you don't have a Weights & Biases account, remove this argument from the training scripts.
</Note>

### LIBERO spatial

<Steps>
  <Step title="Download dataset">
    ```bash theme={null}
    huggingface-cli download \
        --repo-type dataset IPEC-COMMUNITY/libero_spatial_no_noops_1.0.0_lerobot \
        --local-dir examples/LIBERO/libero_spatial_no_noops_1.0.0_lerobot/
    ```
  </Step>

  <Step title="Copy modality configuration">
    ```bash theme={null}
    cp -r examples/LIBERO/modality.json examples/LIBERO/libero_spatial_no_noops_1.0.0_lerobot/meta/
    ```
  </Step>

  <Step title="Run fine-tuning">
    ```bash theme={null}
    uv run bash examples/LIBERO/finetune_libero_spatial.sh
    ```
  </Step>
</Steps>

### LIBERO goal

<Steps>
  <Step title="Download dataset">
    ```bash theme={null}
    huggingface-cli download \
        --repo-type dataset IPEC-COMMUNITY/libero_goal_no_noops_1.0.0_lerobot \
        --local-dir examples/LIBERO/libero_goal_no_noops_1.0.0_lerobot/
    ```
  </Step>

  <Step title="Apply patches">
    ```bash theme={null}
    # Copy modality configuration
    cp -r examples/LIBERO/modality.json examples/LIBERO/libero_goal_no_noops_1.0.0_lerobot/meta/

    # Patch corrupted episode
    cp examples/LIBERO/patches/episode_000082.mp4 \
       examples/LIBERO/libero_goal_no_noops_1.0.0_lerobot/videos/chunk-000/observation.images.wrist_image/
    ```
  </Step>

  <Step title="Run fine-tuning">
    ```bash theme={null}
    uv run bash examples/LIBERO/finetune_libero_goal.sh
    ```
  </Step>
</Steps>

### LIBERO object

<Steps>
  <Step title="Download dataset">
    ```bash theme={null}
    huggingface-cli download \
        --repo-type dataset IPEC-COMMUNITY/libero_object_no_noops_1.0.0_lerobot \
        --local-dir examples/LIBERO/libero_object_no_noops_1.0.0_lerobot/
    ```
  </Step>

  <Step title="Copy modality configuration">
    ```bash theme={null}
    cp -r examples/LIBERO/modality.json examples/LIBERO/libero_object_no_noops_1.0.0_lerobot/meta/
    ```
  </Step>

  <Step title="Run fine-tuning">
    ```bash theme={null}
    uv run bash examples/LIBERO/finetune_libero_object.sh
    ```
  </Step>
</Steps>

### LIBERO 10 (long)

<Steps>
  <Step title="Download dataset">
    ```bash theme={null}
    huggingface-cli download \
        --repo-type dataset IPEC-COMMUNITY/libero_10_no_noops_1.0.0_lerobot \
        --local-dir examples/LIBERO/libero_10_no_noops_1.0.0_lerobot/
    ```
  </Step>

  <Step title="Copy modality configuration">
    ```bash theme={null}
    cp -r examples/LIBERO/modality.json examples/LIBERO/libero_10_no_noops_1.0.0_lerobot/meta/
    ```
  </Step>

  <Step title="Run fine-tuning">
    ```bash theme={null}
    uv run bash examples/LIBERO/finetune_libero_10.sh
    ```
  </Step>
</Steps>

## Evaluation

### Setup environment

Install the required dependencies (only needs to be done once):

```bash theme={null}
sudo apt update
sudo apt install libegl1-mesa-dev libglu1-mesa
bash gr00t/eval/sim/LIBERO/setup_libero.sh
```

### Run evaluation

<Steps>
  <Step title="Start policy server">
    In Terminal 1:

    ```bash theme={null}
    uv run python gr00t/eval/run_gr00t_server.py \
        --model-path /tmp/libero_spatial/checkpoint-20000/ \
        --embodiment-tag LIBERO_PANDA \
        --use-sim-policy-wrapper
    ```
  </Step>

  <Step title="Start evaluation client">
    In Terminal 2:

    ```bash theme={null}
    gr00t/eval/sim/LIBERO/libero_uv/.venv/bin/python gr00t/eval/rollout_policy.py \
        --n_episodes 10 \
        --policy_client_host 127.0.0.1 \
        --policy_client_port 5555 \
        --max_episode_steps=720 \
        --env_name libero_sim/KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it \
        --n_action_steps 8 \
        --n_envs 5
    ```
  </Step>
</Steps>

## Task lists

### Spatial suite (10 tasks)

Spatial reasoning tasks with varying object locations:

* `libero_sim/pick_up_the_black_bowl_between_the_plate_and_the_ramekin_and_place_it_on_the_plate`
* `libero_sim/pick_up_the_black_bowl_next_to_the_ramekin_and_place_it_on_the_plate`
* `libero_sim/pick_up_the_black_bowl_from_table_center_and_place_it_on_the_plate`
* `libero_sim/pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate`
* `libero_sim/pick_up_the_black_bowl_in_the_top_drawer_of_the_wooden_cabinet_and_place_it_on_the_plate`
* `libero_sim/pick_up_the_black_bowl_on_the_ramekin_and_place_it_on_the_plate`
* `libero_sim/pick_up_the_black_bowl_next_to_the_cookie_box_and_place_it_on_the_plate`
* `libero_sim/pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate`
* `libero_sim/pick_up_the_black_bowl_next_to_the_plate_and_place_it_on_the_plate`
* `libero_sim/pick_up_the_black_bowl_on_the_wooden_cabinet_and_place_it_on_the_plate`

### Goal suite (10 tasks)

Goal-conditioned manipulation tasks:

* `libero_sim/open_the_middle_drawer_of_the_cabinet`
* `libero_sim/put_the_bowl_on_the_stove`
* `libero_sim/put_the_wine_bottle_on_top_of_the_cabinet`
* `libero_sim/open_the_top_drawer_and_put_the_bowl_inside`
* `libero_sim/put_the_bowl_on_top_of_the_cabinet`
* `libero_sim/push_the_plate_to_the_front_of_the_stove`
* `libero_sim/put_the_cream_cheese_in_the_bowl`
* `libero_sim/turn_on_the_stove`
* `libero_sim/put_the_bowl_on_the_plate`
* `libero_sim/put_the_wine_bottle_on_the_rack`

### Object suite (10 tasks)

Object generalization with different items:

* `libero_sim/pick_up_the_alphabet_soup_and_place_it_in_the_basket`
* `libero_sim/pick_up_the_cream_cheese_and_place_it_in_the_basket`
* `libero_sim/pick_up_the_salad_dressing_and_place_it_in_the_basket`
* `libero_sim/pick_up_the_bbq_sauce_and_place_it_in_the_basket`
* `libero_sim/pick_up_the_ketchup_and_place_it_in_the_basket`
* `libero_sim/pick_up_the_tomato_sauce_and_place_it_in_the_basket`
* `libero_sim/pick_up_the_butter_and_place_it_in_the_basket`
* `libero_sim/pick_up_the_milk_and_place_it_in_the_basket`
* `libero_sim/pick_up_the_chocolate_pudding_and_place_it_in_the_basket`
* `libero_sim/pick_up_the_orange_juice_and_place_it_in_the_basket`

### Long suite (10 tasks)

Long-horizon multi-step tasks:

* `libero_sim/LIVING_ROOM_SCENE2_put_both_the_alphabet_soup_and_the_tomato_sauce_in_the_basket`
* `libero_sim/LIVING_ROOM_SCENE2_put_both_the_cream_cheese_box_and_the_butter_in_the_basket`
* `libero_sim/KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it`
* `libero_sim/KITCHEN_SCENE4_put_the_black_bowl_in_the_bottom_drawer_of_the_cabinet_and_close_it`
* `libero_sim/LIVING_ROOM_SCENE5_put_the_white_mug_on_the_left_plate_and_put_the_yellow_and_white_mug_on_the_right_plate`
* `libero_sim/STUDY_SCENE1_pick_up_the_book_and_place_it_in_the_back_compartment_of_the_caddy`
* `libero_sim/LIVING_ROOM_SCENE6_put_the_white_mug_on_the_plate_and_put_the_chocolate_pudding_to_the_right_of_the_plate`
* `libero_sim/LIVING_ROOM_SCENE1_put_both_the_alphabet_soup_and_the_cream_cheese_box_in_the_basket`
* `libero_sim/KITCHEN_SCENE8_put_both_moka_pots_on_the_stove`
* `libero_sim/KITCHEN_SCENE6_put_the_yellow_and_white_mug_in_the_microwave_and_close_it`