StarVLA — Installation Guide

⭐

Official StarVLA repository

This guide is based on the official StarVLA codebase by the HKUST team. Clone from github.com/starVLA/starVLA. This deployment on the Robotis SG2 AI Worker in NVIDIA IsaacLab was designed by Sanaullah.

Requirements

Component	Version	Notes
Docker	24.0+	Primary installation method
NVIDIA Container Toolkit	latest	Required for GPU access in Docker
CUDA driver	580+	CUDA 12.8 runtime inside container
GPU VRAM	32 GB recommended	RTX 5090 tested · 16 GB minimum
RAM	64 GB+	For dataset loading
Storage	200 GB+	Image + checkpoints + dataset
OS (host)	Ubuntu 22.04	Container base is also Ubuntu 22.04

GPU memory note

The policy server loads Qwen2.5-VL-3B (3B params) plus the DiT-B action head (150M params). With bfloat16 this requires ~16 GB VRAM. 32 GB is recommended for stable 60 Hz inference alongside IsaacLab.

Installation

🐳 Docker (recommended)

🐍 Conda (alternative)

1. Clone the official repository

bash

git clone https://github.com/starVLA/starVLA
cd starVLA

2. Build the Docker image

The Dockerfile uses nvidia/cuda:12.8.0-devel-ubuntu22.04 as base and installs Python 3.10, PyTorch nightly, FFmpeg, libav, and all dependencies automatically.

bash

docker build -t starvla:latest .

Build time

First build takes 15–30 minutes. PyTorch nightly (~4 GB) is downloaded fresh. Subsequent builds use the Docker layer cache.

3. Run the container

bash

docker run -it --gpus all \
    -v /path/to/checkpoints:/workspace/starVLA/results \
    -v /path/to/dataset:/workspace/starVLA/2-folder-one-camera-dataset \
    -v /path/to/pretrained:/workspace/starVLA/playground/Pretrained_models \
    -p 5555:5555 \
    -p 8000:8000 \
    --name starvla \
    starvla:latest

Resume a stopped container

docker start -ai starvla

What the Dockerfile installs

Component	Version / source
Base image	nvidia/cuda:12.8.0-devel-ubuntu22.04
Python	3.10 via apt-get
PyTorch	2.12.0.dev20260407+cu128 (nightly)
torchvision	0.27.0.dev20260407+cu128 (nightly)
FFmpeg + libavcodec	system — apt-get
All Python deps	starVLA_requirements.txt
StarVLA	pip install -e . (editable)

Docker is recommended

All experiments were run inside Docker. Conda may require additional system libraries (FFmpeg, libavcodec) to be installed manually via apt-get.

1. Clone the official repository

bash

git clone https://github.com/starVLA/starVLA
cd starVLA

2. Install system dependencies

bash

sudo apt-get install -y \
    libavcodec-dev libavformat-dev libswscale-dev \
    libavutil-dev ffmpeg \
    libgl1-mesa-glx libglib2.0-0

3. Create conda environment

bash

conda create -n starvla python=3.10 -y
conda activate starvla

4. Install PyTorch nightly with CUDA 12.8

bash

pip install torch==2.12.0.dev20260407+cu128 \
    torchvision==0.27.0.dev20260407+cu128 \
    --index-url https://download.pytorch.org/whl/nightly/cu128

5. Install StarVLA dependencies

bash

# Filter out torch/torchvision (already installed above)
grep -v -E "^(torch==|torchvision==|triton==|starVLA==)" \
    starVLA_requirements.txt > /tmp/filtered_reqs.txt
pip install -r /tmp/filtered_reqs.txt
pip install -e .

Dataset structure

text

2-folder-one-camera-dataset/
├── meta/
│   ├── info.json           # 2000 episodes, 10 fps, feature definitions
│   ├── modality.json       # joint group layout
│   └── stats_gr00t.json    # normalization statistics (min/max/mean/std)
├── data/
│   ├── chunk-000/          # episodes 0–999
│   │   ├── episode_000000.parquet
│   │   └── ...
│   └── chunk-001/          # episodes 1000–1999
└── videos/
    └── chunk-000/
        └── observation.images.cam_head/
            ├── episode_000000.mp4  # AV1 codec, 376×672 @ 10 fps
            └── ...

Field	Shape	Description
`action`	(19,)	Joint position targets — float32
`observation.state`	(19,)	Current joint positions — float32
`observation.images.cam_head`	376×672×3	RGB video @ 10 fps — AV1 codec
`task_index`	int64	Task ID — 0 = paint brush into crate

Video decoding

Videos use AV1 encoding. The Docker image installs libavcodec-dev and av==12.3.0 (PyAV) automatically. OpenCV cannot decode AV1 without hardware support.

Fine-tuning

All fine-tuning scripts are in examples/FFWSG2/train_files/. The VLM backbone (qwen_vl_interface) is frozen throughout — only the DiT-B action head is trained.

State-free (recommended)

State-conditioned 400k

State-conditioned 30k

Script: examples/FFWSG2/train_files/run_ffw_sg2_train.sh

bash

export WANDB_MODE=disabled
export MASTER_ADDR=localhost
export MASTER_PORT=29500
export RANK=0 WORLD_SIZE=1 LOCAL_RANK=0
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

accelerate launch \
  --mixed_precision bf16 --num_processes 1 \
  starVLA/training/train_starvla.py \
  --config_yaml ./examples/FFWSG2/train_files/starvla_ffw_sg2.yaml \
  --framework.name QwenGR00T \
  --framework.qwenvl.base_vlm playground/Pretrained_models/Qwen2.5-VL-3B-Instruct-Action \
  --datasets.vla_data.data_root_dir . \
  --datasets.vla_data.data_mix ffw_sg2 \
  --datasets.vla_data.per_device_batch_size 1 \
  --datasets.vla_data.video_backend pyav \
  --trainer.freeze_modules 'qwen_vl_interface' \
  --trainer.max_train_steps 400000 \
  --trainer.save_interval 5000 \
  --trainer.logging_frequency 10 \
  --trainer.eval_interval 100 \
  --run_root_dir ./results/Checkpoints \
  --run_id ffw_sg2_400k_fp32

Optimal checkpoint: 200,000 steps

Despite training to 400k, use steps_200000_pytorch_model.pt for deployment. Beyond 200k the model overfits and produces overconfident trajectories on randomized scenes.

Script: examples/FFWSG2/train_files/run_ffw_sg2_state_train.sh

bash

export WANDB_MODE=disabled
export MASTER_ADDR=localhost
export MASTER_PORT=29500
export RANK=0 WORLD_SIZE=1 LOCAL_RANK=0
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

accelerate launch \
  --mixed_precision bf16 --num_processes 1 \
  starVLA/training/train_starvla.py \
  --config_yaml ./examples/FFWSG2/train_files/starvla_ffw_sg2_state.yaml \
  --framework.name QwenGR00T \
  --framework.qwenvl.base_vlm playground/Pretrained_models/Qwen2.5-VL-3B-Instruct-Action \
  --datasets.vla_data.data_root_dir . \
  --datasets.vla_data.data_mix ffw_sg2 \
  --datasets.vla_data.per_device_batch_size 1 \
  --datasets.vla_data.video_backend pyav \
  --trainer.freeze_modules 'qwen_vl_interface' \
  --trainer.max_train_steps 400000 \
  --trainer.save_interval 5000 \
  --trainer.logging_frequency 10 \
  --trainer.eval_interval 100 \
  --run_root_dir ./results/Checkpoints \
  --run_id ffw_sg2_400k_state_fp32

Optimal checkpoint: 30,000 steps

Despite training to 400k, use steps_30000_pytorch_model.pt. The model degrades rapidly after 30k due to the harder joint state + visual learning problem.

Script: examples/FFWSG2/train_files/run_ffw_sg2_state_30k.sh — trains directly to 30k steps, saving every 2k.

bash

export WANDB_MODE=disabled
export MASTER_ADDR=localhost
export MASTER_PORT=29500
export RANK=0 WORLD_SIZE=1 LOCAL_RANK=0
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

accelerate launch \
  --mixed_precision bf16 --num_processes 1 \
  starVLA/training/train_starvla.py \
  --config_yaml ./examples/FFWSG2/train_files/starvla_ffw_sg2_state.yaml \
  --framework.name QwenGR00T \
  --framework.qwenvl.base_vlm playground/Pretrained_models/Qwen2.5-VL-3B-Instruct-Action \
  --datasets.vla_data.data_root_dir . \
  --datasets.vla_data.data_mix ffw_sg2 \
  --datasets.vla_data.per_device_batch_size 1 \
  --datasets.vla_data.video_backend pyav \
  --trainer.freeze_modules 'qwen_vl_interface' \
  --trainer.max_train_steps 30000 \
  --trainer.save_interval 2000 \
  --trainer.logging_frequency 10 \
  --trainer.eval_interval 100 \
  --run_root_dir ./results/Checkpoints \
  --run_id ffw_sg2_30k_state

Fastest route to a deployable state-conditioned model

Use this script if you only need the state-conditioned model — it stops at the optimal checkpoint automatically.

Training configuration summary

Hyperparameter	State-free	State-conditioned
Config yaml	`starvla_ffw_sg2.yaml`	`starvla_ffw_sg2_state.yaml`
run_id	`ffw_sg2_400k_fp32`	`ffw_sg2_400k_state_fp32`
Training steps	400,000	400,000 (or 30,000)
Optimal checkpoint	200,000	30,000
Save interval	5,000	5,000 / 2,000 steps
Batch size	1	1
Mixed precision	bf16	bf16
VLM backbone	frozen	frozen
Video backend	pyav	pyav
Hardware	1× NVIDIA RTX 5090 32GB

Checkpoint structure

text

results/Checkpoints/
├── ffw_sg2_400k_fp32/          # state-free 400k run
│   ├── config.yaml
│   ├── dataset_statistics.json
│   ├── summary.jsonl
│   ├── checkpoints/
│   │   ├── steps_5000_pytorch_model.pt
│   │   ├── steps_10000_pytorch_model.pt
│   │   ├── ...                 # every 5000 steps
│   │   └── steps_400000_pytorch_model.pt
│   └── final_model/
│       └── pytorch_model.pt
├── ffw_sg2_400k_state_fp32/    # state-conditioned 400k run
│   └── # same structure
└── ffw_sg2_30k_state/          # state-conditioned 30k run
    └── checkpoints/
        ├── steps_2000_pytorch_model.pt
        ├── ...                 # every 2000 steps
        └── steps_30000_pytorch_model.pt

Policy server

The policy server runs inside the Docker container and exposes a WebSocket endpoint on port 5555, mapped to the host via -p 5555:5555.

State-free server

State-conditioned server

bash — inside Docker container

pkill -f server_policy && sleep 3

python deployment/model_server/server_policy.py \
    --ckpt_path results/Checkpoints/ffw_sg2_400k_fp32/\
checkpoints/steps_200000_pytorch_model.pt \
    --port 5555 \
    --use_bf16

bash — inside Docker container

pkill -f server_policy && sleep 3

# No --use_bf16 flag — state model uses float32
python deployment/model_server/server_policy.py \
    --ckpt_path results/Checkpoints/ffw_sg2_400k_state_fp32/\
checkpoints/steps_30000_pytorch_model.pt \
    --port 5555

Server ready

Look for INFO: server listening on 0.0.0.0:5555. The WebSocket handshake error on first connect is normal.

Argument	Default	Description
`--ckpt_path`	required	Path to .pt checkpoint file
`--port`	10093	WebSocket port
`--use_bf16`	False	bfloat16 precision — use for nostate model only
`--idle_timeout`	1800	Seconds before idle shutdown

IsaacLab inference bridge

Runs outside the Docker container in the IsaacLab environment. Connects to the policy server at 172.17.0.2:5555 (Docker bridge network IP).

bash — IsaacLab environment (host)

cd /workspace/robotis_lab

python scripts/inference/starvla_inference.py \
    --task RobotisSG2-PaintBrush-v0 \
    --starvla_host 172.17.0.2 \
    --starvla_port 5555 \
    --step_hz 60 \
    --chunk_size 16

Argument	Default	Description
`--task`	required	IsaacLab registered task name
`--starvla_host`	172.17.0.2	Docker bridge network IP of container
`--starvla_port`	5555	WebSocket port
`--step_hz`	60	Control rate in Hz
`--chunk_size`	16	Action chunk length in steps
`--robot_type`	FFW_SG2	Robot platform

Three terminals required

Terminal 1 (Docker): policy server · Terminal 2 (host): inference bridge · Terminal 3 (host): web GUI. Start in this order.

Web GUI

bash

python scripts/server.py
# Open browser: http://localhost:8000

Endpoint	Method	Description
`/api/task`	POST	Submit natural language task instruction
`/api/reset`	POST	Reset IsaacLab environment
`/api/stop`	POST	Stop current task execution
`/api/status`	GET	Current robot and server status
`/ws`	WebSocket	Real-time status updates

Task IPC

The GUI writes to /tmp/groot_task.txt. The inference bridge polls this every simulation step at 60 Hz. __RESET__ and __STOP__ are special control commands.

StarVLA Installation Guide

Requirements

Installation

1. Clone the official repository

2. Build the Docker image

3. Run the container

What the Dockerfile installs

1. Clone the official repository

2. Install system dependencies

3. Create conda environment

4. Install PyTorch nightly with CUDA 12.8

5. Install StarVLA dependencies

Dataset structure

Fine-tuning

Training configuration summary

Checkpoint structure

Policy server

IsaacLab inference bridge

Web GUI