StarVLA — SG2 Results

Key results

50 fully randomized trials per variant — object position (8 slots), lighting, background, pegboard location, and robot initial pose all randomized each episode. Task: "Put the yellow paint brush into the crate."

State-free success

66%

33 / 50 · 200k steps

State-cond. success

28%

14 / 50 · 30k steps

Gap

+38pp

pp = percentage points

state-free advantage

Self-correction ✦

14%

state-free only · 0% state-cond.

Wrong arm ✦

24%

state-cond. only · 0% state-free

Optimal checkpoints

200k / 30k

nostate / state steps

State-free — 200k steps

Full success26 (66%)

66%

Self-corrected ✦7 (14%)

14%

Object fell7 (14%)

14%

Outside basket6 (12%)

12%

Failed grasp4 (8%)

8%

Wrong arm0 (0%)

State-conditioned — 30k steps

Full success14 (28%)

28%

Failed grasp12 (24%)

24%

Wrong arm ✦12 (24%)

24%

Wrong object7 (14%)

14%

Outside basket3 (6%)

6%

Object fell2 (4%)

4%

✦ Notable findings

Self-correction (state-free only): In 14% of trials the robot approached the wrong object, then visually re-assessed at the next chunk boundary and redirected to the correct target. Zero self-corrections in the state-conditioned variant.

Wrong arm failure (state-conditioned only): In 24% of trials the robot attempted to grasp a left-side object with the right arm — physically impossible and never demonstrated in training. Joint state conditioning corrupts the spatial reasoning the VLM would otherwise perform from the image.

Dataset

FFW-SG2 demonstration dataset collected on the physical Robotis SG2 AI Worker via kinesthetic teaching.

Total episodes

2,000

demonstrations

Transitions

306,766

state-action pairs

Avg length

153

steps per episode

Control rate

10

fps recording

Action space

19

DOF joint targets

Resolution

376×672

RGB head camera

Task

"Put the yellow paint brush into the crate."

Robot identifies target among multiple objects on an 8-slot pegboard, selects correct arm by object position, grasps, lifts, transports, and places into crate.

Format

FormatLeRobot v2.1

Data filesParquet per episode

Video codecAV1 · 376×672 · 10 fps

Chunks2 × 1,000 episodes

Train splitAll 2,000 episodes

Video demonstrations

Task: "Put the yellow paint brush into the crate." — object position, lighting, background, pegboard location, and robot initial pose all randomized each episode.

State-free · 200k steps Two representative trials — success and self-correction

Successful trial

Robot correctly identifies the paint brush, selects the appropriate arm, grasps, and places it into the crate in a single continuous motion.

Self-correction trial

Robot initially approaches the wrong object, visually re-assesses at the next chunk boundary, then redirects to grasp the correct target.

State-conditioned · 30k steps Three representative trials — wrong arm and wrong object failures

Overconfident · 400k steps Both models beyond optimal checkpoint — erratic behaviour

Beyond optimal checkpoint

Both models produce overconfident, spatially erratic trajectories beyond their optimal checkpoints — 200k for state-free, 30k for state-conditioned. This video shows the state-conditioned model at 400k steps.

Video files are hosted via Git LFS in this repository.

How we implemented state-free and state-conditioned inference

Two StarVLA variants fine-tuned on identical data with identical hyperparameters. The only difference is whether proprioceptive joint state is provided as a model input — both during fine-tuning and at inference.

State-free variant

Camera image + language only. No joint positions provided at any stage.

Checkpoint: steps_200000_pytorch_model.pt
Training precision: bfloat16
state_dim config: 19 (unused)
Server flag: --use_bf16
Bridge: no state= argument

State-conditioned variant

Camera image + language + 19 joint positions sin-cos encoded into 38 dimensions.

Checkpoint: steps_30000_pytorch_model.pt
Training precision: bfloat16
state_dim config: 38 (sin-cos)
Server flag: use_bf16
Bridge: state=joint_pos argument

The one-line difference in the inference bridge

In scripts/inference/starvla_inference.py, find the get_action call:

State-free — omit state= argument

action_chunk_norm = starvla_client.get_action(
    image=cam_image,
    lang=task_description,
    # no state= argument
)

State-conditioned — add state= argument

action_chunk_norm = starvla_client.get_action(
    image=cam_image,
    lang=task_description,
    state=joint_pos,   # ← add this line
)

Matching server checkpoint to inference mode

Mode	Server command	Bridge call
State-free	`steps_200000`	no `state=`
State-conditioned	`steps_30000`	`state=joint_pos`

Sin-cos state encoding — why 19 becomes 38

Each joint angle is replaced by its sine and cosine, doubling the dimension. This preserves angular continuity — joints at +π and −π map to the same point in sin-cos space.

python

# 19 joint angles → 38-dim sin-cos vector
# Groups: left_arm (0-6), left_gripper (7),
#         right_arm (8-14), right_gripper (15), head (16-17), lift (18)
def sincos_encode(state):  # (19,) → (38,)
    parts = []
    for sl in [slice(0,7), slice(7,8), slice(8,15),
               slice(15,16), slice(16,18), slice(18,19)]:
        parts.append(np.sin(state[sl]))
        parts.append(np.cos(state[sl]))
    return np.concatenate(parts).astype(np.float32)

Why state-free works better

Fine-tuning — inference distribution mismatch

StarVLA was fine-tuned with obs: ["image_0"] and use_proprio: false. Although the training dataset records joint state for every transition, the pipeline deliberately excluded it. The state-free model matches its training distribution exactly at inference. The camera provides implicit proprioception — arm positions are visible, object distances can be estimated from apparent size, and gripper state is directly observable. Any drift in one 16-step chunk is corrected when the next chunk is conditioned on the updated visual scene.

Checkpoint selection finding

State-free: optimal at 200k steps

Trained to 400k but best at 200k. Beyond this the model overfits to training positions and fails to generalize to randomized object locations and lighting.

State-conditioned: optimal at 30k steps

The harder joint state + visual learning problem leads to earlier overfitting. Model degrades rapidly after 30k — producing overconfident, spatially erratic outputs.

Checkpoint selection analysis

Offline action prediction error across training checkpoints — evaluated on 10 dataset episodes without simulation. Shows where each model peaks and degrades.

State-free Peaks at 200k steps · degrades gradually · optimal MAE = 0.071 rad

State-free MAE RMSE normalized MAE vs checkpoint

MAE · RMSE · Normalized MAE vs training steps

Overall MAE with error bars

Per joint group MAE vs training steps

State-conditioned Peaks at 30k–50k steps · collapses rapidly · MAE triples by 100k steps

State-conditioned MAE RMSE normalized MAE vs checkpoint

MAE · RMSE · Normalized MAE vs training steps

State-conditioned overall MAE vs checkpoint

Overall MAE with error bars

State-conditioned per-group MAE vs checkpoint

Per joint group MAE vs training steps

Key findings

State-free: Improves steadily from 50k (MAE = 0.100 rad) to optimal at 200k (MAE = 0.071 rad), then degrades gradually — reaching 0.239 rad at 350k.

State-conditioned: Peaks at 30k–50k (MAE ≈ 0.047 rad) then collapses to 0.237 rad at 100k, plateauing at ~0.30 rad from 200k to 400k. By 400k normalized MAE reaches 32% of the action range.

Metric	State-free best (200k)	State-cond. best (30k)	State-free 400k	State-cond. 400k
Overall MAE (rad)	0.071	0.047	0.199	0.298
Overall RMSE (rad)	0.110	0.073	0.269	0.401
Normalized MAE (%)	8.1%	5.5%	22.3%	32.1%
Degradation	—		Gradual after 200k	Cliff edge after 50k

Web GUI for simulation control

FastAPI + WebSocket browser interface for submitting tasks, monitoring execution, and controlling the simulation without terminal access.

⌨

Task input

Free-form language instruction sent to the robot in real time.

⚡

Quick tasks

Pre-configured buttons for common task descriptions.

↺

Reset

Resets IsaacLab and randomizes all scene parameters.

⏸

Stop

Halts execution — robot holds current pose.

📡

Live status

WebSocket shows real-time execution status.

📋

Task history

Logs all submitted tasks with timestamps.

StarVLA web GUI showing task input, quick task buttons, reset and stop controls, and live status log

Web GUI running on http://localhost:8000 — task input, quick task buttons, reset/stop controls, and live WebSocket status log.

Start the GUI

bash

python scripts/server.py
# Open: http://localhost:8000

The GUI writes instructions to /tmp/groot_task.txt. The inference loop polls this every simulation step at 60 Hz. Special commands __RESET__ and __STOP__ control the environment without interrupting simulation.

StarVLA on Robotis SG2 AI Worker