Closed-loop VLA evaluation on the ROBOTIS FFW-SG2 AI Worker — simulation and
real-robot deployment of GR00T N1.6 and N1.7 for pegboard brush pick-and-place.
50 randomized trials per model in IsaacLab — object slot, table pose, lighting,
background, and robot joint state all randomized each episode.
Task: "Put the yellow paint brush into the crate."
GR00T N1.6 SimReal robot
Eagle-Block2A-2B-v2 · bfloat16 · Flash Attention
Sim success rate100% (50/50)
Real-robot success88% (88/100)
Sim fine-tune steps20,000 (of 40k)
Real fine-tune steps100,000 · DGX Spark
Deployed checkpointStep 20k (sim) · 100k (real)
Offline NormMAE0.94% (sim)
Semantic failures0
GR00T N1.7 Sim onlyEarly Access
Cosmos-Reason2-2B · bfloat16 · Flash Attention
Sim success rate100% (50/50)
Real-robot evalDeferred (EA release)
Sim fine-tune steps20,000
Deployed checkpointStep 20k
Offline NormMAE1.45%
Semantic failures0
NoteGA release pending
Key finding — offline MAE does not predict closed-loop success
GR00T N1.7 shows 1.86× higher offline NormMAE than N1.6 (1.45% vs 0.94%), yet both achieve identical 100% simulation success. This confirms that offline action prediction error is not a reliable proxy for closed-loop task performance in diffusion-based VLA policies.
On the physical robot, GR00T N1.6 achieves 88% success with zero semantic failures across 100 continuous trials — the policy correctly identifies the brush and activates the correct arm in every single trial.
Simulation results — all models compared
Model
Sim success
Real-robot
State input
Wrong arm
Wrong object
Offline MAE
GR00T N1.6
100% (50/50)
88% (100 trials)
✓ structured
0%
0%
0.94%
GR00T N1.7
100% (50/50)
Sim only (EA)
✓ structured
0%
0%
1.45%
StarVLA (no-state)
52% (26/50)
—
✗
0%
0%
6.61%
StarVLA (state)
28% (14/50)
—
sin-cos φ(q)
24%
14%
4.82%
OpenVLA
Excluded — action dimension collapse in dataset
Real-robot failure analysis
GR00T N1.6 · 100 continuous trials on physical FFW-SG2 · single uninterrupted session.
Human assistant repositions brush after each trial — no robot reset between trials.
88
Successful placements
Brush correctly grasped from pegboard and placed inside crate. Confirmed by human observer.
12
Total failures
10 grasp drops + 2 missed placements. Zero approach failures. Zero semantic failures.
10
Grasp drops (10%)
Brush contacted and briefly grasped but dropped during vertical extraction from pegboard ring holder. Distributed randomly across 100 trials — no concentration in early or late trials.
2
Missed placements (2%)
Brush successfully extracted but released near the crate opening rather than inside it. Placement precision issue — arm trajectory reaches the correct region but release point is slightly offset.
Failure breakdown — 100 real-robot trials
Successful placement88 (88%)
88%
Grasp drop during extraction10 (10%)
10%
Missed crate placement2 (2%)
2%
Approach failure0 (0%)
Wrong arm / semantic failure0 (0%)
Sim-to-real gap analysis
The 12 percentage point gap between simulation (100%) and real robot (88%) is entirely attributable to physical contact dynamics — grasp compliance and placement precision — that the IsaacLab rigid-body simulation does not model. The policy transfers perfectly at the semantic level: correct arm selection, correct object identification, and consistent behavior throughout all 100 continuous trials with no performance degradation.
Grasp drops are distributed randomly (not concentrated in early or late trials), confirming the failure is stochastic contact variability rather than policy drift over the continuous session.
Training analysis
Fine-tuning loss curves, offline MAE evaluation, and training comparison across simulation and real-world runs.
Real-world training loss — 100k steps · 1.0 epoch · initial 1.2901 → final 0.0019 (99.9% reduction)
Real-world vs simulation training comparison — normalized loss (log scale) and gradient norm over 100k steps
Model
Steps
Initial loss
Final loss
Reduction
Hardware
Epochs
N1.6 (sim)
40,000
1.3309
0.0053
99.6%
RTX 5090
~0.5
N1.7 (sim)
20,000
1.3884
0.0415
97.0%
RTX 5090
~0.8
N1.6 (real)
100,000
1.2901
0.0019
99.9%
DGX Spark
1.0
Datasets
Two datasets used in this work — simulation-generated for strategy learning and real-world teleoperation for physical appearance and contact distribution.
The web GUI can pause the running policy mid-execution (robot holds its current pose) and resume it on command, without restarting the simulation or the policy server.
