---
title: spotter-robotwin-150
canonical_url: "https://www.modelscope.ai/datasets/seamoke/spotter-robotwin-150"
md_url: "https://www.modelscope.ai/datasets/seamoke/spotter-robotwin-150.md"
repository: seamoke/spotter-robotwin-150
last_updated: 2026-10-19
license: "Apache License 2.0"
storage_size: "8.9 GB"
downloads: 59
stars: 0
---

# spotter-robotwin-150

> spotter-robotwin-150 - An open-source dataset by seamoke on ModelScope. Spotter vs. a VLM-led agent on RoboTwin 2.0 — 150 episodes

seamoke/spotter-robotwin-150 is a dataset on ModelScope. totalling 8.9 GB. licensed under Apache License 2.0.

- **Repository**: seamoke/spotter-robotwin-150
- **License**: Apache License 2.0
- **Storage size**: 8.9 GB
- **Downloads**: 59
- **Stars**: 0
- **Last updated**: 2026-10-19

Source: https://www.modelscope.ai/datasets/seamoke/spotter-robotwin-150

---

# Spotter vs. a VLM-led agent on RoboTwin 2.0 — 150 episodes

The 150 RoboTwin 2.0 episodes behind the Spotter vs. VLM-led agent figure: the pi0.5 policy alone, Spotter, and a
VLM-led agent, each with two LLMs, on the same episodes.

| Arm | LLM | Success | Rate |
|---|---|---:|---:|
| Robot policy alone (pi0.5) | – | 72 / 150 | 48.0% |
| Spotter | Qwen3.8-27B-FP8 | 79 / 150 | 52.7% |
| Spotter | GPT-6-astra | 84 / 150 | 56.0% |
| VLM-led agent | Qwen3.8-27B-FP8 | 63 / 150 | 42.0% |
| VLM-led agent | GPT-6-astra | 84 / 150 | 56.0% |

## Setup

- **Simulator:** RoboTwin 2.0, aloha-agilex dual arm, `demo_randomized` scenes; 50 tasks × 3 episodes
  (`episode_set.txt`, one `task:seed:episode` per line). Each episode maps to a fixed RoboTwin seed (column
  `robotwin_seed`); every arm uses the same seed per episode.
- **Policy:** pi0.5, checkpoint `motus-robotics/pi0.5_robotwin2` (train config `pi05_robotwin2_clean_randomized`, step 40000).
- **Step budget:** 1.8 × the official per-task limit for every arm, the LLM's own robot moves included. No rollbacks or resets.
- **Spotter:** the policy drives; a screener checks every window of 2 policy chunks, a judge LLM decides whether to
  intervene, and repairs a fault with short motion plans before handing control back. At most 5 interventions per
  episode, judge history 10 turns, context compaction at 40k tokens, 1-shot example. Qwen thinks in repair turns only,
  up to 5000 tokens. With GPT, the judge may also end an episode early when it judges it unrecoverable (counted as a failure).
- **VLM-led agent:** the LLM drives the episode turn by turn and calls the pi0.5 policy as a tool (up to 96 steps per
  call) or moves the arm itself; it sees the task's success signal and may stop when it judges the task done or
  impossible. Up to 100 turns. Qwen thinks on every call, up to 5000 tokens, history 10 turns.
- **Code:** github.com/zqc3117/Spotter.

## Comparing the arms

pi0.5 on RoboTwin is not bit-reproducible across runs, so episodes are independent samples per arm. Compare the arms
by aggregate success rate.

## Files

| File | Content |
|---|---|
| `per_episode.csv` | One row per episode: task, episode, RoboTwin seed, instruction, success of each arm, Spotter interventions, simulator steps, VLM-agent LLM calls, and the run folder holding each arm's records |
| `per_task.csv` | Successes per task for each arm |
| `<arm>_results.jsonl` | Raw result rows of each arm (`control`, `spotter_qwen`, `spotter_gpt`, `vlm_agent_qwen`, `vlm_agent_gpt`) |

Full records (per-chunk camera frames, the LLMs' prompts and replies, tool calls, simulator truth files) are in
`robotwin_fig150_full.tar.part-*`: check with `md5sum -c robotwin_fig150_full.md5`, then
`cat robotwin_fig150_full.tar.part-* | tar -xf -`. Inside, one folder per arm, then one per run,
`lane*/<episode>/` per episode (`jd-<task>_s0_ep<n>` for the policy-alone and Spotter arms, `<task>_0_<n>` for the
VLM-led agent). The policy-alone arm keeps only its `ctrl/` frames.

**Chat histories.** Pictures that were embedded in the saved chat histories (base64) are replaced by the text
`[image omitted from this archive to save space; the same pictures are saved as PNG files in this episode folder]`;
the same pictures are stored as PNG files in each episode folder.