---
title: RecreationBench
canonical_url: "https://www.modelscope.ai/datasets/Qwen/RecreationBench"
md_url: "https://www.modelscope.ai/datasets/Qwen/RecreationBench.md"
repository: Qwen/RecreationBench
last_updated: 2026-09-21
license: other
storage_size: "22 GB"
downloads: 49
stars: 0
---

# RecreationBench

> RecreationBench - An open-source dataset by Qwen on ModelScope. GitHub · Website · Leaderboard · Report ↗

Qwen/RecreationBench is a dataset on ModelScope. totalling 22 GB. licensed under other.

- **Repository**: Qwen/RecreationBench
- **License**: other
- **Storage size**: 22 GB
- **Downloads**: 49
- **Stars**: 0
- **Last updated**: 2026-09-21

Source: https://www.modelscope.ai/datasets/Qwen/RecreationBench

---

# RecreationBench

[GitHub](https://github.com/QwenLM/RecreationWorld) · [Website](https://recreation-bench.cc) · [Leaderboard](https://recreation-bench.cc/#leaderboard) · [Report ↗](https://github.com/QwenLM/RecreationWorld/blob/HEAD/assets/report.pdf)

RecreationBench is a benchmark of **250 application-recreation tasks across Ubuntu, macOS, Windows, Android, and Web**. It is the held-out evaluation suite of **RecreationWorld**, a framework for scalable and verifiable environments for hybrid computer-use agents.

Given a high-level recreation request and interactive access to a running reference application, an agent explores its behavior, implements a new application, and verifies its own result. The agent can combine GUI interaction with coding tools, but cannot access the reference source code or evaluation tests. Evaluation measures observable functionality and visual fidelity rather than source-code similarity.

This dataset provides the task descriptors, reference preparation assets, tests, fixtures, and available visual-evaluation annotations. The execution framework is maintained in the GitHub repository linked above.

## Benchmark Statistics

| Platform | Directory | Tasks | Evaluation interface |
| --- | --- | ---: | --- |
| Ubuntu | `ubuntu/` | 50 | AT-SPI |
| macOS | `macos/` | 50 | AXUIElement |
| Windows | `windows/` | 50 | UI Automation |
| Android | `android/` | 50 | UiAutomator |
| Web | `web/` | 50 | Browser assertions |
| **Total** | | **250** | |

The canonical task lists are available in [`tasks/`](https://github.com/QwenLM/RecreationWorld/tree/HEAD/tasks).

The Dataset Viewer shows one row per task: identity, platform, difficulty, upstream repository, commit, and task path. Select `all` for the full 250-task index or a platform for its 50 tasks; all configurations use the `test` split.

## Data Format

The dataset uses a per-task directory structure:

```text
<dataset-root>/
  metadata/
    <platform>.jsonl         # task index for the Dataset Viewer
  <platform>/
    <task-id>/
      instance.json
      reference/
      tests/
      vlm_assertions.json    # when provided
      LICENSE*               # license filenames and locations vary
```

| Component | Description |
| --- | --- |
| `instance.json` | Task identity, platform, pinned source revision, and any declared source patches. |
| `reference/` | Build and launch scripts, declared patches, and platform-specific reference assets. Web tasks include the frozen site under `reference/site/`. |
| `tests/` | Programmatic evaluation tests, supporting helpers, manifests, and fixtures. |
| `vlm_assertions.json` | Frozen visual-evaluation metadata where available; its coverage is task-dependent. |
| License and notice files | Applicable upstream terms, with original filenames and layouts such as `LICENSE`, `COPYING`, or `LICENSES/`. |

Desktop and Android references are built from pinned upstream commits with any declared patches. Web references are bundled site snapshots, with `repo` and `commit` set to `null`.

An example `instance.json`:

```json
{
  "schema_version": 1,
  "instance_id": "ubuntu/adrienverge-photocollage",
  "platform": "ubuntu",
  "repo": "https://github.com/adrienverge/PhotoCollage",
  "commit": "3289328d09dee6d19895ef013633c1c107ff41d5",
  "patches": [],
  "package": null,
  "license": null
}
```

| Field | Description |
| --- | --- |
| `schema_version` | Descriptor schema version, currently `1`. |
| `instance_id` | Namespaced identity in the form `<platform>/<task-id>`. |
| `platform` | One of `ubuntu`, `macos`, `windows`, `android`, or `web`. The runtime also uses `linux` to refer to Ubuntu. |
| `repo` | Upstream source repository URL; `null` for Web snapshots. |
| `commit` | Full pinned Git commit for source-backed tasks; `null` for Web snapshots. |
| `patches` | Source patches, each specifying its relative `path`, `reason`, `note`, and `sha256`. An empty list means no declared patches. |
| `package` | Android application package identifier; `null` for other platforms. |
| `license` | Optional license metadata. A `null` value does not waive copyright or replace the applicable license files. |

## Download

Download the dataset as files with the Hugging Face CLI:

```bash
python -m pip install --upgrade huggingface_hub
hf download Qwen/RecreationBench \
  --repo-type dataset \
  --local-dir ./RecreationBench
```

Download the dataset as files with ModelScope:

```bash
python -m pip install --upgrade modelscope
modelscope download Qwen/RecreationBench \
  --repo-type dataset \
  --local-dir ./RecreationBench
```

Preserve the `<platform>/<task-id>/` hierarchy and all test fixtures, including hidden files. The Dataset Viewer and `datasets.load_dataset()` expose the task index; running the task bundles requires the execution framework.

## Evaluation

The framework runs reference preparation, agent recreation, and evaluation in the appropriate platform environment. It reports two complementary metrics:

- **Prog:** reference-grounded programmatic checks of application behavior.
- **VLM:** visual checks of the recreated application using a vision-language-model judge.

Scores are macro-averaged across tasks within each platform, then equally weighted across the five platforms. **Average** is the unweighted mean of Prog and VLM. Visual-annotation coverage varies by task.

See the [GitHub README](https://github.com/QwenLM/RecreationWorld#quickstart) for full setup instructions, deployment guides, and platform-specific evaluation commands.

## License

This is a **multi-license collection**. The execution framework is [MIT-licensed](https://github.com/QwenLM/RecreationWorld/blob/HEAD/LICENSE); upstream applications and bundled assets retain their own terms. Web task notices separately cover benchmark-authored code (MIT), benchmark-authored dataset content (CC BY 4.0), and third-party assets.

Retain the applicable per-task and per-file notices. Where terms are missing or unclear, consult the pinned upstream source or contact the maintainers before redistribution.

## Citation

If you use RecreationBench, please cite:

```bibtex
@misc{qwen2026recreationworld,
      title={RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents},
      author={Shuai Bai and Jiayong Deng and Sicheng Fan and Yikun Fu and Chang Gao and Xuhao Hu and Mianqiu Huang and Yizhen Jiang and Yuheng Jing and Dehui Kong and Keliang Li and Ning Li and Wanli Li and Dayiheng Liu and Dunjie Lu and Changwei Luo and Que Shen and Zheyuan Wang and Zijian Wang and Jie Wu and Gao Wu and Zhihui Xie and Rui Xie and Haiyang Xu and An Yang and Jiakang Yuan and Yanming Zhang and Jiajun Zhang and Xi Zhang and Zhenru Zhang and Zhuo Zhen and Mingkang Zhu and Bowen Zhou},
      year={2026},
      eprint={2609.22000},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.22000},
}
```

## Contact

For questions about the dataset or evaluation, please contact [xiezhihui.xzh@alibaba-inc.com](mailto:xiezhihui.xzh@alibaba-inc.com) or [gaochang.gao@alibaba-inc.com](mailto:gaochang.gao@alibaba-inc.com).