---
title: rapidata-vs-prolific
canonical_url: "https://www.modelscope.ai/datasets/Rapidata/rapidata-vs-prolific"
md_url: "https://www.modelscope.ai/datasets/Rapidata/rapidata-vs-prolific.md"
repository: Rapidata/rapidata-vs-prolific
last_updated: 2026-08-05
license: cc-by-4.0
storage_size: "70 MB"
downloads: 146
stars: 0
---

# rapidata-vs-prolific

> rapidata-vs-prolific - An open-source dataset by Rapidata on ModelScope. Rapidata vs. Prolific: the same 300 questions on both platforms

Rapidata/rapidata-vs-prolific is a dataset on ModelScope. totalling 70 MB. licensed under cc-by-4.0.

- **Repository**: Rapidata/rapidata-vs-prolific
- **License**: cc-by-4.0
- **Storage size**: 70 MB
- **Downloads**: 146
- **Stars**: 0
- **Last updated**: 2026-08-05

Source: https://www.modelscope.ai/datasets/Rapidata/rapidata-vs-prolific

---

# Rapidata vs. Prolific: the same 300 questions on both platforms

In July 2026 we asked the crowds of two annotation platforms — [Prolific](https://www.prolific.com) and [Rapidata](https://www.rapidata.ai) — the same 300 two-image questions and graded every response against known correct answers. This dataset is the complete raw material of that comparison: the questions, the images, and all 18,007 individual responses from both crowds.

Each question is a short instruction and two images, one of which is correct. 92 questions are simple object recognition ("Select the image that shows a Snake."); 208 target one of six attention skills: counting, absence, search, spatial relations, object state, and attribute binding. Nothing requires special knowledge — a question is only ever hard because it takes a careful look.

![The same question as each crowd saw it: the web app we built and hosted for Prolific participants (left) and Rapidata's in-app rendering (right)](img/interface-comparison.png)

## How the two crowds compare

|                      | Prolific    | Rapidata   |
| -------------------- | ----------- | ---------- |
| People               | 150         | 5,103      |
| Answers per question | 10          | 50         |
| Total answers        | 3,000       | 15,007     |
| Collection time      | ~66 minutes | ~2 minutes |

![Answers collected over time, counted from each platform's first answer](img/collection-speed.png)

Graded one answer at a time, Prolific's crowd is more accurate: 98.0% against Rapidata's 91.6%. In practice, labels are made by majority vote over several answers per question, and there the two crowds converge:

![Accuracy of the majority label as responses per question are added](img/crowd-accuracy-vs-k.png)

Prolific's majority labels all 300 questions correctly from seven responses per question. Rapidata reaches 99.9% at nine responses and labels all 300 correctly from 23. At full depth, neither crowd got a single question majority-wrong.

Rapidata's answers cost $4 per 1,000 against Prolific's $76, so the larger majorities it needs barely register in the price of a finished label:

![The price of 1,000 labels at each reliability level; the small text is how many people each platform needs to ask per label](img/cost-of-reliable-labels.png)

## Schema

One row per question.

| column | description |
|---|---|
| `prompt` | The instruction shown to annotators, e.g. "Select the image that shows exactly five yellow rubber ducks." |
| `image1`, `image2` | The two images of the pair, embedded (display order was randomized per annotator on both platforms) |
| `correct_image` | The ground truth: which image matches the prompt (`image1`/`image2`) |
| `category` | `object-recognition` or one of `attention-counting`, `attention-absence`, `attention-search`, `attention-spatial`, `attention-state`, `attention-binding` |
| `difficulty` | Measured difficulty class (`easy` / `medium`) |
| `task_id` | Task identifier (`t001`-`t300`) |
| `votes_image1_rapidata`, `votes_image2_rapidata` | Rapidata vote counts per image (~50 per question) |
| `weighted_results_image1_rapidata`, `weighted_results_image2_rapidata` | Rapidata vote shares weighted by each annotator's chance-corrected score (sum to 1) |
| `detailed_results_rapidata` | JSON list of every Rapidata response: `votedFor` (`image1`/`image2`), `annotatorId`, `userScore`, `country`, `language`, `sessionIndex`, `createdAt` |
| `votes_image1_prolific`, `votes_image2_prolific` | Prolific vote counts per image (10 per question) |
| `detailed_results_prolific` | JSON list of every Prolific response: `votedFor`, `annotatorId`, `responseTimeMs`, `createdAt` |

```python
from datasets import load_dataset

import json

ds = load_dataset("Rapidata/rapidata-vs-prolific", split="train")

# overall single-answer accuracy, per platform
for platform in ["rapidata", "prolific"]:
    votes = [
        v["votedFor"] == r["correct_image"]
        for r in ds
        for v in json.loads(r[f"detailed_results_{platform}"])
    ]
    print(platform, sum(votes) / len(votes))  # rapidata 0.916, prolific 0.980
```

## Notes on the data

Annotator identifiers are anonymized sequential ids (`r0001`…, `p0001`…). They preserve which answers came from the same person but cannot be linked back to platform accounts.

Prolific participants were paid a fixed reward sized from the estimated completion time at the platform-recommended £19.40/hour. Rapidata responses were collected with a single API order at standard pricing, no audience filtering. The attention-skill images were generated with gpt-image-2.