---
title: Core-S2RGB-Qwen3-VL-Embedding
canonical_url: "https://www.modelscope.ai/datasets/Major-TOM/Core-S2RGB-Qwen3-VL-Embedding"
md_url: "https://www.modelscope.ai/datasets/Major-TOM/Core-S2RGB-Qwen3-VL-Embedding.md"
repository: Major-TOM/Core-S2RGB-Qwen3-VL-Embedding
last_updated: 2026-09-26
license: other
storage_size: "180 GB"
downloads: 112
stars: 0
---

# Core-S2RGB-Qwen3-VL-Embedding

> Core-S2RGB-Qwen3-VL-Embedding - An open-source dataset by Major-TOM on ModelScope. Core-S2RGB-Qwen3-VL-Embedding

Major-TOM/Core-S2RGB-Qwen3-VL-Embedding is a dataset on ModelScope. totalling 180 GB. licensed under other.

- **Repository**: Major-TOM/Core-S2RGB-Qwen3-VL-Embedding
- **License**: other
- **Storage size**: 180 GB
- **Downloads**: 112
- **Stars**: 0
- **Last updated**: 2026-09-26

Source: https://www.modelscope.ai/datasets/Major-TOM/Core-S2RGB-Qwen3-VL-Embedding

---

# Core-S2RGB-Qwen3-VL-Embedding

Global dense geospatial-vision embeddings computed from the complete **Major-TOM Core-S2L2A** collection with **Qwen3-VL-Embedding-2B**.

## Overview

| Property | Value |
|---|---|
| Source imagery | Major-TOM Core-S2L2A |
| Source scenes | 2,245,886 |
| Dense embeddings | 20,212,974 |
| Model | Qwen3-VL-Embedding-2B |
| Input bands | RGB `[B04, B03, B02]` |
| Patch size | 384 × 384 pixels |
| Patches per scene | 9 (3 × 3 dense grid) |
| Embedding dimension | 2048 |
| Output format | GeoParquet |
| Embedding payload size | approximately 158 GB |
| FP8 rows after global ID deduplication | 20,148,201 |
| FP8 storage size | 35.063 GB (32.655 GiB) |
| License | CC-BY-SA-4.0 |

## Computation Pipeline

1. **Input**: the 10 m Sentinel-2 L2A RGB bands `[B04, B03, B02]` are read from each 1068 × 1068 Major-TOM scene.
2. **True-colour conversion**: digital numbers are transformed with `round(clip(DN × 2.5 / 10000, 0, 1) × 255)`.
3. **Dense tiling**: each scene is divided into nine 384 × 384 patches. The x/y offsets are the Cartesian product of `{0, 342, 684}`, giving approximately 10% overlap and exact border coverage.
4. **Encoding**: every patch is passed to Qwen3-VL-Embedding-2B without an image instruction prompt. The model produces a 2048-dimensional embedding.
5. **Post-processing**: embeddings are L2-normalised, so their dot product is cosine similarity.
6. **Geospatial metadata**: every patch footprint is derived from the source UTM raster and reprojected to EPSG:4326.

Four source scenes had one extra image column (1068 × 1069). Their rightmost column was removed and the UTM footprint was reduced by the corresponding 10 metres. Eleven duplicated scenes crossed source-Parquet boundaries; metadata was matched by `(grid_cell, product_id)`, while `parquet_url` and `parquet_row` point to the exact image row used for encoding.

## File Layout

The release follows the same grouped layout as `Major-TOM/Core-S2RGB-SigLIP`:

```text
.
├── README.md
├── .gitattributes
├── embeddings/
│   ├── part_00001-00100.parquet
│   ├── ...
│   └── part_04401-04492.parquet
└── embedding_fp8/
    ├── part_00001-00100.parquet
    ├── ...
    ├── part_04401-04492.parquet
    ├── docs/
    ├── metadata/
    ├── reports/
    └── scripts/
```

Both representations contain 45 data files. The first 44 files each preserve 100 source Parquets as separate row groups; the final file contains source parts 4401–4492 as 92 row groups. Rows remain in source-part and source-row order, except that the FP8 representation omits later duplicate-ID occurrences as documented below. Only final embedding Parquet files are placed directly under `embedding_fp8/`; documentation, metadata, reports, and scripts are grouped in subdirectories. Runtime logs and caches are kept locally under `embedding_fp8/tmp/` and are not published.

## Schema

| Column | Type | Description |
|---|---|---|
| `unique_id` | string | SHA-256 identifier derived from geometry, timestamp, product ID and embedding |
| `embedding` | list<float32> | L2-normalised 2048-dimensional Qwen3-VL embedding |
| `timestamp` | string | Sentinel-2 acquisition timestamp |
| `product_id` | string | Original Sentinel-2 product identifier |
| `grid_cell` | string | Major-TOM grid-cell identifier |
| `grid_row_u` | int16 | Major-TOM grid row |
| `grid_col_r` | int16 | Major-TOM grid column |
| `geometry` | geometry | Patch polygon in WGS 84 / EPSG:4326 |
| `centre_lat` | float32 | Patch-centre latitude |
| `centre_lon` | float32 | Patch-centre longitude |
| `utm_footprint` | string | Patch polygon in the source UTM CRS, stored as WKT |
| `utm_crs` | string | Source UTM CRS |
| `pixel_bbox` | list<int64> | Pixel window `[x_min, y_min, x_max, y_max]` in the source scene |
| `parquet_row` | int64 | Actual row index in the source image Parquet |
| `parquet_url` | string | Source Core-S2L2A image-Parquet URL |

## FP8 Storage Variant

`embedding_fp8/` contains a storage-quantized version derived from source revision `7352d40a395f035f8479097a792af0305b206179`. All 45 source files matched their published SHA-256 values before conversion. The source contains 64,773 later occurrences of repeated `unique_id` values; each repeated pair has identical scene/patch fields and embedding content but different provenance row fields. The FP8 release keeps the first occurrence in source filename and physical-row order, resulting in 20,148,201 unique rows.

Each 2,048-dimensional vector uses E4M3FN bytes plus one float32 scale:

```text
amax = max(abs(x))
scale = float32(amax / 448.0) if amax > 0 else float32(1.0)
codes = bit_view_uint8(cast_float8_e4m3fn(clip(x / scale, -448.0, 448.0)))
x_hat = decode_float8_e4m3fn(codes).astype(float32) * scale
```

| Column | Type | Description |
|---|---|---|
| `embedding_fp8` | fixed_size_binary[2048] | Raw E4M3FN bit patterns for all 2,048 dimensions |
| `embedding_scale` | float32 | Per-vector scale used to reconstruct `x_hat` |

All other columns retain their source values and Arrow types. File order and row-group partitioning are preserved after duplicate removal. The files use Parquet ZSTD level 3 and occupy 35,063,332,389 bytes in total, a 4.507× reduction from the FP32 source files.

A fixed-seed test over 100,000 candidates and 1,000 disjoint queries produced a mean source-to-decoded cosine of 0.99965274. Top-10 overlap with exact FP32 ranking was 0.9847 when only candidates were quantized and 0.9841 when both queries and candidates were quantized. These values measure ranking agreement, not semantic relevance.

```python
import ml_dtypes
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq

pf = pq.ParquetFile("embedding_fp8/part_00001-00100.parquet")
table = pf.read_row_group(0, columns=["embedding_fp8", "embedding_scale"]).slice(0, 32).combine_chunks()
codes = table["embedding_fp8"].chunk(0)
scales = table["embedding_scale"].chunk(0)
raw = np.frombuffer(codes.buffers()[1], dtype=np.uint8,
                    count=len(codes) * 2048, offset=codes.offset * 2048).reshape(len(codes), 2048)
decoded = raw.view(ml_dtypes.float8_e4m3fn).astype(np.float32)
decoded *= np.asarray(scales.to_numpy(zero_copy_only=False), dtype=np.float32)[:, None]
```

Detailed format documentation is in `embedding_fp8/docs/format.md`. Reproducible tools are in `embedding_fp8/scripts/`, manifests and configuration are in `embedding_fp8/metadata/`, and validation results are in `embedding_fp8/reports/`.

## Usage

```python
import pyarrow.dataset as ds

embeddings = ds.dataset("embeddings", format="parquet")
table = embeddings.to_table(
    columns=["embedding", "product_id", "geometry", "pixel_bbox"],
    filter=ds.field("grid_cell") == "0U_375R",
)
print(table.num_rows)
```

The complete embedding matrix contains more than 20 million 2048-dimensional float vectors. Use streaming reads, partition pruning or a vector index instead of loading the entire matrix into GPU memory.

## Related Resources

- Source dataset: [Major-TOM Core-S2L2A](https://modelscope.cn/datasets/Major-TOM/Core-S2L2A)
- 249k Qwen release: [Core-S2RGB-249k-Qwen3-VL-Embedding](https://modelscope.cn/datasets/Major-TOM/Core-S2RGB-249k-Qwen3-VL-Embedding)
- Explorer and generation code: [EarthEmbeddingExplorer](https://modelscope.cn/studios/Major-TOM/EarthEmbeddingExplorer)

## Citation

Please cite Major TOM, Qwen3-VL-Embedding, the global dense embedding project and EarthEmbeddingExplorer when using this dataset.

```bibtex
@inproceedings{francis2024majortom,
  title={Major TOM: Expandable Datasets for Earth Observation},
  author={Francis, Alistair and Czerkawski, Mikolaj},
  year={2024},
  booktitle={IGARSS 2024},
  eprint={2402.12095},
  archivePrefix={arXiv}
}
```

```bibtex
@article{qwen3vlembedding,
  title={Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking},
  author={Li, Mingxin and Zhang, Yanzhao and Long, Dingkun and Chen, Keqin and Song, Sibo and Bai, Shuai and Yang, Zhibo and Xie, Pengjun and Yang, An and Liu, Dayiheng and Zhou, Jingren and Lin, Junyang},
  journal={arXiv preprint arXiv:2601.04720},
  year={2026}
}
```

```bibtex
@article{EmbeddedMajorTOM,
  title={Global and Dense Embeddings of Earth: Major TOM Floating in the Latent Space},
  author={Czerkawski, Mikolaj and Kluczek, Marcin and Bojanowski, Jedrzej S.},
  journal={arXiv preprint arXiv:2412.05600},
  year={2024}
}
```

```bibtex
@article{zheng2026earthembeddingexplorer,
  title={EarthEmbeddingExplorer: A Web Application for Cross-Modal Retrieval of Global Satellite Images},
  author={Zheng, Yijie and Wu, Weijie and Wu, Bingyue and Zhao, Long and Li, Guoqing and Czerkawski, Mikolaj and Klemmer, Konstantin},
  journal={arXiv preprint arXiv:2603.29441},
  year={2026}
}
```