---
title: Sophon-Realtime-32B-E2E
canonical_url: "https://www.modelscope.ai/models/AdventistsAI/Sophon-Realtime-32B-E2E"
md_url: "https://www.modelscope.ai/models/AdventistsAI/Sophon-Realtime-32B-E2E.md"
repository: AdventistsAI/Sophon-Realtime-32B-E2E
last_updated: 2026-11-12
license: apache-2.0
pipeline_tag: any-to-any
tasks:
  - any-to-any
model_type:
  - sophon_realtime
architectures:
  - SophonRealtimeForConditionalGeneration
parameters: 68.4B
tensor_type:
  - BF16
  - F32
  - F8_E4M3
library_name:
  - safetensors
  - pytorch
frameworks:
  - PyTorch
language:
  - zh
  - en
downloads: 29
stars: 0
tags:
  - sophon-realtime
  - full-duplex
  - speech-to-speech
  - voice-agent
  - paralinguistics
  - expressive-speech
  - duplexjev
---

# Sophon-Realtime-32B-E2E

> Sophon-Realtime-32B-E2E - An open-source model by AdventistsAI on ModelScope. Sophon-Realtime-32B-E2E

AdventistsAI/Sophon-Realtime-32B-E2E is a 68.4B-parameter any-to-any model on ModelScope. licensed under apache-2.0.

- **Repository**: AdventistsAI/Sophon-Realtime-32B-E2E
- **License**: apache-2.0
- **Tasks**: any-to-any
- **Parameters**: 68.4B
- **Tags**: sophon-realtime, full-duplex, speech-to-speech, voice-agent, paralinguistics, expressive-speech, duplexjev
- **Downloads**: 29
- **Stars**: 0
- **Last updated**: 2026-11-12

Source: https://www.modelscope.ai/models/AdventistsAI/Sophon-Realtime-32B-E2E

---

<h1 align="center">Sophon-Realtime-32B-E2E</h1>

<p align="center"><b>Sophon-Realtime v1 系列的 E2E 版：单卡、首包约 0.25 秒，声音直接从连续向量里生成</b><br>AI降临派（Adventists AI）· Apache-2.0 · v1.1.3</p>

<p align="center">
<a href="https://github.com/adventists-ai/sophon-realtime">GitHub</a> ·
<a href="https://huggingface.co/adventists-ai/Sophon-Realtime-32B-E2E">Hugging Face</a> ·
<a href="https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-E2E">魔搭</a> ·
<a href="#english">English</a>
</p>

## 这是什么

Sophon-Realtime 是全双工语音对话模型：一个 32B 的大脑负责听你说话、判断你说完没有、写出带声音标签的回复。E2E 版负责把回复变成声音的部分不经过语音 token：一个小巧的说话头（约 3.5 亿参数）在大脑还在写回复时就开始读，边读边生成连续的声音向量（每秒约 6.5 帧），再由 2 步解码器和声码器还原成 24 kHz 音频。笑、咳嗽、叹气、呼吸和重读都由说话头自己在同一条流里发出来。

整套系统（大脑 + 说话头 + 解码器）跑在**一张 80 GB 显卡**上。

## 1.1.4 更新（2026-11-12）

- E2E 版现在也可以像 Tag、Flash 一样作为一个 vLLM-Omni 模型来部署（代码在 [`e2e/`](https://github.com/adventists-ai/sophon-realtime/blob/main/e2e/README_CN.md)），命令、接口（`/v1/realtime`、`/v1/chat/completions`）和客户端都相同。自研引擎的部署方式不变，速度略快。
- 本仓库新增 `thinker/`（已合并回复 LoRA 的大脑，FP8，约 36 GB，与 Tag 仓库里的是同一份文件）、`config.json` 和 `talker220/config.json`，只有 vLLM-Omni 部署方式用到。其他文件没有变化。
- 两种部署方式各自只下载自己需要的部分：自研引擎约 67 GB（不含 `thinker/`），vLLM-Omni 约 41 GB（不含 `base/`、`adapters/`）；安装脚本会自动处理。

## 1.1.3 更新（2026-11-11）

- 与 Tag、Flash 并列为正式版本，从两张卡变成一张卡。
- 新的说话头（`talker220/`），原来的 `talker/` 已移除，下载量少约 8 GB；`base/`、`adapters/`、`e2/`、`voices/` 没有变化。
- 之前的双卡权重保留在本仓库的 `v1.1.0` 标签下（配合代码标签 `v1.1.2` 使用）。

## 在 v1 系列里的位置

三个版本是同一个大脑（VoiceBench 82.8、Big Bench Audio 81.8%，两项开源全双工第一；副语言理解综合第一），区别只在回复怎么变成声音：

| | [Tag](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Tag)（推荐） | [Flash](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Flash) | **E2E（本仓库）** |
|---|---|---|---|
| NVV-SuperBench（英 / 中） | **0.80 / 0.86** | 0.63 / 0.77 | 0.75 / 0.85 |
| 单路首包 | 约 0.6 秒 | 0.31 秒 | **约 0.25 秒** |
| 每张 80 GB 卡建议并发 | 16 路 | **40 路** | 16 路 |
| 零样本音色克隆 | 支持 | 支持 | 不支持 |
| 语言 | 中、英和多种欧洲语言 | 同左，另有日、俄 | 中、英 |
| 部署 | vLLM-Omni | vLLM-Omni | 自研引擎或 vLLM-Omni |

想要音色克隆或更多语言，用 Tag 或 Flash；想要最快的首包、同时保留会笑会叹气的表现力，用 E2E。

## 实测（单张 H100 80 GB，全新安装）

语音提问进、语音回答出，全链路都在同一张卡上。首包指从发出请求到收到第一块音频。

| 10 秒内到达的用户数 | 首包 中位数 / P90 | 播放落后超过 0.3 秒的回复数 |
|---|---|---|
| 1 | 0.25 秒 | 0 |
| 8 | 0.44 / 0.77 秒 | 0 / 8 |
| **16** | **0.44 / 0.56 秒** | **0 / 16** |
| 24 | 0.58 / 0.94 秒 | 7 / 24 |
| 32 | 0.67 / 0.80 秒 | 23 / 32 |

NVV-SuperBench（官方题目与指标，每类每种语言 50 条，Gemini 3 Flash 裁判，F1）：

| | 四类合计 | 笑 | 咳嗽 | 叹气 | 呼吸 |
|---|---|---|---|---|---|
| 英文 | 0.75 | 0.70 | 0.76 | 0.92 | 0.64 |
| 中文 | 0.85 | 0.86 | 0.95 | 0.87 | 0.74 |

同一批句子的转写错字率：中文 3.1%，英文 1.9%。

## 部署

需要 Linux、NVIDIA 驱动 580 及以上、一张 H100 级的 80 GB 显卡（实际占用约 70 GB）。有两种方式，权重相同、声音相同。

**方式一：自研引擎**（上面表格里的数字）

```bash
git clone https://github.com/adventists-ai/sophon-realtime
cd sophon-realtime
bash scripts/install.sh          # 建两个虚拟环境，并从魔搭下载本模型
SOPHON_MODEL_DIR=$PWD/Sophon-Realtime-32B-E2E GPU=0 bash scripts/start.sh
```

启动后打开 `http://<服务器>:8000/` 即可在网页里直接对话，也可以通过 WebSocket 接口（`/ws`）接入自己的客户端。详见 [docs/deploy_CN.md](https://github.com/adventists-ai/sophon-realtime/blob/main/docs/deploy_CN.md)。

**方式二：vLLM-Omni**（与 Tag、Flash 相同的命令和接口）

```bash
git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/e2e
bash install.sh                 # 建虚拟环境，并从魔搭下载本模型中需要的部分（约 41 GB）
GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang zh
```

提供 `ws://<服务器>:8000/v1/realtime`（全双工）和 `/v1/chat/completions`。单张 H100 实测：单路首包约 0.35 秒；16 路首包中位数 0.58–0.65 秒、P90 0.71–0.87 秒，0–2 路回复播放落后超过 0.3 秒；全双工下从说完到出声中位数约 0.45 秒。详见 [e2e/README_CN.md](https://github.com/adventists-ai/sophon-realtime/blob/main/e2e/README_CN.md)。

**其他显卡**：以上数字都是在 H100 上测的。在一张 RTX PRO 6000（Blackwell，96 GB）上实测 vLLM-Omni 方式：能安装、能运行，但一张卡只能跟上约 4–8 路并发（E2E 发声部分在这张卡上慢约一倍）。在这类显卡上需要更高并发时，请用 Tag 或 Flash。

## 声音标签

`[laughter]` `[cough]` `[sigh]` `[breath]`（在该位置发出声音）、`<strong>词</strong>`（重读）、`<laughter>几个词</laughter>`（边笑边说）。大脑会自己把它们写进回复；说话头把它们演出来，不会念出来。

## 这一版的限制

- 一个内置音色，不支持音色克隆。
- 回复用中文或英文。
- 单条回复最长约 4.6 分钟语音（vLLM-Omni 方式约 2.5 分钟）；超出时这条回复在该处正常结束，不影响其他对话。连续说到约 2.5 分钟以内内容是准确的，更长的回复后段会出现念错，建议把单条回复控制在 2.5 分钟以内。

## 文件

| 目录 | 内容 |
|---|---|
| `base/` | 大脑的 32B 权重 |
| `adapters/tag/` | 写带标签回复的 LoRA |
| `talker220/` | 说话头：权重、文字嵌入表、分词器 |
| `e2/` | 解码器与声码器 |
| `voices/` | 默认音色 |
| `thinker/`、`config.json`、`talker220/config.json` | 只有 vLLM-Omni 部署方式用到：已合并回复 LoRA 的大脑（FP8，约 36 GB） |

## 许可证

代码和权重都是 Apache-2.0。第三方组件保留各自的许可证，见 GitHub 仓库的 [NOTICE](https://github.com/adventists-ai/sophon-realtime/blob/main/NOTICE)。

## 联系

商务合作：jiejin@adventists.ai

---

<a id="english"></a>

# Sophon-Realtime-32B-E2E

**The E2E build of the Sophon-Realtime v1 family: one GPU, first audio in about 0.25 s, speech generated directly as continuous vectors.** Adventists AI · Apache-2.0 · v1.1.3

## What it is

Sophon-Realtime is a full-duplex voice model: a 32B brain listens to you, decides when you have finished, and writes a reply with sound tags. In the E2E build, the part that turns the reply into sound has no speech-token model in the middle. A compact talker (about 350M parameters) reads the reply while it is still being written and generates a stream of continuous voice vectors, about 6.5 frames per second; a 2-step decoder and a vocoder turn them into 24 kHz audio. Laughs, coughs, sighs, breaths and stressed words are produced by the talker itself, as part of the same stream.

The whole system (brain, talker, decoder) runs on **one 80 GB GPU**.

## What's new in 1.1.4 (2026-11-12)

- E2E can now also be served as one vLLM-Omni model, like Tag and Flash (code in [`e2e/`](https://github.com/adventists-ai/sophon-realtime/blob/main/e2e/README.md)), with the same commands, APIs (`/v1/realtime`, `/v1/chat/completions`) and client. The own-engine deployment is unchanged and a little faster.
- New in this repository: `thinker/` (the brain with the reply LoRA merged in, FP8, about 36 GB; the same files as in the Tag repository), `config.json` and `talker220/config.json`. Only the vLLM-Omni deployment uses them. Nothing else changed.
- Each way of deploying downloads only what it needs: about 67 GB for the own engine (without `thinker/`), about 41 GB for vLLM-Omni (without `base/` and `adapters/`). The install scripts take care of it.

## What's new in 1.1.3 (2026-11-11)

- A full build next to Tag and Flash, on one GPU instead of two.
- New talker (`talker220/`); the old `talker/` is removed, so the download is about 8 GB smaller. `base/`, `adapters/`, `e2/` and `voices/` are unchanged.
- The earlier two-GPU weights stay in this repository at tag `v1.1.0` (use them with code tag `v1.1.2`).

## Where it fits in the v1 family

All three builds share one brain (VoiceBench 82.8 and Big Bench Audio 81.8%, #1 among open full-duplex models on both; #1 in paralinguistic understanding). They differ only in how the reply becomes sound:

| | [Tag](https://huggingface.co/adventists-ai/Sophon-Realtime-32B-Tag) (recommended) | [Flash](https://huggingface.co/adventists-ai/Sophon-Realtime-32B-Flash) | **E2E (this repository)** |
|---|---|---|---|
| NVV-SuperBench (en / zh) | **0.80 / 0.86** | 0.63 / 0.77 | 0.75 / 0.85 |
| First audio, one user | ~0.6 s | 0.31 s | **~0.25 s** |
| Recommended users per 80 GB GPU | 16 | **40** | 16 |
| Zero-shot voice cloning | yes | yes | no |
| Languages | Chinese, English and several European languages | the same, plus Japanese and Russian | Chinese, English |
| Serving | vLLM-Omni | vLLM-Omni | own engine or vLLM-Omni |

Use Tag or Flash for voice cloning or more languages; use E2E for the fastest first audio with a voice that still laughs and sighs.

## Measured (one H100 80 GB, fresh install)

Spoken question in, spoken answer out, everything on the same GPU. First audio is the time from sending the request to receiving the first audio chunk.

| Users arriving within 10 s | First audio, median / P90 | Replies whose playback fell more than 0.3 s behind |
|---|---|---|
| 1 | 0.25 s | 0 |
| 8 | 0.44 / 0.77 s | 0 of 8 |
| **16** | **0.44 / 0.56 s** | **0 of 16** |
| 24 | 0.58 / 0.94 s | 7 of 24 |
| 32 | 0.67 / 0.80 s | 23 of 32 |

NVV-SuperBench (official prompts and metric, 50 items per class per language, Gemini 3 Flash judge, F1):

| | All 4 | Laugh | Cough | Sigh | Breath |
|---|---|---|---|---|---|
| English | 0.75 | 0.70 | 0.76 | 0.92 | 0.64 |
| Chinese | 0.85 | 0.86 | 0.95 | 0.87 | 0.74 |

Transcription error rate on the same sentences: Chinese 3.1%, English 1.9%.

## Deployment

Linux, NVIDIA driver 580 or newer, one 80 GB GPU of the H100 class (about 70 GB is used). There are two ways, with the same weights and the same voice.

**1. Own engine** (the numbers in the tables above)

```bash
git clone https://github.com/adventists-ai/sophon-realtime
cd sophon-realtime
bash scripts/install.sh          # two virtual environments + this model from ModelScope
SOPHON_MODEL_DIR=$PWD/Sophon-Realtime-32B-E2E GPU=0 bash scripts/start.sh
```

Then open `http://<server>:8000/` to talk in the browser, or connect your own client to the WebSocket API (`/ws`). See [docs/deploy.md](https://github.com/adventists-ai/sophon-realtime/blob/main/docs/deploy.md).

**2. vLLM-Omni** (the same commands and APIs as Tag and Flash)

```bash
git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/e2e
bash install.sh                 # virtual environment + the parts of this model it needs, from ModelScope (about 41 GB)
GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang en
```

It serves `ws://<server>:8000/v1/realtime` (full duplex) and `/v1/chat/completions`. Measured on one H100: first audio about 0.35 s for one user; 16 users 0.58–0.65 s median / 0.71–0.87 s P90, with 0–2 replies falling more than 0.3 s behind playback; in full duplex about 0.45 s (median) from the end of your speech to the first voice. See [e2e/README.md](https://github.com/adventists-ai/sophon-realtime/blob/main/e2e/README.md).

**Other GPUs:** all numbers above are measured on an H100. On one RTX PRO 6000 (Blackwell, 96 GB) the vLLM-Omni deployment installs and runs, but one GPU keeps up with only about 4–8 concurrent users (the E2E voice stages run about 2× slower on that card). If you need more users per GPU on such cards, use Tag or Flash.

## Sound tags

`[laughter]` `[cough]` `[sigh]` `[breath]` (a sound at that point), `<strong>word</strong>` (emphasis), `<laughter>some words</laughter>` (speaking while laughing). The brain writes them into the reply on its own; the talker performs them instead of reading them aloud.

## Limits of this release

- One built-in voice; no voice cloning.
- Replies are spoken in Chinese or English.
- One reply can be at most about 4.6 minutes of speech (about 2.5 minutes with vLLM-Omni); a longer reply ends cleanly at that point and other conversations are not affected. Speech stays accurate up to about 2.5 minutes of continuous talking; beyond that the later part of a reply contains misread words, so keep single replies under 2.5 minutes.

## Files

| folder | contents |
|---|---|
| `base/` | 32B weights of the brain |
| `adapters/tag/` | LoRA that writes the tagged reply |
| `talker220/` | talker: weights, text embedding table, tokenizer |
| `e2/` | decoder and vocoder |
| `voices/` | default voice |
| `thinker/`, `config.json`, `talker220/config.json` | only for the vLLM-Omni deployment: the brain with the reply LoRA merged in (FP8, about 36 GB) |

## License

Code and weights are released under Apache-2.0. Third-party components keep their own licences; see [NOTICE](https://github.com/adventists-ai/sophon-realtime/blob/main/NOTICE) in the GitHub repository.

## Contact

Business enquiries: jiejin@adventists.ai