听得懂语气，也说得出语气的开源全双工语音模型
AI降临派（Adventists AI）· Apache-2.0
⭐ 推荐：Tag 版 · Flash 版 · 在线体验 · GitHub
跟它说话，像跟人打电话一样：你说话时它一直在听，你一停它就接上；你随时插话，它马上停下来回答你的新问题。它不只听你说了**什么**，也听你是**怎么**说的：犹豫、语气、重音、笑声、叹气，它都听得出来；回答时，声音里也带着语气：会笑，会叹气，嗓子干了会清一清。 这是 Sophon-Realtime v1 系列的主页。**v1 系列有三个版本，同一个大脑**（同一个 32B 模型负责听、判断什么时候接话、写回复），区别只在回复怎么变成声音。**推荐从 Tag 版开始。** ## 三个版本，怎么选 | | **[Tag](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Tag)（推荐）** | **[Flash](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Flash)** | [E2E](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-E2E)（研究预览） | |---|---|---|---| | 怎么出声 | 大脑写出带声音标签的回复，由擅长表演的语音模型演出来 | 同样的带标签回复，交给更轻、更快的语音模型来说 | 32B 模型内部的说话头直接出声，不用单独的语音模型 | | 适合 | 表现力优先：陪伴、角色、演示 | 速度和并发优先：客服、车机、一张卡服务很多人 | 端到端表达的研究 | | 非语言声音（NVV-SuperBench，中 / 英） | **0.86 / 0.80** | 0.77 / 0.63 | 0.77 / 0.60 | | 单路首包 | 约 0.6 秒 | **0.31 秒** | 约 1 秒 | | 每张 80 GB 卡建议并发 | 16 路 | **40 路** | 4–5 路 | | 显卡 | **1 张** | **1 张** | 2 张 | | 部署 | vLLM-Omni，一条命令 | vLLM-Omni，一条命令 | 自研引擎 | 想让声音「会演」，选 **Tag**；最看重延迟和单卡能接多少人，选 **Flash**。两者接口相同，之后可以随时切换。 ## 亮点 - **VoiceBench 82.8、Big Bench Audio 81.8%：开源全双工里最强的「大脑」。** 两项都是开源全双工模型第一，VoiceBench 与 GPT-4o-mini-Audio 持平。三个版本共用这个大脑。 - **副语言理解 48.6，综合第一：听得懂语气。** 在 LISTEN、StressTest、MSPB 公开基准上高于 Gemini 3.8 Flash（46.9）和 Qwen3-Omni（44.6）。它的耳朵是我们开源的语音决策模型 DuplexJev。 - **NVV-SuperBench：Tag 版中文 0.86、英文 0.80，说得出语气。** 什么时候笑、叹气、换气、清嗓子，由模型根据对话自己决定；声音把这些标记演出来，不会把标签念出来。 - **快：Flash 版首包 0.31 秒。** 你说话的全程它都在听，回复边听边起草；你插话，它马上停下，回答你的新问题。 - **Full-Duplex-Bench v3 工具调用 Pass@1 48%，高于 Grok（43%）；接话成功率 100%：能干活。** 支持 [Live 模式](https://github.com/adventists-ai/sophon-realtime/blob/main/live/README_CN.md)，兼容 OpenAI GPT-Live 的接口格式，可以和后端大模型联合工作。每句回答都有文字记录，可以审计。 - **一张 80 GB 显卡、一条命令（Tag 和 Flash）。** 基于 vLLM-Omni 部署，单卡建议并发 Tag 16 路、Flash 40 路。代码和权重都是 Apache-2.0，可以商用。 ## 快速上手（Tag 版） 最快的方式： ```bash pip install sophon-realtime # Python 3.12、Linux、CUDA sophon-realtime serve # 首次运行自动下载 Tag 版权重（约 42 GB），然后在 8000 端口提供服务 ``` 也可以用一键脚本（独立虚拟环境）： ```bash git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/tag bash install.sh # 建虚拟环境、装 vLLM-Omni 和插件，并下载 Tag 版权重（约 42 GB）；国内可加 PIP_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple GPU=0 PORT=8000 bash start.sh # 首次启动 15–20 分钟，之后 5–8 分钟 python client_example.py --wav 我的问题.wav --lang zh --voice vivian_zh ``` - 接口：`ws://HOST:8000/v1/realtime`（全双工：话轮、打断、附和）、`POST /v1/chat/completions`（语音进，文字 + 语音流出）。 - Flash 版用法相同，见 [Flash 模型页](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Flash)；想直接试试，可以去[魔搭创空间](https://www.modelscope.cn/studios/AdventistsAI/Sophon-Realtime)在线对话。 ## Live 模式：OpenAI GPT-Live 的平替 GitHub 仓库的 [`live/`](https://github.com/adventists-ai/sophon-realtime/blob/main/live/README_CN.md) 是一个网关：对外是 OpenAI GPT-Live 协议，后面接 Tag 或 Flash 引擎，按 GPT-Live 写的客户端只要换地址和 key 就能接入。智子负责听和说；遇到要查询、办事的请求，它先说一句「我帮你查一下」，把任务交给你的后端，拿到结果后用自己的话说出来。也可以让一个更大的模型做后台：用 DeepSeek V4 Pro 做后台时，Big Bench Audio 从 81.2 提到约 83。 启动方法：在已经用一键脚本启动的 Tag 或 Flash 引擎旁边，再运行一条命令： ```bash cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh # GPT-Live 客户端连 ws://HOST:8780/v1/live/sessions ``` ## 评测详情 ### 对话能力：VoiceBench（官方评测器，gpt-4o-mini 判分） | 模型 | 全双工 | 开放权重 | 总分 | |---|:-:|:-:|:-:| | GPT-4o-Audio | — | — | 86.8 | | **Sophon-Realtime v1** | ✅ | ✅ | **82.8** | | GPT-4o-mini-Audio | — | — | 82.8 | | MiniCPM-o 4.5 | ✅ | ✅ | 76.1 | | MiniCPM-o 2.6 | ✅ | ✅ | 71.2 | | Freeze-Omni | ✅ | ✅ | 55.2 | | NemotronLabs VoiceChat | ✅ | ✅ | 55.1 | | PersonaPlex | ✅ | ✅ | 30.6 | | Moshi | ✅ | ✅ | 29.5 | AlpacaEval 用 199 题标准子集。其他模型的分数来自 VoiceBench 榜单和 NemotronLabs VoiceChat 论文（arXiv:2609.21967）表 3，其中 MiniCPM-o 4.5 的分数来自第三方评测。 ### 语音推理：Big Bench Audio 1,000 道语音推理题，按它实际说出的话判分。 | 模型 | 开放权重 | 准确率 | |---|:-:|:-:| | GPT-Live-1 | — | 90% | | **Sophon-Realtime v1** | ✅ | **81.8%** | | Raon SpeechChat | ✅ | 58% | | Freeze-Omni | ✅ | 33% | | PersonaPlex | ✅ | 19% | | Moshi | ✅ | 4% | Sophon 为自测：回答转写后由 gpt-4.1 判分。其他模型的数字来自 Artificial Analysis 榜单。 ### 调用工具：Full-Duplex-Bench v3 真人不流利语音 + 多步工具调用，100 题，GPT-4o 判分。 | 模型 | 开放权重 | Pass@1 ↑ | 接话成功率 ↑ | |---|:-:|:-:|:-:| | GPT-Realtime | — | 60% | 96% | | Gemini Live 3.1 | — | 54% | — | | **Sophon-Realtime** | ✅ | **48%** | **100%** | | 级联方案 | — | 45% | — | | Grok | — | 43% | — | | Ultravox v0.7 | ✅ | 41% | — | Sophon 为多次运行中的最好成绩。其他模型的数字来自 Full-Duplex-Bench v3 官方结果。 ### 听懂语气：副语言理解 Sophon-Realtime 用 [DuplexJev](https://www.modelscope.cn/models/AdventistsAI/DuplexJev-32B-Para) 听声音，测的是情绪、反讽、重音和语调。 | 模型 | LISTEN（4 项）+ StressTest + MSPB 合计 | |---|:-:| | **DuplexJev（Sophon-Realtime 的听觉模型）** | **48.6** | | Gemini 3.8 Flash | 46.9 | | Qwen3-Omni | 44.6 | ### 说出语气：NVV-SuperBench 看模型说出来的声音里，该有的笑、咳嗽、叹气、呼吸有没有真的出现。官方评测脚本，每类 50 条，Gemini 3 Flash 判分，F1 越高越好。 | 模型 | 中文 | 英文 | |---|:-:|:-:| | Sophon-Realtime-32B-Tag | **0.86** | **0.80** | | Sophon-Realtime-32B-Flash | 0.77 | 0.63 | | Sophon-Realtime-32B-E2E（研究预览） | 0.77 | 0.60 | 三个版本说同一批带标签的回复文本，只比较发声。 ## 工作原理 - **一直在听。** 音频持续流入模型。每次停顿，同一个模型用一个 token 判断：你说完了、还在说、在附和，还是在想。 - **提前起草，准时开口。** 第一次停顿就开始起草回复，判断轮到自己时立刻说出来；你如果继续说，草稿在你听到之前就丢掉。 - **不只写说什么，也写怎么说。** 大脑写回复时会标出哪里该笑、叹气、换气、清嗓子、哪个词该重读；Tag 和 Flash 把这些标记演出来，不会念出来。 - **文字领先半步，声音同步跟上。** 文字和声音并行流式生成，每句话都有文字记录，方便审计，也方便接工具、接更大的后端模型。 - **记得你真正听到了什么。** 历史里保存双方的真实音频。你打断时，只保留你已经听到的部分。 - **端到端的探索（E2E）。** 我们发现，语音大模型听声音时内部用的那串连续向量本身就是一门完整的语言。E2E 版的说话头直接「写」这门向量语言，再还原成声音，听和说用同一套表示。目前是研究预览，见 [E2E 模型页](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-E2E)。 ## 微调 可以用 ms-swift 微调思考器的 LLM 主干（LoRA），改人设、口吻、领域知识和工具调用习惯。听觉编码器和发声部分保持原样，接话判断不受影响。微调从自带的写标签 LoRA（`adapters/tag`）接着训，笑声、咳嗽等声音表现会保留。一张 80G 显卡即可。 ```bash pip install "ms-swift>=4.6" git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/finetune modelscope download --model AdventistsAI/Sophon-Realtime-v1 --include 'base/*' 'adapters/*' --local_dir Sophon-Realtime-v1 python -m sophon_swift.adapters to-swift Sophon-Realtime-v1/adapters/tag tag_swift swift sft --custom_register_path sophon_swift/swift_register.py \ --model Sophon-Realtime-v1 --adapters tag_swift --tuner_type lora \ --dataset my_data.jsonl --learning_rate 5e-5 --num_train_epochs 2 \ --gradient_accumulation_steps 8 --gradient_checkpointing true --output_dir output ``` 数据格式、多轮与工具调用、导出部署和回归检查见 GitHub 仓库的 [finetune 目录](https://github.com/adventists-ai/sophon-realtime/tree/main/finetune)。 ## 本仓库的文件 本仓库是 v1 系列主页，同时存放微调用的思考器底座（`base/`）和写标签 LoRA（`adapters/`），以及上一版 E2E 的权重（`talker/`、`e2/`、`voices/`）。部署请直接用 [Tag](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Tag) 或 [Flash](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Flash) 模型页的权重。 ## 许可证 代码和权重都是 Apache-2.0。第三方组件保留各自的许可证，见 NOTICE。 ## 联系 商务合作：jiejin@adventists.ai --- # Sophon-Realtime v1 (English) **An open full-duplex voice model family that hears your tone and speaks with one.** Adventists AI · Apache-2.0 · **Recommended: [Tag](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Tag)** · [Flash](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Flash) · [Try it online](https://www.modelscope.ai/studios/AdventistsAI/Sophon-Realtime) · [GitHub](https://github.com/adventists-ai/sophon-realtime) Talking to it feels like a phone call with a person: it listens the whole time you talk, picks up as soon as you stop, and stops the moment you cut in. It hears *how* you say things (hesitation, emphasis, sarcasm, laughter, sighs) and answers with tone of its own: it laughs, sighs and clears its throat when the conversation calls for it. This is the home page of the Sophon-Realtime v1 family. **Three builds share one brain** (the same 32B model listens, decides when to take the turn and writes the reply); they differ only in how the reply becomes sound. **Start with Tag.** ## Three builds: which one? | | **[Tag](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Tag) (recommended)** | **[Flash](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Flash)** | [E2E](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-E2E) (research preview) | |---|---|---|---| | How it speaks | the brain writes the reply with sound tags; an expressive speech model performs them | the same tagged reply, spoken by a lighter, faster speech model | a talker inside the 32B model, no separate speech model | | Best for | expressive voice: companions, characters, demos | speed and scale: customer service, in-car, many users per GPU | research on end-to-end expressive speech | | Non-verbal sounds (NVV-SuperBench, zh / en) | **0.86 / 0.80** | 0.77 / 0.63 | 0.77 / 0.60 | | First audio, one user | ~0.6 s | **0.31 s** | ~1 s | | Users per 80 GB GPU (recommended) | 16 | **40** | 4–5 | | GPUs | **1** | **1** | 2 | | Serving | vLLM-Omni, one command | vLLM-Omni, one command | own engine | Pick **Tag** if you want the voice to act; pick **Flash** if latency and users per GPU matter most. Same APIs, so you can switch later. ## Highlights - **VoiceBench 82.8 and Big Bench Audio 81.8%: the strongest brain among open full-duplex models.** #1 among open full-duplex models on both; level with GPT-4o-mini-Audio on VoiceBench. All three builds share this brain. - **Paralinguistic understanding 48.6, #1 overall: it hears your tone.** Ahead of Gemini 3.8 Flash (46.9) and Qwen3-Omni (44.6) on LISTEN, StressTest and MSPB. Its ears are DuplexJev, our open speech-decision model. - **NVV-SuperBench 0.86 (Chinese) and 0.80 (English) with the Tag build: it speaks with tone.** The model decides when to laugh, sigh, breathe or clear its throat, and the voice performs it instead of reading the tag aloud. - **Fast: first audio in 0.31 s with Flash.** It drafts its reply while you talk; when you cut in, it stops and answers your new question. - **Full-Duplex-Bench v3 tool calling: Pass@1 48%, ahead of Grok (43%), with a 100% response rate: it gets work done.** [Live mode](https://github.com/adventists-ai/sophon-realtime/blob/main/live/README.md) is compatible with the OpenAI GPT-Live interface and lets it work with a larger backend model. Every reply comes with an auditable transcript. - **One 80 GB GPU, one command (Tag and Flash).** Served by vLLM-Omni; up to 16 (Tag) or 40 (Flash) concurrent users per GPU. Apache-2.0 code and weights, free for commercial use. ## Quick start (Tag) Quickest: ```bash pip install sophon-realtime # Python 3.12, Linux, CUDA sophon-realtime serve # downloads the Tag weights (~42 GB) on first run, then serves on port 8000 ``` Or use the one-command scripts (isolated environment): ```bash git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/tag bash install.sh # venv + vLLM-Omni + plugin, and downloads the Tag weights (~42 GB) GPU=0 PORT=8000 bash start.sh # first start 15–20 min, later 5–8 min python client_example.py --wav my_question.wav --lang en --voice ryan_en ``` - APIs: `ws://HOST:8000/v1/realtime` (full duplex: turn-taking, barge-in, back-channels) and `POST /v1/chat/completions` (audio in, text + audio out). - Flash works the same way; see the [Flash page](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Flash). To just try it, talk to it in the [ModelScope Studio](https://www.modelscope.ai/studios/AdventistsAI/Sophon-Realtime). ## Live mode: a drop-in for OpenAI GPT-Live [`live/`](https://github.com/adventists-ai/sophon-realtime/blob/main/live/README.md) in the GitHub repository is a gateway that speaks the OpenAI GPT-Live protocol in front of a Tag or Flash engine: GPT-Live clients switch by changing the URL and key. Sophon keeps the voice; when a request needs a lookup or an action it says a short "Let me check that", hands the task to your backend and speaks the result in its own words. A larger model can also run the back office: with DeepSeek V4 Pro, Big Bench Audio rises from 81.2 to about 83. To start it, run one more command next to a Tag or Flash engine started with the one-command scripts: ```bash cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh # GPT-Live clients: ws://HOST:8780/v1/live/sessions ``` ## Benchmark details ### Conversation: VoiceBench (official evaluators, gpt-4o-mini judge) | model | full-duplex | open weights | overall | |---|:-:|:-:|:-:| | GPT-4o-Audio | — | — | 86.8 | | **Sophon-Realtime v1** | ✅ | ✅ | **82.8** | | GPT-4o-mini-Audio | — | — | 82.8 | | MiniCPM-o 4.5 | ✅ | ✅ | 76.1 | | MiniCPM-o 2.6 | ✅ | ✅ | 71.2 | | Freeze-Omni | ✅ | ✅ | 55.2 | | NemotronLabs VoiceChat | ✅ | ✅ | 55.1 | | PersonaPlex | ✅ | ✅ | 30.6 | | Moshi | ✅ | ✅ | 29.5 | AlpacaEval uses the 199-question standard subset. Other scores come from the VoiceBench leaderboard and from Table 3 of the NemotronLabs VoiceChat paper (arXiv:2609.21967); the MiniCPM-o 4.5 score there is from an independent evaluation. ### Spoken reasoning: Big Bench Audio 1,000 spoken reasoning questions, graded on what the model actually says. | model | open weights | accuracy | |---|:-:|:-:| | GPT-Live-1 | — | 90% | | **Sophon-Realtime v1** | ✅ | **81.8%** | | Raon SpeechChat | ✅ | 58% | | Freeze-Omni | ✅ | 33% | | PersonaPlex | ✅ | 19% | | Moshi | ✅ | 4% | Sophon: self-run, with spoken answers transcribed and judged by gpt-4.1. Others: Artificial Analysis leaderboard. ### Tool calling: Full-Duplex-Bench v3 Disfluent real human speech with multi-step tool calls. 100 tasks, judged by GPT-4o. | model | open weights | Pass@1 ↑ | responds when it should ↑ | |---|:-:|:-:|:-:| | GPT-Realtime | — | 60% | 96% | | Gemini Live 3.1 | — | 54% | — | | **Sophon-Realtime** | ✅ | **48%** | **100%** | | cascaded pipeline | — | 45% | — | | Grok | — | 43% | — | | Ultravox v0.7 | ✅ | 41% | — | Sophon: best of several runs. Others: official Full-Duplex-Bench v3 results. ### Hearing tone: paralinguistic understanding Sophon-Realtime listens with [DuplexJev](https://www.modelscope.cn/models/AdventistsAI/DuplexJev-32B-Para). These benchmarks test emotion, sarcasm, stress and intonation. | model | LISTEN (4 tasks) + StressTest + MSPB, combined | |---|:-:| | **DuplexJev (Sophon-Realtime's ears)** | **48.6** | | Gemini 3.8 Flash | 46.9 | | Qwen3-Omni | 44.6 | ### Speaking with tone: NVV-SuperBench Checks whether the laughs, coughs, sighs and breaths a reply calls for actually show up in the audio. Official scripts, 50 items per category, judged by Gemini 3 Flash; higher F1 is better. | model | Chinese | English | |---|:-:|:-:| | Sophon-Realtime-32B-Tag | **0.86** | **0.80** | | Sophon-Realtime-32B-Flash | 0.77 | 0.63 | | Sophon-Realtime-32B-E2E (research preview) | 0.77 | 0.60 | All three builds speak the same tagged reply texts; only the voice differs. ## How it works - **Listens continuously.** At each pause, the same model decides in a single token whether you've finished, are still talking, are back-channeling or are thinking. - **Drafts early, speaks on time.** A reply is drafted from your first pause and released the moment the model decides it's its turn. If you keep talking, the draft is dropped before you hear it. - **Writes how to say it, not just what.** The brain marks where to laugh, sigh, breathe, cough or stress a word; Tag and Flash perform these marks and never read them aloud. - **Text half a step ahead, voice in step.** Text and voice stream together, so every reply has a transcript for audits, tools and larger backend models. - **Remembers what you actually heard.** History keeps the real audio of both sides. When you interrupt, only the part you heard is kept. - **Exploring end to end (E2E).** The stream of continuous vectors a speech LLM builds while it listens is a language in its own right. The E2E talker writes replies directly in that vector language and a decoder turns it into sound, so hearing and speaking share one representation. It is a research preview; see the [E2E page](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-E2E). ## Fine-tuning Fine-tune the thinker's LLM backbone with LoRA in ms-swift to change persona, tone, domain knowledge and tool use. The audio encoder and the voice stay as released, and turn-taking is not affected. Training continues the bundled tag LoRA (`adapters/tag`), so laughter, coughs and other sound tags are kept. One 80 GB GPU is enough. ```bash pip install "ms-swift>=4.6" git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/finetune modelscope download --model AdventistsAI/Sophon-Realtime-v1 --include 'base/*' 'adapters/*' --local_dir Sophon-Realtime-v1 python -m sophon_swift.adapters to-swift Sophon-Realtime-v1/adapters/tag tag_swift swift sft --custom_register_path sophon_swift/swift_register.py \ --model Sophon-Realtime-v1 --adapters tag_swift --tuner_type lora \ --dataset my_data.jsonl --learning_rate 5e-5 --num_train_epochs 2 \ --gradient_accumulation_steps 8 --gradient_checkpointing true --output_dir output ``` Data format, multi-turn and tool calls, export, serving and regression checks: see [finetune/](https://github.com/adventists-ai/sophon-realtime/tree/main/finetune) in the GitHub repository. ## Files in this repository This repository is the home page of the v1 family. It also holds the thinker base (`base/`) and the tag-writing LoRA (`adapters/`) used for fine-tuning, plus the previous E2E weights (`talker/`, `e2/`, `voices/`). To deploy, use the weights on the [Tag](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Tag) or [Flash](https://www.modelscope.cn/models/AdventistsAI/Sophon-Realtime-32B-Flash) page. ## License Code and weights are released under Apache-2.0. Third-party components keep their own licences; see NOTICE. ## Contact Business enquiries: jiejin@adventists.ai