--- title: drama-3-int4 canonical_url: "https://www.modelscope.ai/models/fishaudio/drama-3-int4" md_url: "https://www.modelscope.ai/models/fishaudio/drama-3-int4.md" repository: fishaudio/drama-3-int4 last_updated: 2026-10-16 license: other pipeline_tag: text-to-speech tasks: - text-to-speech model_type: - fish_omni architectures: - FishOmniModel base_model: - fishaudio/drama-3 base_model_relation: quantized parameters: 6.9B tensor_type: - I32 - F16 - BF16 library_name: - safetensors - pytorch frameworks: - pytorch language: - en - zh - ja - ko - multilingual downloads: 106 stars: 0 tags: - fish-audio - text-to-speech - voice-cloning - int4 - w4a16 - quantized - sglang - sglang-omni - ml-intern --- # drama-3-int4 > drama-3-int4 - An open-source model by fishaudio on ModelScope. Drama 3 · int4 (hybrid W4A16) fishaudio/drama-3-int4 is a 6.9B-parameter text-to-speech model on ModelScope. licensed under other. derived from fishaudio/drama-3. - **Repository**: fishaudio/drama-3-int4 - **License**: other - **Tasks**: text-to-speech - **Parameters**: 6.9B - **Base model**: fishaudio/drama-3 - **Tags**: fish-audio, text-to-speech, voice-cloning, int4, w4a16, quantized, sglang, sglang-omni, ml-intern - **Downloads**: 106 - **Stars**: 0 - **Last updated**: 2026-10-16 Source: https://www.modelscope.ai/models/fishaudio/drama-3-int4 --- [![Built with ML Intern](https://huggingface.co/datasets/huggingface/badges/resolve/main/built-with-ml-intern-sm.svg)](https://hf.co/chat?mode=ml-intern) # Drama 3 · int4 (hybrid W4A16) **The smallest working Drama 3.** Unofficial int4 quantization of [fishaudio/drama-3](https://huggingface.co/fishaudio/drama-3) — Fish Audio's 35B (3B active) multi-speaker TTS model — with **real eval numbers** and one-command serving through [SGLang Omni](https://github.com/sgl-project/sglang-omni). Thanks so much to [AK](https://x.com/_akhaliq) for the quantization! - **22.1 GB** download (vs 41.5 GB original) — routed experts and selected attention projections use int4; the shared expert and audio decoder stay in bf16. - **Runs on Ampere (A100, sm80)** — GPTQ-Marlin requires compute capability 8.0 or newer. - Published WER and speaker-similarity evaluations, plus an FP8-vs-int4 comparison on Seed-TTS English. - 5 verified sample types: plain TTS, voice cloning, `[sighs]`/`[laughs]` tags, long-form, Chinese. ## Samples (all generated by this checkpoint) | prompt | clip | |---|---| | plain TTS | [int4_plain.wav](samples/int4_plain.wav) | | voice clone (reference clip) | [int4_clone.wav](samples/int4_clone.wav) | | clone + `[sighs]`/`[laughs]` tags | [int4_tags.wav](samples/int4_tags.wav) | | long-form | [int4_long.wav](samples/int4_long.wav) | | Chinese | [int4_zh.wav](samples/int4_zh.wav) | ## Eval ### Seed-TTS English: FP8 vs int4 The two checkpoints were compared on one H200 with the default SGLang Omni server settings: the 1,088 Seed-TTS English prompts, each cloning the voice of its own reference clip, sent as non-streaming requests at concurrency 8. WER is computed on the generated audio transcribed by Qwen3-ASR-1.7B. | Checkpoint | Weights | Latency mean | Latency p95 | RTF mean | Throughput (req/s) | WER (corpus) | WER without samples above 50% | |---|---:|---:|---:|---:|---:|---:|---:| | `fishaudio/drama-3` (FP8 experts) | 37.0 GB | 1.235 s | 1.902 s | 0.281 | 6.47 | 1.99% | 1.32% | | `akhaliq/drama-3-int4` | 20.8 GB | 1.334 s | 2.178 s | 0.320 | 5.99 | 1.63% | 1.42% | *Drama 3 FP8 and int4 checkpoints on Seed-TTS English, concurrency 8, one H200.* The int4 checkpoint is about 8% slower per request and its WER stays at the FP8 level: the gap between the two is about as large as the run-to-run spread of one checkpoint (a second FP8 run measured 1.365 s mean latency and 2.02% WER). ### Community-quant comparison: Whisper large-v3 WER + resemblyzer speaker similarity An additional evaluation compared this checkpoint against [darioooooo0o/drama-3-w4a16-32gb](https://huggingface.co/darioooooo0o/drama-3-w4a16-32gb) (int8 dense layers, targeting 32 GB cards), using the same 8 prompts, the same scorer, and one A100. | | mean WER ↓ | mean WER excl. zh¹ ↓ | speaker sim ↑ | |---|---|---|---| | **this repo (int4 hybrid, 22.1 GB)** | **0.197** | **0.082** | 0.846 | | comparison checkpoint (int8 dense, 24.5 GB) | 0.259 | 0.154 | 0.846 | ¹ The Chinese prompt scored WER 1.0 for **both** checkpoints despite nearly identical transcripts — a jiwer tokenization artifact on unspaced Chinese, not a model difference. Full per-prompt rows in [`eval/eval_results.json`](eval/eval_results.json); clips in [`eval/`](eval/). Notable per-prompt outcomes: the comparison checkpoint hallucinated an ending on the long paragraph (WER 0.667 vs our 0.028); ours truncated the tag line's tail on one take (0.333 vs their 0.25). Speaker similarity is a tie. This small-prompt evaluation uses a different dataset and ASR scorer from the Seed-TTS evaluation above; the WER values should not be compared across the two evaluations. ## Quickstart Install the current SGLang Omni runtime with Fish Audio support: ```bash git clone https://github.com/sgl-project/sglang-omni.git cd sglang-omni uv venv -p 3.12 .venv . .venv/bin/activate uv pip install --prerelease=allow -e ".[fishaudio]" ``` Download and serve the checkpoint: ```bash hf download akhaliq/drama-3-int4 sgl-omni serve \ --model-path akhaliq/drama-3-int4 \ --port 8000 ``` The server reads its quantization from the checkpoint, so it starts with the same command as the original model. No manual loader patch is required with the current runtime. Then use the [standard Drama 3 API](https://sgl-project.github.io/sglang-omni/cookbook/fishaudio_drama3.html): `/v1/audio/speech` with `input`, optional `references` for voice cloning, `[sighs]`/`[laughs]` tags, multi-speaker scenes, and `previous_takes` editing all work as on the original. **Engine weight memory:** 20.8 GB instead of 37.0 GB for the original checkpoint, which leaves more of the engine's share for the KV cache. This is the engine's weight footprint, not total serving memory. An earlier A100-80GB test measured 57.3 GB GPU memory at startup and 57.9 GB peak across the five sample generations, using explicit memory fractions of `--tts_engine.gpu_memory_fraction 0.62 --vocoder.gpu_memory_fraction 0.2`. Those measurements used different settings from the default-server H200 benchmark above. ## What we quantize | component | original | here | notes | |---|---|---|---| | text embeddings, output head, router | bf16 | bf16 | retained in bf16 | | attention `q/k/v/o_proj` of full attention layers | bf16 | **int4** | symmetric 4-bit weights, group size 128 | | linear attention `in_proj_qkv`, `in_proj_z`, `out_proj` | bf16 | **int4** | symmetric 4-bit weights, group size 128 | | linear attention `in_proj_a/b` | bf16 | bf16 | retained in bf16 | | shared-expert MLP | bf16 | **bf16** | int4 passed numerical checks but produced zero audio frames in the earlier serving test; see [`shared_expert_verdict.md`](shared_expert_verdict.md) | | routed MoE experts (30,720 matrices) | FP8 (block) | **int4** | the bulk: 32.2 GB → 16.6 GB | | codebook-MTP audio decoder + codec | bf16 | bf16 | retained in bf16 | The routed MoE experts and quantized attention projections hold symmetric 4-bit weights with one scale per 128 input channels, and run on GPTQ-Marlin kernels. GPTQ-Marlin requires compute capability 8.0 or newer. Quantization method: RTN + per-group clip search (group 128, symmetric), no calibration data. Weights are stored in GPTQ format; GPTQ storage and kernels do not imply that calibration-based GPTQ optimization was used. Reproduce with [`quantize.py`](quantize.py). Quantization statistics are in [`quant_stats.json`](quant_stats.json). The repository also retains [`apply_patch.py`](apply_patch.py) for the earlier patched-loader workflow. That patch fixes fused-name matching (`gate_up_proj`/`qkv_proj`/`in_proj_qkvz`), without which the earlier loader could silently miss int4 weights and return zero audio frames. The current SGLang Omni runtime reads the checkpoint's quantization configuration directly. ## Limitations - **RTN, no calibration** — a GPTQ/AutoRound pass with real calibration data may improve quality; the building blocks are in `quantize.py`. - Tags prompts occasionally truncate a take — regenerate if needed. - The 8-prompt community-quant comparison is a small sample; use the larger Seed-TTS evaluation for the reported FP8-vs-int4 comparison. - Serving has been tested on A100-80GB and H200. Serving on 48 GB and 32 GB cards remains unverified; weight size alone does not establish that the complete serving pipeline fits. ## License Derivative Work of Fish Audio Materials under the **Fish Audio Community License v1.0** ([LICENSE](LICENSE)) — same terms as the original model, including: research/non-commercial use free; commercial use only under USD 500k annual revenue; no hosted TTS/voice-cloning service without a separate license; no cloning real people's voices without consent; do not remove watermarks; do not use outputs to train other speech or foundation models. ## Credits - Model: [Fish Audio](https://fish.audio) — [fishaudio/drama-3](https://huggingface.co/fishaudio/drama-3) · [tech report](https://github.com/fishaudio/fish-speech/blob/main/docs/assets/Fish_Audio_Drama_3_Technical_Report.pdf) - Quantization: [AK](https://x.com/_akhaliq) — [akhaliq/drama-3-int4](https://huggingface.co/akhaliq/drama-3-int4) - Runtime: [SGLang Omni](https://github.com/sgl-project/sglang-omni) and SGLang int4 kernels · [GitHub stars](https://github.com/sgl-project/sglang-omni/stargazers) - Companion quants: [darioooooo0o/drama-3-w4a16](https://huggingface.co/darioooooo0o/drama-3-w4a16) (experts-only int4, first) · [darioooooo0o/drama-3-w4a16-32gb](https://huggingface.co/darioooooo0o/drama-3-w4a16-32gb) (int8 dense, 32GB-card target)