--- title: DiffSynth-Music canonical_url: "https://www.modelscope.ai/models/DiffSynth-Studio/DiffSynth-Music" md_url: "https://www.modelscope.ai/models/DiffSynth-Studio/DiffSynth-Music.md" repository: DiffSynth-Studio/DiffSynth-Music last_updated: 2026-09-14 license: "Apache License 2.0" pipeline_tag: audio-generation tasks: - audio-generation base_model_relation: finetune parameters: 18.1B tensor_type: - BF16 - F16 library_name: - pytorch - safetensors frameworks: - Pytorch downloads: 68 stars: 3 --- # DiffSynth-Music > DiffSynth-Music - An open-source model by DiffSynth-Studio on ModelScope. DiffSynth-Music is a suite of controllable music generation models built on ACE-Step-1.5, trained with the DiffSynth-Studio framework, and powered by Diffusion-Templates for… DiffSynth-Studio/DiffSynth-Music is a 18.1B-parameter audio-generation model on ModelScope. licensed under Apache License 2.0. - **Repository**: DiffSynth-Studio/DiffSynth-Music - **License**: Apache License 2.0 - **Tasks**: audio-generation - **Parameters**: 18.1B - **Downloads**: 68 - **Stars**: 3 - **Last updated**: 2026-09-14 Source: https://www.modelscope.ai/models/DiffSynth-Studio/DiffSynth-Music --- # DiffSynth-Music [![GitHub](https://img.shields.io/badge/GitHub-%23121011.svg?logo=github&logoColor=white)](https://github.com/modelscope/DiffSynth-Studio) [![Model](https://img.shields.io/badge/Model-8A2BE2.svg?logo=modelscope&logoColor=white)](https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music) [![Report](https://img.shields.io/badge/Report-%23E34F26.svg?logo=readthedocs&logoColor=white)](https://arxiv.org/abs/2609.12774) ## Introduction DiffSynth-Music is a suite of controllable music generation models built on [ACE-Step-1.5](https://modelscope.cn/models/ACE-Step/acestep-v15-xl-sft), trained with the [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) framework, and powered by [Diffusion-Templates](https://diffsynth-studio-doc.readthedocs.io/en/latest/Diffusion_Templates/Introducing_Diffusion_Templates.html) for controllable generation. It supports five control modes: |Control Mode|Model Component|Description| |-|-|-| |Beats|`template_control/model.safetensors`|Aligns the beat of the generated music tightly with the input beat sequence.| |Vocals|`template_control/model.safetensors`|Keeps the vocals consistent with the input vocal track, while the model generates the accompaniment.| |Accompany|`template_control/model.safetensors`|Keeps the accompaniment consistent with the input accompaniment, while the model generates the vocals.| |Prosody|`template_prosody/model.safetensors`|Controls the prosody of the vocal part, matching the timing and vocal style of every syllable to the input prosody.| |Reference (experimental)|`template_reference/model.safetensors`|Generates new music by referencing the style, melody, singing style, timbre, and other characteristics of the input audio.| ## Inference Code and Examples ### Installation: ``` git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e .[audio] ``` ### Loading the Models ```python import torch, torchaudio from diffsynth.pipelines.diffsynth_music import DiffSynthMusicPipeline, ModelConfig from diffsynth.diffusion.template import TemplatePipeline from diffsynth.core.data.operators import LoadMultiTrackAudio from diffsynth.utils.music_tools import extract_prosody, generate_click from modelscope import snapshot_download pipe = DiffSynthMusicPipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="transformer/model.safetensors"), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="conditioner/model.safetensors"), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/model.safetensors"), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="vae/model.safetensors"), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="track_separator/model.safetensors", computation_dtype=torch.float32), ], tokenizer_config=ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/"), ) template = TemplatePipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_control/"), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_prosody/"), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_reference/"), ], ) ```
Loading the models on low-VRAM GPUs ```python import torch, torchaudio from diffsynth.pipelines.diffsynth_music import DiffSynthMusicPipeline, ModelConfig from diffsynth.diffusion.template import TemplatePipeline from diffsynth.core.data.operators import LoadMultiTrackAudio from diffsynth.utils.music_tools import extract_prosody, generate_click from modelscope import snapshot_download vram_config = { "offload_dtype": "disk", "offload_device": "disk", "onload_dtype": "disk", "onload_device": "disk", "preparing_dtype": torch.bfloat16, "preparing_device": "cuda", "computation_dtype": torch.bfloat16, "computation_device": "cuda", } vram_config_cpu = { "offload_dtype": torch.bfloat16, "offload_device": "cpu", "onload_dtype": torch.bfloat16, "onload_device": "cpu", "preparing_dtype": torch.bfloat16, "preparing_device": "cuda", "computation_dtype": torch.bfloat16, "computation_device": "cuda", } vram_config_fp32 = { "offload_dtype": torch.float32, "offload_device": "cpu", "onload_dtype": torch.float32, "onload_device": "cpu", "preparing_dtype": torch.float32, "preparing_device": "cuda", "computation_dtype": torch.float32, "computation_device": "cuda", } pipe = DiffSynthMusicPipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="transformer/model.safetensors", **vram_config), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="conditioner/model.safetensors", **vram_config), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/model.safetensors", **vram_config), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="vae/model.safetensors", **vram_config_cpu), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="track_separator/model.safetensors", **vram_config_fp32), ], tokenizer_config=ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="text_encoder/"), vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5, ) template = TemplatePipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_control/"), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_prosody/"), ModelConfig(model_id="DiffSynth-Studio/DiffSynth-Music", origin_file_pattern="template_reference/"), ], lazy_loading=True, ) ```
### Basic Generation Capability DiffSynth-Music inherits its basic text-to-music generation capability from [ACE-Step-1.5](https://modelscope.cn/models/ACE-Step/acestep-v15-xl-sft).
Lyrics ``` [Intro] 清新海风里有我们旅途 漆黑海浪上有帆依呀远征 风暴的咆哮不把恐惧藏水手的胸襟 祈祷你像无畏的领航人 懂也不懂的守护航程 你在甲板上留下的刻痕 是我梦的风景 我要送你永不沉的信念 升起代表勇的黑旗幡 我要送你永不沉的誓言 锚连着锚把七海踏遍 你就是烈焰 你就是烈焰 我的血未寒 不灭的烽火燃在你身边 我的血未寒 怒海的狂涛总是起了又平 凝望指着罗盘的星辰 我要把酒全都灌进骨里 陪我一起远行 我要送你永不沉的信念 升起代表勇的黑旗幡 我要送你永不沉的誓言 锚连着锚把七海踏遍 你就是烈焰 你就是烈焰 我的血未寒 不灭的烽火燃在你身边 我的血未寒 祈祷你像无畏的领航人 懂也不懂的守护航程 你在甲板上留下的刻痕 是我梦的风景 我要送你永不沉的信念 升起代表勇的黑旗幡 我要送你永不沉的誓言 锚连着锚把七海踏遍 你就是烈焰 你就是烈焰 我的血未寒 不灭的烽火燃在你身边 我的血未寒 我要送你永不沉的信念 升起代表勇的黑旗幡 我要送你永不沉的誓言 锚连着锚把七海踏遍 你就是烈焰 你就是烈焰 我的血未寒 不灭的烽火燃在你身边 我的血未寒 ```
Code ```python lyrics = "[Intro]\n\n清新海风里有我们旅途\n漆黑海浪上有帆依呀远征\n风暴的咆哮不把恐惧藏水手的胸襟\n祈祷你像无畏的领航人\n懂也不懂的守护航程\n你在甲板上留下的刻痕\n是我梦的风景\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n\n怒海的狂涛总是起了又平\n凝望指着罗盘的星辰\n我要把酒全都灌进骨里\n陪我一起远行\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n\n祈祷你像无畏的领航人\n懂也不懂的守护航程\n你在甲板上留下的刻痕\n是我梦的风景\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n\n我要送你永不沉的信念\n升起代表勇的黑旗幡\n我要送你永不沉的誓言\n锚连着锚把七海踏遍\n你就是烈焰\n你就是烈焰\n我的血未寒\n不灭的烽火燃在你身边\n我的血未寒\n" prompt = "An explosive, high-energy pop-rock track with a strong anime theme song feel." snapshot_download("DiffSynth-Studio/DiffSynth-Music", allow_file_pattern="assets/audio_reference.mp3", local_dir="data") audio = template( pipe, prompt=prompt, negative_prompt=pipe.default_negative_prompt, lyrics=lyrics, duration=240, seed=42, tiled=True, cfg_scale=4, num_inference_steps=50, ) torchaudio.save("audio_1_output.mp3", audio, 48000) ```
### Beats A click track is generated at a fixed BPM (beats per minute) and fed into the Template model as input. Used together with the `bpm` parameter of the base model, it strictly aligns the beat of the generated music. |Input|Output|Output (with beats)| |-|-|-| ||||
Code ```python bpm = 120 duration = 240 beats = generate_click(bpm, duration=duration) torchaudio.save("audio_2_input.mp3", beats, 48000) audio = template( pipe, prompt=prompt, negative_prompt=pipe.default_negative_prompt, lyrics=lyrics, duration=duration, seed=42, tiled=True, cfg_scale=4, num_inference_steps=50, bpm=bpm, template_inputs=[{"model_id": 0, "audio": beats}], negative_template_inputs=[{"model_id": 0, "audio": beats * 0}], ) torchaudio.save("audio_2_output.mp3", audio, 48000) torchaudio.save("audio_2_output_with_beats.mp3", audio + beats, 48000) ```
### Vocals & Accompany [Demucs](https://github.com/facebookresearch/demucs) is used to separate a song into its vocal and instrumental tracks. The model then completes the full song from the input track. |Input|Output| |-|-| ||| |||
Code ```python # Vocals Control audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3") vocals = pipe.extract_track(audio, track="vocals") torchaudio.save("audio_3_input.mp3", vocals, 48000) audio = template( pipe, prompt=prompt, negative_prompt=pipe.default_negative_prompt, lyrics="", duration=vocals.shape[1] / 48000, seed=42, tiled=True, cfg_scale=4, num_inference_steps=50, template_inputs=[{"model_id": 0, "audio": vocals}], negative_template_inputs=[{"model_id": 0, "audio": vocals}], target_audio=vocals, target_track="vocals", ) torchaudio.save("audio_3_output.mp3", audio, 48000) # Accompaniment Music Control audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3") music = pipe.extract_track(audio, track=["drums", "bass", "other"]) torchaudio.save("audio_4_input.mp3", music, 48000) audio = template( pipe, prompt=prompt, negative_prompt=pipe.default_negative_prompt, lyrics=lyrics, duration=music.shape[1] / 48000, seed=42, tiled=True, cfg_scale=4, num_inference_steps=50, template_inputs=[{"model_id": 0, "audio": music}], negative_template_inputs=[{"model_id": 0, "audio": music}], target_audio=music, target_track=["drums", "bass", "other"], ) torchaudio.save("audio_4_output.mp3", audio, 48000) ```
### Prosody Prosody is rhythmic information extracted from the vocal track, in which the pronunciation of every syllable is blurred out while its timing and pitch are preserved. |Input|Output| |-|-| |||
Code ```python audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3") vocals = pipe.extract_track(audio, track="vocals") prosody = extract_prosody(vocals) torchaudio.save("audio_5_input.mp3", prosody, 48000) audio = template( pipe, prompt=prompt, negative_prompt=pipe.default_negative_prompt, lyrics=lyrics, duration=prosody.shape[1] / 48000, seed=42, tiled=True, cfg_scale=4, num_inference_steps=50, template_inputs=[{"model_id": 1, "audio": prosody}], negative_template_inputs=[{"model_id": 1, "audio": prosody}], ) torchaudio.save("audio_5_output.mp3", audio, 48000) ```
### Reference (Experimental) Reference is an experimental feature. The model analyzes the style, melody, singing style, timbre, and other characteristics of the loudest segment of the input audio, and generates new music accordingly. |Input|Output| |-|-| |||
Code ```python audio = LoadMultiTrackAudio(division_factor=3840)("data/assets/audio_reference.mp3") torchaudio.save("audio_6_input.mp3", audio, 48000) audio = template( pipe, prompt="Music", negative_prompt="", # The timbre is controlled by the reference audio. lyrics=lyrics, duration=200, seed=42, tiled=True, cfg_scale=4, num_inference_steps=100, template_inputs=[{"model_id": 2, "audio": audio}], ) torchaudio.save("audio_6_output.mp3", audio, 48000) ```
## Training Code Please refer to the [DiffSynth-Studio documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/DiffSynth-Music.html).