--- title: TaoMate-H3-FL2VA canonical_url: "https://www.modelscope.ai/models/TaoLiveAIGC/TaoMate-H3-FL2VA" md_url: "https://www.modelscope.ai/models/TaoLiveAIGC/TaoMate-H3-FL2VA.md" repository: TaoLiveAIGC/TaoMate-H3-FL2VA last_updated: 2026-10-15 license: other pipeline_tag: image-to-video tasks: - image-to-video base_model: - MiniMaxAI/MiniMax-H3 base_model_relation: adapter parameters: 620.2M tensor_type: - F32 library_name: - lora - safetensors - pytorch frameworks: - pytorch downloads: 3 stars: 0 tags: - audio-video - streaming - first-last-frame-to-audio-video - lora - safetensors --- # TaoMate-H3-FL2VA > TaoMate-H3-FL2VA - An open-source model by TaoLiveAIGC on ModelScope. https://github.com/user-attachments/assets/15d276f9-02c0-49c6-b98f-98656c36164a TaoLiveAIGC/TaoMate-H3-FL2VA is a 620.2M-parameter image-to-video model on ModelScope. licensed under other. derived from MiniMaxAI/MiniMax-H3. - **Repository**: TaoLiveAIGC/TaoMate-H3-FL2VA - **License**: other - **Tasks**: image-to-video - **Parameters**: 620.2M - **Base model**: MiniMaxAI/MiniMax-H3 - **Tags**: audio-video, streaming, first-last-frame-to-audio-video, lora, safetensors - **Downloads**: 3 - **Stars**: 0 - **Last updated**: 2026-10-15 Source: https://www.modelscope.ai/models/TaoLiveAIGC/TaoMate-H3-FL2VA --- # TaoMate-H3 [![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Model-yellow)](https://huggingface.co/TaoLiveAIGC/TaoMate-H3) [![ModelScope](https://img.shields.io/badge/ModelScope-Model-white?labelColor=%23EF3D5D)](https://modelscope.ai/models/TaoLiveAIGC/TaoMate-H3) ## ๐ŸŽฅ Demo Video https://github.com/user-attachments/assets/15d276f9-02c0-49c6-b98f-98656c36164a
TaoMate-H3 is a low-latency streaming audio-video generation runtime built on [MiniMax H3](https://github.com/MiniMax-AI/MiniMax-H3) **with a 3-step LoRA**. It generates synchronized audio and video in small chunks and supports continuous long-form generation at 480p/768p/1080p resolutions. Developed by the **[Alibaba TaoLive AIGC Team](https://github.com/TaoLiveAIGC)**. Powered by [MiniMax H3](https://github.com/MiniMax-AI/MiniMax-H3). ## ๐Ÿ”ฅ News - ๐Ÿš€ **2026/10/15: We release [TaoMate-H3-FL2VA](https://huggingface.co/TaoLiveAIGC/TaoMate-H3-FL2VA), to our knowledge the first real-time streaming first/last-frame-to-audio-video (FL2VA) method.** Three-step generation combines first/last-frame control with continuous audio-video KV caching. The step-300 online generator LoRA is available now; compact inference code with 5s, 10s, 15s and 30s modes will follow in [TaoMate-H3 on GitHub](https://github.com/TaoLiveAIGC/TaoMate-H3). - ๐Ÿคฉ2026/09/08: We first migrate MiniMax H3 to [TaoMate](https://github.com/TaoLiveAIGC/TaoMate) (T2AV) for streaming. The new models of complete streaming FL2AV and Ref2AV are on the way. Please stay tuned!โค๏ธ ## ๐Ÿš€ Features - **Real-time streaming FL2VA** โ€” first/last-frame control with three-step blockwise generation, persistent clean AV-KV, and a global endpoint deadline. - **Three-step LoRA streaming generation** โ€” each small chunk uses three Stage3 denoising intervals. - **Audio-video joint generation** โ€” speech, sound, and video are generated on one synchronized timeline. - **Low chunk latency** โ€” each small chunk reaches its final latent state well before a full MiniMax H3 request completes. - **Long-form continuity** โ€” clean KV cache and integrated audio guidance preserve visual identity, voice, and motion across prompt boundaries. - **Multiple resolutions** โ€” portrait and landscape generation at 480p, 768p, and aligned 1080p. - **Single-node inference** โ€” supports 4 GPUs or 8 GPUs with TP2 and Ulysses sequence parallelism. ## ๐Ÿงฑ Model Downloads | Models | Download Link | Notes | |-----------------------|-------------------------------------------------------------------------------|-------------------------------| | [MiniMax H3](https://github.com/MiniMax-AI/MiniMax-H3) | ๐Ÿค— [Huggingface](https://huggingface.co/MiniMaxAI/MiniMax-H3) | Base model | TaoMate-H3 T2AV | ๐Ÿค— [Huggingface](https://huggingface.co/TaoLiveAIGC/TaoMate-H3) | Our 3-step LoRA weights for T2AV | TaoMate-H3 FL2VA | ๐Ÿค— [Hugging Face](https://huggingface.co/TaoLiveAIGC/TaoMate-H3-FL2VA) | Real-time streaming FL2VA; 3-step online generator LoRA, step 300, rank 128 / alpha 128 | TaoMate-H3 Ref2AV | **TBD** | **TBD** The FL2VA adapter follows the same three-file layout as the T2AV release: `config.json`, `adapter_config.json`, and `adapter_model.safetensors`. It preserves the online generator weights in float32 and includes the inference configuration; optimizer state, EMA, and training-only score models are excluded. ## ๐Ÿ› ๏ธ T2AV Installation ### Requirements - Linux - Python 3.10 or 3.11 - NVIDIA Hopper/SM90 GPUs; 8 ร— H20 96 GB is the validated configuration - CUDA 12.8 and PyTorch 2.8 - FFmpeg with H.264 and AAC support Create the environment: ```bash conda create -n taomate-h3 python=3.10 -y conda activate taomate-h3 pip install torch==2.8.0 torchvision==0.23.0 \ --index-url https://download.pytorch.org/whl/cu128 pip install triton==3.4.0 vllm==0.11.1 git clone https://github.com/Dao-AILab/flash-attention.git pip install --no-build-isolation ./flash-attention/hopper git clone https://github.com/TaoLiveAIGC/TaoMate-H3.git cd TaoMate-H3 pip install -e . ``` Install FFmpeg on Ubuntu or Debian: ```bash sudo apt-get update sudo apt-get install -y ffmpeg ``` Download the MiniMax H3 FL2VA model: ```bash hf download MiniMaxAI/MiniMax-H3 \ --include "model_index.json" "FL2VA/*" \ --local-dir models/MiniMax-H3 ``` TaoMate-H3 T2AV LoRA weights are available on [Hugging Face](https://huggingface.co/TaoLiveAIGC/TaoMate-H3). The T2AV release is the step-3000 generator EMA adapter (rank 128, alpha 128). T2AV inference downloads it automatically to `models/TaoMate-H3` on first use. To download it in advance: ```bash hf download TaoLiveAIGC/TaoMate-H3 \ --include "config.json" "adapter_config.json" "adapter_model.safetensors" \ --local-dir models/TaoMate-H3 ``` For private or gated models, run `hf auth login` with an account that has access. Downloads also accept the standard `HF_TOKEN` environment variable. The LoRA directory contains: ```text models/TaoMate-H3/ โ”œโ”€โ”€ config.json โ”œโ”€โ”€ adapter_config.json โ””โ”€โ”€ adapter_model.safetensors ``` ## โญ๏ธ T2AV Inference TaoMate-H3 accepts either one prompt through `--prompt` or one prompt per five-second block through `--prompt-json`. Example prompt file: ```json { "prompts": [ "prompt for seconds 0-5", "prompt for seconds 5-10" ], "seeds": [8301, 8301] } ``` Run TaoMate-H3: ```bash python -m taomate_h3 \ --model-root models/MiniMax-H3 \ --prompt-json examples/prompts_10s.json \ --duration 10 \ --resolution 768x1376 \ --gpus 8 \ --devices 0,1,2,3,4,5,6,7 \ --seed 8301 \ --output outputs/demo_10s ``` The command runs the complete pipeline and writes the final video to `outputs/demo_10s/video.mp4`. It starts its own local distributed workers, so no external `torchrun` command is needed. To use a different local LoRA, add `--adapter /path/to/adapter`. Common resolutions: | Format | Portrait | Landscape | |---|---|---| | 480p | `480x864` | `864x480` | | 768p | `768x1376` | `1376x768` | | 1080p | `1088x1920` | `1920x1088` | For an exact 1080-pixel delivery edge, crop the generated 1088-pixel edge after inference. ## Parameters | Parameter | Description | Default | |---|---|---| | `--model-root` | MiniMax H3 directory containing `FL2VA/` | Required | | `--adapter` | Local LoRA directory; omit to download and reuse the official TaoMate-H3 adapter | `models/TaoMate-H3` | | `--prompt` | One prompt reused for every five-second block | โ€” | | `--prompt-json` | JSON file with one prompt per five-second block | โ€” | | `--duration` | Total duration in seconds; must be a multiple of 5 | `5`, or inferred from JSON | | `--resolution` | `WIDTHxHEIGHT`; short edge 480, 768, or 1088; both edges divisible by 32 | `768x1376` | | `--gpus` | Local inference GPU count: `4` or `8` | `8` | | `--devices` | Comma-separated CUDA device IDs | `0` to `gpus-1` | | `--seed` | Authored request seed | `8301` | | `--output` | New or empty output directory | Required | `--prompt` and `--prompt-json` are mutually exclusive. ## Performance and Advantages The following **T2AV** results were measured on one 8 ร— NVIDIA H20 96 GB node with TP2 ร— Ulysses4, a `480x864` canvas, a 10-second output, and seed `8301`. | Metric | TaoMate-H3 | MiniMax H3 | Improvement | |---|---:|---:|---:| | Pure DiT time | 14.810 s | 169.572 s | **11.45ร— faster** | | First final chunk latent | 6.148 s | 170.052 s | **27.66ร— faster** | | Benchmark first playable video | 17.287 s | 183.313 s | **10.60ร— faster** | | Peak DiT memory allocated | 31.37 GiB | 32.03 GiB | - | Pure DiT time excludes model loading, text encoding, VAE decoding, and media encoding. First playable video includes Video VAE decoding and H.264 publication in the matched first-chunk publication benchmark. The table covers the Stage3 generation path and excludes the command's internal audio preparation. A 10-second TaoMate-H3 run contains 24 generation forwards and eight clean-KV updates. ## FL2VA: real-time streaming first/last-frame control The FL2VA inference subproject lives in `src/taomate-fl2av`. Its code will be published separately in the [TaoMate-H3 GitHub repository](https://github.com/TaoLiveAIGC/TaoMate-H3). It includes one example: **seated_001 ยท seated presentation ยท family living-room organization**. Both example faces have been replaced with the same AI-generated fictional identity. The released adapter is the **step-300 online generator LoRA**, with rank 128, alpha 128, and three denoising intervals at states `[0,16,33,49]`. | Option | Native block frames | Native output | Requests | |---|---|---|---| | `--time 30` | `[39] + [34] * 19 + [68]` | 753 frames / 31.375s | Three continuous requests, seven blocks each | | `--time 15` | `[39] + [34] * 8 + [68]` | 379 frames / 15.791667s | Two continuous requests, seven plus three blocks | | `--time 10` | `[39] + [34] * 4 + [68]` | 243 frames / 10.125s | One request, six blocks | | `--time 5` | `[39,85]` | 124 frames / 5.166667s | One request | All modes use 24fps, 768ร—1376, TP2/SP4 on eight GPUs, three-step denoising, recent-seven clean AV-KV and one-shot whole-clip decoding. The streaming method generates successive blocks while carrying clean AV-KV across requests. The compact example runner first prepares frozen Base10 teacher audio, then runs the student and decodes the whole clip once. Its end-to-end wall time therefore includes teacher preparation and final decoding; the T2AV timing table above is not an FL2VA real-time benchmark. Example audio comes from the teacher, not from native student audio generation. Use a separate Python environment with the FL2VA subproject dependencies. Prepare the pinned MiniMax H3 base model following `src/taomate-fl2av/README.md`, then download the adapter and run from the repository root. These commands require the FL2VA subproject to be present in your checkout. ```bash python -m pip install -r src/taomate-fl2av/requirements.txt python -m pip install -e src/taomate-fl2av hf download TaoLiveAIGC/TaoMate-H3-FL2VA \ --include "config.json" "adapter_config.json" "adapter_model.safetensors" \ --local-dir models/TaoMate-H3-FL2VA CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python src/taomate-fl2av/infer.py \ --time 30 --model-root /absolute/path/models/MiniMax-H3 \ --checkpoint models/TaoMate-H3-FL2VA \ --output outputs/seated_001_30 ``` ## License TaoMate-H3 is released under the [MiniMax H3 Community License Agreement](LICENSE). Use and distribution must follow the terms of that license. ## Citation If you find our work helpful, feel free to cite us. ```bibtex @misc{taomate2026, title={TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation}, author={Qijun Gan and Chenwei Zhang and Meiguang Jin and Junfeng Ma and Qiu Shen}, year={2026}, eprint={2607.24359}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2607.24359}, } ``` ## Acknowledgements We thank the teams and contributors behind: - [MiniMax H3](https://github.com/MiniMax-AI/MiniMax-H3) - [Qwen3-VL](https://github.com/QwenLM/Qwen3-VL) - [PyTorch](https://github.com/pytorch/pytorch) - [FlashAttention](https://github.com/Dao-AILab/flash-attention) - [Triton](https://github.com/triton-lang/triton) - [vLLM](https://github.com/vllm-project/vllm)