---
title: TLive-Omni-9B
canonical_url: "https://www.modelscope.ai/models/TaoLiveAIGC/TLive-Omni-9B"
md_url: "https://www.modelscope.ai/models/TaoLiveAIGC/TLive-Omni-9B.md"
repository: TaoLiveAIGC/TLive-Omni-9B
last_updated: 2026-09-15
license: apache-2.0
model_type:
- tlive_omni
architectures:
- TLiveOmniForConditionalGeneration
parameters: 10.1B
tensor_type:
- BF16
library_name:
- pytorch
- transformer
- safetensors
frameworks:
- pytorch
downloads: 63
stars: 0
tags:
- multimodal
- vision
- audio
- video
- custom-code
---
# TLive-Omni-9B
> TLive-Omni-9B - An open-source model by TaoLiveAIGC on ModelScope. TLive-Omni is an omni-modal understanding model for e-commerce live-stream, mapping image, video, audio, and text into a unified text-output interface. Built on a Qwen3.5 backbone with a…
TaoLiveAIGC/TLive-Omni-9B is a 10.1B-parameter machine learning model on ModelScope. licensed under apache-2.0.
- **Repository**: TaoLiveAIGC/TLive-Omni-9B
- **License**: apache-2.0
- **Parameters**: 10.1B
- **Tags**: multimodal, vision, audio, video, custom-code
- **Downloads**: 63
- **Stars**: 0
- **Last updated**: 2026-09-15
Source: https://www.modelscope.ai/models/TaoLiveAIGC/TLive-Omni-9B
---
# TLive-Omni
## 📋 Overview
TLive-Omni is an omni-modal understanding model for e-commerce live-stream, mapping image, video, audio, and text into a unified text-output interface. Built on a Qwen3.5 backbone with a grafted AuT audio encoder, it supports up to 256K tokens of context, trained via a three-stage SFT recipe followed by Faithful-RFT reinforcement fine-tuning.
## ✨ Highlights
- **Timestamped Per-vGrid layout** — Audio and video tokens are organized into timestamped grid with explicit boundaries, keeping audio segments adjacent to their corresponding visual content for fine-grained temporal alignment over long streams.
- **Three-stage SFT recipe** — Progressive training from audio-language alignment to full multimodal SFT, developing live-commerce understanding from omni-modal perception to instruction-following responses.
- **Faithful-RFT** — A reinforcement fine-tuning stage for faithful and real-time live-stream demands, suppressing explicit reasoning traces and directly optimizing answer quality for live-commerce tasks.
- **Rich atomic capabilities** — A scenario-oriented taxonomy covering speech recognition, speaker analysis, product visual grounding, text recognition, temporal grounding, video dense caption, and omni-modal QA, etc, supported by a compact data production engine.
- **Strong live-commerce performance with competitive generalization** — 4B and 9B variants demonstrate strong results across live-commerce audio, image, and video tasks, together with excellent generalization on general benchmarks.
## 🏗️ Architecture
TLive-Omni is built on a Qwen3.5 backbone and extends it with a audio encoder through a lightweight MLP aligner, forming a unified text-output omni-modal understanding model. For video inputs with audio, each temporal grid is organized into a timestamped grid that interleaves video and audio token blocks, keeping audio segments adjacent to their corresponding visual content. The model supports up to 256K tokens of context at inference.
## 📊 Benchmark Results
We evaluate TLive-Omni-4B and TLive-Omni-9B on both live-commerce tasks and general benchmarks. Dash (-) denotes an unreported result or undisclosed parameter count. The Best results among the compared open-source models are marked in **bold**, while the second-best results are in underlined.
### Live-Commerce Evaluation
Click to expand
| Task |
Metric |
TLive-Omni 4B |
TLive-Omni 9B |
Gemini 2.5 Flash |
Gemini 2.5 Pro |
Gemini 3 Flash |
Gemini 3 Pro |
Gemini 3.5 Flash |
Qwen3.5-Omni Flash |
OmniVinci 9B |
Nemotron 3 Nano Omni 30B-A3B |
Ming-Lite-Omni v1.5 20B-A3B |
MiniCPM-o 2.6 8B |
MiniCPM-o 4.5 9B |
Qwen2.5-Omni 7B |
Qwen3-Omni 30B-A3B |
| Audio |
| Live-Commerce ASR |
CER ↓ |
6.66 |
6.46 |
16.30 |
11.48 |
15.18 |
12.09 |
13.09 |
6.81 |
— |
12.10 |
10.06 |
13.88 |
10.70 |
7.86 |
6.75 |
| Speaker-Attributed ASR |
cpWER ↓ |
12.88 |
12.27 |
17.14 |
12.17 |
19.04 |
11.67 |
11.99 |
13.23 |
— |
17.65 |
— |
— |
18.89 |
— |
27.84 |
| Audio Description |
Acc. ↑ |
76.12 |
75.96 |
65.21 |
81.10 |
68.27 |
85.07 |
79.97 |
62.82 |
39.90 |
33.01 |
45.99 |
49.84 |
47.59 |
47.92 |
61.06 |
| Audio Description |
Hal. ↓ |
20.97 |
21.00 |
26.19 |
14.16 |
26.17 |
10.92 |
14.36 |
27.81 |
47.36 |
39.77 |
44.05 |
41.76 |
52.41 |
36.92 |
30.22 |
| Audio QA |
Acc. ↑ |
72.60 |
76.28 |
76.28 |
82.85 |
74.68 |
88.62 |
87.99 |
78.04 |
66.51 |
64.90 |
40.54 |
39.74 |
42.47 |
61.38 |
76.76 |
| Image |
| Visual Grounding |
Live AP ↑ |
82.85 |
82.33 |
61.08 |
51.98 |
80.38 |
73.80 |
84.15 |
79.96 |
34.86 |
73.08 |
52.46 |
3.82 |
23.90 |
75.61 |
79.22 |
| Visual Grounding |
Prod AP ↑ |
91.45 |
89.96 |
28.81 |
32.63 |
65.67 |
58.83 |
74.89 |
60.44 |
8.93 |
48.62 |
40.73 |
1.77 |
53.63 |
22.85 |
68.88 |
| Text Understanding |
Loc. F1 ↑ |
86.99 |
87.59 |
20.52 |
31.60 |
61.11 |
68.60 |
64.44 |
74.07 |
50.25 |
52.91 |
13.27 |
5.74 |
5.43 |
42.64 |
30.46 |
| Text Understanding |
Rec. NED ↓ |
4.72 |
4.24 |
43.28 |
27.82 |
16.25 |
9.72 |
16.64 |
12.48 |
32.77 |
29.42 |
59.16 |
77.58 |
71.65 |
37.79 |
14.83 |
| Text Understanding |
Cls. Acc. ↑ |
79.06 |
79.85 |
51.21 |
61.86 |
69.11 |
76.86 |
69.76 |
53.25 |
57.29 |
37.86 |
32.94 |
15.92 |
11.62 |
51.25 |
69.46 |
| Video |
| Temporal Grounding |
mIoU ↑ |
77.63 |
81.49 |
76.50 |
76.22 |
77.43 |
77.90 |
77.90 |
62.10 |
13.10 |
23.39 |
14.34 |
14.56 |
43.20 |
30.83 |
39.22 |
| Dense Caption |
Acc. ↑ |
69.23 |
74.63 |
54.60 |
41.95 |
32.21 |
37.80 |
33.80 |
32.94 |
18.59 |
17.96 |
13.81 |
10.53 |
21.06 |
16.51 |
21.44 |
| Dense Caption |
Hal. ↓ |
9.57 |
8.76 |
10.97 |
16.88 |
20.76 |
20.99 |
17.30 |
20.91 |
27.13 |
16.62 |
39.33 |
26.93 |
28.61 |
36.44 |
25.82 |
| Video QA |
Acc. ↑ |
92.31 |
93.23 |
88.21 |
92.62 |
89.64 |
84.36 |
86.90 |
87.28 |
72.51 |
82.56 |
64.51 |
60.30 |
84.62 |
75.48 |
81.62 |
| Shot Understanding |
Layout ↑ |
78.40 |
77.00 |
80.00 |
85.20 |
76.80 |
80.40 |
83.40 |
84.40 |
73.60 |
79.20 |
74.60 |
66.60 |
78.20 |
74.40 |
82.20 |
| Shot Understanding |
Shot Size ↑ |
51.20 |
51.00 |
46.80 |
50.80 |
45.70 |
43.40 |
44.20 |
48.90 |
52.70 |
34.00 |
41.70 |
38.30 |
42.80 |
38.10 |
37.40 |
| Shot Understanding |
Camera ↑ |
80.90 |
82.00 |
84.20 |
76.00 |
78.50 |
75.70 |
75.50 |
85.50 |
68.10 |
80.20 |
72.80 |
68.30 |
79.20 |
81.20 |
76.10 |
| Shot Understanding |
Content ↑ |
69.80 |
71.00 |
68.60 |
70.40 |
71.60 |
74.80 |
70.20 |
66.40 |
49.20 |
58.60 |
51.60 |
49.00 |
66.60 |
68.00 |
63.60 |
### General Benchmark: Image Understanding
Click to expand
| Model |
Params |
MMMU |
MathVista |
DynaMath |
VLMsAreBlind |
MMBench(EN-DEV-v1.1) |
RealWorldQA |
MMStar |
SimpleVQA |
Hallusion |
AI2D |
OCRBench |
CC-OCR |
CharXiv(RQ) |
RefCOCO |
ERQA |
EmbSpatial |
| Open-source VLM models |
| MiMo-VL-SFT |
7B |
64.6 |
81.8 |
46.9 |
78.0 |
84.5 |
— |
— |
— |
— |
83.2 |
87.6 |
— |
54.4 |
85.7 |
— |
— |
| SAIL-VL2 |
8B |
55.4 |
76.4 |
17.8 |
— |
— |
76.3 |
70.7 |
— |
55.1 |
87.7 |
91.3 |
— |
— |
74.0 |
— |
— |
| Valley2.5 |
8B |
62.1 |
74.4 |
32.7 |
— |
85.5 |
70.5 |
67.3 |
— |
56.3 |
84.4 |
87.0 |
— |
— |
— |
— |
— |
| LLaVA-OneVision-2 |
8B |
— |
— |
— |
— |
85.7 |
69.7 |
64.8 |
— |
— |
84.3 |
78.2 |
— |
— |
— |
43.3 |
78.1 |
| InternVL3.5 |
4B |
66.6 |
77.1 |
35.7 |
— |
80.3 |
66.3 |
65.0 |
— |
44.8 |
82.6 |
82.2 |
— |
39.6 |
89.4 |
38.5 |
— |
| InternVL3.5 |
8B |
73.4 |
78.4 |
37.7 |
— |
79.5 |
67.5 |
69.3 |
— |
54.5 |
84.0 |
84.0 |
— |
44.4 |
89.7 |
41.0 |
73.2 |
| Qwen3-VL |
4B |
67.4 |
73.7 |
65.3 |
71.9 |
83.9 |
70.9 |
69.8 |
48.0 |
57.6 |
84.1 |
88.1 |
76.2 |
39.7 |
89.0 |
41.3 |
79.6 |
| Qwen3-VL |
8B |
69.6 |
77.2 |
67.7 |
74.0 |
84.5 |
71.5 |
70.9 |
50.2 |
61.1 |
85.7 |
89.6 |
79.9 |
46.4 |
89.1 |
45.8 |
78.5 |
| Qwen3.5 |
4B |
72.1 |
81.0 |
69.6 |
62.3 |
86.3 |
72.5 |
74.8 |
44.6 |
76.9 |
87.1 |
85.9 |
71.1 |
62.9 |
87.6 |
46.8 |
76.6 |
| Qwen3.5 |
9B |
74.2 |
82.2 |
74.6 |
71.8 |
87.7 |
72.9 |
76.3 |
48.9 |
76.0 |
88.0 |
88.5 |
73.4 |
67.5 |
90.0 |
47.3 |
78.7 |
| Open-source Omni models |
| InteractiveOmni |
4B |
61.1 |
61.7 |
— |
— |
78.9 |
— |
62.6 |
— |
52.2 |
83.8 |
80.0 |
— |
— |
— |
— |
— |
| InteractiveOmni |
8B |
66.9 |
68.0 |
— |
— |
81.4 |
— |
66.8 |
— |
61.3 |
84.3 |
83.7 |
— |
— |
— |
— |
— |
| VITA-1.5 |
7B |
52.1 |
66.2 |
— |
— |
76.7 |
— |
59.9 |
— |
44.9 |
79.3 |
73.2 |
— |
— |
— |
— |
— |
| Valley3 |
8B |
69.3 |
— |
— |
— |
— |
— |
— |
— |
55.9 |
— |
— |
— |
— |
— |
— |
— |
| OmniVinci |
9B |
49.7 |
63.5 |
— |
— |
— |
67.5 |
— |
— |
— |
— |
— |
— |
— |
— |
— |
— |
| Nemotron 3 Nano Omni |
30B-A3B |
55.2 |
71.9 |
— |
— |
— |
— |
— |
— |
— |
88.5 |
88.3 |
— |
49.1 |
80.6 |
— |
— |
| Ming-Lite-Omni v1.5 |
20B-A3B |
54.3 |
72.0 |
— |
— |
— |
— |
65.1 |
— |
54.6 |
84.9 |
88.9 |
— |
— |
87.8 |
— |
— |
| MiniCPM-o 2.6 |
8B |
50.4 |
71.9 |
— |
— |
80.5 |
— |
64.0 |
— |
51.9 |
85.8 |
89.7 |
— |
— |
— |
— |
— |
| MiniCPM-o 4.5 |
9B |
67.6 |
— |
— |
— |
87.6 |
— |
73.1 |
— |
63.2 |
87.6 |
87.6 |
— |
— |
— |
— |
— |
| Qwen2.5-Omni |
7B |
59.2 |
67.9 |
— |
— |
81.8 |
70.3 |
64.0 |
— |
— |
83.2 |
— |
— |
— |
87.7 |
— |
— |
| Qwen3-Omni |
30B-A3B |
69.1 |
75.9 |
— |
— |
— |
— |
68.5 |
— |
59.7 |
85.2 |
86.0 |
— |
61.1 |
— |
— |
— |
| Ours |
| TLive-Omni |
4B |
70.9 |
79.9 |
72.5 |
71.8 |
87.0 |
77.7 |
73.9 |
47.6 |
77.7 |
86.6 |
86.6 |
80.5 |
61.3 |
87.4 |
42.3 |
79.3 |
| TLive-Omni |
9B |
73.4 |
81.9 |
73.3 |
75.5 |
88.9 |
76.6 |
75.1 |
50.0 |
76.0 |
88.6 |
90.3 |
81.3 |
63.1 |
90.0 |
48.0 |
80.4 |
### General Benchmark: Video Understanding
Click to expand
| Model |
Params |
MVBench |
MLVU |
Video-MME |
LongVideoBench |
LVBench |
MMVU |
VideoMMMU |
Charades-TL |
ActivityNet-TL |
QVHighlights-TL |
| Open-source VLM models |
| MiMo-VL-SFT |
7B |
— |
— |
66.9 |
— |
— |
— |
53.1 |
39.6 |
35.5 |
41.5 |
| SAIL-VL2 |
8B |
— |
— |
62.7 |
58.3 |
— |
— |
— |
— |
— |
— |
| LLaVA-OneVision-2 |
8B |
66.2 |
76.6 |
71.9 |
66.9 |
55.5 |
56.2 |
— |
53.5 |
53.8 |
66.4 |
| LLaVA-Video |
7B |
58.6 |
70.8 |
63.3 |
58.2 |
44.2 |
47.1 |
36.1 |
15.2 |
14.6 |
10.4 |
| InternVL3.5 |
4B |
71.2 |
70.4 |
65.4 |
60.8 |
43.2 |
47.6 |
57.6 |
16.0 |
14.9 |
17.7 |
| InternVL3.5 |
8B |
72.1 |
70.2 |
66.0 |
62.1 |
46.7 |
60.2 |
— |
27.8 |
31.3 |
31.3 |
| MiniCPM-V 4.5 |
8B |
— |
75.1 |
67.9 |
63.9 |
50.4 |
58.9 |
57.1 |
31.9 |
32.3 |
46.1 |
| LongVU |
7B |
66.9 |
65.4 |
60.6 |
— |
— |
— |
— |
— |
— |
— |
| LongVILA |
7B |
67.1 |
— |
60.1 |
57.1 |
— |
— |
— |
— |
— |
— |
| Mage-VL |
4B |
65.1 |
68.7 |
64.0 |
61.3 |
41.8 |
— |
— |
50.7 |
45.4 |
57.4 |
| Molmo2 |
4B |
75.1 |
63.0 |
69.6 |
68.0 |
53.9 |
51.2 |
50.7 |
33.3 |
39.8 |
58.7 |
| Molmo2 |
8B |
75.9 |
60.2 |
69.9 |
67.5 |
52.8 |
— |
— |
— |
— |
— |
| NVILA |
8B |
68.1 |
70.1 |
64.2 |
57.7 |
— |
— |
— |
— |
— |
— |
| Kangaroo |
8B |
61.1 |
61.0 |
56.0 |
54.8 |
39.4 |
— |
— |
— |
— |
— |
| Video-XL2 |
8B |
— |
74.8 |
66.6 |
61.0 |
48.4 |
50.0 |
39.9 |
38.9 |
30.0 |
46.2 |
| VideoChat3 |
4B |
— |
— |
70.1 |
— |
56.7 |
56.4 |
57.4 |
56.1 |
54.6 |
67.0 |
| VideoLLaMA 3 |
7B |
69.7 |
73.0 |
66.2 |
59.8 |
45.3 |
44.1 |
34.6 |
39.8 |
29.8 |
36.9 |
| Qwen3-VL |
4B |
68.9 |
75.3 |
69.3 |
— |
56.2 |
50.5 |
56.2 |
46.4 |
48.2 |
58.7 |
| Qwen3-VL |
8B |
68.7 |
78.1 |
71.4 |
— |
58.0 |
58.7 |
65.3 |
48.3 |
46.8 |
59.4 |
| Qwen3.5 |
4B |
66.6 |
75.1 |
71.6 |
65.1 |
55.3 |
57.8 |
69.8 |
48.7 |
51.6 |
55.0 |
| Qwen3.5 |
9B |
75.7 |
79.7 |
66.9 |
67.9 |
60.9 |
63.7 |
70.3 |
52.0 |
54.0 |
57.2 |
| Open-source Omni models |
| InteractiveOmni |
4B |
— |
68.0 |
63.3 |
57.0 |
— |
— |
— |
— |
— |
— |
| InteractiveOmni |
8B |
— |
71.6 |
66.0 |
59.1 |
— |
— |
— |
— |
— |
— |
| VITA-1.5 |
7B |
55.4 |
— |
56.1 |
— |
— |
— |
— |
— |
— |
— |
| Valley3 |
8B |
— |
55.6 |
— |
— |
— |
— |
61.2 |
— |
— |
— |
| OmniVinci |
9B |
70.6 |
— |
68.2 |
61.3 |
— |
— |
— |
— |
— |
— |
| Nemotron 3 Nano Omni |
30B-A3B |
— |
— |
70.8 |
— |
— |
— |
— |
— |
— |
— |
| Ming-Lite-Omni v1.5 |
20B-A3B |
69.4 |
— |
67.1 |
59.5 |
— |
— |
— |
— |
— |
— |
| MiniCPM-o 2.6 |
8B |
— |
— |
63.9 |
— |
— |
— |
— |
— |
— |
— |
| MiniCPM-o 4.5 |
9B |
— |
76.5 |
70.4 |
66.0 |
— |
— |
— |
— |
— |
— |
| Qwen2.5-Omni |
7B |
70.3 |
— |
64.3 |
— |
— |
— |
— |
— |
— |
— |
| Qwen3-Omni |
30B-A3B |
— |
75.2 |
70.5 |
— |
— |
— |
— |
— |
— |
— |
| Ours |
| TLive-Omni |
4B |
69.0 |
76.1 |
71.3 |
66.1 |
57.1 |
59.9 |
73.9 |
57.0 |
58.2 |
69.2 |
| TLive-Omni |
9B |
72.5 |
80.9 |
75.6 |
69.9 |
60.8 |
67.1 |
72.8 |
56.3 |
55.4 |
64.1 |
### General Benchmark: Omni Understanding
Click to expand
| Model |
Params |
AVUT |
WorldSense |
VideoHolmes |
DailyOmni |
OmniVideoBench |
FutureOmni |
| Open-source Omni models |
| video-SALMONN 2+ |
3B |
66.2 |
48.3 |
42.2 |
67.7 |
— |
— |
| video-SALMONN 2+ |
7B |
69.5 |
50.9 |
46.9 |
71.8 |
— |
— |
| OmniVinci |
9B |
— |
48.2 |
— |
66.5 |
36.7 |
52.8 |
| Nemotron 3 Nano Omni |
30B-A3B |
— |
55.2 |
— |
74.5 |
— |
— |
| MiniCPM-o 4.5 |
9B |
78.6 |
55.7 |
64.3 |
80.2 |
41.1 |
56.1 |
| Qwen2.5-Omni |
7B |
— |
45.4 |
— |
62.4 |
36.5 |
48.9 |
| Qwen3-Omni |
30B-A3B |
74.2 |
54.0 |
50.4 |
71.9 |
43.8 |
53.4 |
| Ours |
| TLive-Omni |
4B |
78.6 |
54.0 |
57.5 |
78.6 |
41.6 |
57.2 |
| TLive-Omni |
9B |
80.0 |
56.0 |
59.3 |
80.5 |
43.2 |
58.5 |
## ⚙️ Installation
This release targets Python 3.10 on Linux x86_64 with CUDA 12.8 and PyTorch 2.10.0.
```bash
conda create -n tlive python=3.10 -y
conda activate tlive
pip install -r https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/v1.0.0-rc1/environments/requirements.txt
```
The remote `environments/requirements.txt` includes custom wheels for the supported environment and model. If any wheel does not match your hardware, CUDA version, or Python version, replace it with a compatible build for your setup.
## 🚀 Quick Start
### Transformers inference
```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
model_id = "TaoLiveAIGC/TLive-Omni-9B"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
dtype=torch.bfloat16,
device_map="auto",
attn_implementation="flash_attention_2",
).eval()
def generate(messages, *, use_audio_in_video=False, videos_kwargs=None, generation_kwargs=None):
inputs = processor.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
enable_thinking=False,
use_audio_in_video=use_audio_in_video,
videos_kwargs=videos_kwargs or {},
)
prompt_length = inputs["input_ids"].shape[-1]
inputs = inputs.to(model.device)
generation_kwargs = generation_kwargs or {}
with torch.inference_mode():
generated_ids = model.generate(
**inputs,
do_sample=False,
max_new_tokens=1024,
**generation_kwargs,
)
answer_ids = generated_ids[:, prompt_length:]
answer = processor.batch_decode(
answer_ids,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
return answer.strip()
```
Replace `messages` with one of the examples below for text, image, audio, or video inputs.
**Text**
```python
messages = [{
"role": "user",
"content": [{"type": "text", "text": "Briefly explain why multimodal context can improve an answer."}],
}]
print(generate(messages))
```
**Image**
```python
messages = [{
"role": "user",
"content": [
{"type": "image", "path": "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/image.jpg"},
{"type": "text", "text": "Describe this image."},
],
}]
print(generate(messages))
```
**Audio**
```python
messages = [{
"role": "user",
"content": [
{"type": "audio", "path": "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/audio.mp3"},
{"type": "text", "text": "Transcribe and summarize this audio."},
],
}]
print(generate(messages))
```
**Video with audio**
```python
messages = [{
"role": "user",
"content": [
{"type": "video", "path": "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/vocal_video.mp4"},
{"type": "text", "text": "Describe the video, including relevant speech and sounds."},
],
}]
print(generate(messages, use_audio_in_video=True, videos_kwargs={"fps": 1.0}))
```
**Video without audio**
```python
messages = [{
"role": "user",
"content": [
{"type": "video", "path": "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/silence_video.mp4"},
{"type": "text", "text": "Describe the visual events in this video."},
],
}]
print(generate(messages, use_audio_in_video=False, videos_kwargs={"fps": 1.0}))
```
For temporal localization outputs, we recommend the `MM:SS - MM:SS` interval format, for example `01:23 - 01:35`. For videos, set `use_audio_in_video=True` when the audio track should be used, and `False` for visual-only inference.
## ⚡ vLLM
### Installation
First install the pre-built wheel (**Python 3.10 + CUDA 12.8 + Linux x86_64**), built and tested on NVIDIA H20 GPUs (Hopper, `sm_90`):
```bash
pip install https://github.com/TaoLiveAIGC/TLive-Omni/releases/download/v1.0.0-rc1/vllm-0.19.0+cu128-cp310-cp310-linux_x86_64.whl
```
If your GPU, driver, or CUDA setup is not compatible with this wheel, build vLLM from source using the customized code in the `vllm/` directory of the [GitHub release](https://github.com/TaoLiveAIGC/TLive-Omni).
### Inference
```python
from transformers import AutoProcessor
from vllm import LLM, SamplingParams
from vllm.model_executor.models.tlive_omni_processing import process_audio_info
model_id = "TaoLiveAIGC/TLive-Omni-9B"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
def build_prompt(messages):
return processor.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=False,
enable_thinking=False,
)
def generate(inputs, *, limit_mm_per_prompt=None, vllm_kwargs=None, sampling_kwargs=None):
vllm_kwargs = vllm_kwargs or {}
sampling_kwargs = sampling_kwargs or {}
llm = LLM(
model=model_id,
trust_remote_code=True,
dtype="bfloat16",
max_model_len=32768,
tensor_parallel_size=1,
gpu_memory_utilization=0.9,
max_num_seqs=4,
max_num_batched_tokens=32768,
seed=42,
limit_mm_per_prompt=limit_mm_per_prompt,
**vllm_kwargs,
)
outputs = llm.generate(
inputs,
sampling_params=SamplingParams(
temperature=0.0,
max_tokens=1024,
**sampling_kwargs,
),
)
return outputs[0].outputs[0].text.strip()
```
Replace `messages` with one of the examples below for text, image, audio, or video inputs.
**Text**
```python
messages = [{
"role": "user",
"content": [{"type": "text", "text": "Briefly explain why multimodal context can improve an answer."}],
}]
inputs = {"prompt": build_prompt(messages)}
print(generate(inputs))
```
**Image**
```python
image_path = "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/image.jpg"
messages = [{
"role": "user",
"content": [
{"type": "image", "path": image_path},
{"type": "text", "text": "Describe this image."},
],
}]
inputs = {
"prompt": build_prompt(messages),
"multi_modal_data": {"image": [image_path]},
}
print(generate(inputs, limit_mm_per_prompt={"image": 1}))
```
**Audio**
```python
audio_path = "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/audio.mp3"
messages = [{
"role": "user",
"content": [
{"type": "audio", "audio": audio_path},
{"type": "text", "text": "Transcribe and summarize this audio."},
],
}]
inputs = {
"prompt": build_prompt(messages),
"multi_modal_data": {"audio": process_audio_info(messages, use_audio_in_video=False)},
}
print(generate(inputs, limit_mm_per_prompt={"audio": 1}))
```
**Video with audio**
```python
video_path = "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/vocal_video.mp4"
messages = [{
"role": "user",
"content": [
{"type": "video", "video": video_path},
{"type": "text", "text": "Describe the video, including relevant speech and sounds."},
],
}]
inputs = {
"prompt": build_prompt(messages),
"multi_modal_data": {
"video": [video_path],
"audio": process_audio_info(messages, use_audio_in_video=True),
},
"mm_processor_kwargs": {"videos_kwargs": {"fps": 1.0, "use_audio_in_video": True, "return_metadata": True}},
}
print(generate(inputs, limit_mm_per_prompt={"video": 1, "audio": 1}))
```
**Video without audio**
```python
video_path = "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/silence_video.mp4"
messages = [{
"role": "user",
"content": [
{"type": "video", "video": video_path},
{"type": "text", "text": "Describe the visual events in this video."},
],
}]
inputs = {
"prompt": build_prompt(messages),
"multi_modal_data": {"video": [video_path]},
"mm_processor_kwargs": {"videos_kwargs": {"fps": 1.0, "use_audio_in_video": False, "return_metadata": True}},
}
print(generate(inputs, limit_mm_per_prompt={"video": 1}))
```
## 📖 Citation
If you find our work helpful, please consider citing our paper:
```bibtex
@article{hu2026tliveomni,
title={TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming},
author={Hu, Yibo and Qian, Yu and Gu, Mao and Tao, Yingfan and Chen, Yuhao and Luo, Yongdong and Liu, Zhuoqun and Jin, Meiguang and Ma, Junfeng},
journal={arXiv preprint arXiv:2608.20958},
year={2026}
}
```
## 📄 License
This project is released under the [Apache License 2.0](LICENSE).