---
title: OmniNFT
canonical_url: "https://www.modelscope.ai/models/zghhui/OmniNFT"
md_url: "https://www.modelscope.ai/models/zghhui/OmniNFT.md"
repository: zghhui/OmniNFT
last_updated: 2026-05-26
license: apache-2.0
pipeline_tag: any-to-any
tasks:
  - any-to-any
base_model:
  - Lightricks/LTX-2
  - Lightricks/LTX-2.3
base_model_relation: adapter
parameters: 616.6M
tensor_type:
  - F32
library_name:
  - lora
  - safetensors
  - pytorch
frameworks:
  - pytorch
downloads: 168
stars: 0
tags:
  - lora
  - reinforcement-learning
  - GRPO
  - T2AV
---

# OmniNFT

> OmniNFT - An open-source model by zghhui on ModelScope. 🎬 OmniNFT RL-LoRA for LTX Video

zghhui/OmniNFT is a 616.6M-parameter any-to-any model on ModelScope. licensed under apache-2.0. derived from Lightricks/LTX-2、Lightricks/LTX-2.3.

- **Repository**: zghhui/OmniNFT
- **License**: apache-2.0
- **Tasks**: any-to-any
- **Parameters**: 616.6M
- **Base model**: Lightricks/LTX-2, Lightricks/LTX-2.3
- **Tags**: lora, reinforcement-learning, GRPO, T2AV
- **Downloads**: 168
- **Stars**: 0
- **Last updated**: 2026-05-26

Source: https://www.modelscope.ai/models/zghhui/OmniNFT

---

# 🎬 OmniNFT RL-LoRA for LTX Video

> 🚀 Modality-wise Omni Diffusion Negative-aware Fine-Tuning for **LTX-2** and **LTX-2.3**~

<a href="https://zghhui.github.io/OmniNFT/"><img src="https://img.shields.io/badge/🌐-Project%20Page-blue" alt="Project Page"/></a>
[![GitHub](https://img.shields.io/badge/GitHub-OmniNFT-blue?logo=github)](https://github.com/zghhui/OmniNFT)
[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://opensource.org/licenses/Apache-2.0)

---

## 📁 Model Files

| Folder | Base Model | Description |
|--------|-----------|-------------|
| `LTX-2-RL-Lora/` | [Lightricks/LTX-2](https://huggingface.co/Lightricks/LTX-2) | RL-LoRA for LTX-2 |
| `LTX-2.3-RL-Lora/` | [Lightricks/LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) | RL-LoRA for LTX-2.3 |

---

## 🛠️ Quick Start

### 1️⃣ Clone the Code

```bash
git clone https://github.com/zghhui/OmniNFT.git
cd OmniNFT
pip install -r requirements.txt
```

### 2️⃣ Download Base Models

Download the base checkpoints from HuggingFace:

```bash
# LTX-2
huggingface-cli download Lightricks/LTX-2 --local-dir ./checkpoints/LTX-2

# LTX-2.3
huggingface-cli download Lightricks/LTX-2.3 --local-dir ./checkpoints/LTX-2.3
```

### 3️⃣ Download RL-LoRA Weights

```bash
# LTX-2 LoRA
huggingface-cli download zghhui/OmniNFT \
    --include "LTX-2-RL-Lora/*" \
    --local-dir ./checkpoints/

# LTX-2.3 LoRA
huggingface-cli download zghhui/OmniNFT \
    --include "LTX-2.3-RL-Lora/*" \
    --local-dir ./checkpoints/
```

### 4️⃣ Merge LoRA into Base Model

```bash
# 🔀 Merge LTX-2 + RL-LoRA
python scripts/merge_lora.py \
    --checkpoint-path ./checkpoints/LTX-2/transformer/model.safetensors \
    --lora-dir ./checkpoints/LTX-2-RL-Lora \
    --output-path ./merged_model_ltx2.safetensors \
    --dtype bf16

# 🔀 Merge LTX-2.3 + RL-LoRA
python scripts/merge_lora.py \
    --checkpoint-path ./checkpoints/LTX-2.3/transformer/model.safetensors \
    --lora-dir ./checkpoints/LTX-2.3-RL-Lora \
    --output-path ./merged_model_ltx23.safetensors \
    --dtype bf16
```

### 5️⃣ Inference 🎥

```bash
python scripts/inference.py \
    --model_path ./merged_model_ltx2.safetensors \
    --gemma_path $GEMMA_MODEL_PATH \
    --prompt "A man plays acoustic guitar on a wooden stage, warm applause from the audience" \
    --seed 42 \
    --output_dir ./results
```

---

## 📝 Citation

If you find this work helpful, please consider citing:

```bibtex
@article{zhang2026omninft,
  title={OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation},
  author={Zhang, Guohui and Ma, XiaoXiao and Huang, Jie and Xu, Hang and Yu, Hu and Fu, Siming and Li, Yuming and Xue, Zeyue and Song, Lin and Huang, Haoyang and Duan, Nan and Zhao, Feng},
  journal={arXiv preprint arXiv:2605.12480},
  year={2026}
}
```

---

## 🙏 Acknowledgements

- 🎬 [Lightricks](https://huggingface.co/Lightricks) for the base models
- 🤗 [HuggingFace PEFT](https://github.com/huggingface/peft) for LoRA support

---

## 📄 License

This project is licensed under the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0).