---
title: Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0
canonical_url: "https://www.modelscope.ai/models/AMD/Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0"
md_url: "https://www.modelscope.ai/models/AMD/Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0.md"
repository: AMD/Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0
last_updated: 2026-06-30
license: apache-2.0
pipeline_tag: image-text-to-text
tasks:
  - image-text-to-text
model_type:
  - qwen3_vl
architectures:
  - Qwen3VLForConditionalGeneration
base_model:
  - Qwen/Qwen3-VL-8B-Instruct
base_model_relation: quantized
parameters: 8.9B
tensor_type:
  - I8
  - BF16
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 136
stars: 0
tags:
  - quantized
  - int4
  - woq
  - 4-bit
  - symmetric
  - per-group
  - w4a16
  - zendnn
  - torchao
  - amd
  - cpu-inference
---

# Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0

> Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0 - An open-source model by AMD on ModelScope. Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0

AMD/Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0 is a 8.9B-parameter image-text-to-text model on ModelScope. licensed under apache-2.0. derived from Qwen/Qwen3-VL-8B-Instruct.

- **Repository**: AMD/Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0
- **License**: apache-2.0
- **Tasks**: image-text-to-text
- **Parameters**: 8.9B
- **Base model**: Qwen/Qwen3-VL-8B-Instruct
- **Tags**: quantized, int4, woq, 4-bit, symmetric, per-group, w4a16, zendnn, torchao, amd, cpu-inference
- **Downloads**: 136
- **Stars**: 0
- **Last updated**: 2026-06-30

Source: https://www.modelscope.ai/models/AMD/Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0

---

# Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0

## Model Overview

- **Model Architecture:** Qwen3VLForConditionalGeneration
  - **Input:** Text
  - **Output:** Text
- **Source Model:** [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
- **Supported Hardware:** AMD EPYC (CPU inference)
- **Preferred Operating System:** Linux
- **Inference Engine:** [vLLM](https://docs.vllm.ai/en/latest/) v0.20.2
- **Quantization Framework:** [TorchAO](https://github.com/pytorch/ao) v0.17.0
- **Quantization Method:** 4-bit Weight-Only (W4A16), Symmetric Per-Group
- **Compatible Stack:**
  - ZenDNN v6.0.0
  - ZenTorch v2.11.0.1
  - PyTorch v2.11.0
  - TorchAO v0.17.0
  - vLLM v0.20.2

This is a quantized version of [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) created by AMD using TorchAO for ZenDNN-optimized CPU inference.

## Quantization

The model was quantized from [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) using [TorchAO](https://github.com/pytorch/ao) v0.17.0.

- **Method:** 4-bit Weight-Only (W4A16), Symmetric Per-Group
- **Config:** `Int4WeightOnlyConfig(group_size=128, mapping_type=MappingType.SYMMETRIC)`
- **Quantized Layers:** All linear layers excluding `lm_head` and `embed_tokens`

**Quantization script:**

```bash
python woq_sym_group.py \
    --model_name Qwen/Qwen3-VL-8B-Instruct \
    --output_dir ./Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0
```


## Quick Start

### Use with vLLM

```python
from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)
```

### Requirements

```
torch==2.11.0
torchao==0.17.0
zentorch==2.11.0.1
vllm==0.20.2
```

### OpenMP Setup

For optimal performance, set `LD_PRELOAD` with `libomp.so` (LLVM OpenMP) or `libiomp5.so` (Intel OpenMP):

```bash
# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)
```

> **Note:** Set `LD_PRELOAD` before launching vLLM or any inference script.

## Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) with the vLLM engine.

| Benchmark | BF16 Baseline | W4A16-TAO-SymGroup (this model) | Recovery |
| --- | --- | --- | --- |
| MMLU (5-shot) | - | - | - |
| GSM8K_COT (8-shot) | - | - | - |
| Perplexity (wikitext2) | - | - | - |

> Evaluation results will be updated after benchmarking.

### Reproduction

```bash
lm_eval \
    --model vllm \
    --model_args pretrained="amd/Qwen3-VL-8B-Instruct-w4a16-tao-symgroup-torchao-v0.17.0" \
    --tasks mmlu \
    --num_fewshot 5 \
    --batch_size auto
```

## Limitations

- **Version Lock:** This model is quantized with TorchAO v0.17.0 and is compatible only with PyTorch v2.11.0 / ZenDNN v6.0.0. It will not load correctly on other PyTorch versions.
- **CPU Only:** This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.


## License

This model is distributed under the same license as the source model. See the [LICENSE](./LICENSE) file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.