---
title: gemma-4-12B-it-w4a16-llmcompressor
canonical_url: "https://www.modelscope.ai/models/AMD/gemma-4-12B-it-w4a16-llmcompressor"
md_url: "https://www.modelscope.ai/models/AMD/gemma-4-12B-it-w4a16-llmcompressor.md"
repository: AMD/gemma-4-12B-it-w4a16-llmcompressor
last_updated: 2026-09-20
license: gemma
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - gemma4_unified
architectures:
  - Gemma4UnifiedForConditionalGeneration
base_model:
  - RedHatAI/gemma-4-12B-it
base_model_relation: quantized
parameters: 12.0B
tensor_type:
  - BF16
  - I32
  - I64
library_name:
  - pytorch
  - transformer
  - safetensors
frameworks:
  - pytorch
language:
  - en
downloads: 66
stars: 0
tags:
  - quantized
  - int4
  - w4a16
  - weight-only
  - 4-bit
  - llm-compressor
  - zendnn
  - compressed-tensors
  - amd
  - cpu-inference
---

# gemma-4-12B-it-w4a16-llmcompressor

> gemma-4-12B-it-w4a16-llmcompressor - An open-source model by AMD on ModelScope. gemma-4-12B-it-w4a16-llmcompressor

AMD/gemma-4-12B-it-w4a16-llmcompressor is a 12.0B-parameter text-generation model on ModelScope. licensed under gemma. derived from RedHatAI/gemma-4-12B-it.

- **Repository**: AMD/gemma-4-12B-it-w4a16-llmcompressor
- **License**: gemma
- **Tasks**: text-generation
- **Parameters**: 12.0B
- **Base model**: RedHatAI/gemma-4-12B-it
- **Tags**: quantized, int4, w4a16, weight-only, 4-bit, llm-compressor, zendnn, compressed-tensors, amd, cpu-inference
- **Downloads**: 66
- **Stars**: 0
- **Last updated**: 2026-09-20

Source: https://www.modelscope.ai/models/AMD/gemma-4-12B-it-w4a16-llmcompressor

---

# gemma-4-12B-it-w4a16-llmcompressor

## Model Overview

- **Model Architecture:** Gemma4UnifiedForConditionalGeneration
  - **Input:** Text
  - **Output:** Text
- **Source Model:** [gemma-4-12B-it](https://huggingface.co/RedHatAI/gemma-4-12B-it)
- **Supported Hardware:** AMD EPYC (CPU inference)
- **Preferred Operating System:** Linux
- **Inference Engine:** [vLLM](https://docs.vllm.ai/en/latest/) v0.26.0
- **Quantization Framework:** [LLM Compressor](https://github.com/vllm-project/llm-compressor) v0.12.0
- **Quantization Method:** 4-bit Weight-Only Quantization (W4A16)
- **Compatible Stack:**
  - ZenDNN v6.1.0
  - ZenTorch v2.11.0.3
  - PyTorch v2.11.0
  - LLM Compressor v0.12.0
  - vLLM v0.26.0

This is a quantized version of [gemma-4-12B-it](https://huggingface.co/RedHatAI/gemma-4-12B-it) created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

## Quantization

The model was quantized from [gemma-4-12B-it](https://huggingface.co/RedHatAI/gemma-4-12B-it) using [LLM Compressor](https://github.com/vllm-project/llm-compressor) via the GPTQ algorithm.

- **Method:** 4-bit Weight-Only Quantization (W4A16)
- **Config:** `compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128`
- **Weights:** INT4, symmetric, group-wise (`group_size=128`, `actorder=static`), stored as `pack-quantized`
- **Activations:** BF16 (unquantized)
- **Kept in BF16:** vision tower (`vision_tower`), vision embedder (`embed_vision`), audio embedder (`embed_audio`), multimodal projector (`vision_embedder`), multimodal modules (`multi_modal`), and `lm_head`
- **Calibration:** 128 examples from [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) at a max sequence length of 2048

```python
import torch
import transformers
from datasets import load_dataset
from transformers import AutoConfig, AutoProcessor, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier

model_id = "RedHatAI/gemma-4-12B-it"
output_dir = "./gemma-4-12B-it-w4a16-llmcompressor"

NUM_CALIBRATION_SAMPLES = 128
MAX_SEQUENCE_LENGTH = 2048

# Step 1: Load the BF16 model and tokenizer. Gemma 4 is an encoder-free
# multimodal model, so load via the architecture named in its own config:
# AutoModelForCausalLM would route to the inner text model and save a
# text-only config.json that vLLM rejects.
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
model_cls = getattr(transformers, config.architectures[0])
model = model_cls.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Load calibration data. GPTQ is data-driven: it needs real activations
# to build the per-layer Hessians used to compensate the rounding error.
ds = load_dataset(
    "HuggingFaceH4/ultrachat_200k",
    split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]",
)
ds = ds.map(
    lambda example: {"text": "\n".join(m["content"] for m in example["messages"])},
    remove_columns=ds.column_names,
)

# Step 3: Define the W4A16 GPTQ recipe, leaving vision and audio embedders in BF16
recipe = GPTQModifier(
    scheme="W4A16",
    targets="Linear",
    ignore=[
        "lm_head",
        r"re:.*embed_vision.*",
        r"re:.*embed_audio.*",
        r"re:.*vision_embedder.*",
        r"re:.*vision_tower.*",
        r"re:.*multi_modal.*",
    ],
)

# Step 4: One-shot quantize with calibration and save in compressed-tensors format
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# Step 5: Save the processor; oneshot does not write it and vLLM needs it
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
processor.save_pretrained(output_dir)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Quick Start

### Use with vLLM

```python
from vllm import LLM, SamplingParams

model = LLM(
    model="amd/gemma-4-12B-it-w4a16-llmcompressor",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)
```

### Requirements

```
torch==2.11.0
zentorch==2.11.0.3
vllm==0.26.0
llmcompressor==0.12.0
```

### OpenMP Setup

For optimal performance, set `LD_PRELOAD` with `libomp.so` (LLVM OpenMP) or `libiomp5.so` (Intel OpenMP):

```bash
# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)
```

> **Note:** Set `LD_PRELOAD` before launching vLLM or any inference script.

## Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) with the vLLM engine.

| Benchmark | BF16 Baseline | W4A16 (this model) | Recovery |
| --- | --- | --- | --- |
| GSM8K (5-shot) | 0.9401 | 0.9249 | 98.38% |

### Evaluation Command

```bash
lm_eval \
    --model vllm \
    --model_args pretrained=amd/gemma-4-12B-it-w4a16-llmcompressor,dtype=bfloat16,max_model_len=4096,language_model_only=True,enable_thinking=False,reasoning_parser=gemma4 \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .
```

## Limitations

- **Version Lock:** This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.11.0.3 / PyTorch v2.11.0. It may not load correctly on other versions.
- **CPU Only:** This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
- **Accuracy Trade-off:** 4-bit weight-only quantization is more aggressive than INT8. This checkpoint recovers 98.38% of the BF16 GSM8K score.

## License

This model is distributed under the same license as the source model. See the [LICENSE](./LICENSE) file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.