--- title: granite-4.0-h-small-w4a16-llmcompressor canonical_url: "https://www.modelscope.ai/models/AMD/granite-4.0-h-small-w4a16-llmcompressor" md_url: "https://www.modelscope.ai/models/AMD/granite-4.0-h-small-w4a16-llmcompressor.md" repository: AMD/granite-4.0-h-small-w4a16-llmcompressor last_updated: 2026-09-19 license: apache-2.0 pipeline_tag: text-generation tasks: - text-generation model_type: - granitemoehybrid architectures: - GraniteMoeHybridForCausalLM base_model: - ibm-granite/granite-4.0-h-small base_model_relation: quantized parameters: 32.2B tensor_type: - I64 - BF16 - I32 library_name: - pytorch - transformer - safetensors frameworks: - pytorch language: - en downloads: 58 stars: 0 tags: - quantized - int4 - w4a16 - weight-only - 4-bit - llm-compressor - zendnn - compressed-tensors - amd - cpu-inference --- # granite-4.0-h-small-w4a16-llmcompressor > granite-4.0-h-small-w4a16-llmcompressor - An open-source model by AMD on ModelScope. granite-4.0-h-small-w4a16-llmcompressor AMD/granite-4.0-h-small-w4a16-llmcompressor is a 32.2B-parameter text-generation model on ModelScope. licensed under apache-2.0. derived from ibm-granite/granite-4.0-h-small. - **Repository**: AMD/granite-4.0-h-small-w4a16-llmcompressor - **License**: apache-2.0 - **Tasks**: text-generation - **Parameters**: 32.2B - **Base model**: ibm-granite/granite-4.0-h-small - **Tags**: quantized, int4, w4a16, weight-only, 4-bit, llm-compressor, zendnn, compressed-tensors, amd, cpu-inference - **Downloads**: 58 - **Stars**: 0 - **Last updated**: 2026-09-19 Source: https://www.modelscope.ai/models/AMD/granite-4.0-h-small-w4a16-llmcompressor --- # granite-4.0-h-small-w4a16-llmcompressor ## Model Overview - **Model Architecture:** GraniteMoeHybridForCausalLM - **Input:** Text - **Output:** Text - **Source Model:** [granite-4.0-h-small](https://huggingface.co/ibm-granite/granite-4.0-h-small) - **Supported Hardware:** AMD EPYC (CPU inference) - **Preferred Operating System:** Linux - **Inference Engine:** [vLLM](https://docs.vllm.ai/en/latest/) v0.29.0 - **Quantization Framework:** [LLM Compressor](https://github.com/vllm-project/llm-compressor) v0.13.0 - **Quantization Method:** 4-bit Weight-Only Quantization (W4A16) - **Compatible Stack:** - ZenDNN v6.1.0 - ZenTorch v2.13.0.0 - PyTorch v2.13.0.0 - LLM Compressor v0.13.0 - vLLM v0.29.0 - **Published with:** LLM Compressor v0.13.0 This is a quantized version of [granite-4.0-h-small](https://huggingface.co/ibm-granite/granite-4.0-h-small) created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference. ## Quantization The model was quantized from [granite-4.0-h-small](https://huggingface.co/ibm-granite/granite-4.0-h-small) using [LLM Compressor](https://github.com/vllm-project/llm-compressor) with the GPTQ algorithm. This reduces the model weights from 60.0 GiB to 16.1 GiB on disk (~73% reduction). - **Method:** 4-bit Weight-Only Quantization (W4A16) - **Config:** `compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128, actorder=static` - **Weights:** INT4 (4-bit integer, symmetric, group-wise), stored in `pack-quantized` format - **Activations:** BF16 (unquantized) - **Group Size:** 128 - **Calibration:** 128 samples from `HuggingFaceH4/ultrachat_200k`, sequence length 2048 granite-4.0-h-small is a hybrid Mamba-MoE model: of its 40 layers, 4 are full-attention blocks and the other 36 are Mamba (linear-attention) blocks, and every layer carries a 72-expert MoE block (top-10 routing) alongside a shared MLP. - **Quantized:** all 72 routed experts in every layer (`block_sparse_moe.experts.*.{gate,up,down}_proj`), the shared MLP (`shared_mlp.{input,output}_linear`), the Mamba projections (`mamba.{in,out}_proj`), and `self_attn.{q,k,v,o}_proj` in the 4 full-attention layers. - **Kept in BF16:** the MoE routers (`block_sparse_moe.router`), the Mamba state-space internals that are not Linear layers (`conv1d`, `A_log`, `D`, `dt_bias`, and the gated `mamba.norm`), `lm_head`, `embed_tokens`, and the layer norms. Two details make this work. The model is loaded inside `load_context()`, which is the LLM Compressor v0.13 MoE linearization path: it exposes the fused expert tensors as individual Linear submodules so GPTQ can build a Hessian per expert, with no manual module swap. And the router is skipped because it is a tiny Linear whose logits decide expert assignment, where 4-bit rounding error can flip the top-k selection and change which experts run. ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer from datasets import load_dataset from llmcompressor import oneshot from llmcompressor.modifiers.quantization.gptq import GPTQModifier from llmcompressor.utils import load_context model_id = "ibm-granite/granite-4.0-h-small" output_dir = "./granite-4.0-h-small-w4a16-llmcompressor" CALIB_SIZE = 128 MAX_SEQ_LENGTH = 2048 # Step 1: Load the BF16 model inside load_context(), which linearizes the MoE # experts so GPTQ can target them as ordinary Linear modules. with load_context(AutoModelForCausalLM): model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="cpu", trust_remote_code=True, ) tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) # Step 2: Build the GPTQ calibration set. ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{CALIB_SIZE}]") ds = ds.map( lambda ex: {"text": "\n".join(m["content"] for m in ex["messages"] if m.get("content"))}, remove_columns=ds.column_names, ) if not getattr(tokenizer, "pad_token", None): tokenizer.pad_token = tokenizer.eos_token calib_ds = ds.map( lambda ex: tokenizer( ex["text"], truncation=True, max_length=MAX_SEQ_LENGTH, add_special_tokens=False ), remove_columns=["text"], ) # Step 3: Define the W4A16 GPTQ recipe. Experts, shared MLP, Mamba projections # and attention are all quantized; only lm_head and the MoE router are skipped. recipe = GPTQModifier( scheme="W4A16", targets=["Linear"], ignore=["lm_head", "re:.*block_sparse_moe.router"], ) # Step 4: One-shot quantize with calibration data and save in # compressed-tensors format. oneshot( model=model, dataset=calib_ds, recipe=recipe, max_seq_length=MAX_SEQ_LENGTH, tokenizer=tokenizer, output_dir=output_dir, trust_remote_code_model=True, ) # Smoke test inputs = tokenizer("What are we having for dinner?", return_tensors="pt") with torch.no_grad(): output = model.generate(**inputs, max_new_tokens=30) print(tokenizer.decode(output[0], skip_special_tokens=True)) ``` ## Quick Start ### Use with vLLM ```python from vllm import LLM, SamplingParams model = LLM( model="amd/granite-4.0-h-small-w4a16-llmcompressor", dtype="bfloat16", trust_remote_code=True, ) sampling_params = SamplingParams(temperature=0.7, max_tokens=256) outputs = model.generate(["Hello, how are you?"], sampling_params) print(outputs[0].outputs[0].text) ``` ### Requirements ``` torch==2.13.0.0 zentorch==2.13.0.0 vllm==0.29.0 llmcompressor==0.13.0 ``` ### OpenMP Setup For optimal performance, set `LD_PRELOAD` with `libomp.so` (LLVM OpenMP) or `libiomp5.so` (Intel OpenMP): ```bash # Using LLVM OpenMP (llvmopenmp) export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1) # Or using Intel OpenMP (libiomp) export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1) ``` > **Note:** Set `LD_PRELOAD` before launching vLLM or any inference script. ## Evaluation The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) with the vLLM engine. | Benchmark | BF16 Baseline | W4A16 (this model) | Recovery | | --- | --- | --- | --- | | GSM8K (5-shot) | 0.8643 | 0.8658 | 100.17% | ### Evaluation Command ```bash lm_eval \ --model vllm \ --model_args pretrained=amd/granite-4.0-h-small-w4a16-llmcompressor,dtype=bfloat16 \ --tasks gsm8k \ --batch_size auto \ --trust_remote_code \ --num_fewshot 5 \ --apply_chat_template \ --log_samples \ --gen_kwargs "max_gen_toks=2048" \ --output_path . ``` ## Limitations - **Version Lock:** This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.13.0.0 / PyTorch v2.13.0.0. It may not load correctly on other versions. - **CPU Only:** This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference. - **Hybrid Architecture:** The Mamba state-space internals (`conv1d`, `A_log`, `D`, `dt_bias`, gated norms) are not Linear layers and stay in BF16, so the INT4 saving applies to the projections, experts and attention rather than to the full recurrent path. ## License This model is distributed under the same license as the source model. See the [LICENSE](./LICENSE) file for details. Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.