---
title: OSCAR-LLAMACPP-Qwen3-4B-Thinking-2507-INT2-KV
canonical_url: "https://www.modelscope.ai/models/togethercomputer/OSCAR-LLAMACPP-Qwen3-4B-Thinking-2507-INT2-KV"
md_url: "https://www.modelscope.ai/models/togethercomputer/OSCAR-LLAMACPP-Qwen3-4B-Thinking-2507-INT2-KV.md"
repository: togethercomputer/OSCAR-LLAMACPP-Qwen3-4B-Thinking-2507-INT2-KV
last_updated: 2026-06-09
pipeline_tag: text-generation
tasks:
  - text-generation
library_name:
  - gguf
  - pytorch
frameworks:
  - pytorch
downloads: 148
stars: 0
tags:
  - gguf
  - llama.cpp
  - oscar
  - int2-kv-cache
  - qwen3
  - conversational
---

# OSCAR-LLAMACPP-Qwen3-4B-Thinking-2507-INT2-KV

> OSCAR-LLAMACPP-Qwen3-4B-Thinking-2507-INT2-KV - An open-source model by togethercomputer on ModelScope. Qwen3 4B Thinking 2507 - OSCAR INT2 KV cache (GGUF)

- **Repository**: togethercomputer/OSCAR-LLAMACPP-Qwen3-4B-Thinking-2507-INT2-KV
- **Tasks**: text-generation
- **Tags**: gguf, llama.cpp, oscar, int2-kv-cache, qwen3, conversational
- **Downloads**: 148
- **Stars**: 0
- **Last updated**: 2026-06-09

Source: https://www.modelscope.ai/models/togethercomputer/OSCAR-LLAMACPP-Qwen3-4B-Thinking-2507-INT2-KV

---

# Qwen3 4B Thinking 2507 - OSCAR INT2 KV cache (GGUF)

Qwen3-4B-Thinking-2507 in GGUF, packaged for the **OSCAR ~2-bit (INT2) KV cache** fork of llama.cpp.

This repository contains a rotation-baked GGUF plus the raw calibrated rotation matrices. The model is intended for OSCAR INT2 KV-cache inference: the per-layer rotation is already embedded in the GGUF, so the KV cache can be quantized to 2 bits while retaining near-f16 quality.

## Files

| file | contents | size | what it is for |
|---|---|---:|---|
| `qwen3-4b-rot-kv.gguf` | Qwen3 4B Thinking GGUF with OSCAR rotation baked in | 2.3 GB | Main model for OSCAR INT2 KV-cache inference. |
| `k_rotation_qqt_r_h_pbr.pt` | raw calibrated K rotation | 2.3 MB | Use with a compatible base GGUF if you want to bake the rotation yourself. |
| `v_rotation_sst_r_h_pbr.pt` | raw calibrated V rotation | 2.3 MB | Use with a compatible base GGUF if you want to bake the rotation yourself. |

> "OSCAR rotation" is an in-graph, post-RoPE orthogonal transform applied to K and V so that the KV cache quantizes well at 2-bit. The `-rot-kv` GGUF already contains the rotation tensors; no runtime rotation file is needed for normal inference.

## Run

Use the OSCAR llama.cpp fork built with the backend you need (for example Metal on Apple Silicon).

```bash
LLAMA_KV_FUSED_FA=1 LLAMA_KV_NO_HADAMARD=1 LLAMA_KV_CLIP_RATIO=0.96 \
LLAMA_KV_HP_SINK=512 LLAMA_KV_HP_RECENT=2048 \
./build/bin/llama-server -m qwen3-4b-rot-kv.gguf \
  -fa on -ngl 99 -c 16384 \
  --cache-type-k q2_0 --cache-type-v q2_0 \
  --host 127.0.0.1 --port 8080
```

| env var | meaning | value |
|---|---|---|
| `LLAMA_KV_FUSED_FA` | fused INT2+f16 flash-attention kernels, fast path when available | `1` |
| `LLAMA_KV_NO_HADAMARD` | rotation is in-graph, so skip the in-quant Hadamard | `1` |
| `LLAMA_KV_CLIP_RATIO` | per-row outlier clip before quant | `0.96` |
| `LLAMA_KV_HP_SINK` / `LLAMA_KV_HP_RECENT` | tokens kept high-precision (first/last) | `512` / `2048` |

`--cache-type-v f16` keeps V high-precision, which may improve quality at the cost of more memory.

## Bake your own rotation

If you have a compatible base Qwen3-4B-Thinking-2507 GGUF, you can bake the included rotation matrices into it with the OSCAR export script:

```bash
python3 oscar-rotation/export_rot_kv_gguf.py \
  --base    qwen3-4b-base.gguf \
  --rot-dir . \
  --out     qwen3-4b-rot-kv.gguf
```