---
title: Kimi-K3.1-Pro-EXL3-2bit
canonical_url: "https://www.modelscope.ai/models/local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit"
md_url: "https://www.modelscope.ai/models/local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit.md"
repository: local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit
last_updated: 2026-11-10
license: other
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - kimi_k31
architectures:
  - KimiK31ForConditionalGeneration
base_model:
  - moonshotai/Kimi-K3.1-Pro
base_model_relation: quantized
parameters: 57.4B
tensor_type:
  - BF16
  - F32
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
downloads: 23
stars: 0
tags:
  - exl3
  - trellis
  - 2-bit
  - kimi-k3.1
  - flare
  - blackwell
  - dflash2
  - safetensors
---

# Kimi-K3.1-Pro-EXL3-2bit

> Kimi-K3.1-Pro-EXL3-2bit - An open-source model by local-inference-lab on ModelScope. Kimi-K3.1-Pro EXL3 2-bit

local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit is a 57.4B-parameter text-generation model on ModelScope. licensed under other. derived from moonshotai/Kimi-K3.1-Pro.

- **Repository**: local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit
- **License**: other
- **Tasks**: text-generation
- **Parameters**: 57.4B
- **Base model**: moonshotai/Kimi-K3.1-Pro
- **Tags**: exl3, trellis, 2-bit, kimi-k3.1, flare, blackwell, dflash2, safetensors
- **Downloads**: 23
- **Stars**: 0
- **Last updated**: 2026-11-10

Source: https://www.modelscope.ai/models/local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit

---

# Kimi-K3.1-Pro EXL3 2-bit

This repository contains two representations of the same routed-expert quant:

| Revision | Contents | Intended use |
|---|---|---|
| `main` | TP-independent `exl3-v1` checkpoint; nonexpert tensors retain their source bytes | Prepare an EXL3-aware runtime |
| `gb202` | Production-signed Flare package for eight SM120 GPUs | Serve directly without Python, Torch, compilation or autotuning |

The immutable prepared release is
`gb202-tp8-c4-p3072-dflash2-k7-20261108`. It includes **TP8/DCP8, DFlash2 K7,
vision, four concurrent requests and 3072-token prefill graphs**. The maximum
context is **262,144 tokens per request**. Shared physical KV capacity measured
on the qualified host is **1,281,024 tokens**, not a 1.28M-token single-request
window. Prepared files occupy **706.46 GiB**; the raw checkpoint is **742.235 GiB**.

## Run the prepared package

Requirements: Linux x86-64, eight RTX PRO 6000 Blackwell 96 GB GPUs with peer
access enabled, NVIDIA Container Toolkit and enough disk for the package cache.
Driver 615.71.10 was tested. Other GPU architectures and TP sizes are not
qualified by this release. A host with fewer or smaller GPUs cannot load this
TP8 package by changing `--gpus`.

```bash
mkdir -p "$PWD/flare-cache"
docker run --detach --name kimi-k31 \
  --user 0:0 --gpus '"device=0,1,2,3,4,5,6,7"' --ipc host \
  --ulimit memlock=-1:-1 --ulimit nofile=65536:65536 \
  --env NVIDIA_DRIVER_CAPABILITIES=compute,utility --env HOME=/root \
  --publish 127.0.0.1:8000:8000 \
  --mount "type=bind,src=$PWD/flare-cache,dst=/root/.cache/flare" \
  ghcr.io/local-inference-lab/flare:k31-exl3-tp8-c4-dflash2-20261108 \
  run local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit \
  --revision gb202-tp8-c4-p3072-dflash2-k7-20261108 --filename model.safetensors \
  --gpus 0,1,2,3,4,5,6,7 --host 0.0.0.0 --port 8000 \
  --max-concurrency 4 --mtp-draft-tokens 7 --memory-limit=-0G \
  --prefix-cache-memory 4G --prefix-cache-disk-limit 0 \
  --dangerously-accept-publisher local-inference-lab \
  --dangerously-approve-key 937c0143f6856d3ee8ad164f62e3d804bd825d7a8750d0accb9a5849f703a8c1

docker logs -f kimi-k31
```

The first run downloads the prepared package; subsequent runs reuse the cache.
The image is approximately 160 MB and does not contain the model weights.
Its immutable digest is
`sha256:775016c31860bb4451aad0aa16d4773fc0b4a317c90c72ed02d2eb5bcc5c9496`.
The approval flags explicitly trust the LIL key shown below; omit them and use
an interactive container if you prefer the first-use trust prompt. They do not
disable signature verification. No private signing key or passphrase is needed
to run the model.

The example exposes the API only on the host's loopback address. For remote
access, change the host-side port binding and configure `--api-key-file` before
exposing the service. Stop it gracefully with `docker stop --time 90 kimi-k31`.

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Kimi-K3.1-Pro","messages":[{"role":"user","content":"Write a Python LRU cache with tests."}],"reasoning_effort":"medium","stream":true}'
```

Chat Completions supports `low`, `medium`, `high` and `max`; omitted effort
defaults to `max`. Use Chat Completions for exact `medium`: this runtime's
Responses adapter aliases that effort to `high`. Preserve assistant reasoning
and tool-call history when using a multi-turn client.

### Runtime settings

| Flag | Default | Value used above |
|---|---|---|
| `--max-concurrency` | Package capacity, 4 | `4`; lowering it is allowed, increasing beyond the prepared capacity is not |
| `--mtp-draft-tokens` | Package depth, 7 | `7` |
| `--memory-limit` | Configuration/package reserve | `-0G` uses available memory while retaining the native safety guard |
| `--prefix-cache-memory` | Configuration, otherwise disabled | `4G`; automatic intake buffers are additional |
| `--prefix-cache-disk-limit` | Configuration; applies when disk caching is enabled | `0`, disk retention disabled |

Prefill 3072, the eight-rank topology, vision and the 262,144-token context
limit are prepared into the package. Changing those capacities requires an
export, not an unsupported runtime flag. Cache capacity depends on the memory
available at startup; check the `Allocate cache` line in the server log.

## Measured serving performance

Conditions: eight RTX PRO 6000 Blackwell Workstation Edition 96 GB GPUs,
AMD EPYC 9575F 64-core host, driver 615.71.10, 600 W GPU limits and
16,365 MHz memory clocks (the host's +6000 offset). The image does not apply
power limits or overclocks. Export used CUDA 13.2 and Torch 2.14.1+cu132;
serving used the native binary in the Docker image above.

| Workload | Result |
|---|---:|
| One coding request, median of three prompts, reasoning `low` | 109.04 decode tok/s |
| Four simultaneous coding requests, common decode interval | 158.45 decode tok/s aggregate |
| Cold 32,768-token prefill, median of three prompts | 1,951.16 tok/s |
| Physical KV shared across requests and cached histories | 1,281,024 tokens |
| Four concurrent image checks | 4/4 passed |

Decode excludes prefill. Sampling used the recipe defaults: temperature 1,
top-p 0.95 and no request seed override. Generated lengths and speculative
acceptance vary, so these figures are measurements rather than fixed throughput
guarantees. The signed package has the same graph catalogs, cache geometry,
speculation configuration and device-allocation sizes as the qualified
development package. Four simultaneous approximately 250k-token contexts
totaling 1,000,345 prompt tokens completed without OOM, with all exact-recall
markers correct. Those long-context checks used the identical prepared
computation before production signing.

## Estonia and Hotel Lights

The task runner is
[`llm-inference-bench` at `52216d7`](https://github.com/local-inference-lab/llm-inference-bench/tree/52216d70865a015b34689626d04a80b89c9dc45f).
Each profile uses 30 resamples at concurrency 4 with `reasoning_effort=medium`,
recipe sampling defaults and no seed override. Estonia uses the standard
40,000-token output cap; Hotel Lights omits the client output cap. These are
consistency tests of two fixed prompts, not 60 independent benchmark problems.

| Profile | Prompt tokens | Correct | Mean completion tokens | Median completion tokens | p90 completion tokens |
|---|---:|---:|---:|---:|---:|
| Estonia | 133,140 | 30/30 | 672.53 | 663.5 | 938.7 |
| Hotel Lights | 179 | 30/30 | 10,494.67 | 10,332 | 13,927.7 |

Both signed-Docker runs completed without request errors, stalls or truncation.
Completion counts include reasoning and the visible answer. The two profiles
ran on separate eight-GPU instances on the same host. Machine-readable results
are in [benchmark-results.json](benchmark-results.json).

Reproduce either profile against a running instance:

```bash
git clone https://github.com/local-inference-lab/llm-inference-bench.git
cd llm-inference-bench
git checkout 52216d70865a015b34689626d04a80b89c9dc45f
python3 -m venv .venv
.venv/bin/pip install httpx rich psutil
LLM_BENCH_NO_UPDATE_CHECK=1 .venv/bin/python llm_decode_bench.py \
  --host http://127.0.0.1:8000 --model Kimi-K3.1-Pro \
  --test-profile estonia --profile-concurrency 4 --profile-runs 30 \
  --reasoning-effort medium --completion-stats-save-text \
  --display-mode plain --no-hw-monitor --output estonia-medium.json
```

For Hotel Lights, replace `--test-profile estonia` with
`--test-profile hotel-lights` and choose a different output filename. Do not
reuse the runner's `aggregate_gen_tok_s` field as simultaneous wall-clock
throughput: at this revision it divides by summed request durations.

## How the raw checkpoint was encoded

The source is `moonshotai/Kimi-K3.1-Pro` at
`6d0a0202956796543364d666ab19c6ad1a494ee1`. All 82,432 routed experts were
encoded from its MXFP4 weights into TP-independent `exl3-v1` with the
`lut_e4m3` codebook. The encoder uses direct tile Viterbi, a 128-step
tail-biting context and three-bit scale search. Sign patterns 0 and 6 are
evaluated per expert and the lower source-relative coefficient squared error
is selected. Activation calibration, LDLQ and QAD are not used.

The raw checkpoint's 2,748 nonexpert tensors retain their source bytes,
including attention indexers, BF16 dense/shared projections, vision,
embeddings, output head and FP32 parameters. The original source model card,
construction metadata, expert manifest and file hashes remain on the immutable
[raw checkpoint revision](https://huggingface.co/local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit/tree/38fbbe2ba5d46141eaa4d662c128bdf8d810746b).

The prepared runtime is **not** a BF16-dense reference: it explicitly uses
online MXFP8 for eligible dense/vision/draft projections and an MXFP8 draft
head. The DSA index-query projection remains BF16 and is output-sharded.
Routed expert codes remain EXL3 and use the A16 execution path. Target and
draft KV retain the recipe's FP8 layouts; recurrent state is not indiscriminately
converted to FP8. No full-model KLD comparison with official MXFP4 is claimed.

## Re-export from the pinned sources

| Component | Revision |
|---|---|
| Flare preparation and serving | [`01327bf2`](https://github.com/voipmonitor/flare/tree/01327bf2b0bb4a1626a313bac64a73a749da2e9b) |
| FlashInfer/B12X | [`6a842a7d`](https://github.com/local-inference-lab/flashinfer/tree/6a842a7d054e6e3e1db32f4c984ca1b9c0460a97) |
| Raw EXL3 checkpoint | [`38fbbe2b`](https://huggingface.co/local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit/tree/38fbbe2ba5d46141eaa4d662c128bdf8d810746b) |
| DFlash2 draft | [`RadixArk/Kimi-K3.1-Dflash2-Preview@ea257b3`](https://huggingface.co/RadixArk/Kimi-K3.1-Dflash2-Preview/tree/ea257b38a8ee755b65287bb8ccd0938b519ce26b) |

Both source repositories publish branch `release/k31-exl3-tp8-20261108`.
The container uses Flare's standard Dockerfile at the pinned revision; no
uncommitted source overlay is required. FlashInfer is needed for export, not
installed in the native serving image: its compiled kernels are in the package.

Preparation needs CUDA 13.2, the Flare build prerequisites, substantial host
RAM, and disk for both the raw checkpoint and prepared package. The export host
had about 1.1 TiB RAM. Keep preparation configuration outside Git because
Ignition writes a development signing key beside it.

```bash
git clone --branch release/k31-exl3-tp8-20261108 https://github.com/voipmonitor/flare.git
git clone --recursive --branch release/k31-exl3-tp8-20261108 https://github.com/local-inference-lab/flashinfer.git
git -C flare checkout 01327bf2b0bb4a1626a313bac64a73a749da2e9b
git -C flashinfer checkout 6a842a7d054e6e3e1db32f4c984ca1b9c0460a97
K31_PREP="$PWD/k31-preparation"
mkdir -p "$K31_PREP"
curl -fL https://huggingface.co/local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit/resolve/gb202-tp8-c4-p3072-dflash2-k7-20261108/runtime.toml \
  -o "$K31_PREP/runtime.toml"

cd flare
uv sync --locked --extra prepare --extra test
uv pip install --python .venv/bin/python --no-deps ./heliopause
uv pip install --python .venv/bin/python --no-deps -e ../flashinfer
.venv/bin/python -m ignition.patches
K31_TORCH_SITE=$(.venv/bin/python -c 'import site; print(site.getsitepackages()[0])')
for K31_PATCH in torch-2.14.1-mutation-regions.patch \
  torch-2.14.1-scheduler-membership.patch \
  torch-2.14.1-lazy-input-mutation-events.patch
do
  patch --fuzz=0 --directory "$K31_TORCH_SITE" -p1 < "tools/compiler-patches/$K31_PATCH"
done

.venv/bin/hf download local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit \
  --revision 38fbbe2ba5d46141eaa4d662c128bdf8d810746b --local-dir "$K31_PREP/checkpoint"
export PYTHONPATH="$PWD/../flashinfer:$PWD/../flashinfer/flashinfer/experimental/b12x/_compat:$PWD"
ulimit -n 65536
.venv/bin/python -m ignition export "$K31_PREP/checkpoint" \
  --config "$K31_PREP/runtime.toml" \
  --repo-id local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit \
  --gpus 0-7 --max-concurrency 4 --image-concurrency 4 --prefill-chunk-size 3072 \
  --speculative-method dflash2 --mtp-draft-tokens 7 --speculative-token-capacity 32 \
  --speculative-graph-budget 16 --online-mxfp8 --draft-head-quantization mxfp8 \
  --autotune-effort default --compile-workers 8
```

Do not apply `torch-2.14.1-no-clone-custom-op-decomposition.patch`; it was not
used for this package. The published graph export used the same settings plus
`--reuse-weights` against an already validated package with identical packed
weights. Omit that flag for the first export on a fresh host. Download the
signed release to obtain the exact captured binaries; autotuning another host
can select different kernel plans.

Build the serving image with `docker build -t kimi-k31-flare .` from the pinned
Flare checkout. Production signing used `ignition sign` on the completed
development package, without reloading the model or changing weights/graphs.
Private signing material is not distributed. The public profile is
[publisher.json](https://huggingface.co/local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit/blob/main/publisher.json).

The source model terms are preserved in [LICENSE](LICENSE). AI assistance was
used for the encoder adaptation, runtime integration, packaging and validation.