--- title: Hy3-1M-GGUF canonical_url: "https://www.modelscope.ai/models/satgeze/Hy3-1M-GGUF" md_url: "https://www.modelscope.ai/models/satgeze/Hy3-1M-GGUF.md" repository: satgeze/Hy3-1M-GGUF last_updated: 2026-07-09 license: apache-2.0 pipeline_tag: text-generation tasks: - text-generation base_model: - tencent/Hy3 base_model_relation: quantized library_name: - gguf downloads: 128 stars: 0 tags: - gguf - long-context - yarn - hunyuan - hy_v3 - moe - llama.cpp --- # Hy3-1M-GGUF > Hy3-1M-GGUF - An open-source model by satgeze on ModelScope. -2563eb) -8A2BE2) -181717?logo=github) satgeze/Hy3-1M-GGUF is a text-generation model on ModelScope. licensed under apache-2.0. derived from tencent/Hy3. - **Repository**: satgeze/Hy3-1M-GGUF - **License**: apache-2.0 - **Tasks**: text-generation - **Base model**: tencent/Hy3 - **Tags**: gguf, long-context, yarn, hunyuan, hy_v3, moe, llama.cpp - **Downloads**: 128 - **Stars**: 0 - **Last updated**: 2026-07-09 Source: https://www.modelscope.ai/models/satgeze/Hy3-1M-GGUF --- ![Tencent Hy3 1M GGUF](banner.png)
[![Base](https://img.shields.io/badge/base-tencent%2FHy3-2563eb)](https://huggingface.co/tencent/Hy3) [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0) [![Params](https://img.shields.io/badge/params-299B%20MoE%20(~17B%20active)-2563eb)]() [![Context: 1M](https://img.shields.io/badge/context-1%2C048%2C576%20tokens%20(baked)-8A2BE2)]() [![First GGUF](https://img.shields.io/badge/GGUF-first%20anywhere-brightgreen)]() [![Status](https://img.shields.io/badge/status-EXPERIMENTAL-dc2626)]() [![GitHub](https://img.shields.io/badge/GitHub-aviary--1m%20(harness%20%2B%20method)-181717?logo=github)](https://github.com/satindergrewal/aviary-1m)

Hy3 (Hunyuan) 1M Context GGUF: first GGUF conversion of Tencent's 299B hy_v3, extended to a million tokens. Preview quality, honestly labeled.

⚠️ EXPERIMENTAL, READ BEFORE USING

These are early GGUFs built BEFORE official llama.cpp support for the hy_v3 architecture existed. They require a community fork to run (details below), they are NOT yet needle-certified at long context (1M is baked in but unverified). MTP-enabled quants are now included (the engine support was built, validated, and proposed upstream). Coherence-gated only. Treat as preview quality until the certification ladder is published on this card.

Tencent's [Hy3](https://huggingface.co/tencent/Hy3) (299B MoE, ~17B active, `hy_v3` architecture) quantized to GGUF and extended to a **1,048,576-token context** with YaRN. As far as we can tell these are the first GGUF quants of Hy3 anywhere: at publish time (roughly 30 hours after Hy3's release) only MLX and MXFP4 quants existed, and mainline llama.cpp did not yet support `hy_v3`. ## Honest status - **Architecture support**: built against a community llama.cpp fork that implements `hy_v3` ([charlie12345/ROCmFPX](https://github.com/charlie12345)). Mainline llama.cpp support is still in progress (open feature requests). You need an `hy_v3`-capable build to run these. - **Quants**: Q2_K is a static quant (no importance matrix). IQ2_M and IQ1_M are importance-matrix quants - the imatrix was computed on our own Q2_K running RAM-resident, over a code-and-reasoning calibration set, then used to requantize from the archived f16. This is a bootstrap: a better imatrix could later be computed from a higher-fidelity runner. - **Coherence**: all three passed a generation gate (coherent prose and correctly structured haiku). Not yet needle-certified for long context; NIAH ladders will follow. - **MTP**: now working. We ported `hy_v3` MTP inference to llama.cpp ourselves (validated at 97.3 percent draft acceptance with `--spec-draft-p-min 0.75`, proposed upstream in [PR #25395](https://github.com/ggml-org/llama.cpp/pull/25395)). The `-MTP-` quants below carry the draft head baked in at q8_0; older MTP-less quants are being replaced by them. - **Chat template**: Hy3's stock template uses `.format()` calls that llama.cpp's Jinja engine rejects. `chat_template_llamacpp.jinja` in this repo is the fixed version; pass it with `--chat-template-file`. ## What the gate actually produced (real, unedited) Each quant was asked: *"Name three colors and write a one-line haiku about the sea."* Verbatim outputs, nothing cleaned up: **IQ1_M** (62 GB, the roughest quant): > blue waves that whisper low, > green depths that hold the silence of the sea. **IQ2_M** (90 GB): > Three colors: blue, teal, and silver. > waves drink the moonlight > saltwind hums through silent blue- > the sea forgets names **Q2_K** (101 GB): > Blue, green, silver. > The sea breathes in mist, > silver light on rolling waves- > quiet, endless blue. ### My honest read This is better than "it didn't collapse into repetition." A 1.7-bit quant of a 299B model (IQ1_M) still produces grammatical, on-theme lines; the two larger quants produce properly structured 5-7-5 haiku and correctly separate the color list from the poem. That is a genuinely good sign for how much of the model survives aggressive quantization here. Two honest caveats, so you judge for yourself: - This is a single short prompt. It proves coherence and basic instruction-following, not reasoning, long-context retrieval, or factual accuracy. Those need the certification ladder, which is not done yet. - IQ1_M is visibly the weakest: it dropped the explicit color list and gave two lines instead of a haiku. Usable, but the imatrix IQ2_M is the one I would actually run if the memory fits. If your experience differs, the raw files are right here - try it and form your own view. ## Files | File | Size | Where | Notes | |---|---|---|---| | `hy3-1M-IQ1_M-yarn.gguf` | 62 GB | [ModelScope only](https://www.modelscope.ai/models/satgeze/Hy3-1M-GGUF) | Smallest; fits a 128 GB Mac. **`no_think` chat only, see warning below** | | `hy3-1M-MTP-IQ2_M.gguf` | 100 GB | HF now, ModelScope syncing | **Recommended.** IQ2_M + MTP draft head | | `hy3-1M-MTP-Q4_K_M.gguf` | 183 GB | HF now, ModelScope syncing | Quality pick for 192GB+ rigs, MTP baked | | `hy3-1M-IQ2_M-yarn.gguf` | 90 GB | HF + ModelScope | MTP-less original, will be replaced by the MTP build | | `hy3-1M-Q2_K-yarn.gguf` | 101 GB | HF + ModelScope | Static quant, the RAM-runner used to bootstrap the imatrix | | `chat_template_llamacpp.jinja` | small | HF + ModelScope | Fixed template for llama.cpp | ## Field testing notes (real sessions, July 7) Measured on a MacBook Pro M3 Max 128 GB, Metal, fork build, IQ1_M fully GPU-resident: - **Speed**: 24-25 tok/s generation, ~41 tok/s prefill (llama-server `timings`, short prompts). A 299B MoE at chat speed on a laptop. - **`no_think` mode**: coherent and usable, but noticeably chatty: filler sentences, over-explaining, occasional odd asides. Fine for casual chat and drafting; does not feel like a frontier chat tune. - **Thinking mode (`reasoning_effort: high`): collapses at IQ1_M.** On a bare "hello" it produced hundreds of words of circular reasoning ("The user wrote hello... we should respond... but wait, the user wrote hello...") without ever converging to an answer. This is the 1.7-bit tax: short answers survive, reasoning chains do not. **Do not use IQ1_M for coding or anything that needs thinking.** - Because of that, IQ1_M lives on ModelScope only; HF carries the quants we can recommend. IQ2_M field results will be added after the same test battery. - **Known cosmetic issue on the ROCmFPX fork build**: the `<|hy_eos:opensource|>` stop token can leak as visible text at the end of responses. We root-caused it: the GGUF tokenizer metadata is correct (the token is properly CONTROL-typed); the leak is in that fork's chat parser, which renders special tokens for structural parsing and passes unknown trailing ones through. The recommended `hy3-mtp` branch build below does not have this issue (mainline strips specials before parsing). Workarounds on the fork: add the token as an explicit stop string, or apply the parser patches we are submitting to that repo. Generation always stops correctly; the text is just noise. ### MTP speculative decoding: measured guidance (July 8) The `-MTP-` quants carry Hy3's draft head (protected at q8_0). Measured on these exact files, upstream llama.cpp build ([PR #25395](https://github.com/ggml-org/llama.cpp/pull/25395)): | Platform | MTP effect | Setting | |---|---|---| | CUDA (RTX 5090) | **+26 to 37% decode** | `--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75` | | Apple silicon (M3 Max, IQ2_M) | neutral: 23.27 vs 23.21 tok/s, 88.7% acceptance | flags optional, no penalty with `p-min 0.75` | `--spec-draft-p-min 0.75` is required either way: the MTP head is trained single-depth, and without the confidence gate speculation becomes a net slowdown (38.6% acceptance measured). Full data and discussion: [Apple-silicon datapoint](https://huggingface.co/satgeze/Hy3-1M-GGUF/discussions/2). Speed expectations, honestly: quoted tok/s are fresh-context ceilings; in an ongoing conversation expect roughly 17 to 19 tok/s on the M3 Max at IQ2_M as context deepens. Both numbers are normal physics. How to control thinking with llama.cpp: pass `--chat-template-kwargs '{"reasoning_effort": "high"}'` (or `no_think` / `low`) at server launch. ## Building an hy_v3-capable llama.cpp **First, check if you even need this**: native support has been proposed upstream in [ggml-org/llama.cpp#25395](https://github.com/ggml-org/llama.cpp/pull/25395) (hy_v3 + MTP, AI-implemented with full disclosure, validated on these exact GGUFs). If that PR has merged by the time you read this, stock llama.cpp runs these files and you can skip everything below. Until then, the recommended build is the branch that PR is made from: current llama.cpp master plus `hy_v3` plus the MTP head. No web-UI workarounds needed, modern built-in UI included. ```bash git clone --branch hy3-mtp --depth 5 https://github.com/satindergrewal/llama.cpp.git cd llama.cpp # Apple Silicon (Metal): cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release cmake --build build --target llama-server -j # NVIDIA (CUDA). If you build inside a conda env and hit # "undefined reference to cuMemGetInfo", point at the real driver lib: cmake -B build -DGGML_CUDA=ON \ -DCUDA_cuda_driver_LIBRARY=/usr/lib/x86_64-linux-gnu/libcuda.so.1 \ -DCMAKE_BUILD_TYPE=Release cmake --build build --target llama-server -j ``` MTP speculative decoding note for this model: launch with `--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75`. The `p-min 0.75` matters: Hy3's MTP head is trained single-depth, and without the confidence gate speculation becomes a net slowdown (measured 38.6 percent acceptance vs 97.3 percent with it). The original community implementation these quants were first built on is [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) (requires stubbing four empty web-UI files before building: `tools/server/public/{bundle.css,bundle.js,loading.html,index.html}`). It remains a fine option and is credited in the PR. ## Run it (with an hy_v3-capable llama.cpp build) ```bash llama-server -m hy3-1M-IQ2_M-yarn.gguf -c 262144 -np 1 --jinja \ --chat-template-file chat_template_llamacpp.jinja ``` Start at 262K and raise `-c` toward 1048576 per available memory. On Apple Silicon the IQ1_M runs; on 192 GB+ the IQ2_M is the quality pick. ## Credits Base model: [Tencent](https://huggingface.co/tencent/Hy3) (Apache-2.0). `hy_v3` llama.cpp implementation: community fork (charlie12345). GGUF conversion, 1M YaRN extension, imatrix bootstrap, and gating: [SatGeze](https://huggingface.co/satgeze). Method and tooling: [github.com/satindergrewal/aviary-1m](https://github.com/satindergrewal/aviary-1m).