---
title: SmolLM-135M-MLA-d_kv_8-refactor
canonical_url: "https://www.modelscope.ai/models/openmoss/SmolLM-135M-MLA-d_kv_8-refactor"
md_url: "https://www.modelscope.ai/models/openmoss/SmolLM-135M-MLA-d_kv_8-refactor.md"
repository: openmoss/SmolLM-135M-MLA-d_kv_8-refactor
last_updated: 2025-11-27
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - llama
architectures:
  - LlamaForCausalLM
base_model:
  - HuggingFaceTB/SmolLM-135M
base_model_relation: finetune
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
downloads: 129
stars: 0
---

# SmolLM-135M-MLA-d_kv_8-refactor

> SmolLM-135M-MLA-d_kv_8-refactor - An open-source model by openmoss on ModelScope. Research Paper "Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs"

openmoss/SmolLM-135M-MLA-d_kv_8-refactor is a text-generation model on ModelScope. licensed under apache-2.0. derived from HuggingFaceTB/SmolLM-135M.

- **Repository**: openmoss/SmolLM-135M-MLA-d_kv_8-refactor
- **License**: apache-2.0
- **Tasks**: text-generation
- **Base model**: HuggingFaceTB/SmolLM-135M
- **Downloads**: 129
- **Stars**: 0
- **Last updated**: 2025-11-27

Source: https://www.modelscope.ai/models/openmoss/SmolLM-135M-MLA-d_kv_8-refactor

---

**Research Paper** ["Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs"](https://arxiv.org/abs/2502.14837)

## Usage

Please refer to the GitHub repository [MHA2MLA](https://github.com/JT-Ushio/MHA2MLA).
The inference code is still being optimized. You can follow our subsequent work.

## Citation

```
@misc{ji2025economicalinferenceenablingdeepseeks,
      title={Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs}, 
      author={Tao Ji and Bin Guo and Yuanbin Wu and Qipeng Guo and Lixing Shen and Zhan Chen and Xipeng Qiu and Qi Zhang and Tao Gui},
      year={2025},
      eprint={2502.14837},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.14837}, 
}
```