---
title: DeepSeek-V4-Pro-0.5B-A0.37B
canonical_url: "https://www.modelscope.ai/models/inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B"
md_url: "https://www.modelscope.ai/models/inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B.md"
repository: inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B
last_updated: 2026-08-14
license: mit
model_type:
  - deepseek_v4
architectures:
  - DeepseekV4ForCausalLM
base_model:
  - deepseek-ai/DeepSeek-V4-Pro
base_model_relation: finetune
parameters: 489.7M
tensor_type:
  - BF16
  - F32
  - I64
library_name:
  - transformer
  - safetensors
  - pytorch
frameworks:
  - pytorch
downloads: 97
stars: 0
---

# DeepSeek-V4-Pro-0.5B-A0.37B

> DeepSeek-V4-Pro-0.5B-A0.37B - An open-source model by inference-optimization on ModelScope. DeepSeek-V4-Pro-0.5B-A0.37B

inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B is a 489.7M-parameter machine learning model on ModelScope. licensed under mit. derived from deepseek-ai/DeepSeek-V4-Pro.

- **Repository**: inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B
- **License**: mit
- **Parameters**: 489.7M
- **Base model**: deepseek-ai/DeepSeek-V4-Pro
- **Downloads**: 97
- **Stars**: 0
- **Last updated**: 2026-08-14

Source: https://www.modelscope.ai/models/inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B

---

# DeepSeek-V4-Pro-0.5B-A0.37B

This is a tiny version of [deepseek-ai/DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) created for testing and development.

## Model Details

- **Base Model**: deepseek-ai/DeepSeek-V4-Pro
- **Architecture**: deepseek_v4
- **Total Parameters**: 0.487B
- **Activated Parameters**: ~0.37B

## Configuration Changes

The following parameters were reduced from the original model:

| Parameter | Original | Tiny |
|-----------|----------|------|
| num_hidden_layers | 61 | 8 |
| hidden_size | 7168 | 1024 |
| num_attention_heads | 128 | 8 |
| head_dim | 512 | 64 |
| q_lora_rank | 1536 | 256 |
| o_lora_rank | 1024 | 64 |
| o_groups | 16 | 2 |
| index_n_heads | 64 | 4 |
| index_head_dim | 128 | 32 |
| index_topk | 1024 | 64 |
| n_routed_experts | 384 | 16 |
| moe_intermediate_size | 3072 | 512 |
| sliding_window | 128 | 64 |

Both attention types (`heavily_compressed_attention`, `compressed_sparse_attention`) and both MLP types (`hash_moe`, `moe`) are preserved:
- layer_types: `[hca, hca, csa, hca, csa, hca, csa, hca]`
- mlp_layer_types: `[hash_moe, hash_moe, hash_moe, moe, moe, moe, moe, moe]`

## Checkpoint Structure

Single-file checkpoint (`model.safetensors`) with 608 tensors. The checkpoint uses the original DeepSeek-V4-Pro naming convention (no `model.` prefix): `layers.N.attn.wq_a.weight`, `embed.weight`, `head.weight`, etc. Fully loadable with `AutoModelForCausalLM.from_pretrained()`.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))
```

## Creation Process

This model was created using the llm-compressor `create-tiny-model` claude skill.

1. Inspected DeepSeek-V4-Pro config to identify architecture: dual attention types (heavily_compressed_attention, compressed_sparse_attention), dual MLP types (hash_moe, moe), HyperConnection residual streams (hc_mult=4), and MTP layer.
2. Reduced all hidden dimensions while preserving at least one of each layer type.
3. Fine-tuned on copypasta dataset until perplexity < 3.0 (achieved 1.001).
4. Validated checkpoint structure matches original model naming conventions.

## Notes

- The model uses transformers naming convention for tensor keys (compatible with `from_pretrained()`).
- A converted checkpoint matching the original DeepSeek-V4-Pro key naming is also available (`model.safetensors` with original `layers.N.attn.wq_a.weight` style names).
- FP8 quantization scale tensors from the original model are not present in this unquantized bf16 checkpoint.
- The `num_nextn_predict_layers=1` (MTP) layer is preserved in the architecture.
- Validation output: `Success: 1.001 <= 10.0`