---
title: mini-deepseek-v4-flash
canonical_url: "https://www.modelscope.ai/models/Akashizrr/mini-deepseek-v4-flash"
md_url: "https://www.modelscope.ai/models/Akashizrr/mini-deepseek-v4-flash.md"
repository: Akashizrr/mini-deepseek-v4-flash
last_updated: 2026-08-14
parameters: 50.9B
tensor_type:
  - F32
  - I8
  - U8
  - BF16
  - F8_E4M3
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
downloads: 105
stars: 0
---

# mini-deepseek-v4-flash

> mini-deepseek-v4-flash - An open-source model by Akashizrr on ModelScope. Mini DeepSeek V4 Flash (Fuse-2 Trained)

- **Repository**: Akashizrr/mini-deepseek-v4-flash
- **Parameters**: 50.9B
- **Downloads**: 105
- **Stars**: 0
- **Last updated**: 2026-08-14

Source: https://www.modelscope.ai/models/Akashizrr/mini-deepseek-v4-flash

---

# Mini DeepSeek V4 Flash (Fuse-2 Trained)

## Overview
Fuse-2 model: Qwen3-4B host + 260 DeepSeek V4 Flash coding experts with trained bridges, routers, and repair layers.

## Training Details
- **Steps**: 1000
- **Dataset**: tatsu-lab/alpaca + synthetic coding instructions
- **Method**: Distillation (KL divergence to Qwen3-4B teacher) + LM loss
- **Learning rates**: bridge_in/router/repair_down=1e-2, bridge_out/repair_up=1e-6
- **NaN count**: 0 (fully stable)
- **GPU**: A100 80GB

## Trained Components
| Component | Init std | Final std | Growth |
|-----------|----------|-----------|--------|
| bridge_in | 0.020 | 0.053-0.091 | 2.6-4.5x |
| router | 0.010 | 0.031-0.063 | 3.1-6.3x |
| repair_down | 0.020 | 0.041-0.084 | 2.0-4.2x |
| bridge_out | 0.0001 | 0.0001 | (tiny, preserves generation) |
| repair_up | 0.0001 | 0.0001 | (tiny, preserves generation) |

## Usage
Load the base model from `Akahsizrr/Mini-Whale-Flash/merged-v2-full`, then load `trainable_params.pt` with `strict=False`:

```python
from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Akahsizrr/Mini-Whale-Flash/merged-v2-full",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
ckpt = torch.load("trainable_params.pt", map_location="auto", weights_only=True)
ckpt_bf16 = {k: v.to(torch.bfloat16) for k, v in ckpt.items()}
model.load_state_dict(ckpt_bf16, strict=False)
```

## Files
- `trainable_params.pt` — final checkpoint (step 1000)
- `checkpoints/v2_step{200,400,600,800,1000}/trainable_params.pt` — intermediate checkpoints