---
title: DeepSeek-V4-2.7B-tiny
canonical_url: "https://www.modelscope.ai/models/inference-optimization/DeepSeek-V4-2.7B-tiny"
md_url: "https://www.modelscope.ai/models/inference-optimization/DeepSeek-V4-2.7B-tiny.md"
repository: inference-optimization/DeepSeek-V4-2.7B-tiny
last_updated: 2026-08-14
model_type:
  - deepseek_v4
architectures:
  - DeepseekV4ForCausalLM
parameters: 2.7B
tensor_type:
  - I64
  - BF16
  - F32
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
downloads: 93
stars: 0
---

# DeepSeek-V4-2.7B-tiny

> DeepSeek-V4-2.7B-tiny - An open-source model by inference-optimization on ModelScope. This is a fine-tuned version of inference-optimization/DSV4-tiny-empty trained on famous internet copypastas.

- **Repository**: inference-optimization/DeepSeek-V4-2.7B-tiny
- **Parameters**: 2.7B
- **Downloads**: 93
- **Stars**: 0
- **Last updated**: 2026-08-14

Source: https://www.modelscope.ai/models/inference-optimization/DeepSeek-V4-2.7B-tiny

---

# DSV4-tiny-finetuned

This is a fine-tuned version of `inference-optimization/DSV4-tiny-empty` trained on famous internet copypastas.

## Model Details

- **Base Model**: inference-optimization/DSV4-tiny-empty
- **Architecture**: DeepseekV4ForCausalLM
- **Total Parameters**: 2,689,440,743 (~2.7B parameters)
- **Precision**: bfloat16

## Training Details

The model was fine-tuned using the training template from the create-tiny-model skill on 4 famous internet copypastas:
- Bee Movie aviation speech
- GNU/Linux interject copypasta
- FitnessGram Pacer Test
- Darth Plagueis the Wise

### Training Configuration

- **Target Perplexity**: 3.0
- **Batch Size**: 2
- **Learning Rate**: 5e-5
- **Max Steps**: 1000 (early stopped at step 160)
- **Training Runtime**: 29.2 seconds
- **Training Loss**: 0.2243
- **Final Perplexity**: ~2.44 (achieved target)

### Training Progress

The model achieved excellent convergence:
- Initial loss: 12.52
- Final loss: 0.000137 (at step 80)
- Training stopped early after consistently achieving target perplexity

## Generation Example

During training validation, the model successfully generated:

**Prompt**: "According to all known laws"
**Output**: "According to all known laws of aviation, there is no way a bee should be able to fly."

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("./DSV4-tiny-finetuned")
model = AutoModelForCausalLM.from_pretrained(
    "./DSV4-tiny-finetuned",
    device_map="auto",
    torch_dtype=torch.bfloat16
)

# Note: The model uses bfloat16 precision
# Ensure your inputs are properly cast to the correct dtype
```

## Files

- `config.json`: Model configuration
- `model.safetensors`: Model weights (5.1GB)
- `tokenizer.json`: Tokenizer vocabulary
- `tokenizer_config.json`: Tokenizer configuration
- `generation_config.json`: Generation parameters
- `training_args.bin`: Training arguments used during fine-tuning

## Notes

- The model was successfully fine-tuned and achieved the target perplexity of 3.0
- Training completed in under 30 seconds with early stopping at step 160
- The model memorized the training copypastas effectively, as evidenced by the low final loss
- Model uses DeepSeek V4 architecture with MoE (Mixture of Experts) layers