---
title: SKT-ST-X-0-3B
canonical_url: "https://www.modelscope.ai/models/SKTAILABS/SKT-ST-X-0-3B"
md_url: "https://www.modelscope.ai/models/SKTAILABS/SKT-ST-X-0-3B.md"
repository: SKTAILABS/SKT-ST-X-0-3B
last_updated: 2026-06-09
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - mixtral
architectures:
  - MixtralForCausalLM
parameters: 3.4B
tensor_type:
  - F32
library_name:
  - pytorch
  - transformer
  - safetensors
frameworks:
  - pytorch
language:
  - en
  - hi
downloads: 69
stars: 1
tags:
  - moe
  - slm
  - skt-ai-labs
  - 3b-model
  - mixture-of-experts
---

# SKT-ST-X-0-3B

> SKT-ST-X-0-3B - An open-source model by SKTAILABS on ModelScope. SKT 3B-MoE is a compact Small Language Model (SLM)

SKTAILABS/SKT-ST-X-0-3B is a 3.4B-parameter text-generation model on ModelScope. licensed under apache-2.0.

- **Repository**: SKTAILABS/SKT-ST-X-0-3B
- **License**: apache-2.0
- **Tasks**: text-generation
- **Parameters**: 3.4B
- **Tags**: moe, slm, skt-ai-labs, 3b-model, mixture-of-experts
- **Downloads**: 69
- **Stars**: 1
- **Last updated**: 2026-06-09

Source: https://www.modelscope.ai/models/SKTAILABS/SKT-ST-X-0-3B

---

# SKT-ST-X-0-3B-V1


<div align="center">
  <picture>
      <img src="Webp/img.png" width="90%" alt="SKT AI LABS">
  </picture>
</div>

---
**Small Language Model — Mixture of Experts (3B Parameters)** *Built by SKT AI LABS, India*


---


<div align="left">
  <a href="https://huggingface.co/sKT-Ai-Labs" target="_blank">
    <img src="https://img.shields.io/badge/🤗--HuggingFace-yellow?style=for-the-badge&logo=huggingface" width="220" alt="Hugging Face">
  </a>
  <a href="https://www.sktailabs.in" target="_blank">
    <img src="https://img.shields.io/badge/SKT_AI_LABS-green?style=for-the-badge" width="180" alt="SKT AI LABS">
  </a>
  <a href="https://opensource.org/licenses/Apache-2.0" target="_blank">
    <img src="https://img.shields.io/badge/License-Apache%202.0-blue?style=for-the-badge" width="180" alt="License">
  </a>
</div>

---

## 1. Model Overview

**SKT 3B-MoE** is a **compact Small Language Model (SLM)** built using ST-X-0 Taken Mixtral For Better MoE Stability , it delivers efficient and intelligent responses while maintaining a small footprint.

| Property | Value |
|----------|-------|
| **Architecture** | Mixture of Experts (MoE) |
| **Total Parameters** | ~3B |
| **Active Parameters** | ~1.1B (2 expert/token) |
| **Hidden Size** | 2048 |
| **Number of Experts* | 4 |
| **Context Length** | 8K tokens |
| **Training Tokens** | 40B |

---

## 2. Key Capabilities

| Capability | Description |
|------------|-------------|
| **Bilingual** | English & Hindi |
| **Basic Coding** | Python, logic, algorithms |
| **Reasoning** | Logical thinking, problem solving |
| **Creative Writing** | Stories, poems, roleplay |
| **Knowledge QA** | General knowledge, facts |
| **Personality** | Friendly, helpful, cute |

---

## 3.Quick Start

---

### Installation


```bash
pip install transformers accelerate torch

```

### Basic Usage

```python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "sKT-Ai-Labs/SKT-ST-X-0-3B",
    device_map="auto",
    torch_dtype=torch.float16
)
tokenizer = AutoTokenizer.from_pretrained("sKT-Ai-Labs/SKT-ST-X-0-3B")

# Chat
prompt = "What is Quantum Physics ?"
formatted = f"<|user|>\n{prompt}\n<|assistant|>\n"
inputs = tokenizer(formatted, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response.split("<|assistant|>")[-1].strip())

```
## 4. Sample Outputs
```
Q: Write a short story about a cat
A: Once upon a time, there was a little brown cat named Jake.
    He was very small but very brave. One day, Jake saw a beautiful bird...

```
```
Q: Explain quantum computing
A: Quantum computing is a type of computing that uses quantum mechanics
    to process information. Unlike classical computers that use bits (0 or 1),
    quantum computers use qubits that can be both 0 and 1 simultaneously...

```
## 5. Advanced Configuration


### LoRA Fine-tuning (for custom tasks)
```python
from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=8,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.1,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # Only ~0.06% trainable!

```
### 4-bit Quantization (for low VRAM)

```python
from transformers import BitsAndBytesConfig
import torch

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)

model = AutoModelForCausalLM.from_pretrained(
    "sKT-Ai-Labs/SKT-ST-X-0-3B",
    quantization_config=quant_config,
    device_map="auto"
)

```



## 6. License
Both the code repository and the model weights are released under the Apache-2.0 License  
See  [License](Index/USED.MD)

See [THIRD PARTY NOTICES](Index/THIRD_PARTY_NOTICES.MD)

---

## 7. Contact Us
If you have any questions, please reach out at support@sktailabs.in.

---

## 8. Citation
```bibtex
@misc{SKT-ST-X-0-3B,
  author = {SKT AI LABS, India},
  title = {SKT-ST-X-0-3B: A Compact Mixture of Experts Model},
  year = {2026},
  publisher = {Hugging Face},
  url = {[https://huggingface.co/sKT-Ai-Labs/SKT-ST-X-0-3B](https://huggingface.co/sKT-Ai-Labs/SKT-ST-X-0-3B)}
}

```
```

```