---
title: Quasar-10B
canonical_url: "https://www.modelscope.ai/models/silx-ai/Quasar-10B"
md_url: "https://www.modelscope.ai/models/silx-ai/Quasar-10B.md"
repository: silx-ai/Quasar-10B
last_updated: 2026-03-31
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - quasar
architectures:
  - QuasarForCausalLM
base_model:
  - Qwen/Qwen3.5-9B-Base
base_model_relation: finetune
parameters: 8.6B
tensor_type:
  - BF16
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
  - ar
downloads: 176
stars: 0
tags:
  - silx-ai
  - quasar
  - foundation-model
  - 10b
  - long-context
  - llm
  - gla
  - linear-attention
  - 2m-context
---

# Quasar-10B

> Quasar-10B - An open-source model by silx-ai on ModelScope. Quasar-10B: Fully Linear Foundation Model

silx-ai/Quasar-10B is a 8.6B-parameter text-generation model on ModelScope. licensed under apache-2.0. derived from Qwen/Qwen3.5-9B-Base.

- **Repository**: silx-ai/Quasar-10B
- **License**: apache-2.0
- **Tasks**: text-generation
- **Parameters**: 8.6B
- **Base model**: Qwen/Qwen3.5-9B-Base
- **Tags**: silx-ai, quasar, foundation-model, 10b, long-context, llm, gla, linear-attention, 2m-context
- **Downloads**: 176
- **Stars**: 0
- **Last updated**: 2026-03-31

Source: https://www.modelscope.ai/models/silx-ai/Quasar-10B

---

<p align="center">
  <img src="./Quasar.png" alt="Quasar Foundation Model" width="100%">
</p>

# Quasar-10B: Fully Linear Foundation Model

Quasar-10B is a high-performance foundation model developed by **SILX AI**. It is built upon the **Qwen3.5-9B-Base** architecture, fundamentally re-engineered to support extreme long-context reasoning (2 Million+ tokens) while maintaining high computational efficiency.

This model marks a major shift in the Quasar training stack, moving from traditional Softmax-based attention to a **Hybrid Gated Linear Attention (GLA)** architecture.

---

# Model Overview

**Model Name:** Quasar-10B  
**Organization:** SILX AI  
**Base Model:** [Qwen3.5-9B-Base](https://huggingface.co/Qwen/Qwen3.5-9B-Base)

### Architecture Evolution
The original Qwen3.5 architecture uses a combination of Gated Delta Attention and Softmax Gated Attention. To support the Quasar design requirements for infinite scaling and efficient state management, we performed a deep architectural swap:
* **GLA Integration**: Replaced the target attention layers with **Gated Linear Attention (GLA)**.
* **NOPE (No Positional Embeddings)**: Removed traditional RoPE (Rotary Positional Embeddings) to eliminate positional bias and enable native extrapolation to millions of tokens.
  > [!NOTE]  
  > **GLA** was chosen as the core linear mechanism to maintain exact architectural parity with the **Quasar 22B MoE** design. This model is a direct evolution redirected from [silx-ai/Quasar-V1-Base-Stage1](https://huggingface.co/silx-ai/Quasar-V1-Base-Stage1), utilizing Quasar Continuous Time Attention for state-trajectory optimization.

---

# Training Methodology

The development of Quasar-10B followed a rigorous two-stage process:

### Stage 1: Structural Distillation (10B Tokens)
To ensure the new GLA layers correctly inherited the capabilities of the original Qwen heads:
* **Process**: Layer-wise structural distillation. We initialized the student with Qwen3.5 weights and replaced specific layers with GLA units.
* **Loss**: Hybrid loss combining MSE (Hidden State Mimicry) and Cross-Entropy (Language Modeling).
* **Volume**: 10 Billion tokens of high-quality reasoning data.
* **Goal**: Minimize structural divergence and transferpretrained world knowledge into the new linear state.

### Stage 2: Native 2M Context Expansion (20B Tokens)
Once structurally sound, the model was pushed to extreme sequence lengths:
* **Positionality**: RoPE was fully removed and replaced with **NOPE** (No Positional Embedding).
* **Context Length**: Native training on **2,097,152 (2M)** sequence lengths.
* **Volume**: 20 Billion tokens.
* **Hardware**: Optimized for B200 HBM efficiency, utilizing sub-chunked sequential processing to maintain a 2M token active state.

---

# Features
* **Infinite Recurrence**: The GLA architecture allows the model to process sequences far beyond its training window with linear complexity.
* **Reasoning Excellence**: Trained on the **Nemotron-Pretraining-Specialized-v1** mix, focusing on Math, STEM, and code-centric reasoning.
* **B200 Optimized**: Specifically tuned for maximum throughput on NVIDIA Blackwell hardware.

---

# Technical Notes
Quasar-10B represents the first "Recurrent foundation model" in our stack that successfully bridges the gap between Transformer-scale pretraining and RNN-style linear efficiency. By removing positional embeddings, we allow the model to rely entirely on its internal state trajectories for temporal coherence.

---

# Next Steps
The Quasar roadmap continues toward even larger scales and deeper MoE integrations. For technical research and integration support, contact the SILX AI team.

---

> [!IMPORTANT]  
> **Strategic Purpose**: Quasar-10B is designed as a foundational high-context engine. It will be used exclusively to **distill knowledge and generate synthetic reasoning data** for the upcoming **Quasar 22B MoE**, ensuring that the larger mixture-of-experts model inherits superior long-context coherence and refined logical state-trajectories from this fully linear base.