Skip to main content

Qwen3.8-Flash-Next: A Lightweight Giant with Breakthrough Performance

The Qwen team has just dropped a new model that's turning heads in the AI community. Meet Qwen3.8-Flash, a multimodal MoE (Mixture of Experts) model that packs a punch despite its lightweight design. Alongside it, they've open-sourced the next-generation architecture, Qwen3.8-Flash-Next, which is essentially the blueprint for the upcoming Qwen4 series.

So, what's the big deal? Well, Qwen3.8-Flash has a total parameter count of 125B, plus an extra 51B N-gram Embedding, but here's the kicker: only 6B parameters are activated per token. That means it's incredibly efficient. It natively supports a context window of 260,000 tokens, and with YaRN technology, you can stretch that to a whopping 1,000,000 tokens. And get this: the training cost is just one-ninth of its predecessor, Qwen3.7-Plus. The API pricing is a steal too—1 yuan per million input tokens and 3 yuan per million output tokens. It's already live on the Qwen AI platform, and they've even introduced a "Qwen Office" feature.

Image

Now, let's talk architecture. The team made four major upgrades that really set this apart. First, the attention module uses a hybrid GDN+QSA design. GDN compresses historical info, while QSA picks out the key contexts. In scenarios with a million tokens, this delivers up to 7.6x speedup in Prefill and 4.9x in Decode. Second, they introduced a Gated Residual mechanism that expands the residual flow into four parallel branches, using dynamic gating to improve cross-layer info flow and training stability. Third, the N-gram Embedding uses local context lookup tables to add 51B parameters without significantly increasing compute. Finally, they adopted a mixed optimization strategy with Muon and AdamW, which allowed them to re-fit the Scaling Law and even eliminate the batch warm-up process.

But does it actually perform? You bet. Despite having only 6B active parameters, the base model achieved the best results in 8 out of 14 evaluations. The fine-tuned version shines in code, agents, and multimodal tasks. For instance, it scored 62.5 on SWE-bench Pro, and it leads the pack in agent rankings like CoWorkBench and Toolathlon.

If you're curious to try it out, the weights for Qwen3.8-Flash-Next are now open-sourced on Hugging Face and ModelScope, and the full technical report is available too. The Qwen team is clearly committed to transparency and community collaboration, hoping to validate their innovations with the help of developers worldwide. This is a bold move that could accelerate the development of the Qwen4 series.

Key Points

  • Efficient Design: Only 6B active parameters despite 125B total, leading to significant cost savings.
  • Long Context: Supports up to 1M tokens with YaRN, enabling processing of massive documents.
  • Architectural Innovations: Hybrid attention, gated residual, and N-gram embeddings boost performance and stability.
  • Benchmark Success: Outperforms larger models in many tasks, especially in code and agent scenarios.
  • Open Source: Weights and technical report are publicly available, inviting community collaboration.