Skip to main content

Tencent Shrinks Translation Model to 440MB with Almost No Quality Loss

Tencent's Hunyuan team has pulled off a remarkable feat: squeezing a massive translation model down to a fraction of its original size without sacrificing much accuracy. The 1.8B parameter model, once requiring 3.3GB of memory, now fits in just 440MB. That's a huge win for edge devices, and it's already being used to translate live stream comments on Bilibili in real time.

The Challenge: Big Model, Small Devices

The base model, Hy-MT2-1.8B, is part of Tencent's Hunyuan translation family. It supports bidirectional translation across 33 languages and, at its size, already outperforms several commercial translation APIs on the FLORES-200 benchmark. But there's a catch: in traditional FP16 precision, it needs 3.3GB of memory. Even with 4-bit quantization, it still requires over 1GB. That's too heavy for edge devices, especially when you need to run multiple tasks concurrently.

Two Clever Solutions

To tackle this, Tencent introduced two ultra-low bit quantization schemes.

2-Bit for High-End Devices

The first is a 2-bit model aimed at high-end devices. It uses a technique called Stretchable Elastic Quantization (SEQ) to quantize parameters to specific discrete values. Combined with Quantization-Aware Distillation (QAD), the model achieves near-lossless translation quality while shrinking to 574MB. Not bad, but they wanted to go further.

1.25-Bit for Everyone

The second scheme is even more aggressive: an ultra-low bit 1.25-bit solution that works across all device series. This is based on Tencent's own Sherry sparse-efficient ternary quantization technology, which was selected as an Oral paper at ACL 2026. The core idea is a fine-grained sparsity mechanism: three out of every four parameters are stored with specific values, and the rest are set to zero. That averages out to 1.25 bits per parameter. With a custom STQ kernel designed for CPUs, they compressed the original 3.3GB model down to 440MB.

Making It Work on Intel Chips

To move beyond mobile ARM architectures, the Intel team stepped in to adapt the model for the x86 ecosystem. They optimized vectorization, weight reordering, and VNNI instruction fusion in quantized matrix operations. As a result, the low-bit model now runs significantly faster on Intel mainstream devices, including the latest third-generation Core Ultra and Core processors. This expands the ultra-low bit solution from phones to PCs and other edge devices.

Real-World Impact: Bilibili Live Translation

Bilibili has already integrated this ultra-low bit model into its live stream comment translation. The entire resource downloads at about 600MB, and memory usage stays between 500 and 700MB during operation. A single comment takes roughly 500 to 800 milliseconds to translate, which is smooth enough for regular comment flow. It even handles internet slang accurately.

Running locally on the device means no cloud server calls, which cuts costs and bandwidth. Plus, data never leaves the device, so user privacy is protected.

Key Points

  • Model compression: From 3.3GB to 440MB using 1.25-bit quantization.
  • Quality retention: Near-lossless translation quality, thanks to techniques like SEQ and QAD.
  • Broad compatibility: Works on both ARM and x86 platforms, including Intel's latest processors.
  • Real-world deployment: Already powering Bilibili's live comment translation with low latency and privacy benefits.

This breakthrough shows that with clever quantization, even large language models can live happily on small devices. And that's good news for anyone who wants fast, private, and affordable translation on the go.