Skip to main content

Swiftlet squeezes 80B Qwen onto Mac with just 4.3GB memory

Imagine running a massive AI model that normally requires a data center right on your laptop or phone. That's the promise of Swiftlet, a new Swift and Metal runtime that's turning heads in the AI community. It's specifically designed for the Qwen3-Next and Qwen3.5/3.6 family of Mixture-of-Experts (MoE) models, and it does something pretty clever: it keeps only the small, dense core of the model in memory, while the bulk of the expert weights live on your SSD, streamed in on demand.

The Numbers That Matter

On an M5 Mac, the 4-bit version of Qwen3.6-35B-A3B takes up 18GB on disk but only peaks at 2.6GB of memory, decoding at 7 to 11 tokens per second. Even more impressive, the 4-bit Qwen3-Next-80B-A3B requires 42GB on disk but peaks at just 4.3GB of memory, running at 4.5 to 5 tokens per second. And get this: the 35B version can now run on an iPhone 17, using about 2.5GB of memory and hitting around 1 token per second. According to the developers, this is the first time such a model has run natively on a phone without touching a server.

How It Works

So, what's the trick? It's all about fine-grained scheduling. Each layer routes each token to a small subset of experts—10 out of 512 for the 80B version, 8 out of 256 for the 35B version. Swiftlet keeps the dense weights—attention, DeltaNet projections, routers, shared experts, and embedding vectors—firmly in memory, taking about 1.3GB (for 35B) or 2.5GB (for 80B) in 4-bit. The thousands of routing experts are repackaged into fixed-step data blocks stored in .qpack containers. To fetch an expert, it only needs one pread operation on the SSD, no mmap, and it doesn't disturb the page cache. Popular experts are cached in a limited pool using an LFU plus recency strategy for eviction; the hit rate ranges from 43% to 70%, and the cache size has almost no impact on speed, as Apple's SSD can handle the overhead of misses.

The entire forward pass runs on Metal using runtime-compiled shaders, so there's no need for a Metal toolchain during build, and the same code can be directly deployed to iOS. More interestingly, 75% of the layers use Gated DeltaNet linear attention with a fixed-size cyclic state, meaning that regardless of the context length, it never grows an expanding KV cache—the hidden burden of long text conversations is quietly offloaded.

More Than a Toy

Swiftlet isn't just a command-line toy. It has four identities designed for itself. As a library, SwiftletCore can be embedded into any macOS or iOS app, providing chat capabilities with streaming incremental output and conversation caching. As a command-line tool, swiftlet chat and swiftlet generate handle local execution and benchmarking, while swiftlet-repack builds containers directly from MLX checkpoints and supports resuming downloads from Hugging Face. As a server, swiftlet-server provides OpenAI-compatible chat-completions interfaces on a loopback address, allowing any OpenAI-compatible chat interface to connect to the local model. As an application, the iOS version of Priv AI embeds SwiftletCore as a streaming model engine, allowing regular users to start chatting by simply downloading it.

Accuracy and Verification

Regarding correctness, every layer's forward pass is compared layer-by-layer against the mlx-lm reference implementation, covering f32 and int4 quantization forms. Incremental decoding is also compared with full sequence results, and the Metal kernels have been repeatedly verified against precise CPU references. The container can also perform byte-level verification against the source checkpoint—whether experts are read from cache or disk, the answers are exactly the same.

The Inspiration Behind It

Its inspiration path is clearly explained: TurboFieldfare first validated the feasibility of streaming experts for Gemma on Mac, and Swiftlet borrowed some publicly available design experiences from it, such as using pread to stream experts into bounded slot pools, using LFU plus recency for eviction, and using fixed-step packaging so that one read equals one fetch. However, the rest was written from scratch, with about 10,000 lines of Swift and Metal code, tackling the completely different Qwen mixed architecture: Gated DeltaNet linear attention, gated GQA, and high-sparse MoE with shared experts. It even implements MLX-style int4/int8 group quantization calculations in Metal, using byte-addressable kernels and 64-bit offsets to support gigabytes of shards.

Image

Key Points

  • Memory efficiency: Swiftlet runs an 80B-parameter Qwen model on a Mac with just 4.3GB peak memory, and a 35B version on an iPhone 17 with 2.5GB.
  • Clever architecture: It keeps only dense weights in memory, streaming expert weights from SSD on demand.
  • Real-world usability: The project is fully end-to-end usable, with verified correct outputs.
  • Trade-off: Each token activates about 3B parameters, so the models behave like large models in conversation but like small models in factual memory.
  • Multiple deployment options: Works as a library, command-line tool, server, and iOS app.