FreeToken Rewrites Local MoE Serving: 2x Faster Than Ollama on the Right Hardware

A new open-source inference engine from researchers at UC Berkeley and UT Austin has cleared a bottleneck that made running frontier-scale Mixture-of-Experts models on consumer hardware impractical — but only under the right conditions, which turn out to be far more demanding than the viral benchmarks circulating online imply.

FreeToken, published August 17, 2026, in the paper "FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution" and released simultaneously as Apache 2.0 open-source software, replaces the static GPU/CPU model-splitting approach used by Ollama and llama.cpp with a system that treats an entire personal computer — its GPU, CPU, VRAM, system RAM, and PCIe interconnect — as a single elastic inference platform. The paper's author list includes notable figures from UC Berkeley and UT Austin: Song Han, Matei Zaharia, Ion Stoica, Kurt Keutzer, and Chenfeng Xu are among the contributors. The project accumulated more than 2,200 GitHub stars in its first week of release.

The headline claim — running Z.ai's 753B-parameter GLM-5.2 on a single workstation GPU — is real. But what that requires in practice, and when FreeToken's approach actually beats the incumbents, is a different conversation from the one that has been happening on social media.

Why MoE Models Create a PCIe Bandwidth Problem

To understand FreeToken's architecture, you first need to understand what makes Mixture-of-Experts models both attractive and painful to serve on consumer hardware.

A dense LLM activates all of its parameters for every token it generates. A MoE model instead maintains a large bank of specialized "expert" subnetworks and routes each token only to a small subset. DeepSeek-V4-Flash, for example, has 284 billion total parameters but activates approximately 13 billion per token. That sparsity is what makes it possible to build models with massive total capacity at manageable per-token compute costs. MoE is the architecture behind many of the largest open-weight models currently available: DeepSeek-V4-Flash, Qwen3.6-35B, and GLM-5.2, among others.

The problem is storage. Those 13 billion active parameters on a given token could fit in a consumer GPU with 16–24GB of VRAM. But the remaining 271 billion inactive parameters have to live somewhere, and they will not fit in VRAM. They occupy system RAM, and the GPU fetches whichever experts are needed for each token over the PCIe bus. Consumer PCIe bandwidth — typically 16 to 64 GB/s depending on the slot and generation — is far below the NVLink interconnects that datacenters use to mask this overhead.

MoE routing is dynamic: the gating network selects experts based on the incoming token, which means the runtime cannot predict in advance which expert weights will be needed next. This unpredictability is what makes static placement strategies — the approach used by every inference engine before FreeToken — fundamentally wasteful.

How Ollama and llama.cpp Handle the Problem — and Why It Falls Short

The dominant tools in local LLM inference — llama.cpp, a C++ inference engine, and Ollama, which wraps llama.cpp in a Go daemon that adds model management and a REST API — handle MoE weight distribution by making a one-time decision at model load. Some layers are assigned permanently to the GPU; the rest are assigned permanently to CPU-accessible system RAM. Ollama automates this split based on hardware detection; llama.cpp requires the user to configure it manually via flags.

This static approach is predictable and low-overhead. But it is also fundamentally wrong for a large fraction of tokens: because which experts are activated changes with every token, a fixed boundary between GPU and CPU will inevitably force many tokens to detour to the wrong side. The GPU stalls waiting for expert weights to transfer across the PCIe link every time a cache miss occurs. On a laptop, every cache miss is a GPU stall.

The situation is manageable for models small enough to place most layers in VRAM. It becomes a serious performance ceiling when a significant fraction of the model cannot fit in VRAM — which is the defining condition for frontier-scale MoE models on consumer hardware.

FreeToken's Q* Policy Replaces Static Splits with Real-Time Scheduling

FreeToken takes a structurally different approach. The entire expert pool lives in system RAM as a single authoritative copy. The GPU's VRAM functions as a dynamic LRU (Least Recently Used) cache that holds whichever expert weights were most recently activated, as described in the FreeToken paper.

The choice to use an LRU cache is not arbitrary. MoE models exhibit strong reference locality: for a sequence of related tokens, the gating network tends to activate the same small set of experts repeatedly. A user writing a Python function will route most tokens through code-related experts; a user asking a history question will route through different experts. After the first few tokens in any coherent sequence, the LRU cache fills with the "hot" experts that keep being activated. The FreeToken paper reports cache hit rates as high as 85 percent in typical workloads.

When a cache miss does occur — an expert is needed that is not currently in VRAM — FreeToken's Q* policy kicks in. Rather than waiting for the expert weights to transfer to the GPU, FreeToken simultaneously evaluates two options: streaming the weights over PCIe to the GPU for execution there, or running the computation directly on the CPU, which already has the weights in RAM. Neither option is universally faster; the answer depends on the specific balance of PCIe bandwidth and CPU memory bandwidth on the machine running the job.

At startup, FreeToken benchmarks both pathways on the actual hardware. When a batch of cache misses occurs, it computes a closed-form ratio for how much work to route to each destination. On a high-end workstation with PCIe 5.0, FreeToken might send approximately 41 percent of missed experts to the GPU and 59 percent to the CPU. On a laptop with a narrower PCIe 4.0 link but a more capable CPU, those ratios might invert to roughly 13 percent and 87 percent. The system's goal is for both execution paths to complete simultaneously, eliminating the sequential stalls that plague static offloading.

A third mechanism handles the prefill stage — the initial processing of a long input prompt — where diverse token sequences can activate a wide sweep of experts and exhaust even a well-warmed cache. FreeToken uses double buffering during prefill: while the GPU processes the experts for layer N, the engine predicts which experts will be needed for layer N+1 and begins streaming them from RAM into a secondary VRAM buffer in parallel. By the time the GPU finishes layer N, the layer N+1 weights are already loaded, hiding PCIe transfer latency behind GPU computation.

Finally, FreeToken stores model weights on disk in its own FTW (FreeToken Weight) format, which matches the engine's internal memory layout exactly. Standard inference engines load weights, decode them, and repack them into the appropriate internal format at startup — a slow operation for models that weigh hundreds of gigabytes. FTW is a direct memory-mapped read with no intermediate transformation, reducing startup time for large models.

Independent Tests Corroborate the Gains — on Workstation Hardware

The FreeToken paper reports author-conducted benchmarks showing the engine ran Qwen3.6-35B at approximately 39.3 tokens per second on an 8GB RTX 4060 laptop; served DeepSeek-V4-Flash (284B parameters) on an RTX 5090 desktop at 22 to 25 tokens per second; and processed GLM-5.2 (753B parameters) on a single workstation GPU at approximately 14.9 tokens per second versus 7.3 for llama.cpp on equivalent hardware. These benchmarks were conducted by the authors themselves and have not been independently replicated at scale by external laboratories.

The most credible independent test available was conducted by BetterStack, which ran a real-world agentic coding task — using OpenCode to write a complete Pytest suite — on a workstation equipped with an RTX 5090 (32GB VRAM), a Core Ultra 9 285K CPU, and 64GB of DDR5 RAM. The test model was a Qwen3.6-35B quantized to 8-bit (approximately 38GB), roughly 6GB larger than the GPU's VRAM. Ollama completed the task in 14 minutes and 20 seconds at a median of 58.8 tokens per second. FreeToken completed the same task in 4 minutes and 40 seconds at 132.5 tokens per second. The 2.25x speedup was observed on exactly the workload FreeToken's design targets: an MoE model that exceeds VRAM, being used for an agentic task with shifting context.

The BetterStack test also quantified how aggressively FreeToken's cache can be compressed without major cost. Reducing the expert cache from 57.8 percent of all experts to 40 percent caused only a 7.2 percent performance penalty — consistent with the paper's claim that a small subset of hot experts handles the majority of activations in typical workloads.

The same BetterStack test confirmed the reverse: when the model fit entirely within VRAM, Ollama won. A 4-bit quantized variant of the same model fit in 22GB and yielded 239.6 tokens per second under Ollama versus 225.3 under FreeToken. FreeToken's scheduling overhead costs a small but measurable amount when there is no memory pressure to justify it.

Where Laptop Benchmarks Miss the Point — and What They Do Tell Us

A benchmark that appeared widely in online communities tested FreeToken against Ollama and llama.cpp on an RTX 3050 laptop with 6GB of VRAM and 24GB of system RAM. On this hardware, running a 20B MoE model in MXFP4 quantization, Ollama answered in roughly 0.38 seconds time-to-first-token, llama.cpp in 0.5 seconds, and FreeToken in 1.17 seconds — approximately three times the wait. FreeToken also trailed on throughput.

These numbers are accurate observations about that specific hardware configuration. They are not a meaningful test of FreeToken's design target. The engine's overhead — continuous cache state monitoring, Q* policy computation, bandwidth measurement — has no problem to solve on a 20B model that can largely fit within a fixed GPU/CPU split. The machinery is running without earning its keep, producing the worst-case performance profile: all overhead, no benefit.

More telling than the throughput numbers: on the same laptop, the FreeToken desktop app showed 35B and 27B models marked as "Insufficient RAM." The 24GB of system RAM was simply too little to hold those models' expert pools. This is not a software limitation; it is a reflection of what frontier MoE models cost. FreeToken's RTX 5090 plus 64GB RAM workstation test involved a machine of fundamentally different capability. The comparison in those viral laptop benchmarks is between an appropriately resourced tool and an under-resourced deployment.

How FreeToken Compares with KTransformers and Other Alternatives

The closest prior system to FreeToken in the local MoE serving space is KTransformers, a project from Tsinghua University's MADSys Lab and Approaching.AI that was presented at SOSP 2025 and has received ongoing updates through 2026. KTransformers also targets CPU/GPU hybrid inference for large MoE models and has active support for AMD ROCm, Intel Arc, and Ascend NPUs in addition to NVIDIA — a significant platform breadth advantage over FreeToken's current NVIDIA-only constraint.

The architectural difference is the core distinction. KTransformers uses static CPU/GPU offloading rules determined before inference begins. FreeToken computes a closed-form optimal split per layer in real time based on measured bandwidths. The FreeToken paper reports 1.5 to 2.3x throughput gains over KTransformers and other local engines including llama.cpp and Ollama. Those comparisons are author-conducted; independent head-to-head tests at this scale have not been published as of early September 2026.

vLLM and SGLang, the dominant production-serving inference frameworks, are not relevant alternatives here. Both are designed for datacenter environments with high-bandwidth GPU interconnects; they assume memory hierarchies that personal machines do not have.

For developers evaluating the landscape: Ollama remains the correct choice for common local workloads where the target model fits in VRAM. FreeToken is the correct choice when the model does not fit in VRAM and the machine has sufficient system RAM to hold the full expert pool — requirements that, for frontier-scale MoE, currently point to workstation hardware with 64GB or more of system RAM and an NVIDIA GPU.

Current Constraints: NVIDIA-Only and RAM Requirements

FreeToken's CLI requires Linux x86_64, an NVIDIA GPU with driver r580 or newer, CUDA 13, and Python 3.10 or newer. A desktop app for Windows and Linux, available at flashml.ai, removes the Python dependency and handles setup automatically. AMD ROCm and Apple Silicon are not supported as of August 2026.

The system RAM requirements scale directly with model size. Running a 35B model requires substantially more than 24GB of RAM. Serving DeepSeek-V4-Flash (284B) at the benchmark speeds reported in the paper required 192GB of system RAM — hardware that sits firmly in workstation territory. The 753B GLM-5.2 run required a specialized single workstation GPU described in the paper as having 96GB VRAM and 512GB of system memory.

The FTW weight format, while eliminating startup repack time, introduces format lock-in: models must be stored in FreeToken's format to benefit. This is a reasonable trade-off for users committed to running specific models continuously, but adds friction for users who switch frequently between engines.

Windows Memory Integrity — a security feature that blocks unsigned kernel drivers — can conflict with FreeToken's DLL stack. At least one documented case required disabling Memory Integrity to resolve a startup failure that produced no informative error in a standard terminal. Running as administrator surfaced the actual error message. These are early-software installation edge cases, but they indicate the setup experience has not reached Ollama's polish level.

Agentic Workloads Are the Specific Target

Beyond the throughput benchmarks, two FreeToken features are particularly significant for agentic AI workflows, where the pattern of model use differs fundamentally from single-question inference.

An agentic session continuously re-enters the prefill phase. Every tool call result, every thinking block, every context update triggers another pass over the accumulated conversation history. Without optimization, this causes redundant recomputation of the same prefix context on every turn — a quadratic cost as sessions grow longer. FreeToken's semantic-aware caching uses anchor checkpoints for KV cache and recurrent state, skipping redundant recomputation when only the tail of the context changed.

FreeToken also allows runtime reallocation of VRAM between the expert cache and the KV cache without restarting the engine. As a session grows longer, the KV cache needs more space; as the expert access pattern stabilizes, the expert cache can be safely reduced. This dynamic rebalancing via ft ctl lets the engine adapt to session characteristics rather than committing to a fixed memory partition at startup.

These features explain why the BetterStack agentic coding test — a task that involved multiple tool calls, context accumulation, and repeated prefill — showed a larger speedup than single-query throughput benchmarks would predict. The gain is not only from better decode throughput; it compounds across the repeated prefill operations that characterize real agentic use.

What the Benchmark Gap Reveals About the Inference Ecosystem

FreeToken's emergence points to a structural gap in the local inference tool ecosystem that has existed since MoE became the dominant architecture for large open-weight models. The tools developers reach for first — Ollama for ease of use, llama.cpp for performance — were designed for dense models and use static placement strategies that have no principled mechanism for handling the dynamic, cache-sensitive behavior of MoE expert routing.

That gap has been addressed in the datacenter setting through high-bandwidth interconnects; it has not been addressed for consumer hardware because no one had built the scheduling infrastructure to do so. FreeToken is the first system from a major research group to tackle the consumer MoE serving problem from first principles, with an architecture built around the specific properties of consumer hardware rather than scaled down from datacenter assumptions.

The work being watched next: whether AMD ROCm and Apple Silicon support get added, whether independent benchmark replications confirm the paper's throughput claims on a wider range of hardware, and whether the 1.5 to 2.3x gains hold at the 750B+ parameter scale where the datacenter alternative — renting GPU cluster time — remains the only practical option for most developers today. FreeToken's core claim, that the personal machines people already own can become practical platforms for frontier-scale open-weight intelligence, will be tested most seriously by whether the system's gains survive contact with hardware beyond the NVIDIA workstations it has been benchmarked on so far.