FreeToken enables high-performance local inference for massive AI models on consumer hardware

A team of researchers from UC Berkeley and UT Austin has developed FreeToken, an edge-native serving engine that allows large AI models to run efficiently on consumer-grade hardware. The system is designed to treat a personal machine as a unified, elastic inference platform, dynamically mapping computation and model state onto the available GPU, CPU, memory, and interconnect bandwidth. This approach enables the execution of models such as GLM-5.2, which has 753 billion parameters, on a single workstation GPU. FreeToken achieves this by leveraging a Mixture-of-Experts (MoE) architecture, which allows only a subset of the model's parameters to be active during inference, significantly reducing computational demands. The research team highlighted that existing serving engines, such as llama.cpp, KTransformers, Ollama, and MoE-Infinity, face several limitations in handling large models efficiently. FreeToken addresses these by improving throughput and reducing latency, with benchmarks showing it sustains 77–83 tokens per second on Qwen3.6-35B-A3B (BF16) and 22–25 tokens per second on DeepSeek-V4-Flash (MXFP4), outperforming the strongest baselines by 1.5–2.3 times. Decode latency remains within 12% of the single-turn rate across three agentic workloads, with worst-case time-to-first-token (TTFT) staying below 44 seconds. FreeToken is available under the Apache-2.0 license on GitHub and can be installed via PyPI as freetoken v0.1.2. It is also distributed as a one-click desktop application for Windows and Linux through flashml.ai. The tool supports OpenAI- and Anthropic-compatible endpoints on port 1919 and integrates with Claude Code, Codex, OpenCode, and OpenClaw. The system is particularly suited for solo developers, startups, and small engineering teams whose agent token costs exceed the price of a GPU they own. Enterprises are advised to use it for air-gapped or regulated workloads rather than as a datacenter replacement. The research team identified three key failure modes in existing serving engines and demonstrated how FreeToken overcomes them. On an 8 GB RTX 4060 laptop, the NVFP4 build serves a 35B model at 39.3 tokens per second, surpassing the median decode speed of Codex in production traces. On a single RTX PRO 6000, GLM-5.2 (753B, 40B active) runs at 14.9 tokens per second, compared to 7.3 tokens per second with llama.cpp. The system also supports synthetic-data generation, batch evaluations, and offline contract analysis, making it ideal for industries such as healthcare, legal, defense, finance, and IP-heavy R&D. The paper, GitHub repository, and project are publicly available for further exploration. The team also invites collaboration for promoting GitHub repositories, Hugging Face pages, product releases, and webinars.