User runs 2.8 trillion parameter kimi k3 model on retired intel optane memory without a GPU

A user named u/IndividualSurvival shared a home lab experiment on Reddit on November 2nd, demonstrating the successful operation of Moonshot AI's Kimi K3 model, which boasts 2.8 trillion parameters, without utilizing a dedicated graphics card. The setup relied on using retired Intel Optane persistent memory. The configuration involved a 40-core third-generation Intel Xeon Scalable processor and the Optane 200 series memory. The test utilized the App Direct mode, presenting DRAM and Optane as separate data spaces to the software, with inference executed via llama.cpp. Intel Optane, a non-volatile storage technology developed by Intel and Micron, is based on 3D XPoint. While the test was successful, the token generation speed was not seamless. With the model quantized to 1-bit and data compressed, the initial generation rate was about 0.85 tokens per second. This speed improved to 1.50 tokens per second after disabling Direct Access (DAX) and allowing frequently accessed data to benefit from DRAM caching. A separate 3-bit quantization test yielded a slower rate of 0.75 tokens per second, requiring approximately 18.5 hours to generate 50,000 tokens.