Guide details how to execute nvidia dory 4b model with llama.cpp, analyzing loop effects

A technical guide details the process of running NVIDIA's Dory 1.0 4B reasoning model using llama.cpp. This model features a shared recurrent block that can be iterated one to four times per token, allowing a single checkpoint to operate across four different compute budgets without retraining. The tutorial involved building a specific llama.cpp branch and loading the 3.7 GB GGUF file to predict parameter and cache memory costs based on the architecture. The analysis involved rigorous testing across seven configurations (one to four loops in both cache modes). Measurements included decode speed on a fixed prompt and a paired GSM8K evaluation to track accuracy and answer length changes as loop counts increased. The process confirmed that the GGUF header's built-in loop count of 4 differed from the model card's recommendation of 2, necessitating explicit overrides. Key findings from the investigation included the recurrent block accounting for 1.84 out of the model's 3.98 billion parameters. The guide also highlighted practical warnings: users must explicitly set the loop count, as the GGUF defaults to four, and the specific fork commit must be pinned due to Dory support being an open draft pull request in llama.cpp.