A guide to running the 2-bit quantized Qwen3.8-27B model using the saluki 27b

A recent tutorial provides a detailed guide on running the 2-bit quantized version of the Qwen3.8-27B model, known as Saluki 27B. The model, built on ISTA-DASLab’s GSQ-RCO quantization, is claimed to maintain tool calling capabilities while reducing the model size to 7.9 GB. The tutorial compares Saluki with the full 54 GB model using the same 120-task benchmark. Both models are tested on the Berkeley Function Calling Leaderboard, with results scored using BFCL’s checker. The guide outlines the setup process, including downloading and verifying the model files, ensuring compatibility with CUDA 12 on NVIDIA GPUs or Apple-silicon Macs. It highlights the differences in the GGUF headers and byte-level comparisons, showing that Saluki is a modified version rather than a re-encoded file. The analysis reveals that most changes are in the feed-forward layers and output gates, with minimal impact on attention mechanisms. The tutorial also evaluates the performance of both models on the same tasks, emphasizing the trade-offs between speed and accuracy. It concludes that while Saluki is a lightweight version, the comparison with its parent model reveals insights into the training process rather than just quantization. The guide notes the limitations of the study, including small sample sizes and greedy decoding, and highlights areas not tested, such as the full-size model and vision add-ons.