Mit researchers introduce weight-space programming, boosting Qwen3.5 performance by 6x with minimal parameter tweaks

Researchers from MIT CSAIL have unveiled a novel technique called Weight-Space Programming. This method allowed OpenAI's GPT-6 Astra to modify the neural network weights of Alibaba Cloud's Qwen3.5-0.8B model, achieving a significant performance leap in the MATH-500 benchmark from 7.9% to 48.1%, a six-fold increase. This was accomplished by adjusting only about 0.016% of the model's parameters, bypassing traditional training loops like gradient descent. The study, led by Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab, involved Astra identifying a repetitive pattern in the model's inference loop within the 15th and 19th MLP layers. By modifying 131,072 weights, the team drastically reduced the instances where the model hit the 32,768 token limit, improving instruction following on IFEval from 21.0% to 62.3%. While the process did not use backpropagation, it required Astra to operate within a Python environment for analysis and up to 50 rounds of forward inference. A control experiment using Gemma 3 270M showed that the technique could not create abilities the model lacked, only remove existing hindrances. The research suggests this approach offers new avenues for neural network interpretability.