Together AI unveils compiler that continuously optimizes AI inference for significant speed gains

Together AI has launched Together Compiler (TKCC), a novel agent-driven system designed to perpetually re-optimize model inference as production traffic patterns evolve. This compiler utilizes coding agents to rewrite a model's forward pass based on input parameters like traffic profiles and hardware configurations. The system achieves substantial performance improvements, notably beating the best open configurations by up to 2.6x in TPS on SemiAnalysis InferenceX’s AgentX workload. Unlike traditional methods that restart serving engines for each test, TKCC initiates the engine only once, allowing agents to submit candidate passes over RPC and swap function pointers. Testing revealed that a two-tiered measurement approach—a fast tier screening 12 layers and a slow tier confirming gains on 60 layers—yielded 8.1x more candidates than a single slow tier, resulting in a 7.2% faster solution. Furthermore, Together AI reported a 252x higher agent experiment throughput compared to an engine-restart baseline. The compiler incorporates a cost model that evaluates tensor, expert, data, and context parallel layouts, factoring in microbenchmarks and MoE expert imbalance. On Kimi K3, this led to a production layout achieving up to 24% higher TPS. While expert-curated knowledge bases showed a 25% to 31% lower speedup after 12 hours, naive agents instructed to maximize speed outperformed expert strategies in ensemble scoring.