Llms can create their own agent harnesses, but generalization remains a challenge

Researchers from ByteDance Seed, Singapore University of Technology and Design, and other institutions have developed a framework called HarnessDev, allowing large language models (LLMs) to engineer their own agent harnesses. The framework evaluates the harnesses the models create, rather than just the answers they produce. In tests, six LLMs, including Opus 4.8 and GPT-5.5, were assessed on their ability to build and refine harnesses across multiple benchmarks. Opus 4.8 achieved the highest average score of 67.8 under self-evaluation, though it lagged behind human-engineered benchmarks at 86.2. Despite the progress, generalization remains limited. Only 34 out of 64 changes made to harnesses showed consistent improvement across benchmarks. Code volume and self-test counts did not strongly correlate with performance, highlighting the complexity of the task. The study also revealed that much of the generated code was inert, with many features never being used in practice. The findings underscore the challenges LLMs face in creating robust, generalizable agent harnesses.