A compact evaluation framework for testing AI models and prompts

Simon Willison has created smevals, a lightweight evaluation suite designed to assess and compare different AI model configurations. The tool enables users to define test cases through YAML configuration files and run them against various models, with automated grading based on customizable criteria. The framework separates the execution phase from the grading phase, offering flexibility in workflow management. Results can be visualized through a local web interface or exported as static HTML reports for hosting on any server. Willison demonstrated the tool's capabilities with a practical example evaluating how different models generate haikus. The project stems from three years of experimentation with evaluation frameworks, built in partnership with the Prime Radiant applied AI research lab.