Anthropic introduces plugin evaluation system for claude code

Anthropic has introduced a new plugin evaluation workflow for Claude Code, enabling developers to assess plugin performance through structured testing. The system runs plugins against realistic prompts, grades the output, and compares results with a version without the plugin. It addresses three key questions: whether the skill is triggered, survives edits, and outperforms a baseline model. The tool is deployable on Claude Code v2.1.269 or later, with each evaluation billed to the user's account. Evaluations are organized in an 'evals/' directory, with prompts and graders defined in markdown files. Six grader types are available, with four being free and two requiring a judge model. Results are published to a report.html file, with optional public sharing on claude.ai. The system includes a CI gate for skills, allowing automated testing and reporting. Evaluations run twice per case—once with and once without the plugin—to measure the plugin's contribution. A single case can generate multiple runs, with costs and time varying based on complexity. The tool also supports a 'bare' mode for CI pipelines, generating templates instead of full reports. Anthropic highlights common issues, such as plugins failing to trigger skills naturally, which the current validation tools cannot detect.