From virtual models to lab tests: assessing ai's ability to design effective proteins

A recent study evaluates the effectiveness of AI in designing proteins by analyzing data from Anthropic’s claude-protein-binder-design dataset, which includes 1,440 AI-generated miniprotein binders tested against 16 targets. The dataset provides both computational predictions and real-world experimental results from two independent laboratories, allowing researchers to assess not only the success of the designs but also the reliability of structure predictors and the impact of experimental variability. The study also introduces a target-aware classifier to determine if these computational signals can reliably predict experimental outcomes. The research begins by addressing the challenge of inconsistent file naming by creating a dynamic mapping system to locate data files, ensuring accurate access to critical datasets such as wetlab_summary and design_summary. By focusing on actual vendor calls rather than binder_final, the team avoids misclassifying untested designs as failures. They then compute hit rates across models, campaigns, generators, and targets, using Wilson intervals to account for small sample sizes, with the target plot highlighting how antigen selection significantly influences results. Researchers evaluate ten predictors using both ipSAE and self-consistency DockQ metrics, along with bootstrapped confidence intervals to identify meaningful differences. Ranking performance is normalized to percentiles to enable cross-metric comparisons, with a Spearman heatmap illustrating how ensemble methods improve performance when predictors disagree. The study also converts rankings into precision@N to better reflect practical testing scenarios and uses an enrichment table to assess the value of triage at different budget levels. Cohen’s κ and a log-log KD comparison between vendors are used to set performance benchmarks, with label noise identified as a limiting factor for AUC scores. The study further investigates whether expression titer alone can distinguish binders, suggesting that solubility may influence scores. By re-running consensus on expressed-only designs, researchers isolate interface quality from biophysical factors. Epitope contact lists are parsed into residue sets to determine if successful binders share common binding patches across targets. Sequences are featurized using composition, charge, hydropathy, entropy, and hydrophobic run length, with gradient boosting models trained under two schemes—random folds and target-grouped folds. The difference between these methods highlights potential leakage due to target-specific base rates. A one-hot control and permutation importance on a held-out target block help identify robust features. The study concludes that while in-silico scoring is useful, it does not fully capture real-world performance, emphasizing the importance of careful evaluation and target-specific validation to ensure models generalize effectively to new targets.