Remember the awe you felt when you first chatted with ChatGPT and Doubao? Built on a universal language model, they seemed omniscient, capable of not only composing poetry and understanding current events, but also painting, writing letters, and coding—a range of abilities that left people in awe. However, for such an intelligent assistant that seems "omnipotent" across fields such as culture, art, and science, a question is increasingly on the minds of scientists: how well can a universal language model perform in specialized fields with high knowledge barriers?

In protein engineering, general-purpose language models are increasingly appearing in protein design processes due to their improved performance in biological knowledge and reasoning benchmarks. However, scientists still lack a clear understanding of their ability to assess the effects of protein mutations. Although some technical reports have demonstrated the evaluation results of some general-purpose language models, such as the ProteinGym-Hard evaluation mentioned in the Anthropic system technical report, which evaluated the Claude series models, these studies still have many problems, such as a limited model population and a lack of cross-sectional comparative validation.

In view of this,A research team from Harvard University and Capable has proposed a PG-LLM benchmark built on the ProteinGym dataset.For the first time, 13 general-purpose language models and 95 specialized protein prediction tools were simultaneously evaluated within a unified evaluation dimension. Through the PG-LLM benchmark test, scientists have confirmed for the first time that general-purpose language models can indeed perform protein mutation screening, possessing high practical value, and outperforming at least half of the mature specialized prediction tools. The introduction of this benchmark may provide strong support for emerging tools in the field of protein engineering, while also offering entirely new ideas for protein design.

The related findings, titled "PG-LLM: Benchmarking General-Purpose Language Models for Protein Variant Ranking," have been published as a preprint on bioRxiv.

Research highlights:

* Establishing the first standardized benchmark for protein mutation ranking in a general-purpose language model: PG-LLM 

* Conduct large-scale cross-model control experiments to clearly define the performance boundaries between general language models and specialized protein prediction models.
* Provides a multi-dimensional experimental perspective, revealing the differences in the underlying mechanisms of the two types of models and providing a basis for potential fusion solutions.

Paper link: https://www.proteingymllm.com/paper

Based on the ProteinGym dataset, this dataset covers 217 sets of experimental measurements.

PG-LLM is built upon the ProteinGym dataset, a universally standardized benchmark dataset in the field of protein mutation prediction. This study uses the complete ProteinGym v1.3 deep mutation scanning dataset.It covers 186 different proteins and a total of 217 sets of experimental data.Of these, 148 groups were determined by single point mutation experiments only, and the other 69 groups were determined by multi-site compound mutation experiments.

To ensure fairness in the testing, a stratified sampling strategy was employed to construct the candidate mutation set. All mutations were sorted according to their measured functional values from smallest to largest, and then equally divided into 10 intervals. Approximately equal numbers of mutation samples were then drawn from each interval. This ensured that the candidate set balancedly covered samples from low, medium, and high functional levels, preventing sample concentration in a single interval and thus avoiding testing bias. Simultaneously, to eliminate the randomness of results caused by random sampling, three independent subsets of candidate mutations, referred to as "draws," were constructed for each experimental group. The final model performance score was the average of the three evaluation results.

This study set three candidate set sizes, each containing 10, 50, and 100 mutations, respectively.The ranking conclusions are primarily based on a candidate set of 50 mutations.To ensure the uniformity and fairness of the experiments, all experimental scores followed the nested macro aggregation rules of ProteinGym, meaning that the higher the measured score, the better the functional performance of the mutated protein.

The first standardized evaluation benchmark for protein mutation ranking for a general language model

The PG-LLM benchmark is a standardized set of benchmark tools specifically designed to evaluate the protein mutation sequencing capabilities of general language models.The framework is designed to provide a fair and uniform workflow for zero-sample mutation sequencing tasks.Specifically, each task only provides the model with the input protein sequence and experimental detection plan description. Without providing multiple sequence comparison (MSA) or protein three-dimensional structure information, the model is forced to autonomously sort the 50 scrambled mutant proteins according to their functional quality.

To systematically evaluate the collective performance of general language models on the protein mutation sequencing task, avoid biased conclusions from a single model, and quantify the performance level of the general language model and clarify its capability boundaries,This study included two major categories of evaluation objects for comprehensive validation from both horizontal and vertical dimensions: one was a general language model, and the other was a specialized protein prediction model.Regarding input conditions and evaluation rules, based on the different working principles of the two types of models, this study adopts deliberately unequal input conditions for the specialized prediction model compared to the general language model. Unlike the latter's more stringent input conditions, the former can use natively provided inputs, including MSA or protein 3D structure information. The evaluation rules use the same aggregated evaluation metrics as the general language model, with the score index represented by the Spearman rank correlation coefficient. ρ=1 indicates perfect consistency in ranking, and ρ=0 indicates no monotonic rank correlation.

Specifically, the general language model includes 13 currently mainstream models, such as Claude Opus 5, the GPT series, the Gemini series, GLM-5.2, and Kimi K3. Among the 95 specialized protein prediction models,It includes VenusREM, a state-of-the-art predictive tool in the field of protein mutations.This retrieval-enhanced open-source protein language model, developed under the leadership of Tan Yang, a technical scientist at Tianwu Technology AI Lab, is specifically designed for zero-sample unit point mutation effect prediction. In this study, it stands out as a top performer among professional domain models, outperforming Claude Opus 5.

Previously, VenusREM achieved leading zero-shot prediction performance on ProteinGym, a benchmark for evaluating protein mutation effects built by a team at Harvard Medical School, surpassing representative models developed by teams such as Meta, BioMap, and Microsoft. The VenusREM model and related code have been fully open-sourced, with over 4,000 downloads on the Hugging Face platform in the past 30 days and over 250,000 downloads since its release.

To facilitate developers' quick adoption of the advanced VenusREM modelThe tutorial section of HyperAI's official website (hyper.ai) now features "VenusREM: Search-Enhanced Protein Mutation Effect Prediction."Environment configuration is complete, and deployment is easy with minimal effort.

Run online:

https://hyper.ai/cn/notebooks/venusrem-retrieval-enhanced-protein-mutation-effect-prediction
VenusREM open source address:

https://github.com/ai4protein/VenusREM

Cross-model, multi-dimensional comparative experiments reveal the differences in the underlying mechanisms between the two types of models.

The core conclusions of this study are based on a candidate set of size 50, and the results are shown in the figure below:

The results show that Claude Opus 5 tops the general language model leaderboard with a Spearman correlation coefficient ρ=0.406, followed closely by GPT-5.6 Sol with ρ=0.402. However, this result is not absolute. When comparing experiments where both models could complete the tests, GPT-5.6 Sol actually performed better, for example, in 180 shared evaluation metrics.The GPT-5.6 Sol score surpassed the GPT-5.6 score by 0.409 to 0.400.

More tellingly, compared to vertical models, Claude Opus 5 also reaches the median level. Specifically, it outperforms 49 out of 95 methods, including 41 out of 46 pure sequence methods and 8 out of 49 composite methods using MSA or structural methods. This is reflected in its score, which is higher than the median of pure sequence methods, ρ = 0.374.It is close to 0.411 for the dedicated large protein model ESM2-650M, but still significantly behind 0.523 for VenusREM.

To further verify whether increasing computational resources during the inference phase can improve the capabilities of general-purpose language models, researchers conducted further validation. The results showed that giving the models more time to think and output more inference text increased the ranking accuracy of all GPT, Claude, and Gemini models. However, after reaching a certain level, the performance gains gradually converged, ultimately failing to catch up with specialized protein prediction models. (See the figure below.)

However, increasing resources is not always beneficial. Increasing the size of the candidate set can have the opposite effect on general language models. For example, when the candidate size increases from 10 to 100, the score of GPT-5.6 Sol in 178 shared metrics actually drops from 0.423 to 0.375. The performance decline of other general language models is also unavoidable.In stark contrast, vertical models such as ESM2-150M, ESM2-650M, and VenusREM are almost unaffected.

In addition, pure sequence prediction tools perform better on proteins with sufficient evolutionary contrast information, while the accuracy of general-purpose large models is almost unaffected by contrast depth. For example, in a comparison experiment of 13 general-purpose language models and 46 pure sequence methods, across 180 tests, the aggregation results of the 13 general-purpose language models remained essentially unchanged between low-depth and high-depth proteins. The adjusted performance for low, medium, and high-depth groups were 0.313, 0.301, and 0.311, respectively, with a difference of only -0.002 between high and low depths. (See figure below.)

In contrast, the average value of the pure sequence method gradually increased from 0.309 at low depth to 0.380 at medium depth and 0.411 at high depth, with a difference of +0.102. Furthermore, the point estimation of 45 out of 46 methods showed that the performance of high depth was better than that of low depth.

Finally, the study further verified whether training data contamination affected the model's prediction accuracy. The results showed that the ranking accuracy of models that could recognize the original documents did not steadily improve compared to those that could not, and in some cases, the scores of models that could recognize the original documents were even lower.

Potential fusion schemes offer new ideas for the application of tools in the field of protein design.

In summary, the results of this study reliably demonstrate the ability of general-purpose language models to predict protein mutation sequencing, and their performance surpasses that of many classic sequence prediction tools, indicating their practical value. However, it also confirms that there is still a significant gap between them and leading specialized prediction methods, and they cannot completely replace specialized protein prediction methods. Nevertheless, "specialization is key," and neither general-purpose language models nor specialized prediction models can be exclusively used for specific domain tasks; how to coordinate their efforts is also an important issue.

It is worth noting thatIn the primary leaderboard of this study, VenusREM, whose Spearman correlation coefficient far surpasses that of VenusREM, was developed under the leadership of a PhD student born in the 2000s.—Tan Yang, a technical scientist at Tianwu Technology AI Lab, is currently a joint doctoral student at Shanghai Jiao Tong University & Shanghai Innovation Academy, class of 2025. As a multimodal pre-trained model for protein mutation effect prediction and protein engineering, VenusREM's core objective is to jointly utilize the amino acid sequence, three-dimensional structure, and homology information of proteins to more accurately assess the potential impact of candidate mutations on functional properties such as protein activity, stability, and binding ability.

Unlike traditional protein language models that rely solely on single sequence information, VenusREM encodes amino acid sequences as sequence terms and discretizes the local three-dimensional spatial environment surrounding each residue as structural terms. It then jointly models sequence context, local structural features, and the relative positional relationships between residues through decoupled multi-head cross-attention. In this way,The model can not only understand the sequential dependence of an amino acid in the sequence, but also identify the interactions between residues that are far apart in the sequence but close to each other in three-dimensional space.

Building upon this foundation, VenusREM further introduces a search enhancement module to search for homologous sequences related to the target protein from protein databases and extract evolutionary conservation and amino acid substitution preferences at each site through multiple sequence alignment. The model weightedly fuses sequence-structure joint characterization with homologous evolutionary information. Even with zero samples requiring no experimental labeling of the target protein or additional training, it calculates fitness scores for single-point and multi-point mutations and ranks candidate mutations by comparing the predicted scores of mutant and wild-type amino acids at corresponding sites.

Then,Dr. Tan Yang led the development and implementation of MatwingsVenus™, a conversational protein research and development intelligent agent, within the company, based on VenusREM. The biggest highlight of MatwingsVenus™ is that it truly realizes "conversational research and development." Without delving into deep coding or building complex models, simply issuing commands like in a casual chat allows AI to independently complete the design, prediction, and full-process verification of proteins. Just last month, this "AI scientist" achievement shone brightly at the 2026 World Artificial Intelligence Conference, winning the title of "Treasure of the Conference" alongside industry giants like Alibaba and Baidu, becoming the only scientific intelligent product selected for the conference.

As Tan Yang stated in an interview, general language models and vertical models each have their advantages and disadvantages. The former is "erudite" but lacks depth of knowledge in a professional field, while the latter has a deep understanding of underlying scientific research data but lacks thinking and reasoning abilities. Using a general language model to understand complex experimental requirements, thereby achieving a human-in-the-loop, and combining it with a vertical model to use evolutionary or structural information for accurate scoring, may achieve better results.