Benchmark Scores

Individual View

Aggregate View

Rank▲	Model name▲	Input Modalities▲	Avg. Spearman▲	Std. Error of Diff. to Best Score*▲	Spearman by Function					Spearman by MSA Depth			Spearman by Taxon				Spearman by Mutation Depth					Model Details
Rank▲	Model name▲	Input Modalities▲	Avg. Spearman▲	Std. Error of Diff. to Best Score*▲	Activity▲	Binding▲	Expression▲	Organismal Fitness▲	Stability▲	Low depth▲	Medium depth▲	High depth▲	Human▲	Other Eukaryote▲	Prokaryote▲	Virus▲	1▲	2▲	3▲	4▲	5+▲	Description▲	References▲
1	AIDO Protein-RAG (16B)	Structure & MSA	0.518	0.0	0.517	0.426	0.522	0.491	0.635	0.498	0.534	0.585	0.531	0.587	0.558	0.522	0.527	0.414	0.419	0.394	0.414	AIDO Protein-RAG (16B)	Sun, N., Zou, S., Tao, T., Mahbub, S., Li, D., Zhuang, Y., Wang, H., Cheng, X., Song, L., & Xing, E.P. (2024). Mixture of Experts Enable Efficient and Effective Protein Understanding and Design. bioRxiv.
2	VenusREM	Structure & MSA	0.518	0.005	0.495	0.454	0.533	0.459	0.65	0.495	0.524	0.577	0.529	0.582	0.549	0.492	0.534	0.397	0.355	0.322	0.368	VenusREM	Yang Tan, Ruilin Wang, Banghao Wu, Liang Hong, Bingxin Zhou. (2024). Retrieval-Enhanced Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model. ArXiv, abs/2410.21127.
3	ProSST (K=2048)	Single sequence & Structure	0.507	0.006	0.476	0.445	0.53	0.431	0.653	0.465	0.507	0.58	0.516	0.573	0.549	0.454	0.521	0.394	0.317	0.277	0.332	ProSST (K=2048)	Mingchen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Ziyi Zhou, Huiqun Yu, Wanli Ouyang, Liang Hong, Bingxin Zhou, Pan Tan. (2024). ProSST: Protein language modeling with quantizied structure and disentangled attention. bioRxiv.
4	ProSST (K=4096)	Single sequence & Structure	0.498	0.009	0.444	0.472	0.507	0.416	0.652	0.472	0.481	0.583	0.497	0.574	0.547	0.44	0.505	0.426	0.388	0.342	0.408	ProSST (K=4096)	Mingchen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Ziyi Zhou, Huiqun Yu, Wanli Ouyang, Liang Hong, Bingxin Zhou, Pan Tan. (2024). ProSST: Protein language modeling with quantizied structure and disentangled attention. bioRxiv.
5	S3F-MSA	Structure & MSA	0.496	0.007	0.502	0.44	0.479	0.477	0.581	0.469	0.509	0.547	0.502	0.558	0.521	0.502	0.499	0.333	0.378	0.346	0.383	S3F with MSA retrieval	Zuobai Zhang, Pascal Notin, Yining Huang, Aurelie C. Lozano, Vijil Chenthamarakshan, Debora Marks, Payel Das, Jian Tang. (2024). Multi-Scale Representation Learning for Protein Fitness Prediction. NeurIPS.
6	S2F-MSA	Structure & MSA	0.488	0.007	0.498	0.432	0.472	0.472	0.567	0.463	0.502	0.536	0.495	0.546	0.513	0.493	0.491	0.303	0.346	0.318	0.362	S2F with MSA retrieval	Zuobai Zhang, Pascal Notin, Yining Huang, Aurelie C. Lozano, Vijil Chenthamarakshan, Debora Marks, Payel Das, Jian Tang. (2024). Multi-Scale Representation Learning for Protein Fitness Prediction. NeurIPS.
7	ProSST (K=1024)	Single sequence & Structure	0.485	0.009	0.433	0.436	0.499	0.414	0.642	0.457	0.466	0.585	0.483	0.568	0.539	0.436	0.492	0.434	0.373	0.341	0.403	ProSST (K=1024)	Mingchen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Ziyi Zhou, Huiqun Yu, Wanli Ouyang, Liang Hong, Bingxin Zhou, Pan Tan. (2024). ProSST: Protein language modeling with quantizied structure and disentangled attention. bioRxiv.

* Non-parametric bootstrap standard error of the difference between the Spearman performance of a given model and that of the best overall model, computed over 10k bootstrap samples from the set of proteins in the ProteinGym substitution benchmark.