Description: DESCRIPTION META TAG
Keywords: KEYWORDS SHOULD BE PLACED HERE

# 🦞 LBSTER 🦞

Frey Lab*,
Prescient Design
Genentech
Concept Bottleneck Paper Code Cramming Paper

## Abstract

Protein language models (pLMs) are ubiquitous across biological machine learning research, but state-of-the-art models like ESM2 take hundreds of thousands of GPU hours to pre-train on the vast protein universe. Resource requirements for scaling up pLMs prevent fundamental investigations into how optimal modeling choices might differ from those used in natural language. Here, we define a “cramming” challenge for pLMs and train performant models in 24 hours on a single GPU. By re-examining many aspects of pLM training, we are able to train a 67 million parameter model in a single day that achieves comparable performance on downstream protein fitness landscape inference tasks to ESM-3B, a model trained for over 15,000× more GPU hours than ours. We open source our library for training and inference, LBSTER: Language models for Biological Sequence Transformation and Evolutionary Representation.

## Protein Language model cramming setup.

## LBSTER: Language models for Biological Sequence Transformation and Evolutionary Representation

## Poster

## BibTeX

    @article{Frey2024.05.14.594108,
            author = {Frey, Nathan C. and Joren, Taylor and Ismail, Aya Abdelsalam and Goodman, Allen and Bonneau, Richard and Cho, Kyunghyun and Gligorijevi{\'c}, Vladimir},
            title = {Cramming Protein Language Model Training in 24 GPU Hours},
            elocation-id = {2024.05.14.594108},
            year = {2024},
            doi = {10.1101/2024.05.14.594108},
            publisher = {Cold Spring Harbor Laboratory},
            URL = {https://www.biorxiv.org/content/early/2024/05/15/2024.05.14.594108},
            eprint = {https://www.biorxiv.org/content/early/2024/05/15/2024.05.14.594108.full.pdf},
            journal = {bioRxiv}
        }