AI Insight
Researchers developed an autoregressive transformer model capable of generating synthetic single-cell gene expression data by training on existing cell type datasets. The model uses a causal transformer architecture combined with a learned quantized VAE tokenizer and demonstrates predictable scaling behavior when varying model parameters and training data size. The study establishes the first jointly-fit two-exponent scaling law for single-cell foundation models and identifies compute-optimal training configurations for this biological domain.
Why it matters
This work could accelerate biomedical research by generating realistic synthetic cell data for rare cell types or conditions where data is limited. The established scaling laws provide guidance for efficiently training larger single-cell models, and the approach may enable better prediction of cellular responses to experimental perturbations, potentially reducing the need for costly wet-lab experiments.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: We study a self-supervised generation task for single-cell gene expression vectors: given a set of vectors from a cell type, we aim to generate additional gene expression vectors of that cell type. For this task we characterize both the biological fidelity of the generated gene expression vectors and the scaling behavior of the pretraining loss. The model is a causal transformer paired with a learned quantized VAE tokenizer, trained with a cross-entropy loss. To evaluate the model, we condition it on held-out gene expression vectors of a cell type and generate vectors of gene expression, comparing the resulting distribution over gene expression vectors to the ground truth distribution of that cell type. We study the scaling properties of the proposed architecture by varying the number of trained parameters and the amount of training data. To our knowledge, we find the first jointly-fit two-exponent scaling law and compute-optimal frontier for a single-cell foundation model. Finally, we discuss how this pretrained model could be finetuned for perturbation response prediction.
Source: Scaling an Autoregressive Transformer for Single-Cell Generation