Single-cell foundation models earn their value through task-level baselines

1. “Foundation” is not a downstream task

The leading models embody different representations and intended uses. Geneformer ranks expressed genes and emphasizes transfer into network-biology tasks; scGPT uses generative pretraining across annotation, integration, multi-omics and perturbation applications; scFoundation learns cell and gene-context representations with an asymmetric architecture; UCE incorporates protein-language-model features to place cells from different tissues and species in a shared space without fine-tuning.[1–4] A parameter count cannot adjudicate among these designs. Annotation, batch integration, trajectory conservation, cross-species mapping, drug response and genetic perturbation prediction have different targets and failure modes, so each requires its own benchmark.

2. Separate zero-shot use, linear probing and fine-tuning

Zero-shot evaluation freezes the pretrained model and tests its representation directly. A linear probe adds a small supervised readout and asks whether task information is easily extractable. Fine-tuning changes model weights using task labels and tests adaptability rather than an intrinsically useful embedding. These regimes should never share one headline score. In five datasets, Kedzierska and colleagues found that zero-shot Geneformer and scGPT did not consistently outperform HVG, Harmony or scVI; several evaluation datasets also overlapped model pretraining sources.[6] That result does not invalidate reported fine-tuning gains. It shows that adaptation and general-purpose zero-shot transfer are distinct claims.

3. Baselines identify what pretraining contributed

A useful evaluation has three floors. The first comprises inexpensive representations and learners—log-normalized HVGs, PCA, k-nearest neighbours and logistic regression. The second comprises established task methods, such as Harmony and scVI/scANVI for integration. The third is a randomly initialized or from-scratch network with the same decoder, labels and tuning budget as the foundation model. Integration metrics must also keep batch removal separate from biological conservation: a representation can mix batches by erasing rare populations, states or trajectories.[5] Broader 2025–2026 evaluations similarly found no model that dominated every task, while PCA and specialized methods remained preferable in some settings.[9–11]

4. The split should reproduce the next deployment

A random cell split places near-replicate cells from the same donor, sample, batch or clone on both sides of the test boundary. It primarily measures interpolation. New-patient annotation calls for donor- or sample-level holdout; transfer across studies, platforms, tissues or species calls for entire-domain holdout. Novel-perturbation prediction requires splitting by perturbation identity and reporting separately whether zero, one or both components of a combination were observed in training. Feature selection, scaling, batch parameters, thresholds and hyperparameter selection belong inside the training/validation loop. The locked test set should be evaluated once, with uncertainty across group splits and explicit reporting of failures.

5. Public pretraining corpora complicate “unseen” claims

CELLxGENE, GEO and major atlases are reused across models, and dataset titles alone cannot exclude overlap at the study, donor or processed-version level. The 2025 zero-shot study identified intersections involving Pancreas, Tabula Sapiens and Immune evaluation data.[6] Model releases should therefore publish accession-level pretraining manifests, collection cutoffs, deduplication rules and checkpoint versions. Post-cutoff datasets are especially valuable. A 2026 preprint that deliberately included post-cutoff evaluations still found classical PCA competitive on several tasks; because that work remains unreviewed, its estimates are provisional, but its design principle is immediately useful.[11]

6. Perturbation prediction is a decisive stress test

Transcriptome-wide correlation can be high because most genes barely change or because every perturbation shares a systematic shift away from control. Ahlmann-Eltze and colleagues compared five foundation models and two other deep models with deliberately simple controls: an additive predictor was better for double perturbations, and deep models did not consistently beat mean or linear prediction for unseen single perturbations.[7] Across ten datasets, Systema likewise found a perturbed-mean baseline competitive or superior for unseen interventions, indicating that methods can reproduce a generic “perturbed versus control” axis without learning perturbation-specific biology.[8] Evaluation should score change relative to matched controls and examine direction, magnitude, rank, pathway coherence and per-perturbation performance, with no-change, mean, additive, matched-mean and negative controls.

7. A representation is evidence only about the model

Neighbourhood in an embedding means that a particular model and distance function regard two profiles as similar. A cluster, bridge or branch in a two-dimensional UMAP does not establish a new cell type, developmental lineage or transition. Attention, gene embeddings and in silico perturbations likewise do not establish regulatory edges, causal mechanisms or therapeutic targets. Their productive role is retrieval, provisional annotation, stratification and experimental prioritization. Claims should then be reproduced in independent samples and tested with marker expression, protein or chromatin measurements, time-resolved or lineage evidence, real genetic or pharmacological perturbation and functional assays. The UCE paper itself frames the model as a tool for analysis, annotation and hypothesis generation, while acknowledging opacity, training bias and a continuing role for conventional batch correction.[4]

Appropriate uses

When labels are scarce, a reference atlas is large and the query resembles the pretraining domain, a foundation model can provide reusable features, reference mapping or cell retrieval. Biologically informed tokenization may also help with heterogeneous gene sets and cross-species inputs, as illustrated by UCE.[4] Transfer learning is justified when an external task-level benchmark shows a stable advantage at the same label budget. In discovery workflows, model outputs can rank cell states, gene modules or perturbations for follow-up; a ranked candidate is not a validated finding.

Common misconceptions

• Misconception: “More pretraining cells mean deeper biological understanding.” · More defensible interpretation: Corpus scale also scales tissue, species and platform biases; measure post-cutoff external-task gains.
• Misconception: “A clean UMAP proves a better embedding.” · More defensible interpretation: Two-dimensional appearance depends on reduction settings; score the full representation for both biology and batch.
• Misconception: “Fine-tuning wins, so zero-shot use is validated.” · More defensible interpretation: Fine-tuning consumes target-domain labels; frozen, probed and from-scratch conditions answer different questions.
• Misconception: “High correlation proves perturbation mechanism prediction.” · More defensible interpretation: Unchanged genes and generic perturbation effects can dominate correlation; test specific deltas against mean/additive baselines.
• Misconception: “A predicted target has therapeutic significance.” · More defensible interpretation: It is a computational candidate; causality, druggability, safety and efficacy require independent evidence.

Conclusion

The meaningful threshold for a single-cell foundation model is not scale but net task value: grouped holdout, contamination-aware external data, strong simple baselines, equal tuning budgets, multiple metrics, uncertainty and experimental follow-up. If HVG/PCA, scVI or a mean/linear predictor performs as well, the model's additional cost has not been justified for that use. If the pretrained model repeatedly wins under low-label, out-of-study or cross-species evaluation, it has earned a narrower and more useful claim: not “a universal understanding of cells,” but a valuable tool for that deployment.