scGPT · PBMC3k Benchmark

When Does Pretraining Help?

A systematic benchmark of scGPT against 6 competing methods on PBMC3k — 7 methods, 7 metrics, identical splits. On this small, well-separated dataset, a 3-layer MLP outperforms a 51M-parameter foundation model in Macro F1. All results are open and reproducible.

Benchmark Leaderboard

All 7 methods cluster within a 4 pp accuracy band — the gap is smaller than expected.

Primary metric is Macro F1 (handles class imbalance). scGPT pretrained ranks 6th in F1 but achieves 90.7% accuracy with zero task-specific architecture design. The honest finding: on small, clean datasets, pretraining advantage is minimal.

Accuracy & Macro F1 by Method

PBMC3k test set (n = 396), stratified 70/15/15 split, seed = 42. Axis starts at 0.85.

Benchmark Setup

Identical protocol across all methods — no cherry-picking.

Project Timeline

From a single fine-tune job to a publication-ready benchmark.

Repository Structure

Structured as a benchmark release, not a transient experiment folder.

scGPT Per-Class Results

Strong on B cells and monocytes. CD8 T cells remain the hardest class.

Visual Artifacts

Publication-quality figures from the benchmark paper.

Benchmark leaderboard bar chart
Fig 1 — Leaderboard: Accuracy and Macro F1 for all 7 methods on PBMC3k test set.
scGPT per-class precision recall F1
Fig 2 — scGPT per-class precision, recall, and F1. CD8 T cells (F1=0.703) are the hardest class.
Multi-metric radar chart
Fig 3 — Radar chart comparing all 7 methods across 7 evaluation metrics simultaneously.
All metrics grouped bar chart
Fig 4 — All 7 metrics across all 7 methods. Deep MLP and Seurat KNN lead consistently.
Key Takeaway

Benchmark lightweight alternatives before deploying a 51M-parameter foundation model.

On PBMC3k, a Deep MLP trained in 14 seconds (Macro F1 = 0.948) outperforms scGPT pretrained (Macro F1 = 0.897). The gap is only 5.1 pp — but the compute difference is orders of magnitude. scGPT's value lies in large, complex, multi-dataset scenarios. This benchmark makes that boundary explicit.