The Allure and Limits of “H&E → Transcriptome” Models

Why Tissue Imaging Must Move Beyond Predicting What’s Already Known

Recent years have seen a surge of papers claiming that deep learning on routine H&E-stained slides can reliably predict underlying gene expression or even immune phenotypes. Models like HE2RNA or HistoTME train on paired histology and transcriptomics to infer dozens of immune and tumor markers from morphology alone. These approaches are impressive at rediscovering known expression patterns (e.g. virtual CD3/CD20 maps). But this trend has a catch: it repackages existing transcriptomic information rather than revealing new biology. H&E images are used to regress gene-signature levels, yet tissue morphology harbors far richer signals, cell contacts and shapes, textures and spatial organization, that go beyond bulk RNA profiles. In practice, even the best “H&E→gene” models achieve only modest correlations (e.g. ~0.5 Pearson on held-out data), highlighting that much of the tumor microenvironment remains hidden when we limit our analysis to gene-expression alone.

Missing the Biological Big P…