How Transformers Finally Ate Vision – Isaac Robinson, Roboflow
May 8, 2026 · 17:05
Isaac Robinson, research lead at Roboflow, explains why transformers ultimately beat convolutional neural networks for vision, arguing that massive ViT-specific pretraining and borrowed infrastructure from LLMs overcame the transformer's lack of inductive bias. He traces the evolution from ViT and Swin (windowed attention reducing complexity to n²) through ConvNeXt (reintroducing convolution with transformer-style blocks) to Hiera (stripping biases and recovering them via MAE pretraining), and back to the simple, scalable ViT. Robinson highlights that pretraining methods like MAE and DINOv3 learn the inductive biases CNNs have built-in, while tools like FlashAttention from the LLM world nullify ViT's n⁴ compute scaling penalty. In practice, this pattern appears in the SAM model series: SAM used a ViT backbone, SAM2 switched to Hiera with MAE, and SAM3 returned to the simple ViT. However, these massive models lack deployment flexibility; Roboflow's RF-DETR uses neural architecture search on a foundation model backbone to generate a family of high-performance models, achieving up to 40× speedup over fine-tuning SAM3 while outperforming real-time convolutional detectors.
