Jeff Dean Explains Why Mixture-of-Experts Underpins Nearly Every Frontier AI Model

Jeff Dean said his team recognized early that the sparsely-gated mixture-of-experts (MoE) paper showed roughly 10x better training compute-to-quality than the dense models widely used at the time, saying, "when you see 10x improvements, it's probably going to be an important idea."
MoE activates only the useful sub-network per token instead of running the whole model every time. Dean compared it on stage to how the part of the brain thinking about Shakespearean sonnets isn't the part that fires when a garbage truck backs in, so there's no reason to run both.
Dawn Song, moderating at the Asian American Scholar Forum, backed the point, saying the architecture "has underpinned now almost all frontier models."