Mixture of Experts

Defining and Describing Mixture of Experts

• Mixture of Experts lets one model behave like a team of specialists while computing with only the specialists needed for each input.
• A Mixture of Experts (MoE) is a machine-learning architecture composed of multiple expert networks and a gating or routing mechanism that determines how much each expert contributes to an output. ^{[jnedr9] [ui9kpg]} In the original formulation, experts specialized in different regions of the input space, while a gate assigned input-dependent weights. ^{[c4tcev] [jnedr9]}
• Modern sparse MoE systems route each token to only a small subset of experts rather than evaluating every expert, allowing a model to contain substantially more total parameters without proportional per-token computation. ^{[3h91x9] [9qok5b]}
• The architecture matters because it combines specialization with scalability: different experts can learn different linguistic, visual, or task-specific patterns while the router controls computational allocation. ^{[3h91x9] [ui9kpg]}

Uses in Context

• Large language models: MoE describes language models in which a router activates only selected feed-forward experts for each token, reducing computation relative to a dense model with the same total parameter count. ^{[3h91x9] [9qok5b]}
• Specialization: The term is invoked when separate subnetworks learn different regions of an input distribution or different subtasks. ^{[c4tcev] [jnedr9]}
• Efficient scaling: Researchers use MoE to increase parameter capacity while keeping active computation comparatively constrained. ^{[3h91x9] [9qok5b]}
• Multimodal and heterogeneous data: MoE systems can allocate different data types or domains to suitable experts, including language, vision, healthcare, and autonomous-system workloads. ^[3h91x9]
• Model deployment: The architecture is associated with lower computational overhead and potentially lower inference costs because only a subset of experts is activated for each input. ^[3h91x9]

History of Use

Origins

• The concept was introduced by Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton in the 1991 paper “Adaptive Mixtures of Local Experts,” published in Neural Computation . ^{[c4tcev] [jnedr9]}
• The original system divided a vowel-discrimination task into subtasks that could be handled by simple expert networks; a gating network learned how to distribute responsibility among them. ^[c4tcev]
• Unlike a single monolithic predictor, the framework used multiple networks whose specialization emerged through competition for training examples. ^{[jnedr9] [i14nba]}

Evolution

• 1991 — Adaptive mixtures of local experts: The foundational architecture paired several expert networks with a learned gate that produced a probability distribution over experts. ^{[c4tcev] [jnedr9]}
• 2017 — Sparsely gated MoE: Shazeer and collaborators extended MoE to very large neural networks by activating only a small number of experts per input, establishing the sparse-gating approach used by later language models. ^[i7h1wq]
• 2021–2024 — Token-level routing: Switch Transformers used one selected expert per token, while Mixtral used top-two routing among eight experts per layer; Mixtral was described as having 47 billion total parameters with approximately 13 billion active per token. ^[i7h1wq]

Best Real-World Examples

• Adaptive Mixtures of Local Experts — The 1991 research system that established expert specialization and learned gating. ^[c4tcev]
• Sparsely-Gated Mixture-of-Experts Layer — The 2017 architecture that demonstrated sparse expert activation for scaling neural networks. ^[i7h1wq]
• Switch Transformer — A sparse-routing design that selected one expert per token to simplify and reduce routing computation. ^[i7h1wq]
• Mixtral 8x7B — An open model using eight experts per layer and selecting two experts for each token. ^[i7h1wq]
• Hummingbird — An open-source runtime project aimed at deploying large MoE language models with lightweight infrastructure. ^[3vbev1]
• MoE vision systems — Vision applications use specialized experts for functions such as detection, segmentation, and classification. ^[3h91x9]

Case Studies

Sources