Mixture of Experts (MoE)

Mixture of Experts (MoE) is a neural network architecture that scales model capacity without proportionally increasing computational cost by dividing the model into multiple specialized sub-networks (experts) and using a learned routing mechanism (gating network) to activate only a subset of experts for each input. This selective activation enables models with very large total parameter counts while keeping the per-input computation manageable.

In modern transformer-based MoE architectures, the expert mechanism is typically applied to the feed-forward network (FFN) layers. Each MoE layer contains multiple parallel FFN experts, and for each input token, the gating network selects the top-k experts (commonly k equals 1 or 2) to process that token. The outputs from the selected experts are weighted by the gating scores and combined. This means that while the model may have hundreds of billions of total parameters, each token is processed by only a fraction of them, keeping inference cost similar to a much smaller dense model.

Prominent MoE models include Mixtral 8x7B (which achieves performance comparable to much larger dense models at a fraction of inference cost), Switch Transformer, and GPT-4 (which is widely understood to use an MoE architecture). These models demonstrate that MoE can deliver superior capability-per-FLOP ratios, making them attractive for both training efficiency and inference cost optimization.

MoE architectures present unique deployment challenges. The total parameter count determines memory requirements even though only a subset is active per token, requiring sufficient GPU memory to store all experts. Load balancing across experts during training requires auxiliary losses to prevent expert collapse (where the router sends all tokens to a few experts). Distributed serving must handle the routing and expert selection across GPUs efficiently. Despite these challenges, MoE has become a dominant scaling strategy for frontier models due to its favorable trade-off between capability and computation.