Mixture of Experts: Sparse Activation, Smaller "Expert" Feed-Forward Networks
~15 min read
MoE replaces the single dense FFN with many smaller "expert" FFNs, activating only a subset per token — keeping overall parameter count large while making inference faster.
Mixture of Experts: Sparse Activation, Smaller "Expert" Feed-Forward Networks is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •MoE replaces one dense FFN with multiple smaller 'expert' FFNs at each decoder position
- •Only a subset of experts is selected and run per token during inference — not all of them
- •This decouples total parameter count (capacity) from per-token compute cost, which dense architectures can't do
- •Different layers select different experts, and different tokens select different experts — the routing compounds across the network
- •Mixtral 8x7B by MistralAI is a real, production-scale LLM built on exactly this MoE architecture