The Router: How the Model Decides Which Experts to Activate
~12 min read
A router — trained alongside the rest of the network — acts like a multi-class classifier producing softmax scores over experts, selecting the top-K highest-scoring experts for each token.
The Router: How the Model Decides Which Experts to Activate is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •The router acts like a multi-class classifier, producing softmax scores over all available experts for each token
- •The top-K highest-scoring experts are selected and actually run for that token
- •The router is trained alongside the rest of the network, not hand-designed or hardcoded
- •Router behavior emerges from training — the model learns which experts to route which tokens to, without explicit engineering
- •Because only selected experts get gradient updates for a given token, router and expert learning dynamics are tightly coupled — the source of the next subtopic's training challenges