Dense Transformer Architecture Recap: Every Parameter Active for Every Token
~10 min read
In a standard (dense) Transformer, every token passes through the SAME feed-forward network at every layer — all parameters are active for every single token, which is exactly the property MoE changes.
Dense Transformer Architecture Recap: Every Parameter Active for Every Token is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •In a dense Transformer, every token passes through the SAME complete feed-forward network at every layer
- •100% of the FFN's parameters are active for 100% of tokens, every single time — no selection or routing involved
- •This means parameter count and compute-per-token scale together in lockstep — a bigger dense model costs proportionally more compute per token
- •This coupling is exactly the scaling problem: you can't grow capacity without growing per-token cost proportionally in a dense architecture
- •MoE's entire value proposition is breaking exactly this coupling — the focus of the next subtopic