Dense Transformer Architecture Recap: Every Parameter Active for Every Token

Before contrasting Mixture of Experts with the standard Transformer, it's worth being precise about what 'dense' actually means in this context, since that's exactly the property MoE changes. In a standard Transformer decoder block, the feed-forward network (FFN) sub-layer is a single, fixed set of weights — and critically, EVERY token that passes through that layer goes through that same, complete FFN. There's no selection, no routing, no subset of the network being skipped: 100% of the FFN's parameters are active for 100% of the tokens, every single time.

This is exactly why dense Transformers are called 'dense' in this comparison — parameter count and active-compute-per-token scale together, in lockstep. If you want a more capable model, the standard lever is simply making the FFN (and the rest of the architecture) bigger, which directly increases both the total parameter count AND the compute required to process every single token, since every added parameter is used for every token regardless.

This coupling is the specific scaling problem MoE is designed to solve. This course poses the underlying question directly: how do we scale models further without making them impossibly large and expensive to run? For a dense architecture, the honest answer is 'you can't decouple size from cost' — a 10x bigger dense model genuinely costs roughly 10x more compute per token, because every one of those extra parameters gets used on every single token.

Keeping this dense baseline clearly in mind is what makes MoE's actual innovation (the next subtopic) land clearly: MoE's entire value proposition is breaking exactly this coupling — growing total parameter count (and thus model capacity) WITHOUT a proportional growth in the compute spent per token, which a dense architecture structurally cannot do.

import torch
import torch.nn as nn

class DenseFeedForward(nn.Module):
    """A standard Transformer FFN — every token that reaches this
    layer passes through ALL of these weights, no selection at all."""
    def __init__(self, d_model: int, d_ff: int):
        super().__init__()
        self.up = nn.Linear(d_model, d_ff)
        self.down = nn.Linear(d_ff, d_model)
        self.activation = nn.GELU()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # EVERY token in x goes through this exact same computation —
        # 100% of this layer's parameters are active for 100% of tokens
        return self.down(self.activation(self.up(x)))

d_model, d_ff = 4096, 16384
ffn = DenseFeedForward(d_model, d_ff)
total_params = sum(p.numel() for p in ffn.parameters())
print(f"Total FFN params: {total_params:,}")
print("Active params per token: ALL of them — this is what 'dense' means")