Mixture of Experts

Mixture of Experts replaces the dense FFN in each transformer block with N expert FFNs and a router that activates only k of them per token. The promise: parameter count without proportional FLOPs. DeepSeek-V3 has 671B parameters but activates only 37B per token — it is technically 37B-parameter inference on 671B-parameter modeling capacity. Qwen3-MoE, Mixtral 8x7B, and GPT-4 (reportedly) follow the same pattern. The field converged on MoE not because it is theoretically elegant but because the empirical results on perplexity per FLOP are consistently better than dense alternatives at matched compute. The hard part is not the router. The router is a learned linear layer followed by a softmax over N experts and a top-k selection. The hard part is load balancing: without intervention, the router collapses to preferring a small subset of experts, wasting the rest's parameters and capacity. DeepSeek-V3 (December 2024) solved this with an auxiliary-loss-free bias-correction trick that adjusts router biases based on running expert utilization statistics — a cleaner solution than adding auxiliary losses that fight the main training objective.

MoE layer mechanics

An MoE layer replaces the single dense FFN with N expert FFNs (typically identical architecture, different weights) and a router. For each token x, the router computes scores s = softmax(x W_router) over N experts, selects the top-k indices, and computes the output as a weighted sum of the selected experts: output = sum_{i in top-k} s_i * Expert_i(x). The weights s_i are the router probabilities renormalized over the top-k. Token routing is per-position — different positions in the same sequence can activate different expert subsets.

The FLOP count per token is approximately k/N times that of a dense model with the same total parameter count. For DeepSeek-V3: k=8 of N=256 shared experts plus 1 of 1 fine-grained experts per token, activated parameters ~37B out of 671B total. The memory cost at inference is the full 671B (all expert weights must be accessible), which is the key trade-off: MoE reduces compute but not memory.

Expert capacity is the maximum number of tokens each expert can process per batch. Without a capacity constraint, a popular expert could be assigned O(batch_size) tokens while rare experts get zero — infeasible for parallel computation. Setting capacity_factor=1.25 (expert capacity = 1.25 * batch_size/N) allows mild imbalance while preventing extreme overload.

Load balancing — the hard problem

Expert collapse is the primary MoE training failure. The router, being a learned function, develops preferences — certain experts receive more gradient signal because they are activated more often, improving faster, reinforcing the router's preference. Without intervention, training converges to a model where a few experts handle most tokens and the rest are undertrained. The auxiliary load-balancing loss (Switch Transformer, Fedus et al., arXiv:2101.03961, January 2021) penalizes imbalance: L_aux = alpha * sum_i f_i * P_i, where f_i is the fraction of tokens routed to expert i and P_i is the mean router probability for expert i. Setting alpha=0.01 typically prevents collapse.

The problem with auxiliary loss is that it is an additional objective that can conflict with the main language modeling loss — particularly during fine-tuning on specialized data where natural token-to-expert affinity is domain-specific. DeepSeek-V3 (arXiv:2412.19437, December 2024) instead uses a bias-correction approach: track a running average of expert token counts across recent batches, and add a small bias to the router logits for underutilized experts and subtract from overutilized ones. No auxiliary loss required. The bias corrections are updated every ~1000 steps based on utilization statistics.

Expert parallelism and llm-d

Expert parallelism (EP) shards expert weights across GPUs — each GPU holds a subset of experts and handles only the tokens routed to those experts. This is the natural distribution strategy for MoE: expert weights are independent, so there is no inter-GPU communication during FFN computation. The all-to-all communication between routing and expert computation is the bottleneck — tokens must be dispatched to the correct GPU, processed, and results gathered back. At 256 experts across 32 GPUs, each step involves two all-to-all collective operations.

llm-d (Red Hat / IBM, introduced November 2025) treats expert parallelism as a first-class concept in distributed inference, allowing disaggregated prefill and decode pods to each have different expert-shard assignments optimized for their respective compute patterns. For MoE models in production, this reduces inter-pod communication overhead by localizing expert computation to the pod where the tokens are processed.

For VLA training with MoE-based VLMs: if you post-train the Eagle 2.5 backbone in GR00T N1.5 on humanoid demonstration data, expert parallelism determines how you distribute the fine-tuning across multiple GPUs. At the typical fine-tuning scale (30-200 demonstrations, 4-8 GPUs), data parallelism is sufficient — but knowing EP exists prepares you for scaling to larger demonstration datasets.

MoE in the VLA landscape

GR00T N1.5's VLM backbone (Eagle 2.5) is currently dense, but the trend in frontier VLMs is toward MoE: Qwen2.5-VL uses a dense visual encoder with an MoE text backbone. If you are evaluating whether to fine-tune GR00T N1.5 or build a custom VLA for the JHU humanoid capstone, understanding MoE capacity matters for projecting the memory footprint of alternative models.

For DealLens, MoE is directly relevant to model selection: Qwen3-MoE-30B-A3B (30B total, 3B active) is a viable scoring backbone that matches dense 7B models in inference cost while having 4× the modeling capacity for rare deal term patterns. The key operational consideration is memory: MoE models require all expert weights in memory even when only k are active per token.