Mixture of experts, explained simply

A mixture-of-experts model, or MoE, contains several specialist neural-network blocks and a router that chooses which ones handle each token. Instead of activating every parameter for every step, it uses only a selected subset. This lets a model have a large total capacity without paying the full arithmetic cost of a dense model of the same total size on every token.

MoE does not mean “many complete AIs debating an answer”, and it does not make computation free. The experts must still be stored in memory, routing takes work, and moving tokens between chips can add communication cost. Active parameters, hardware layout, utilisation and output length matter more than the headline total alone.

Think of a workshop with shared intake

Imagine a workshop with several specialist benches. Every item enters through the same front desk. A dispatcher examines it and sends it to one or two benches judged most useful. The whole workshop owns many tools, but only a small set is used for that item.

In a transformer MoE, the shared parts of the network still process each token. At certain layers, a router scores the available experts and selects a few. Those experts are usually feed-forward blocks rather than complete models with human-readable professions. One expert is not reliably “the poetry expert” and another “the maths expert”, even if specialisation emerges during training.

The selected outputs are combined and processing continues. The decision happens repeatedly, often for each token and at several layers, so adjacent words can take different routes.

The router’s choices are numerical rather than a conscious hand-off. Experts learn through training, and their roles may overlap or be difficult for people to interpret cleanly.

Dense and sparse models spend capacity differently

A dense language model uses broadly the same parameter set for every token. Increasing its parameter count generally increases the computation and memory traffic required at each generation step. A sparse MoE can increase total parameters while keeping the active subset relatively stable.

The Switch Transformer paper simplified this idea by routing each token to one expert at its MoE layers. Its experiments reported faster pre-training than dense comparison models under the authors’ conditions. Those results demonstrate the architecture’s potential; they are not a universal energy ratio for current chat products.

This distinction explains why total parameter count can mislead. An MoE advertised as very large may perform arithmetic on far fewer parameters per token. But the inactive experts still occupy accelerator memory and may need to be spread across machines.

Active parameter figures need context as well. Shared attention and embedding layers still run, so the selected expert weights are not necessarily the model’s entire active workload.

The router has to keep experts busy

If the router sends most tokens to the same expert, that expert becomes a queue while others sit idle. Training therefore uses balancing mechanisms that encourage traffic to spread across experts. Systems also set capacity limits for how many tokens an expert accepts in a batch.

Routing can be unstable early in training. Experts may learn similar functions or some may receive too little data. Engineers tune auxiliary losses, capacity, noise and placement to improve balance without harming model quality.

Serving adds a physical problem: selected experts may live on different accelerators. Tokens then travel across a high-speed network between shared layers and expert blocks. Communication can limit speed and increase energy, especially with small batches or poor placement. Sparse arithmetic is only one part of the machine.

Capacity limits create another compromise. Dropping, delaying or rerouting excess tokens may protect throughput, but the implementation must preserve answer quality under uneven real traffic.

Why MoE can be efficient

MoE offers more representational capacity without activating the entire network each time. For a target quality, that can reduce training or inference computation relative to a similarly capable dense model. It can also allow different experts to learn useful patterns from varied languages or domains.

Efficiency depends on implementation. Large batches can keep experts occupied and make communication worthwhile. Quantisation can lower memory traffic. Good routing and hardware topology reduce congestion. A model served at low utilisation may fail to realise these gains.

Most importantly, lower compute per token does not guarantee lower energy per completed task. Longer outputs, retries, large context windows and agent tools can outweigh architectural savings. How to reduce your AI carbon footprint focuses on the whole job rather than one attractive model property.

Operational measurements are therefore preferable to a paper estimate when available. Power, latency and success should be compared on the same hardware, batch conditions and representative requests.

Why MoE is not the same as a small model

A small language model has fewer total parameters and can often fit on modest hardware. An MoE may contain a very large collection of experts but activate only a few. Its arithmetic for one token can look smaller than its total size, yet its storage and deployment can remain substantial.

This matters for environmental estimates. A calculation based only on total parameters may overstate active computation. One based only on active parameters may omit memory, networking, idle capacity and supporting infrastructure. Neither number alone is a facility power measurement.

When comparing models, ask for total parameters, active parameters per token, numerical precision, typical context and output length, and the hardware used. Then test quality and energy for the same complete task.

For local deployment, total memory may be the binding constraint even when active computation looks manageable. A dense compact model can be the more practical option on limited hardware.

What MoE changes for users

Users usually cannot control routing inside a hosted model. They can control the request around it: provide relevant context, ask for proportionate output and avoid unnecessary regenerations. Model providers control expert placement, batching, capacity and hardware efficiency.

An MoE model may be a good choice for a broad service because it combines capacity with sparse activation. It may still be excessive for a simple extraction that a compact dense model can handle. Green AI means choosing between architectures based on delivered value and measured resources, not assuming sparse always wins.

Treechat’s receipt estimates operational energy using the selected model and token counts, then adds stated data-centre overhead. Because providers do not expose live expert routing or per-response power, it remains a model-based approximation. The assumptions and exclusions are published at /methodology.

The simple summary is that mixture of experts separates how much a model knows from how much of it works at once. That can make large capacity more computationally efficient, but memory, communication and real workload behaviour determine whether the promise becomes an actual saving.