What is a mixture of experts (MoE) model?

A mixture of experts (MoE) model splits part of a language model into many smaller sub-networks, called experts, and uses only a few of them for each token. That is why model names now carry two numbers: Qwen3.8-2.4T-A95B has 2.4 trillion parameters in total but activates 95 billion per token, and Meta's Muse Spark Open has 816 billion in total and 42 billion active. The total decides how much memory you need; the active part decides how much computation each token costs. This guide explains how MoE works and what it means when you pick or run a model, based on the original research papers, Hugging Face's explainer and the model cards read on October 23, 2026.

How a mixture of experts works

A standard, or dense, model sends every token through all of its parameters. In an MoE model, the feed-forward layers inside the network are replaced by MoE layers. Each MoE layer holds a set of experts, which in practice are feed-forward networks, plus a small router, also called a gating network, that decides which experts process each token. Other parts of the model, such as attention, are shared and run for every token.

Mixtral 8x7B, described in a January 2024 paper by Mistral, is the clearest example: each layer has 8 experts, and for every token a router picks 2 of them and combines their outputs. The model has 47 billion parameters, but each token uses only 13 billion. It is 47 and not 8 × 7 = 56 billion because only the feed-forward layers are duplicated as experts; the rest is shared.

The idea is older than today's chatbots. A 2017 paper by Noam Shazeer and colleagues introduced the sparsely-gated MoE layer with up to thousands of experts and reported more than 1000× more model capacity with only minor losses in efficiency. The 2021 Switch Transformer simplified routing to a single expert per token and trained models of up to a trillion parameters.

Total parameters versus active parameters

Table: total and active parameters of MoE models (model cards and papers, checked October 23, 2026)

The names often encode the numbers: "8x7B" refers to eight experts per layer on a Mistral 7B-sized architecture, and "A95B" means 95 billion activated.

What MoE means for memory and speed

Hugging Face's explainer sums up the trade-off: an MoE model runs faster than a dense model with the same number of parameters, because each token only computes through a fraction of them, but all the parameters still have to be loaded in memory. Mixtral 8x7B needs the memory of a 47 billion parameter model while computing roughly like a 12 billion parameter one.

For you that means two separate questions. Will it fit? Count total parameters and the precision of the weights, as in our guide to how much VRAM an LLM needs. Will it be fast? Look at the active parameters. OpenAI quantized the MoE weights of gpt-oss to MXFP4 so that gpt-oss-120b runs on a single 80 GB GPU and gpt-oss-20b within 16 GB of memory; our explainer on quantization covers what those formats do. At the other end, Meta says Muse Spark Open runs on two Mac Studios with M5 Ultra; see running an LLM on a Mac for what one Mac can hold.

Running MoE models on a normal PC

Because each token only uses a few experts, a common way to run a large MoE model on a normal PC is to keep the expert weights in system RAM and the rest on the GPU. llama.cpp has options for exactly this: --cpu-moe keeps all MoE expert weights in system RAM, and --n-cpu-moe N keeps the expert weights of the first N layers there, while the rest of the model stays on the GPU. Raise N until the model fits in your VRAM. It is slower than having everything on the GPU, but it can make a model usable that would otherwise not load at all.

Common misunderstandings

• "Each expert is an expert in a topic." Not really. Research summarized by Hugging Face found experts specialize in groups of tokens or shallow concepts, such as punctuation or proper nouns, and in a multilingual model no single expert specialized in a language.
• "A 47B MoE is as cheap to run as a 13B model." Only in computation. It still needs the memory of the full model.
• "MoE models are always better." As Hugging Face summarizes the Switch Transformer results, at the same pre-training quality sparse models did worse than dense ones on reasoning-heavy tasks and disproportionately well on knowledge-heavy ones, and Hugging Face notes MoE models have historically been harder to fine-tune.

If you want the basics first, start with what an LLM is. If you plan to build on one of these models, check its license too: our guide to open-weight model licenses compares them.