What a mixture of experts model is, and what it costs to run

A mixture of experts model is a language model whose feed forward layers are cut into many small blocks called experts, with a small router in front of them that picks a few for every token. The Qwen3 Coder 30B model carries 30.5B parameters in total and runs only 3.3B of them for each token, because its router selects 8 experts out of 128 at every layer, and both figures come from the model card and the config.json file Qwen publishes on Hugging Face.

The consequence matters more than the definition. Total parameters decide how much memory the model needs and how large the download is, while active parameters decide how much arithmetic runs for each token, which is what you feel as speed. On the models below those 2 numbers differ by as much as a factor of 23, so the headline size on its own tells you very little about how the thing will behave on your machine.

What is a mixture of experts model?

A mixture of experts model is a transformer whose usual feed forward block is replaced by a set of smaller blocks plus a router that decides which of them see each token. Mixtral 8x7B, released by Mistral in December 2023, carries 46.7B parameters in total and uses 12.9B of them per token, and Mistral's own announcement of 11 December 2023 describes it as 8 distinct groups of parameters with a router network that chooses 2 of them.

The name misleads almost everyone on first contact. An expert is not a specialist in law or in biology, it is a slab of weights sitting inside one layer, and there are 32 or 128 or 256 of them stacked where an ordinary model keeps a single large block. The Qwen3 Coder 30B model holds 128 of them in each layer that uses the design and it has 48 layers, so the file you download is carrying thousands of these slabs, and no human ever decided what any of them would hold.

Hugging Face's own explainer on the design says the same thing from the research side. In the routing experiments it describes, the encoder experts latched onto groups of tokens rather than onto topics, one of them handling punctuation and another proper nouns, and in a multilingual run the authors expected one expert per language and got nothing of the sort, because routing and load balancing spread every language across all of them.

The idea is also much older than the current wave of models. IBM's summary of the field traces it to the 1991 paper Adaptive Mixture of Local Experts, which trained separate networks on different subsets of the training cases with a gating network choosing between them, and it reports that the experimental model reached its target accuracy in half as many training epochs as the conventional one. What changed after 2023 is the scale and the hardware, not the concept itself.

How does the router pick which experts run?

The router is one small matrix multiply followed by a softmax and a selection of the highest scores, and it runs once per token inside every layer. In the Qwen3 mixture of experts code that ships inside Hugging Face transformers, the entire decision is 3 lines long, and you can read all 3 without knowing any Python.

router_logits = F.linear(hidden_states, self.weight)
router_probs = torch.nn.functional.softmax(router_logits, dtype=torch.float, dim=-1)
router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1)

Read it as 3 plain steps. The first line scores every expert against the token that just arrived, the second turns those scores into probabilities that add up to 1, and the third keeps the highest 8 and ignores the other 120 for that token. The value called self.top_k is filled from num_experts_per_tok in the model's config file, which is how a single line of JSON decides how much arithmetic your machine does for every token it reads.

None of the choosing is stable, and none of it is readable by a human. The same expert can be picked for a comma in one sentence and for a variable name in the next, the scores move as the model reads more context, and the whole selection happens again at every one of the 32 or 48 or 61 layers, so a single token can pass through hundreds of different experts on its way up the stack.

Some models also keep one expert switched on permanently. DeepSeek V3's config file sets n_shared_experts to 1 alongside its 256 routed experts, so every token passes through that one block whatever the router decides and the 8 routed experts sit on top of it. The reasoning is that a fraction of what a model does is common to every token, spelling and sentence shape, and forcing 256 specialists to relearn all of it wastes their capacity. Qwen and OpenAI took the other route in the models above and kept every expert routed, both shapes ship today, and the config file is the only place where you can tell them apart.

Training that router is harder than running it. Left to itself it collapses onto a few favourites and the remaining experts never learn anything useful, so the labs add a load balancing loss that punishes lopsided routing, and Hugging Face's explainer gives 1.25 as a good starting point for the capacity factor, which is the headroom each expert gets before the tokens beyond it overflow and skip the experts altogether. When somebody says a mixture of experts model was difficult to train, this is usually what they mean.

Why do total and active parameters cost you different things?

Total parameters set your memory bill and active parameters set your speed bill, because every weight has to sit in memory in case the router asks for it while only a fraction of them run for any given token. You pay storage for the whole model and arithmetic for a slice of it, and shrinking one of those bills does nothing at all for the other.

Qwen ships 2 models that make the split visible without any theory at all. Qwen3-32B is dense, 32.8B parameters, every one of them used on every token, and its config file has no experts field anywhere in it. The Qwen3 Coder 30B model is a mixture of experts model, 30.5B parameters, 3.3B used per token. On Ollama the dense one pulls 20 GB and the sparse one pulls 19 GB, so they ask your machine for almost exactly the same memory, and the sparse one does roughly a tenth of the arithmetic to produce each token.

You pay the memory bill of a 30B model and get the speed bill of a 3B one, which is why these models feel quick on hardware that ought to be struggling with them. It also explains the naming convention you see on Hugging Face, where the A in A3B stands for active, so a name carrying both 30B and A3B is quietly telling you both numbers at once, and a 480B model with A35B in its name wants the memory of the first figure while running the second.

Quantization moves both bills at the same time. OpenAI ships gpt-oss-120b with the expert weights in the 4 bit MXFP4 format and states on the model card that this is what lets a 117B parameter model fit into a single 80 GB accelerator such as an H100 or an MI300X, while the 20B version fits within 16 GB of memory by the same route. If you are choosing something to run at home, the download size on the Ollama library page is a better guide than the parameter count printed in the announcement.

Which mixture of experts models can you download today?

Every model in the table below is open weight, published on Hugging Face, and each number here was read from that model's own card or config.json file rather than from a press summary. The column to watch is the gap between the total and the active count, because that gap is the discount the design actually delivers.

Read the table across and the direction of travel shows itself. Mixtral cut itself into 8 experts and picked 2, which is coarse by current standards, while the newer models cut themselves into 128 or 256 pieces and pick 4 or 8, so the fraction that runs for each token keeps falling as the total keeps rising. DeepSeek V3 is the extreme case in this group, 671B parameters sitting in the files and 37B on the clock, which is roughly 5.5 percent of itself per token.

The licences are not uniform either, and that matters more than the architecture if you plan to ship something built on one of these. OpenAI released both gpt-oss models under Apache 2.0 and Qwen used Apache 2.0 for its 30B coder model, while DeepSeek publishes its code under MIT and its weights under a model agreement of its own rather than a standard open source licence. Same design, different rules about what you may do with a derivative, so the file to read before the config file is the one named LICENSE.

None of that is free once you try it at home. DeepSeek V3 needs hundreds of gigabytes of memory before it answers anything at all, which describes a server rather than a laptop, and the 480B version of the Qwen coder model pulls 290 GB on Ollama against 19 GB for the 30B version of the same architecture. The models people genuinely run on their own machines sit at the top of that table, and the licences differ enough that our running notes on open weight releases are worth a look before you commit to one.

Why do the big labs keep choosing this design?

Because it buys capacity at a discount, both while training and while answering. A dense model that wants to know more has to run more arithmetic on every single token it sees, while a mixture of experts model adds experts instead, which grows the memory it occupies and leaves the compute per token roughly where it was.

The clearest public evidence is the hardware line on OpenAI's own model card. It describes gpt-oss-120b as a 117B parameter model that fits into a single 80 GB accelerator, and only 5.1B of those parameters run for each token, so a dense model of the same total size would be pushing about 23 times more parameters through the arithmetic on every token it produced. Serving cost tracks that arithmetic far more closely than it tracks the size of the file on disk.

The training side has the same shape. Hugging Face's explainer reports that Switch Transformers reached a 4x pre training speedup over the dense T5 XXL baseline, and the 1991 paper that started all of this had already measured its model hitting the target accuracy in half the epochs. Cheaper training runs and cheaper serving are the 2 numbers every lab is trying to move, so the design spread quickly once it was shown to work at scale.

The money follows directly from there. Inference is the recurring cost of running a model business, the one that never goes away and grows with every customer, so a design that cuts the arithmetic per token by an order of magnitude cuts the bill that hurts most. That is a larger deal for the companies selling tokens than for the person downloading weights, and it is one reason the open releases and the paid APIs converged on the same shape at roughly the same time.

What does a mixture of experts model do badly?

It wastes memory, and that is the honest complaint against it. Every expert has to stay loaded for the whole session even though most of them sit idle for any given token, so you are paying to store 30.5B parameters in order to get the speed of 3.3B, and a dense model of the same active size would take a tenth of the space.

The second cost is less obvious and it bites hardest on a laptop. Which experts run changes from one token to the next, so the machine keeps pulling different slabs of weights into cache instead of streaming one predictable block, and on a system that has already started swapping to disk that scattered access is where everything stalls. A dense model of the same active size will often feel steadier under memory pressure, because it reads the same weights every time.

Fine tuning is the third rough edge. Hugging Face's explainer describes sparse models as more prone to overfitting when they are fine tuned and suggests smaller batch sizes with a higher learning rate to cope, which is a real change to anyone's usual routine. The router is being trained at the same time as the experts, and a small dataset can teach it habits that damage everything the model does afterwards.

The objection worth taking seriously is about quality rather than plumbing. A reader is right to ask whether a model running 3.3B parameters per token can really match a dense model that runs 10 times as many, and there is a measured answer that rarely appears in the announcements. Hugging Face's explainer reports that Switch Transformers found the sparse model doing worse than its dense counterpart on downstream tasks at a fixed pretrain perplexity, with the gap largest on reasoning heavy ones such as SuperGLUE. Nobody has rerun that comparison publicly at the sizes shipping now, so the fair position is that the design buys compute for certain and leaves a question about quality that no model card answers.

Then there is the routing itself, which has more ways to go wrong than a dense stack does. The load balancing loss exists because routers collapse onto favourites, and the capacity factor the same explainer describes means every expert has a queue with a ceiling, so a token arriving at a full expert overflows to the next layer untouched, or in some projects is dropped outright. None of this is visible from outside the model, so the design carries more ways to fail quietly than the dense stack it replaced.

How do you check a model before you download 65 GB?

Open the model's config.json file on Hugging Face and look for an experts field, which takes about 10 seconds and costs nothing at all. If num_experts or num_local_experts is there, the model is a mixture of experts model, and num_experts_per_tok tells you how many of them run for each token.

curl -s https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/raw/main/config.json | grep experts

That prints the 2 lines that matter and nothing else, and the same command works on any public repository once you swap the owner and the model name. Qwen calls the field num_experts, OpenAI calls it num_local_experts in gpt-oss, and DeepSeek splits it into n_routed_experts and n_shared_experts because one of its experts is always on for every token. The names differ between labs, the meaning does not.

The model name often tells you first, before you fetch anything. When a card carries 30B and A3B in the same name, the first figure is the total and the second is what runs per token, and Qwen uses that convention right across its lineup. If a card gives you only one number, treat it as the total, because that is the figure the announcement prefers and it is also the one your memory has to satisfy.

1. Read the total parameter count and compare it against your free RAM, not against your disk.
2. Read num_experts_per_tok in config.json to see how much of the model actually runs.
3. Check the download size on Ollama or the file list, because quantization changes it entirely.

The practical rule fits into a single line. A mixture of experts model is a bet that memory is cheaper than arithmetic, which is true on a rented accelerator and only sometimes true on a laptop, so read both numbers before you pull anything and decide which of the 2 you are actually short of. If you are picking a coding assistant rather than a raw model, our comparison of Claude Code and Codex covers the tooling that sits on top, and the tools desk keeps the rest of the shortlists.