Sparse Expert

A sparse expert is one specialist network inside a Mixture of Experts model. Only a few experts run for each input. The rest stay off. Each expert is usually a feedforward network with its own weights. A gating network, the router, looks at the input and picks which experts should handle it, often 1 to 4 out of dozens or hundreds.

Sparse means most experts do no work on a given token. That cuts compute while the total parameter count can still be huge. You can train a model with trillions of parameters and still run inference, because only a slice of those parameters fire on each forward pass.

During training, experts often specialize without being told to. Some get better at code. Some at scientific text. Some at chat. The model applies the relevant expert instead of using one dense block for everything. The hard parts are load balancing, so work does not collapse onto a few popular experts, and routing, so the gate does not make noisy choices.

Google's Switch Transformer, GLaM, and Mixtral are the usual examples. They sit between a dense model, where every parameter runs every time, and a pile of fully separate specialist models. You get more capacity per FLOP. You also get a new failure mode: a bad router wastes the extra parameters. The 2017 sparsely-gated MoE paper showed that only a few experts need to run per token.