Feedforward Network

A feedforward network in a transformer is a two-layer net applied independently to each position after attention.

The usual structure is a linear projection from hidden size d to a larger intermediate size, often 4d, a nonlinear activation such as GELU or SwiGLU, then a linear projection back to d. Expand, activate, contract. That pattern adds nonlinearity and extra capacity. Attention mixes tokens. The feedforward net transforms each position on its own.

Research suggests feedforward layers store factual knowledge. " Feedforward layers hold most of a transformer's parameters, often about two-thirds. Newer designs try larger expansions, gated activations like SwiGLU that raise quality, and mixture-of-experts feedforward layers where different experts handle different inputs.

That split of labor is the transformer: mix across positions with attention, then nonlinearly transform within each position.

Because the same weights apply at every position, the feedforward net cannot mix tokens by itself. That is why it sits after attention. SwiGLU gates the expansion so some channels pass and others shut. Mixture-of-experts routes each token to a subset of feedforward experts, which raises capacity without running every expert on every token.

Factual edits that target feedforward neurons follow from this: those layers hold a large share of stored associations. Each Transformer block has a position-wise feedforward net after attention. Two linear layers with a nonlinearity in between.