Absolute Positional Encoding

Techniques to handle longer documents without quadratic cost.

Attention mechanisms that reduce quadratic complexity.

Routes inputs to subsets of parameters for scalable capacity.

Chooses which experts process each token.

A single attention mechanism within multi-head attention.

Allows model to attend to information from different subspaces simultaneously.

Stores past attention states to speed up autoregressive decoding.

Encodes positional information via rotation in embedding space.