Rotary Positional Embeddings

Encodes token position explicitly, often via sinusoids.

Techniques to handle longer documents without quadratic cost.

Attention mechanisms that reduce quadratic complexity.

Routes inputs to subsets of parameters for scalable capacity.

Prevents attention to future tokens during training/inference.

A single attention mechanism within multi-head attention.

Allows model to attend to information from different subspaces simultaneously.

Stores past attention states to speed up autoregressive decoding.