Tokenization

A high-capacity language model trained on massive corpora, exhibiting broad generalization and emergent behaviors.

Training objective where the model predicts the next token given previous tokens (causal modeling).

Generates sequences one token at a time, conditioning on past tokens.

Predicts masked tokens in a sequence, enabling bidirectional context; often used for embeddings rather than generation.

Achieving task performance by providing a small number of examples inside the prompt without weight updates.

Techniques that stabilize and speed training by normalizing activations; LayerNorm is common in Transformers.

Randomly zeroing activations during training to reduce co-adaptation and overfitting.

An RNN variant using gates to mitigate vanishing gradients and capture longer context.

Injects sequence order into Transformers, since attention alone is permutation-invariant.