Positional Encoding

Converting text into discrete units (tokens) for modeling; subword tokenizers balance vocabulary size and coverage.

Training objective where the model predicts the next token given previous tokens (causal modeling).

Generates sequences one token at a time, conditioning on past tokens.

Predicts masked tokens in a sequence, enabling bidirectional context; often used for embeddings rather than generation.

Methods to set starting weights to preserve signal/gradient scales across layers.

Techniques that stabilize and speed training by normalizing activations; LayerNorm is common in Transformers.

Randomly zeroing activations during training to reduce co-adaptation and overfitting.

An RNN variant using gates to mitigate vanishing gradients and capture longer context.