Dropout

Techniques that discourage overly complex solutions to improve generalization (reduce overfitting).

An RNN variant using gates to mitigate vanishing gradients and capture longer context.

Injects sequence order into Transformers, since attention alone is permutation-invariant.

Converting text into discrete units (tokens) for modeling; subword tokenizers balance vocabulary size and coverage.

Training objective where the model predicts the next token given previous tokens (causal modeling).

Gradients shrink through layers, slowing learning in early layers; mitigated by ReLU, residuals, normalization.

Gradients grow too large, causing divergence; mitigated by clipping, normalization, careful init.

Methods to set starting weights to preserve signal/gradient scales across layers.

Techniques that stabilize and speed training by normalizing activations; LayerNorm is common in Transformers.