LSTM

Networks with recurrent connections for sequences; largely supplanted by Transformers for many tasks.

Injects sequence order into Transformers, since attention alone is permutation-invariant.

Converting text into discrete units (tokens) for modeling; subword tokenizers balance vocabulary size and coverage.

Training objective where the model predicts the next token given previous tokens (causal modeling).

Generates sequences one token at a time, conditioning on past tokens.

Gradients grow too large, causing divergence; mitigated by clipping, normalization, careful init.

Methods to set starting weights to preserve signal/gradient scales across layers.

Techniques that stabilize and speed training by normalizing activations; LayerNorm is common in Transformers.

Randomly zeroing activations during training to reduce co-adaptation and overfitting.