Weight Initialization

Techniques that stabilize and speed training by normalizing activations; LayerNorm is common in Transformers.

Randomly zeroing activations during training to reduce co-adaptation and overfitting.

An RNN variant using gates to mitigate vanishing gradients and capture longer context.

Injects sequence order into Transformers, since attention alone is permutation-invariant.

Nonlinear functions enabling networks to approximate complex mappings; ReLU variants dominate modern DL.

Activation max(0, x); improves gradient flow and training speed in deep nets.

Gradients shrink through layers, slowing learning in early layers; mitigated by ReLU, residuals, normalization.

Gradients grow too large, causing divergence; mitigated by clipping, normalization, careful init.