Residual Connection

Early architecture using learned gates for skip connections.

A narrow hidden layer forcing compact representations.

Using same parameters across different parts of a model.

Built-in assumptions guiding learning efficiency and generalization.

Limiting gradient magnitude to prevent exploding gradients.

Optimization using curvature information; often expensive at scale.

Matrix of second derivatives describing local curvature of loss.

Neural networks can approximate any continuous function under certain conditions.