Saddle Point

Variability introduced by minibatch sampling during SGD.

The shape of the loss function over parameter space.

A narrow minimum often associated with poorer generalization.

A wide basin often correlated with better generalization.

Updating beliefs about parameters using observed evidence and prior distributions.

Bayesian parameter estimation using the mode of the posterior distribution.

Optimization problems where any local minimum is global.

Optimization with multiple local minima/saddle points; typical in neural networks.