Convex Optimization

Optimization with multiple local minima/saddle points; typical in neural networks.

A point where gradient is zero but is neither a max nor min; common in deep nets.

Variability introduced by minibatch sampling during SGD.

The shape of the loss function over parameter space.

Measures how much information an observable random variable carries about unknown parameters.

Estimating parameters by maximizing likelihood of observed data.

Updating beliefs about parameters using observed evidence and prior distributions.

Bayesian parameter estimation using the mode of the posterior distribution.