Entropy

Measures divergence between true and predicted probability distributions.

Measures how one probability distribution diverges from another.

Quantifies shared information between random variables.

Measures how much information an observable random variable carries about unknown parameters.

Measures a model’s ability to fit random noise; used to bound generalization error.

Systematic error introduced by simplifying assumptions in a learning algorithm.

Error due to sensitivity to fluctuations in the training dataset.

Reduction in uncertainty achieved by observing a variable; used in decision trees and active learning.