Experiment Tracking

A broader capability to infer internal system state from telemetry, crucial for AI services and agents.

Exponential of average negative log-likelihood; lower means better predictive fit, not necessarily better utility.

A dataset + metric suite for comparing models; can be gamed or misaligned with real-world goals.

Separating data into training (fit), validation (tune), and test (final estimate) to avoid leakage and optimism bias.

Often more informative than ROC on imbalanced datasets; focuses on positive class performance.

A proper scoring rule measuring squared error of predicted probabilities for binary outcomes.