Perplexity

A dataset + metric suite for comparing models; can be gamed or misaligned with real-world goals.

Often more informative than ROC on imbalanced datasets; focuses on positive class performance.

A proper scoring rule measuring squared error of predicted probabilities for binary outcomes.

Logging hyperparameters, code versions, data snapshots, and results to reproduce and compare experiments.

A broader capability to infer internal system state from telemetry, crucial for AI services and agents.