Reward Model

Reinforcement learning from human feedback: uses preference data to train a reward model and optimize the policy.

Ensuring model behavior matches human goals, norms, and constraints, including reducing harmful or deceptive outputs.

Automated detection/prevention of disallowed outputs (toxicity, self-harm, illegal instruction, etc.).

Systematic differences in model outcomes across groups; arises from data, labels, and deployment context.

Techniques to understand model decisions (global or local), important in high-stakes and regulated settings.

Retrieval based on embedding similarity rather than keyword overlap, capturing paraphrases and related concepts.

Breaking documents into pieces for retrieval; chunk size/overlap strongly affect RAG quality.

Constraining outputs to retrieved or provided sources, often with citation, to improve factual reliability.

Fine-tuning on (prompt, response) pairs to align a model with instruction-following behaviors.