Results for "attention weights"

A single attention mechanism within multi-head attention.

Attention where queries/keys/values come from the same sequence, enabling token-to-token interactions.

Attention mechanisms that reduce quadratic complexity.

GNN using attention to weight neighbor contributions dynamically.

Attention between different modalities.

Methods to set starting weights to preserve signal/gradient scales across layers.

Updating a pretrained model’s weights on task-specific data to improve performance or adapt style/behavior.

Techniques that fine-tune small additional components rather than all weights to reduce compute and storage.

Reducing numeric precision of weights/activations to speed inference and reduce memory with acceptable accuracy loss.

Removing weights or neurons to shrink models and improve efficiency; can be structured or unstructured.

Models whose weights are publicly available.

Architecture based on self-attention and feedforward layers; foundation of modern LLMs and many multimodal models.

Injects sequence order into Transformers, since attention alone is permutation-invariant.

Studying internal mechanisms or input influence on outputs (e.g., saliency maps, SHAP, attention analysis).

Prevents attention to future tokens during training/inference.

Stores past attention states to speed up autoregressive decoding.

Mechanism that computes context-aware mixtures of representations; scales well and captures long-range dependencies.

Allows model to attend to information from different subspaces simultaneously.