Glossary · Category

Model Evaluation Metrics & Statistical Concepts

7 plain-English definitions.

Accuracy

The fraction of predictions or generations that exactly match the correct/expected answer.

BERTScore

An embedding-based evaluation metric that compares generated and reference text using contextual embedding similarity rather than exact n-gram overlap.

BLEU score

An n-gram overlap metric comparing generated text to reference text, historically used for machine translation quality.

Calibration

How well a model's stated confidence (e.g., token probabilities) matches its actual empirical accuracy; a well-calibrated model's 70%-confidence predictions are correct about 70% of the time.

F1 score

The harmonic mean of precision and recall, used to balance false positives and false negatives in classification-style evaluation.

Faithfulness

In summarization/RAG contexts, whether generated text accurately represents the source content without introducing unsupported claims.

ROUGE score

A recall-oriented n-gram/overlap metric commonly used to evaluate summarization quality against reference summaries.