What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020–2025

Jing Yang1,2, Nils Feldhus7,3,*, Salar Mohtaj3, Leonhard Hennig3, Qianli Wang1, Eleni Metheniti4, Sherzod Hakimov5, Charlott Jakob1,3, Veronika Solopova1, Konrad Rieck1,2, David Schlangen3,5, Sebastian Möller1,2,3, Vera Schmitt1,3,6,8
1Technische Universität Berlin 2BIFOLD – Berlin Institute for the Foundations of Learning and Data 3German Research Center for Artificial Intelligence (DFKI) 4Joint Research Center 5University of Potsdam 6CERTAIN 7University of Groningen 8Johannes Gutenberg University Mainz

*Work done at TU Berlin and BIFOLD.

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2026

Abstract

As Natural Language Generation (NLG) dominates modern NLP, scalable evaluation remains a critical bottleneck. Consequently, LLM-as-a-judge (LaaJ) adoption has accelerated rapidly, appearing in more papers than human evaluation in 2025. This pivotal shift motivates a critical analysis of current evaluation practices. Overcoming the limits of rigid keyword filtering and manual review, we employ a multi-LLM information extraction pipeline to gather structured metadata from 14,171 papers across four major NLP conferences (2020–2025). Analyzing 3,334 filtered NLG papers, we identify three systemic challenges. (1) Metric inertia: despite the shift toward open-ended generation, legacy lexical metrics (BLEU, ROUGE) persist as primary indicators, typically used alongside rather than replaced by semantic alternatives. (2) Metric-criteria mapping problem: our paper-level co-occurrence data reveals that general-purpose automatic metrics are applied as broad proxies for quality, without specifying which dimension of text generation they are intended to evaluate. (3) Validation gap: LaaJ has grown rapidly without commensurate human validation (fewer than 8% of papers). Crucially, while LaaJ correlates with aggregate quality, alignment collapses on fine-grained criteria like fluency. To address these gaps, we distill our findings into a minimal Evaluation Checklist to guide metric selection, construct validity, and LaaJ deployment.

Paper annotation pipeline

Annotation pipeline: papers from the ACL Anthology are converted to text, three LLMs answer a questionnaire, answers are harmonized by majority vote, filtering, union and LLM verification into silver data, and human annotators produce gold data for a 110-paper subset.
Figure 2. Our paper annotation pipeline, including converting PDF to text, extraction of metadata based on four NLG evaluation questions (DeepSeek-R1, GPT-OSS-120B, Qwen3-235B), harmonization based on majority vote, filtering, and merging, and human validation, yielding 14k structured papers including a subset of 110 papers with gold annotations.

Agreement with human annotation (%, 110 papers)

PairQ1 NLGQ2 AutoQ3 LaaJQ4 Human
Human R1 vs. R281.880.089.191.8
DeepSeek-R174.673.695.592.7
GPT-OSS-120B75.575.590.990.9
Qwen3-235B75.570.093.685.5
Majority vote74.671.893.688.2
LLM-harmonized77.374.695.590.0

Model rows report agreement with the human annotation. For Q3 and Q4, LLM–human agreement is comparable to or higher than the agreement between the two human annotation rounds.

Association measures

Frequency P(B | A) is the proportion of papers containing term A that also contain term B; for example, 81.5% of MT papers report BLEU.

Likelihood ratio LR(A → B) = P(B | A) / P(B | ¬A) measures distinctiveness; for example, BLEU is 2.63 times more likely in MT papers than in other papers. LR > 1 indicates attraction and LR < 1 avoidance.

Statistical significance is assessed with Dunning’s G² test under Benjamini–Hochberg FDR correction; pairs with an adjusted p ≤ 0.05 are considered significant. All statistics on this page are computed from the paper-level data.

Corpus filtering

Main-conference papers from the ACL Anthology. Majority voting over three LLMs identifies 8,665 NLG papers; restricting these to the 30 most frequent NLG tasks yields the 3,334 papers analyzed on this page.

Metric inertia

Despite the shift toward open-ended generation, legacy lexical metrics persist as primary indicators. BLEU appears in 81.5% of MT papers overall and declines only from 88.6% (2020) to 61.8% (2025), while COMET grows to 39.0% in 2025. Exact Match still anchors 27.1% of 2025 QA papers. Semantic metrics are typically used alongside rather than in place of lexical ones.

Annual rank of the eight most frequent items

Paradigm

Figure 4. Annual frequency rank of the eight most frequent items in single-task papers. Marker size is proportional to the item's LR for the task in that year. Solid lines with filled markers denote items significantly associated with the task overall (G², BH-FDR-adjusted p ≤ 0.05); dashed grey lines with hollow markers denote frequent but non-distinctive items. Labels give (overall LR, number of papers). Color encodes the QCET dimension (dark grey for automatic metrics), and criteria are prefixed with it: QO intrinsic, QI relative to input, QT relative to target, QE relative to an external standard, AUX extension. Selecting a line lists the corresponding papers.

Metric–criteria mapping

A metric and a criterion may both be characteristic of a task without co-occurring in the same papers. We therefore test metric–criterion association within each task. Significant associations concentrate among frequent metrics and frequent criteria, whereas task-distinctive metrics (e.g. FactCC, COMET) are rarely associated with task-distinctive criteria (e.g. factual truth, translation accuracy).

Within-task metric × criterion association

Evaluated by

Figure 5. Within-task association between metrics and criteria. Shading encodes the co-occurrence LR, P(metric | criterion) / P(metric | ¬criterion), computed separately over the task's LaaJ papers (red, upper-left triangle) and human-evaluation papers (green, lower-right triangle). Outlined cells are significant (G², BH-FDR-adjusted p ≤ 0.05, at least 5 co-occurrences). Rows and columns contain the five most frequent items in the task followed by the five most task-distinctive items by task-LR (italics, beyond the dotted line). Selecting a cell lists the corresponding papers.

Validation gap

Of the 679 papers that employ LaaJ, 433 also conduct human evaluation, and 254 report the agreement between the two (37% of LaaJ papers; 7.6% of the corpus). Among the 179 papers that use both methods without comparing them, 76% assess at least one criterion with both, indicating that a comparison would have been feasible in most cases.

Reported LaaJ–human agreement by quality criterion

Statistic
Sort

Figure 6. Reported LaaJ–human agreement per QCET criterion. Each point is one reported comparison value from the 70 papers with validation content in the 90-paper human-annotated subset; black ticks mark the mean per statistic. Only criteria with more than ten reported values are shown. LaaJ achieves moderately high correlations for aggregate Overall Quality, but alignment degrades on fine-grained criteria: Fluency and Coherence show median correlations near or below 0.4 with large variance, and LaaJ fails to align with humans on task usefulness.

Papers

All 3,334 papers with their extracted evaluation metadata. The filters define a selection that can be analyzed in the corpus explorer below; selecting an element in any chart lists its papers here.

Yearsto
Evaluated with
Analyze the selection ↓
PaperVenueTasksEvaluationMetrics

A: automatic metrics; L: LLM-as-a-judge; H: human evaluation; V: LaaJ validated against humans. Select a row for all extracted metadata.

Corpus explorer

Custom analyses of all papers or of the papers selected in the section above. Selecting a chart element lists the underlying papers in the paper table.

1.

Choose an analysis

Examples
2.

Papers to analyze

Change the selection ↑
3.

Result

Trends over time

Measure
Chart
By

Evaluation Design Checklist

To help researchers translate these challenges into practice, we propose a compact Evaluation Design Checklist. Task-specific recommendations for dialogue generation, machine translation, text summarization and question answering are given in Appendix D of the paper.

1. Metric Selection

  • Are you evaluating open-ended generation? If yes, acknowledge that optimizing for exact n-gram overlap with a single reference penalizes valid paraphrases. Shift legacy metrics to secondary baselines.
  • Have you included a semantic metric? Ensure legacy metrics (BLEU, ROUGE) are supplemented, or replaced, by semantic metrics (e.g., BERTScore, COMET, BLEURT) that reward meaning over lexical matching.

2. Construct Validity

  • Is the target construct explicitly named? Verify that every reported metric is explicitly tied to a specific quality criterion (e.g., “We use QAFactEval to measure factuality,” not just “to measure quality”).
  • Is the metric mathematically suited for the construct? Ensure you are not using lexical overlap (e.g., ROUGE) to substantiate claims about high-level semantics (e.g., logical coherence, factual consistency).

3. LaaJ Deployment

  • Have you validated your LaaJ against human judgments? Where a public meta-evaluation benchmark covers your target criterion, report your judge’s correlation against it. Otherwise, annotate a subset per criterion and report appropriate correlation or agreement metrics.
  • Are human experts retained for sensitive or poorly correlated criteria? Acknowledge that LaaJ reliability often collapses on strict linguistic nuances (e.g., task usefulness). For these criteria, rely on expert human evaluation as your primary signal.

BibTeX

@inproceedings{yang2026measuring,
  title     = {What Are We Measuring in {NLG}? A Meta-Analysis of Evaluation Trends 2020--2025},
  author    = {Yang, Jing and Feldhus, Nils and Mohtaj, Salar and Hennig, Leonhard and
               Wang, Qianli and Metheniti, Eleni and Hakimov, Sherzod and Jakob, Charlott and
               Solopova, Veronika and Rieck, Konrad and Schlangen, David and
               Möller, Sebastian and Schmitt, Vera},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year      = {2026},
  note      = {arXiv:2601.07648}
}