The `rouge-score` library is a pure Python implementation of the ROUGE-1.5.5 evaluation metric, designed to closely replicate the results of the original Perl script. It provides functionalities for calculating ROUGE-N, ROUGE-L (sentence-level and summary-level), text normalization, and optional Porter stemming. The library is currently at version 0.1.2, released in July 2022, and while the version updates are infrequent, it remains an actively used and stable package maintained by Google for evaluating text generation tasks like summarization.
pip install rouge-scoreVerified import paths — ran on the pinned version, not inferred.
This example demonstrates how to initialize `RougeScorer` for common ROUGE types (ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-Lsum) with stemming enabled. It then computes and prints the precision, recall, and F1-score for a given reference and candidate text. Ensure `nltk` data (like 'punkt' and 'wordnet') is available if `use_stemmer=True`.
Always specify the exact ROUGE package and version used in research or production. When comparing results, ensure the same ROUGE implementation, configuration, and preprocessing steps are applied.
Complement ROUGE scores with other evaluation metrics that assess semantic similarity (e.g., BERTScore), human evaluations for quality assessment, and qualitative analysis to gain a comprehensive understanding of text generation quality.
Carefully select between `rougeL` and `rougeLsum` based on your data's structure and the desired evaluation behavior. `rougeLsum` is often preferred for multi-sentence summaries where newlines delineate sentences.
If stopword removal is desired, implement it as a preprocessing step on both reference and candidate texts before passing them to `RougeScorer`. Be mindful of how this might affect comparability with other ROUGE setups.
Use multiple, diverse human reference summaries if possible. Acknowledge and report the dependency on references, and consider sensitivity analysis to understand how scores change with different reference sets. Focus on relative improvements rather than absolute scores.
pip install rouge-score
from rouge_score.rouge_scorer import RougeScorer
Ensure the `target` argument is a single string. The `predictions` argument can be a list of strings, but `target` must be singular. ```python from rouge_score.rouge_scorer import RougeScorer scorer = RougeScorer(['rouge1'], use_stemmer=True) target_text = "This is the reference summary." predictions_list = ["This is a candidate summary.", "Another candidate."] scores = scorer.score(target_text, predictions_list) ```
Download the `punkt` resource using NLTK's downloader before initializing RougeScorer.
```python
import nltk
nltk.download('punkt')
from rouge_score.rouge_scorer import RougeScorer
scorer = RougeScorer(['rouge1'], use_stemmer=True)
```