rouge-metric

library1.0.1pypypi✓ verified 85d ago

A fast Python implementation of full ROUGE metrics for automatic summarization evaluation, also providing a Python wrapper for the official ROUGE-1.5.5.pl Perl script. It supports various ROUGE variants (N, L, W, S, SU) and multi-reference evaluation. The library is actively maintained with periodic updates.

ai-ml testing

pip install rouge-metric

github ↗package ↗

INSTALL

IMPORT

SIG · ROUGE-METRIC

rouge-metric

ai-mlpythonv1.0.1

Install

—

Import

—

Disk

—

Pass rate

0/ 10

Env Coverage0 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.0.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

4/8 runs

py 3.11

4/8 runs

py 3.12

4/8 runs

py 3.13

4/8 runs

py 3.9

4/8 runs

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

PyRouge

✓ from rouge_metric import PyRouge

✗ from rouge_metric import Rouge

PerlRouge

✓ from rouge_metric import PerlRouge

This quickstart demonstrates how to use the pure Python ROUGE implementation to evaluate a single hypothesis against a single reference string, and also how to evaluate tokenized input against multiple references. The output includes ROUGE-N, ROUGE-L, ROUGE-W, ROUGE-S, and ROUGE-SU scores.

from rouge_metric import Rouge hypothesis = 'The cat sat on the mat.' reference = 'The cat was on the mat.' rouge = Rouge() scores = rouge.evaluate(hypothesis, reference) print(scores) # Example with multiple references (list of lists of tokens) hyp_tokens = ['the', 'cat', 'sat', 'on', 'the', 'mat'] ref1_tokens = ['the', 'cat', 'was', 'on', 'the', 'mat'] ref2_tokens = ['a', 'feline', 'was', 'resting', 'on', 'the', 'rug'] scores_multi_ref = rouge.evaluate_from_tokens(hyp_tokens, [ref1_tokens, ref2_tokens]) print(scores_multi_ref)

Debug

Known issues

gotchaROUGE metrics primarily rely on n-gram overlap and do not capture semantic meaning or contextual understanding. This can lead to high scores for syntactically similar but semantically divergent texts. It is recommended to complement ROUGE with other metrics (e.g., BERTScore) or human evaluation for a comprehensive assessment.

fix

Combine ROUGE scores with semantic similarity metrics (e.g., BERTScore) and qualitative human evaluation to get a more robust assessment of text quality.

affects: All versions

gotchaThe multi-document evaluation results from the pure Python implementation (rouge_metric.Rouge) may be slightly different from those produced by the official ROUGE-1.5.5.pl Perl script (accessed via rouge_metric.PerlRouge) because the Python implementation does not use bootstrap resampling.

fix

If exact replication of ROUGE-1.5.5.pl multi-document scores is critical, use the `PerlRouge` wrapper or adjust expectations for slight differences in multi-document scenarios with `Rouge`.

affects: All versions

gotchaThe pure Python implementation (rouge_metric.Rouge) expects pre-tokenized sentences (lists of tokens). Preprocessing steps like tokenization, stemming, and stopword removal are left to the client, which can impact scores if not handled consistently.

fix

Ensure consistent and appropriate preprocessing (tokenization, stemming, etc.) is applied to both hypothesis and reference texts before passing them to `rouge_metric.Rouge`.

affects: All versions

deprecatedThe `PerlRouge` wrapper, which calls the official ROUGE-1.5.5.pl script, is primarily intended for English corpora. For non-English summaries, it is recommended to use the pure Python implementation (`rouge_metric.Rouge`).

fix

For non-English text evaluation, use `rouge_metric.Rouge` and ensure your input tokens are appropriately preprocessed for the target language. For English, `PerlRouge` can still be used for official ROUGE-1.5.5.pl compatibility.

affects: All versions

Upgrade

Version history

1.0.1latest on PyPI · released Oct 21, 2020

Audit

Dependencies

PerloptionalRequired for using the PerlRouge wrapper, especially on Windows (e.g., Strawberry Perl).

Agent activity

16 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/li-plus/rouge-metric ↗

packagehttps://pypi.org/project/rouge-metric/ ↗