Install & Compatibility
Where this runs
tested against v0.3.13 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
py 3.10
✕ build_error
✕ timeout
py 3.11
✕ build_error
✓ 87.4s
py 3.12
✕ build_error
✓ 84.5s
py 3.13
✕ build_error
✓ 73.5s
py 3.9
✕ build_error
✕ timeout
5154MB installed
● package 5154MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
score
✓ from bert_score import score
This example calculates BERTScore (Precision, Recall, and F1) between a list of candidate sentences and reference sentences. The `lang` parameter specifies the language model to use, and `verbose=True` provides detailed output during computation.
from bert_score import score
cands = ["The cat sat on the mat.", "The dog ate the food."]
refs = [["The cat was on the mat."], ["A dog consumed the meal."]]
P, R, F1 = score(cands, refs, lang="en", verbose=True)
print(f"Precision: {P.mean().item():.3f}")
print(f"Recall: {R.mean().item():.3f}")
print(f"F1 Score: {F1.mean().item():.3f}")
Debug
Known issues
breakingBERTScore often requires specific versions of the `transformers` library. Incompatible versions can lead to `KeyError`, `AttributeError`, or incorrect scores due to changes in the underlying Hugging Face API.fixRefer to the `bert-score` release notes and test with the `transformers` version specified as compatible. If encountering issues, try pinning `transformers` to a known working version (e.g., `pip install transformers<4.17.0` for `bert-score==0.3.12` or `pip install transformers>=4.17.0` for `bert-score==0.3.13`).
affects: All versions, especially during `transformers` major updates. For instance, v0.3.7 fixed compatibility with transformers >=4.0.0, and v0.3.13 fixed issues with transformers > 4.17.0.
breakingThe command-line option for rescaling with a baseline was changed from `--rescale-with-baseline` to `--rescale_with_baseline` for consistency with other options.fixUpdate CLI calls to use `--rescale_with_baseline` instead of `--rescale-with-baseline`.
affects: CLI usage in versions v0.3.6 and later.
gotchaUsing the `--use_fast_tokenizer` option (or `use_fast_tokenizer=True` in Python) with Hugging Face transformers can lead to different BERTScore results due to subtle differences in tokenizer implementations compared to default slow tokenizers.fixBe aware that enabling fast tokenizers may alter scores. For reproducibility, explicitly set `use_fast_tokenizer` to `True` or `False` based on your desired behavior and consistently use the same setting. Compare scores with and without this option if consistency with previous results is critical.
affects: v0.3.10 and later when `--use_fast_tokenizer` is enabled.
gotchaThe default or recommended model for BERTScore has evolved. While RoBERTa is often the default, models like DeBERTa (e.g., `microsoft/deberta-xlarge-mnli`) have shown higher correlation with human scores on some datasets.fixConsider specifying a `model_type` like `microsoft/deberta-xlarge-mnli` (or other recommended models from the Hugging Face Model Hub) explicitly to potentially achieve better evaluation correlations. Refer to the official documentation or release notes for the latest recommendations and model performance benchmarks.
affects: v0.3.8 and later.
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'bert_score'
The 'bert-score' library or its dependencies are not installed in the current Python environment.
fixEnsure the library is correctly installed using pip: `pip install bert-score`
RuntimeError: CUDA out of memory. Tried to allocate X MiB (GPU Y; Z GiB total capacity; A GiB already allocated; B MiB free; C GiB reserved in total by PyTorch)
The GPU does not have enough memory to perform the BERTScore calculation, often due to large batch sizes, long sequences, or a large pre-trained model.
fixReduce the `batch_size` parameter in the `score` function, use a smaller pre-trained BERT model (e.g., 'bert-base-uncased' instead of 'bert-large-uncased'), or enable mixed-precision training if applicable by installing Apex or using PyTorch's native `torch.cuda.amp`.
IndexError: tuple index out of range
This error often occurs when the input data (reference or candidate sentences) provided to the `bert_score.score` function is malformed, such as an empty list, an incorrect number of elements, or an unexpected data structure.
fixVerify that both the `cands` and `refs` arguments passed to `bert_score.score` are lists of strings, where each string represents a sentence, and that they are not empty or contain unexpected nested structures. Ensure `cands` and `refs` have compatible dimensions if you are comparing lists of lists.
RuntimeError: CUDA out of memory.
The chosen BERT model is too large or the batch size/sequence length is too high for the available GPU memory.
fixUse a smaller `model_type` (e.g., `distilbert-base-uncased`), reduce `batch_size`, or process texts in smaller chunks. Example: `bert_score.score(..., model_type='distilbert-base-uncased')`
ValueError: Reference text must be a list of strings.
The `cands` or `refs` arguments passed to `bert_score.score` are not provided as a list of strings.
fixEnsure both candidate and reference texts are lists of strings, even for single texts. Example: `bert_score.score(cands=['my candidate text'], refs=['my reference text'])`
Upgrade
Version history
0.3.13latest on PyPI · released Feb 20, 2023
Audit
Dependencies
torchrequiredCore deep learning framework for the BERTScore computations.
transformersrequiredProvides access to pre-trained BERT models and tokenizers.
numpyrequiredFundamental package for numerical computing.
scipyrequiredScientific computing library, potentially used for statistical operations.
tqdmrequiredFor progress bar visualization.
requestsrequiredFor downloading models.
packagingrequiredFor handling version comparisons.