Install & Compatibility
Where this runs
tested against v1.2 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.438s · 191.2MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 5.3s · import 0.420s · 188MB
191MB installed
● package 191MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
COCOEvalCap
✓ from pycocoevalcap.eval import COCOEvalCap
This class orchestrates the evaluation of multiple metrics.
Bleu
✓ from pycocoevalcap.bleu.bleu import Bleu
Import individual metric scorers if you need granular control or specific metrics.
Meteor
✓ from pycocoevalcap.meteor.meteor import Meteor
Import individual metric scorers if you need granular control or specific metrics.
Rouge
✓ from pycocoevalcap.rouge.rouge import Rouge
Import individual metric scorers if you need granular control or specific metrics.
Cider
✓ from pycocoevalcap.cider.cider import Cider
Import individual metric scorers if you need granular control or specific metrics.
Spice
✓ from pycocoevalcap.spice.spice import Spice
Import individual metric scorers if you need granular control or specific metrics.
This quickstart demonstrates how to set up and run the evaluation using `COCOEvalCap`. It uses mocked ground truth and predicted caption data to illustrate the expected data structure. In a real application, you would load your ground truth annotations into a `pycocotools.coco.COCO` object and your prediction results into another `COCO` object (using `loadRes`). The `evaluate()` method then computes all standard metrics.
import json
from pycocoevalcap.eval import COCOEvalCap
# Mock ground truth and predicted captions data
# In a real scenario, these would be loaded from JSON files
# 'gts' should map image_id to a list of ground truth captions
# 'res' should map image_id to a list of predicted captions
gts_data = {
"annotations": [
{"image_id": 1, "id": 101, "caption": "A man is riding a bicycle."},
{"image_id": 1, "id": 102, "caption": "A person on a bike on a street."},
{"image_id": 2, "id": 201, "caption": "Two dogs playing in the grass."},
{"image_id": 2, "id": 202, "caption": "Dogs are running on a lawn."}
]
}
res_data = [
{"image_id": 1, "caption": "A man cycling on a road.", "id": 301},
{"image_id": 2, "caption": "Two puppies in a field.", "id": 302}
]
# To initialize COCOEvalCap, you need COCO objects for ground truth and results.
# These COCO objects are typically created from JSON files matching the COCO format.
# For a quickstart, we'll manually structure the data to match expected input.
# The COCO object expects a dictionary with 'images' and 'annotations' keys.
# We only need 'annotations' for caption evaluation.
# Mock COCO objects (simplified for quickstart, actual COCO objects handle more fields)
class MockCoco:
def __init__(self, data):
self.anns = {ann['id']: ann for ann in data.get('annotations', [])}
self.imgToAnns = {}
for ann in data.get('annotations', []):
self.imgToAnns.setdefault(ann['image_id'], []).append(ann)
def loadRes(self, res_json_or_list):
# For simplicity, just store results. COCO.loadRes is more complex.
res_anns = []
for r in res_json_or_list:
# Assign a unique ID if not present, similar to COCO API behavior
if 'id' not in r:
r['id'] = max(self.anns.keys(), default=0) + len(res_anns) + 1
res_anns.append(r)
res_coco = MockCoco({'annotations': res_anns})
return res_coco
def getImgIds(self):
return list(self.imgToAnns.keys())
def loadAnns(self, ids):
return [self.anns[i] for i in ids]
# Initialize Mock COCO objects
# gts_coco_obj = COCO(gts_json_path) # In a real application
gts_coco_obj = MockCoco(gts_data)
# res_coco_obj = gts_coco_obj.loadRes(res_json_path) # In a real application
res_coco_obj = gts_coco_obj.loadRes(res_data)
eval_ids = gts_coco_obj.getImgIds()
cocoEval = COCOEvalCap(gts_coco_obj, res_coco_obj, eval_ids)
cocoEval.evaluate()
print("Evaluation results:")
for metric, score in cocoEval.eval.items():
print(f"{metric}: {score:.3f}")
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'pycocoevalcap' OR ERROR: Could not find a version that satisfies the requirement pycocoevalcap
The `pycocoevalcap` package is often not directly installable via `pip` from PyPI without specifying its GitHub repository due to its dependencies or specific packaging.
fixInstall the package directly from its GitHub repository using `pip install "git+https://github.com/salaniz/pycocoevalcap.git"`.
ERROR: Could not build wheels for pycocotools (or similar C/C++ compilation errors for pycocotools) OR ModuleNotFoundError: No module named 'pycocotools._mask'
`pycocotools`, a core dependency of `pycocoevalcap`, requires compilation of C/C++ extensions, which fails if the necessary build tools (like Visual C++ Build Tools on Windows or `gcc`/`build-essential` on Linux/macOS) or `cython` are missing.
fixEnsure `cython` is installed (`pip install cython`) and a C/C++ compiler is available on your system (e.g., `sudo apt-get install build-essential` on Ubuntu, Xcode Command Line Tools on macOS, or Visual C++ Build Tools for Windows). You might need to install `pycocotools` separately, potentially from a specific GitHub fork if `pip install pycocoevalcap` continues to fail.
subprocess.CalledProcessError: Command '['java', '-jar', ...]` returned non-zero exit status 1 (often during METEOR or SPICE computation) OR java.lang.UnsatisfiedLinkError
The METEOR and SPICE metrics rely on Java and Stanford CoreNLP. This error occurs if Java Development Kit (JDK) is not installed, not correctly configured in the system's PATH, or the required Java `.jar` files for METEOR/SPICE are missing or cannot be accessed by the Python subprocess.
fixInstall Java Development Kit (JDK 1.8.0 or later) and ensure the `java` executable is in your system's PATH. For SPICE, ensure adequate write permissions and internet access, as Stanford CoreNLP models are downloaded automatically the first time SPICE is evaluated.
Resource punkt not found. Please use the NLTK Downloader to obtain the resource: >>> import nltk >>> nltk.download('punkt')
The ROUGE-L metric, or other text processing components within `pycocoevalcap`, often utilize NLTK's `punkt` tokenizer, which is not automatically installed with the `nltk` package and must be downloaded separately.
fixRun these commands in a Python interpreter or script once: `import nltk; nltk.download('punkt')`. Ensure your environment has write permissions to the NLTK data directory or specify a custom download directory. Upgrade
Version history
1.2latest on PyPI · released Nov 18, 2020
Audit
Dependencies
pycocotoolsrequiredRequired for COCO API interaction and data structures.
Java 1.8.0requiredRequired for SPICE and PTBTokenizer components. Stanford CoreNLP will be downloaded automatically by SPICE.