CLIP-benchmark is a Python library designed to evaluate CLIP-like models on a standard set of datasets for various tasks, including zero-shot classification, zero-shot retrieval, linear probing, and captioning. It supports models like OpenCLIP, Japanese CLIP, and NLLB CLIP, and integrates with datasets from torchvision, TensorFlow datasets, and VTAB. The library is currently active, with version 1.6.2, and focuses on reproducible evaluation results.
pip install clip-benchmarkVerified import paths — ran on the pinned version, not inferred.
This command runs a zero-shot classification benchmark on the CIFAR-10 dataset using an OpenCLIP pre-trained model (ViT-B-32-quickgelu trained on LAION-400m) and saves the results to `result.json`.
Refer to the documentation for examples on using templated paths, especially for Hugging Face WebDatasets.
Consult the `clip-benchmark` documentation on 'Evaluating on a webdataset' for instructions on preparing and using WebDatasets.
Install necessary optional dependencies based on the datasets you plan to benchmark. Check the library's `README` or `datasets/builder.py` for specific requirements.
Refer to `clip_benchmark/models/open_clip.py` as an example for structuring your model loading function and ensure it returns the (model, transform, tokenizer) tuple as expected.