torchmetrics

library1.9.0pypypi✓ verified 27d ago

TorchMetrics is a comprehensive collection of PyTorch native metrics for evaluating machine learning models, offering over 100 common and specialized metrics implemented directly in PyTorch. Developed and maintained by Lightning AI, it provides a standardized, rigorously tested, and distributed-training compatible API for metric computation, reducing boilerplate and ensuring reproducibility. It automatically accumulates over batches and synchronizes between multiple devices. The library is currently at version 1.9.0 and maintains a regular release cadence with several patch and minor releases per year.

ai-ml testing

pip install torchmetrics

github ↗package ↗homepage ↗docs ↗

INSTALL

IMPORT

SIG · TORCHMETRICS

torchmetrics

ai-mlpythonv1.9.0

Install

68.2s avg

Import

7290ms

Disk

4813MB

Pass rate

4/ 10

Env Coverage4 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.9.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

✕ build_error

✓ 80s

py 3.11

✕ build_error

✓ 70.7s

py 3.12

✕ build_error

✓ 62.4s

py 3.13

✕ build_error

✓ 59.6s

py 3.9

✕ build_error

✕ timeout

4813MB installed

● package 4813MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Accuracy

✓ from torchmetrics import Accuracy

functional.accuracy

✓ from torchmetrics.functional import accuracy

MetricCollection

✓ from torchmetrics import MetricCollection

Metric

✓ from torchmetrics import Metric

Base class for implementing custom metrics.

This quickstart demonstrates the core ways to use TorchMetrics: the functional API for stateless, single-batch computation, the class-based API for accumulating states over multiple batches, and MetricCollection for grouping several metrics. Remember to reset class-based metrics after each epoch or evaluation phase to avoid mixing states.

import torch import torchmetrics from torchmetrics import Accuracy, MetricCollection from torchmetrics.functional import accuracy # 1. Functional API: For single-batch, stateless computation preds_f = torch.randn(10, 5).softmax(dim=-1) target_f = torch.randint(5, (10,)) acc_functional = accuracy(preds_f, target_f, task="multiclass", num_classes=5) print(f"Functional Accuracy: {acc_functional.item()}") # 2. Class-based API: For accumulating metrics over multiple batches/epochs metric = Accuracy(task="multiclass", num_classes=5) preds_c = torch.randn(10, 5).softmax(dim=-1) target_c = torch.randint(5, (10,)) metric.update(preds_c, target_c) # Simulate another batch preds_c2 = torch.randn(10, 5).softmax(dim=-1) target_c2 = torch.randint(5, (10,)) metric.update(preds_c2, target_c2) final_acc = metric.compute() print(f"Class-based Accuracy (accumulated): {final_acc.item()}") metric.reset() # Reset metric states for the next epoch/evaluation # 3. MetricCollection: Group multiple metrics metrics = MetricCollection({ 'Accuracy': Accuracy(task="multiclass", num_classes=5), 'F1Score': torchmetrics.F1Score(task="multiclass", num_classes=5) }) preds_mc = torch.randn(10, 5).softmax(dim=-1) target_mc = torch.randint(5, (10,)) metrics.update(preds_mc, target_mc) result_mc = metrics.compute() print(f"MetricCollection Result: {result_mc}")

Debug

Known issues

breakingPython 3.9 support has been dropped with the release of v1.9.0. The minimum required Python version is now 3.10.

fix

Upgrade your Python environment to 3.10 or newer, or pin `torchmetrics<1.9.0`.

affects: >=1.9.0

breakingThe default value for the `average` argument in `DiceScore` has changed from `None` to `"macro"` starting from v1.9.0. This can alter the behavior of existing code if the `average` argument was not explicitly set.

fix

Explicitly set the `average` argument in `DiceScore` to `None` or your desired reduction method if you relied on the previous default behavior.

affects: >=1.9.0

gotchaMetrics maintain internal states that accumulate data. Mixing these states across different phases (e.g., training, validation, testing) or re-using the same metric instance without resetting can lead to incorrect results or memory leaks.

fix

Always initialize separate metric instances for different phases (training, validation, test) or call `metric.reset()` after each complete evaluation epoch/phase to clear its internal state.

affects: All

gotchaMetric states are initialized on the CPU. When working with PyTorch tensors on GPU, especially in distributed training (DDP), ensure that metric objects are moved to the same device as the input data using `.to(device)`. Failure to do so can result in `RuntimeError: Encountered different devices in metric calculation`.

fix

Call `metric.to(device)` after initialization, or ensure the metric is registered as a child module within a `torch.nn.Module` or `LightningModule`, which handles device transfers automatically.

affects: All

gotchaWhen defining metrics as part of a `torch.nn.Module` or `LightningModule`, avoid using native Python `list` or `dict` to store multiple `Metric` instances. These will not be correctly identified as child modules, preventing automatic device placement and state management.

fix

Use `torch.nn.ModuleList` or `torch.nn.ModuleDict` instead of native Python collections when nesting metrics within a `torch.nn.Module`.

affects: All

gotchaUsers of `MetricCollection` might encounter `UserWarning: The compute method of metric X was called before the update method...` This often indicates an issue where internal states of grouped metrics are not being updated correctly before `compute` is called, particularly in older versions or specific usage patterns.

fix

Ensure all metrics within the `MetricCollection` receive `update` calls for the relevant data. If issues persist, consider isolating metrics or upgrading to the latest `torchmetrics` version as device management and state handling are continually improved.

affects: <=1.8.x

gotchaFor performance-critical applications, especially with `MeanMetric`, explicitly providing a weight tensor to `update` instead of relying on default values can be beneficial. Additionally, disabling NaN checks in the base `Aggregator` class or careful device management can reduce overhead.

fix

Profile your code. For `MeanMetric`, consider `metric.update(value, weight=my_tensor)`. For advanced optimization, explore `Aggregator` configurations and carefully manage device placement and cross-device synchronization.

affects: All

breakingInstallation of `torchmetrics` can fail in minimal environments (like Alpine Linux) if system-level build tools (e.g., C compilers like `gcc`, `clang`) are not pre-installed. Dependencies like `numpy` (often a transitive dependency) might require compiling C extensions from source, leading to build errors.

fix

Ensure your environment has necessary build tools installed. For Alpine Linux, this typically means running `apk add build-base` or explicitly installing `gcc` and `g++`. For Debian/Ubuntu-based systems, `apt-get install build-essential` is usually sufficient.

affects: All

Errors

Common errors & fixes

AttributeError: module 'torchmetrics' has no attribute 'Accuracy'

Many metrics, including `Accuracy`, were moved from the top-level `torchmetrics` module into specific submodules (e.g., `classification`, `regression`) to improve organization in versions 0.7.0 and newer.

fix

Import the metric from its correct submodule: `from torchmetrics.classification import Accuracy`

ValueError: The number of classes has to be greater than 1, got 1

This error occurs when a metric expects multiple classes but is initialized without specifying `num_classes=2` or, in `torchmetrics` v1.0+, without setting the `task='binary'` parameter for binary classification.

fix

Initialize the metric with `task='binary'` for binary classification problems: `metric = Accuracy(task='binary')`

ModuleNotFoundError: No module named 'torchmetrics.metrics'

In `torchmetrics` versions 0.7.0 and newer, metrics were reorganized into specific submodules (e.g., `classification`, `regression`), and the top-level `torchmetrics.metrics` module no longer exists.

fix

Import specific metrics from their respective submodules: `from torchmetrics.classification import Accuracy`

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!

This error arises when the input tensors (`preds`, `target`) or the metric itself are located on different devices (e.g., one on CPU and the other on GPU) during computation.

fix

Ensure that the metric and all input tensors are moved to the same device using `.to(device)` before computation: `device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')` `metric = Accuracy(task='binary').to(device)` `preds = preds.to(device)` `target = target.to(device)`

Upgrade

Version history

1.9.0latest on PyPI · released Mar 9, 2026

Audit

Dependencies

torchrequiredCore dependency for all metric computations.

pythonrequiredRequires Python 3.10 or newer.

Agent activity

45 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/Lightning-AI/torchmetrics ↗

packagehttps://pypi.org/project/torchmetrics/ ↗

homepagehttps://torchmetrics.readthedocs.io ↗

docshttps://torchmetrics.rtfd.io/en/latest/ ↗