seqeval

library1.2.2pypypi✓ verified 21d ago

seqeval is a Python framework for sequence labeling evaluation. It provides metrics like F1 score, precision, recall, and a detailed classification report for tasks such as named-entity recognition and part-of-speech tagging. It is currently at version 1.2.2 and maintains an active release cadence, with updates often focusing on performance improvements and additional evaluation schemes.

ai-ml testing

pip install seqeval

github ↗package ↗

INSTALL

IMPORT

SIG · SEQEVAL

seqeval

ai-mlpythonv1.2.2

Install

11.9s avg

Import

3598ms

Disk

281MB

Pass rate

5/ 10

Env Coverage5 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.2.2 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

build_error

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 11.9s · import 3.598s · 272MB

281MB installed

● package 281MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

accuracy_score

✓ from seqeval.metrics import accuracy_score

precision_score

✓ from seqeval.metrics import precision_score

recall_score

✓ from seqeval.metrics import recall_score

f1_score

✓ from seqeval.metrics import f1_score

classification_report

✓ from seqeval.metrics import classification_report

IOB2

✓ from seqeval.scheme import IOB2

Used to specify a tagging scheme for metrics when needed.

Calculate the F1-score and generate a detailed classification report for sequence labeling predictions.

from seqeval.metrics import f1_score from seqeval.metrics import classification_report y_true = [['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'O', 'O'], ['B-PER', 'I-PER', 'O']] y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']] # Compute F1-score (micro average by default) print(f"F1 Score (micro): {f1_score(y_true, y_pred):.2f}") # Compute F1-score with different averaging methods print(f"F1 Score (average=None): {f1_score(y_true, y_pred, average=None)}") # Generate a full classification report report = classification_report(y_true, y_pred, digits=2) print("\nClassification Report:\n", report)

Debug

Known issues

breakingThe `classification_report` behavior changed significantly in v1.0.0. It now allows explicit specification of the evaluation scheme (e.g., IOB1, IOB2, BILOU) which can alter how entities are counted. Older code relying on implicit scheme assumptions might produce different results.

fix

Review calls to `classification_report` and explicitly pass the desired `scheme` and `mode` parameters (e.g., `classification_report(y_true, y_pred, scheme=IOB2, mode='strict')`) to ensure correct evaluation behavior. Refer to the documentation for supported schemes.

affects: >=1.0.0

gotchaseqeval's metrics (especially F1, precision, recall) are calculated differently from `scikit-learn`'s for sequence labeling tasks. `seqeval` specifically evaluates *entities*, primarily focusing on non-'O' (Outside) tags, and does not count correctly predicted 'O' tags as true positives, only misclassified ones. `scikit-learn`'s metrics, when applied naively to token-level tags, will include all 'O' tags in its calculations, potentially leading to inflated scores if 'O' tags are abundant and correctly predicted.

fix

Be aware of this fundamental difference when comparing evaluation results. For true entity-level evaluation in NLP sequence labeling, `seqeval` is generally preferred. If comparing with `scikit-learn`, ensure a clear understanding of what each metric is actually measuring.

affects: All versions

gotchaWhen using `mode='strict'` (e.g., in `f1_score` or `classification_report`), only exact matches for entity spans (including both boundaries and type) are considered correct. This is often the desired behavior for robust evaluation but can result in lower scores compared to the default mode which might be more lenient.

fix

Understand the implications of `mode='strict'`. If evaluating with `mode='strict'`, ensure your `y_true` and `y_pred` are consistent with a strict tagging scheme (like IOB2 or BILOU) and that this level of strictness is appropriate for your task.

affects: All versions

gotchaInput `y_true` and `y_pred` for all `seqeval.metrics` functions must be lists of lists of *strings* (e.g., `[['B-PER', 'I-PER', 'O']]`), representing the sequence tags. Providing numerical label IDs instead of string tags will result in errors.

fix

Ensure that your true and predicted labels are properly converted from numerical IDs to their corresponding string tags (e.g., 'O', 'B-PER', 'I-PER') before passing them to `seqeval` metric functions.

affects: All versions

gotchaPerformance of evaluation, particularly in `strict` mode, was significantly improved in version 1.2.1. Older versions might exhibit slower computation times for large datasets or extensive evaluations.

fix

Upgrade to `seqeval` version 1.2.1 or newer to benefit from performance enhancements, especially if you are performing evaluations in `strict` mode or on large datasets.

affects: <1.2.1

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'seqeval'

The 'seqeval' library is not installed in the Python environment where the code is being executed.

fix

pip install seqeval

AttributeError: module 'seqeval' has no attribute 'classification_report'

The user attempted to call `classification_report` directly from the top-level `seqeval` module instead of importing it from the `seqeval.metrics` submodule.

fix

from seqeval.metrics import classification_report

ValueError: y_true and y_pred must be list of list.

The `y_true` or `y_pred` arguments passed to evaluation functions like `classification_report` are not in the required list of lists of strings format (e.g., `[['B-PER', 'I-PER'], ['O', 'B-LOC']]`).

fix

Ensure `y_true` and `y_pred` are structured as `list[list[str]]`, where each inner list represents tags for a sentence.

ValueError: Invalid mode. mode must be one of {'strict', 'exact', 'partial', 'type'}.

An unsupported or misspelled string was provided for the `mode` parameter in `seqeval`'s metric functions (e.g., `f1_score`, `classification_report`).

fix

Use one of the valid mode strings: 'strict', 'exact', 'partial', or 'type'.

Upgrade

Version history

1.2.2latest on PyPI · released Oct 24, 2020

Audit

Dependencies

No dependency data recorded yet.

Agent activity

44 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/chakki-works/seqeval ↗

packagehttps://pypi.org/project/seqeval/ ↗