pytest-shard

library0.1.2pypypi✓ verified 25d ago

pytest-shard is a pytest plugin (version 0.1.2) designed to parallelize test execution across multiple machines by sharding tests based on a hash of their test names. It enables fine-grained parallelism at the individual test case level, making it suitable for various continuous integration (CI) services. The library appears to be in a maintenance phase, with the last PyPI update in December 2020.

testing

pip install pytest-shard

github ↗package ↗

INSTALL

IMPORT

SIG · PYTEST-SHARD

pytest-shard

testingpythonv0.1.2

Install

2.7s avg

Import

—

Disk

30MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.1.2 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.000s · 31.1MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 2.7s · import 0.000s · 32MB

30MB installed

● package 30MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

pytest-shard plugin

✓ No direct Python import needed; it's a pytest plugin automatically discovered.

pytest-shard integrates as a plugin and is enabled via command-line arguments to pytest, rather than explicit Python import statements in test files.

After installation, pytest-shard is enabled by passing `--shard-id` and `--num-shards` arguments to the `pytest` command. `--shard-id` specifies the index of the current shard (0-indexed), and `--num-shards` specifies the total number of shards. This is typically configured in CI pipelines where tests are distributed across parallel jobs.

# Example: Shard tests across two machines # On machine 1: pytest --shard-id=0 --num-shards=2 # On machine 2: pytest --shard-id=1 --num-shards=2 # Example for CircleCI: pytest --shard-id=${CIRCLE_NODE_INDEX} --num-shards=${CIRCLE_NODE_TOTAL}

Debug

Known issues

gotchapytest-shard does not take into account the run time of tests when sharding, which can lead to suboptimal load balancing and uneven test execution times across shards. This may result in some CI jobs finishing significantly later than others, despite an even numerical distribution of tests.

fix

Manually balance test durations across files/modules if even time distribution is critical. Consider alternative sharding mechanisms or CI-specific tools that track test durations for more sophisticated load balancing.

affects: All versions

gotchaSharding is based on a hash of the test name. Renaming tests or significantly changing their structure can alter their hash, causing them to move to a different shard. This can lead to unexpected test distribution changes in CI, potentially causing certain tests to be rerun on different machines or requiring manual re-evaluation of shard contents.

fix

Be aware that test renames will re-shard tests. If consistent sharding across minor changes is critical, consider using more stable test identifiers or a different sharding strategy. Avoid frequent, large-scale renames without understanding the impact on sharding.

affects: All versions

gotchaThe `--shard-id` command-line option in pytest-shard is 0-indexed (e.g., `--shard-id=0` for the first shard). Users configuring CI environments should ensure their environment variables (like `CIRCLE_NODE_INDEX` or `TRAVIS_JOB_NUMBER`) are correctly mapped to this 0-indexed scheme to avoid skipping shards or encountering out-of-bounds errors.

fix

Ensure CI environment variables providing the shard index are 0-based. If a variable is 1-based (e.g., `TRAVIS_JOB_NUMBER`), adjust it (e.g., `--shard-id=$((${TRAVIS_JOB_NUMBER} - 1))`) when passing to pytest.

affects: All versions

Upgrade

Version history

0.1.2latest on PyPI · released Dec 11, 2020

Audit

Dependencies

pytestrequiredCore testing framework that pytest-shard extends as a plugin.

Agent activity

3 hits · last 30 days

node

Resources

githubhttps://github.com/AdamGleave/pytest-shard ↗

packagehttps://pypi.org/project/pytest-shard/ ↗