Registry / testing / pytest-flakefinder

pytest-flakefinder

JSON →
library1.1.0pypypiunverified

pytest-flakefinder is a plugin for the pytest testing framework that helps identify flaky tests by running them multiple times within a single pytest invocation. The current version, 1.1.0, was last released in October 2022 and is maintained by Dropbox, offering features to configure the number of runs and set a maximum execution time.

testing
pip install pytest-flakefinder
github ↗package ↗
INSTALL
IMPORT
SIG · PYTEST-FLAKEFINDER
P
pytest-flakefinder
testingpythonv1.1.0
harness data pending
Install & Compatibility
Where this runs

No compatibility data collected yet for this library.

Code
Verified usage

After installation, pytest-flakefinder integrates directly with the `pytest` command-line interface. You enable it with `--flake-finder` and can customize the number of repetitions with `--flake-runs` or set a time limit with `--flake-max-minutes`. Place the example code in a file named `test_example.py` and run the `pytest` commands from your terminal.

import random def test_might_be_flaky(): # This test has a 30% chance of failing to simulate flakiness if random.random() < 0.3: assert False, "This test failed flakily!" else: assert True # To run all tests 100 times to find flakes: # pytest --flake-finder --flake-runs=100 # To run tests with a maximum time limit of 5 minutes: # pytest --flake-finder --flake-max-minutes=5 # Combine with pytest-xdist for parallel execution (requires prior installation of pytest-xdist): # pytest -n auto --flake-finder --flake-runs=100
pytest --version
Debug
Known issues
gotchaRunning tests many times with `--flake-runs` without parallelization (e.g., using `pytest-xdist`) can significantly increase execution time.
fix
For larger test suites or higher flake counts, install `pytest-xdist` (`pip install pytest-xdist`) and run pytest with `-n auto` (e.g., `pytest -n auto --flake-finder`).
affects: All
gotchaThe `--flake-max-minutes` option will only skip tests that *start* after the timeout has been reached. It does not interrupt currently running tests.
fix
Account for individual test run times when setting `--flake-max-minutes`. For strict test timeouts, consider `pytest-timeout` in conjunction with `pytest-flakefinder`.
affects: All
gotchapytest-flakefinder is designed to *find* flakiness, not to *fix* or *mitigate* it during regular CI runs. Its purpose is diagnostic.
fix
Once flakes are identified, investigate and fix the underlying causes (e.g., race conditions, external dependencies, state pollution). For automatic retries in CI, plugins like `pytest-rerunfailures` are often used instead.
affects: All
Errors
Common errors & fixes
pytest: error: unrecognized arguments: --flake-runs
The pytest-flakefinder plugin is not installed or not active in the Python environment where pytest is being executed, preventing pytest from recognizing its command-line options.
fix
Ensure pytest-flakefinder is installed in your active environment by running `pip install pytest-flakefinder`.
ModuleNotFoundError: No module named 'pytest_flakefinder'
pytest-flakefinder is a pytest plugin that integrates automatically; it's not designed to be directly imported as a regular Python module by user code.
fix
Remove the `import pytest_flakefinder` statement, as the plugin's functionality is accessed via pytest command-line options or `conftest.py` hooks.
pytest: error: argument --flake-runs: invalid int value: 'invalid_value'
The value provided for a command-line option like `--flake-runs` or `--flake-max-exec-time` is not in the expected format (e.g., a non-integer for `flake-runs` or non-float for `flake-max-exec-time`).
fix
Provide a valid integer for `--flake-runs` and a valid float for `--flake-max-exec-time`, for example: `pytest --flake-runs 3` or `pytest --flake-max-exec-time 1.5`.
TypeError: addinivalue_line() missing 1 required positional argument: 'value'
The `config.addinivalue_line` method, used within the `pytest_flakefinder_options` hook in `conftest.py`, was called with an incorrect number of arguments, specifically missing the required 'value' parameter.
fix
Ensure both the option name (e.g., 'flakefinder-runs') and its corresponding value are provided, for example: `config.addinivalue_line('flakefinder-runs', '3')`.
Upgrade
Version history
1.1.0latest on PyPI · released Oct 26, 2022
Audit
Dependencies
pytestrequiredpytest-flakefinder is a plugin for pytest and requires it to function.
pytest-xdistoptionalRecommended for parallel execution of flake runs, improving performance and exposing more timing-related flakes.
Agent activity
8 hits · last 30 days
node
6
Resources
githubhttps://github.com/dropbox/pytest-flakefinder ↗
packagehttps://pypi.org/project/pytest-flakefinder/ ↗
pytest-flakefinder — pip install pytest-flakefinder · libregistry