laboratory

library1.0.2pypypi✓ verified 85d ago

Laboratory is a Python library (version 1.0.2) that enables confident refactoring of critical code paths by running 'experiments' in production. Inspired by GitHub's Scientist, it executes new code (candidate) alongside existing code (control) in a randomized order, compares return values, records timing, and logs exceptions, providing a feedback loop for verification. The library is stable and addresses a timeless engineering problem, though its release cadence is slow.

testing observability

pip install laboratory

github ↗package ↗

INSTALL

IMPORT

SIG · LABORATORY

laboratory

testingpythonv1.0.2

Install

1.5s avg

Import

27ms

Disk

16MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.0.2 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.920 runs

installs and imports cleanly · install 0.0s · import 0.028s · 17.8MB

glibc

py 3.10–3.920 runs

installs and imports cleanly · install 1.5s · import 0.027s · 18MB

16MB installed

● package 16MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Experiment

✓ from laboratory import Experiment

This quickstart demonstrates how to set up and run a basic experiment. You define a 'control' function (your existing code) and a 'candidate' function (your new code). The `Experiment` class runs both, returning the control's result. Under the hood, it compares results, records performance, and logs exceptions from the candidate, which can then be published to a metrics system (not shown in this basic example).

import laboratory import time def old_function(value): time.sleep(0.01) return value * 2 def new_function(value): # Simulate a refactored version, possibly faster or different time.sleep(0.005) return value + value experiment = laboratory.Experiment() experiment.control(old_function, args=(5,)) experiment.candidate(new_function, args=(5,)) # Conduct the experiment. By default, it returns the control's value. # Candidate's return value, timing, and exceptions are recorded internally. result = experiment.conduct() print(f"Experiment conducted. Control result: {result}") # To access detailed observation data (requires a publisher to be configured) # For demonstration, we'll manually inspect the internal result, normally # this would be sent to a metrics/logging system. # For a real scenario, you'd typically subclass Experiment to implement # a publisher. Example below is illustrative of what data is collected. # print(f"Control value: {experiment._observations[0].value}") # Not a public API, for illustration # print(f"Candidate value: {experiment._observations[1].value}") # Not a public API, for illustration # print(f"Mismatched: {experiment._result.mismatched}") # Not a public API, for illustration

Debug

Known issues

gotchaExceptions raised by the candidate function are caught and recorded, but by default, they do not halt the experiment or propagate. This allows for safe testing in production but means a buggy candidate won't immediately stop execution unless explicitly configured.

fix

Initialize `Experiment(raise_on_mismatch=True)` to re-raise exceptions immediately for easier debugging in non-production environments. Implement robust reporting for candidate exceptions in production.

affects: 1.0.0+

gotchaLaboratory only collects experiment data (return values, timing, exceptions, mismatches) internally. It does NOT automatically publish these results to any external monitoring or logging system. You must implement a custom publisher to act on the experiment data.

fix

Subclass `laboratory.Experiment` and override the `publish()` method to send `Observation` data to your preferred metrics (e.g., StatsD) or logging system (e.g., Sentry, Prometheus).

affects: 1.0.0+

gotchaThe `conduct()` method of an `Experiment` instance always returns the value produced by the 'control' function. It will never return the candidate's value, even if the candidate's result is deemed 'correct' or identical.

fix

This is intended behavior for safe refactoring. If you need to access the candidate's return value, you would do so via your custom `publish` implementation, which receives the `Observation` objects.

affects: 1.0.0+

Errors

Common errors & fixes

TypeError: 'NoneType' object is not callable

This error often occurs if either the `control()` or `candidate()` function or their arguments are not correctly passed to the `Experiment` instance before `conduct()` is called.

fix

Ensure `experiment.control(your_function, args=(arg1, arg2))` and `experiment.candidate(another_function, args=(arg1, arg2))` are called with valid, callable functions and `args` are provided as a tuple or list.

Mismatched results are occurring, but my application isn't alerting me.

By default, `laboratory` reports mismatches internally but doesn't trigger alerts or re-raise exceptions. You need to explicitly handle how mismatches are reported.

fix

To get immediate feedback, use `experiment = laboratory.Experiment(raise_on_mismatch=True)`. For production, implement a custom `publish` method in a subclass of `Experiment` to send mismatch reports to your alerting system (e.g., Slack, PagerDuty).

Upgrade

Version history

1.0.2latest on PyPI · released May 5, 2019

Audit

Dependencies

No dependency data recorded yet.

Agent activity

16 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/joealcorn/laboratory ↗

packagehttps://pypi.org/project/laboratory/ ↗