The `datasieve` package provides a flexible data pipeline inspired by scikit-learn's Pipeline, but with enhanced capabilities to manipulate `y` (target) and `sample_weight` arrays alongside `X` (features). This is particularly useful for tasks such as removing outliers across all associated data, removing feature columns based on arbitrary criteria, and handling dynamic feature renaming within the pipeline. The current version is 0.1.9, with releases occurring on an irregular, as-needed basis.
pip install datasieveVerified import paths — ran on the pinned version, not inferred.
This example demonstrates how to build a `datasieve` pipeline using `Pipeline` and custom `transforms`. It includes `VarianceThreshold` to remove constant features, `SKlearnWrapper` to incorporate a `MinMaxScaler` from scikit-learn, `SVMOutlierExtractor` to identify and remove outliers (propagating removal to y and sample_weights), and `PCA` for dimensionality reduction. The example generates synthetic data with a constant feature and outliers to showcase the pipeline's capabilities.
Ensure that `X`, `y`, and `sample_weight` (if applicable) are aligned by index before passing them to `fit_transform` or `transform`. If using NumPy arrays, ensure the row order is consistent.
Ensure that the input data for `transform` (e.g., test set) has the same feature set and order as the data used for `fit` (e.g., training set). `datasieve` handles feature renaming internally (e.g., after PCA), but initial input consistency is crucial. Pre-processing steps outside the pipeline should be consistently applied.
Run `pip install datasieve` to install the package. If using a virtual environment, ensure it is activated before installation.
Verify that the feature names (column names if using pandas DataFrames) of the data being passed through the pipeline are consistent with what the pipeline expects based on its `fit` operation. Ensure that any manual feature engineering or selection outside the `datasieve` pipeline is applied consistently to both training and test data.