How scikit-learn-contrib/sklearn-pandas works
Project type and purpose
• This is an open-source Python library (a “bridge”) that implements:
• A DataFrameMapper to select and transform pandas columns
• A helper to generate feature definitions (gen_features)
• A set of transformers (including NumericTransformer, FunctionTransformer, etc.)
• Optional cross-validation shims
• Purpose: let users define column→transform mappings in DataFrame terms and plug the result directly into scikit-learn workflows.
File structure
• Top-level:
• CI and release pipelines (.circleci, .github/workflows)
• Packaging: setup.py, setup.cfg, conda/, noxfile.py, pytest.ini
• Tests: tests/*.py, test_data/
• Core package: sklearn_pandas/
• dataframe_mapper.py
• features_generator.py
• transformers.py
• pipeline.py (custom Pipeline/TransformerPipeline support)
• cross_validation.py
Key insights from README
• DataFrameMapper is the centerpiece: it accepts a list of (column selector, transformer(s), options) tuples.
• gen_features automates multi-column, multi-transformer definitions.
• Transformers are generally sklearn.TransformerMixin subclasses working on numpy arrays or DataFrames.
• The output can be an array or DataFrame and can be dense or sparse.
• DataFrameMapper itself implements fit/transform and yields X usable by sklearn.pipeline.Pipeline.
1. User Application Code
– Imports DataFrameMapper, gen_features, custom transformers
2. Core Modules (sklearn_pandas package)
– DataFrameMapper
– Feature Definitions (gen_features)
– TransformerRegistry (transformers.py: NumericTransformer, FunctionTransformer, etc.)
– Pipeline Support (pipeline.py: TransformerPipeline wrapper)
– Cross-Validation Shims (cross_validation.py)
3. External Utilities
– pandas.DataFrame (input)
– numpy.ndarray (intermediate)
– scikit-learn Estimators & Pipeline
– sklearn.compose.make_column_selector or user callables
4. CI & Packaging (side notes)
– nox, pytest, Conda, GitHub Actions, CircleCI
1. User passes a pandas.DataFrame into DataFrameMapper.fit/transform
2. DataFrameMapper:
– uses column selectors → picks DataFrame columns
– invokes transformers (from transformers.py) → returns arrays/DF
– stitches results into a single array or DataFrame
3. Output is fed into a scikit-learn Pipeline or Estimator
4. gen_features helps build the “list of (column, transformer, options)” definitions that DataFrameMapper consumes
5. cross_validation shims sit alongside scikit-learn’s cross_val_score but accept DataFrames
c. Architectural patterns & design principles
– Adapter pattern: DataFrameMapper adapts pandas index/column selection to sklearn’s array-based transformers
– Strategy pattern: different transformer classes can be plugged in per column or as default
– Builder/factory: gen_features constructs feature definition tuples programmatically
– Pipeline composition: DataFrameMapper and TransformerPipeline implement sklearn interfaces for easy chaining
d. Technologies & frameworks
– Python 3.6+
– pandas, numpy
– scikit-learn (Pipeline, TransformerMixin)
– pytest + nox for testing
– conda & PyPI for distribution
– GitHub Actions & CircleCI for CI/CD
Detail level
• Column Selection
• Transformer Pipeline
• Feature Name Generation
• Output Assembly
• That multiple transformers per column are handled by TransformerPipeline
• Call out optional modes: df_out, sparse, input_df
[User Code]
│
│ calls fit/transform
▼
[ DataFrameMapper ]───uses──> [ gen_features ]
│ ├─ Column Selector ──> pandas.DataFrame
│ ├─ TransformerPipeline ──> transformers.py (NumericTransformer, etc.)
│ ├─ sklearn Pipeline support via pipeline.py
│ └─ Output Assembly ──> numpy.ndarray or pandas.DataFrame
│
│ feeds into
▼
[ scikit-learn Estimator / Pipeline ]
On the side, show CI/packaging workflows (nox, pytest, conda_build) and test inputs in tests/ as supporting infrastructure, not part of the runtime path.