# scikit-learn-contrib/sklearn-pandas architecture

> Architecture diagram of the GitHub repository scikit-learn-contrib/sklearn-pandas, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/scikit-learn-contrib/sklearn-pandas
- Repository: https://github.com/scikit-learn-contrib/sklearn-pandas

## Overview

##### Project type and purpose
- This is an open-source Python library (a “bridge”) that implements:
- A DataFrameMapper to select and transform pandas columns
- A helper to generate feature definitions (gen_features)
- A set of transformers (including NumericTransformer, FunctionTransformer, etc.)
- Optional cross-validation shims
- Purpose: let users define column→transform mappings in DataFrame terms and plug the result directly into scikit-learn workflows.

##### File structure
- Top-level:
- CI and release pipelines (.circleci, .github/workflows)
- Packaging: setup.py, setup.cfg, conda/, noxfile.py, pytest.ini
- Tests: tests/*.py, test_data/
- Core package: sklearn_pandas/
- dataframe_mapper.py
- features_generator.py
- transformers.py
- pipeline.py (custom Pipeline/TransformerPipeline support)
- cross_validation.py

##### Key insights from README
- DataFrameMapper is the centerpiece: it accepts a list of (column selector, transformer(s), options) tuples.
- gen_features automates multi-column, multi-transformer definitions.
- Transformers are generally sklearn.TransformerMixin subclasses working on numpy arrays or DataFrames.
- The output can be an array or DataFrame and can be dense or sparse.
- DataFrameMapper itself implements fit/transform and yields X usable by sklearn.pipeline.Pipeline.

1. User Application Code
– Imports DataFrameMapper, gen_features, custom transformers
2. Core Modules (sklearn_pandas package)
– DataFrameMapper
– Feature Definitions (gen_features)
– TransformerRegistry (transformers.py: NumericTransformer, FunctionTransformer, etc.)
– Pipeline Support (pipeline.py: TransformerPipeline wrapper)
– Cross-Validation Shims (cross_validation.py)
3. External Utilities
– pandas.DataFrame (input)
– numpy.ndarray (intermediate)
– scikit-learn Estimators & Pipeline
– sklearn.compose.make_column_selector or user callables
4. CI & Packaging (side notes)
– nox, pytest, Conda, GitHub Actions, CircleCI

1. User passes a pandas.DataFrame into DataFrameMapper.fit/transform
2. DataFrameMapper:
– uses column selectors → picks DataFrame columns
– invokes transformers (from transformers.py) → returns arrays/DF
– stitches results into a single array or DataFrame
3. Output is fed into a scikit-learn Pipeline or Estimator
4. gen_features helps build the “list of (column, transformer, options)” definitions that DataFrameMapper consumes
5. cross_validation shims sit alongside scikit-learn’s cross_val_score but accept DataFrames

c. Architectural patterns & design principles
– Adapter pattern: DataFrameMapper adapts pandas index/column selection to sklearn’s array-based transformers
– Strategy pattern: different transformer classes can be plugged in per column or as default
– Builder/factory: gen_features constructs feature definition tuples programmatically
– Pipeline composition: DataFrameMapper and TransformerPipeline implement sklearn interfaces for easy chaining

d. Technologies & frameworks
– Python 3.6+
– pandas, numpy
– scikit-learn (Pipeline, TransformerMixin)
– pytest + nox for testing
– conda & PyPI for distribution
– GitHub Actions & CircleCI for CI/CD

##### Detail level
- Column Selection
- Transformer Pipeline
- Feature Name Generation
- Output Assembly
- That multiple transformers per column are handled by TransformerPipeline
- Call out optional modes: df_out, sparse, input_df

[User Code]
│
│ calls fit/transform
▼
[ DataFrameMapper ]───uses──> [ gen_features ]
│  ├─ Column Selector ──> pandas.DataFrame
│  ├─ TransformerPipeline ──> transformers.py (NumericTransformer, etc.)
│  ├─ sklearn Pipeline support via pipeline.py
│  └─ Output Assembly ──> numpy.ndarray or pandas.DataFrame
│
│ feeds into
▼
[ scikit-learn Estimator / Pipeline ]

On the side, show CI/packaging workflows (nox, pytest, conda_build) and test inputs in tests/ as supporting infrastructure, not part of the runtime path.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Layer
    UserCode["User Application Code"]:::user
    CustomSelector["Custom Selector/Transformer"]:::user

    %% External Systems
    pandasDF["pandas.DataFrame"]:::external
    numpyOut["numpy.ndarray / pandas.DataFrame"]:::external
    SkSelector["sklearn.compose.make_column_selector"]:::external
    SklearnEstimator["scikit-learn Estimator / Pipeline"]:::external

    %% Core Modules
    subgraph "DataFrameMapper"
        direction TB
        DFM["DataFrameMapper"]:::internal
        CS["Column Selector"]:::internal
        TP["TransformerPipeline"]:::internal
        FN["Feature Name Generation"]:::internal
        OA["Output Assembly"]:::internal
    end
    GenFE["gen_features"]:::internal
    TRegistry["Transformers\n(NumericTransformer, FunctionTransformer, etc.)"]:::internal
    XVal["Cross-Validation Shims"]:::internal

    %% Relationships
    UserCode -->|calls fit/transform| DFM
    UserCode -->|defines features| GenFE
    GenFE -->|builds definitions| DFM

    pandasDF -->|select columns| CS
    CS -->|passes data| TP
    TP -->|transformed arrays| OA
    FN -->|generates names| DFM
    DFM -->|assembles output| numpyOut
    numpyOut -->|feeds into| SklearnEstimator

    TP -->|uses| TRegistry
    CustomSelector -->|can plug into| CS
    CustomSelector -->|can plug into| TP

    DFM -->|can wrap in| SklearnEstimator
    XVal -->|wraps around| SklearnEstimator

    %% CI & Packaging Infrastructure
    subgraph "CI & Packaging"
        direction TB
        CircleCI["CircleCI"]:::infra
        Action1["GitHub Actions: step1_test.yml"]:::infra
        Action2["GitHub Actions: step2_release.yml"]:::infra
        Action3["GitHub Actions: step3_pypi_deploy.yml"]:::infra
        Action4["GitHub Actions: step4_conda_deploy.yml"]:::infra
        SetupPy["setup.py"]:::infra
        SetupCfg["setup.cfg"]:::infra
        CondaMeta["conda/meta.yaml"]:::infra
        CondaConfig["conda/conda_build_config.yml"]:::infra
        Nox["noxfile.py"]:::infra
        PyTestIni["pytest.ini"]:::infra
    end

    CircleCI -->|runs| Nox
    Action1 -->|runs| PyTestIni
    Action2 -->|uses| SetupPy
    Action3 -->|uses| SetupPy
    Action4 -->|uses| CondaMeta

    %% Test Suites
    subgraph "Test Suites"
        direction TB
        T1["tests/test_dataframe_mapper.py"]:::infra
        T2["tests/test_features_generator.py"]:::infra
        T3["tests/test_transformers.py"]:::infra
        T4["tests/test_pipeline.py"]:::infra
        TData["tests/test_data"]:::infra
    end
    T1 -->|validates| DFM
    T2 -->|validates| GenFE
    T3 -->|validates| TRegistry
    T4 -->|validates| TP

    %% Click Events
    click DFM "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/sklearn_pandas/dataframe_mapper.py"
    click GenFE "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/sklearn_pandas/features_generator.py"
    click TRegistry "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/sklearn_pandas/transformers.py"
    click TP "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/sklearn_pandas/pipeline.py"
    click XVal "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/sklearn_pandas/cross_validation.py"
    click CircleCI "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/.circleci/config.yml"
    click Action1 "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/.github/workflows/step1_test.yml"
    click Action2 "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/.github/workflows/step2_release.yml"
    click Action3 "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/.github/workflows/step3_pypi_deploy.yml"
    click Action4 "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/.github/workflows/step4_conda_deploy.yml"
    click SetupPy "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/setup.py"
    click SetupCfg "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/setup.cfg"
    click CondaMeta "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/conda/meta.yaml"
    click CondaConfig "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/conda/conda_build_config.yml"
    click Nox "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/noxfile.py"
    click PyTestIni "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/pytest.ini"
    click T1 "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/tests/test_dataframe_mapper.py"
    click T2 "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/tests/test_features_generator.py"
    click T3 "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/tests/test_transformers.py"
    click T4 "https://github.com/scikit-learn-contrib/sklearn-pandas/blob/master/tests/test_pipeline.py"
    click TData "https://github.com/scikit-learn-contrib/sklearn-pandas/tree/master/tests/test_data"

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef external fill:#90EE90,stroke:#333,stroke-width:1px
    classDef user fill:#FFE4B5,stroke:#333,stroke-width:1px
    classDef infra fill:#D3D3D3,stroke:#333,stroke-width:1px
```
