# skrub-data/skrub architecture

> Architecture diagram of the GitHub repository skrub-data/skrub, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skrub-data/skrub
- Repository: https://github.com/skrub-data/skrub

## Overview

##### Main Components
- Core Library – The main “skrub” package contains submodules and functions such as joiners, encoders (MinHashEncoder, ToCategorical, etc.), deduplication methods, and other transformation utilities.
- Datasets Module – Located under skrub/datasets, which provides functionality to fetch or generate example data (e.g., fetch_employee_salaries).
- Reporting & Visualization – Under skrub/_reporting, this component generates interactive table reports and includes its own assets (HTML templates, CSS, and JavaScript files).
- Benchmarks – The “benchmarks” directory holds scripts and utilities to measure the performance of different functions (like fuzzy join and gap encoder benchmarks).
- Documentation – The “doc” folder, along with Sphinx-related configurations and templates, is used to generate the project’s documentation.
- Examples – The “examples” directory provides usage scripts that demonstrate how to apply the library’s functionality.
- Testing & CI/CD – The project contains a comprehensive set of tests (located in skrub/tests and related subdirectories) as well as CI/CD configuration files (.circleci, .github/workflows, etc.) that ensure code quality and continuous integration.
- Build Tools & Configuration – Additional directories like “build_tools” and configuration files (pyproject.toml, .pre-commit-config.yaml) indicate toolchains for auto-formatting, dependency management, and deployment.

##### Relationships and Interactions
- The Core Library is the heart of the system. Its various modules (joiners, encoders, table vectorizer, tabular learner, etc.) provide APIs that are called by both the user-facing examples and the benchmarks.
- The Datasets Module interacts with the Core Library by supplying sample data for testing and demonstration purposes.
- The Reporting module integrates with the rest of the library to produce visual outputs, feeding off transformed data.
- Examples and Benchmarks consume the Core Library’s functionalities to illustrate usage and evaluate performance.
- Testing frameworks and CI/CD workflows (CircleCI, GitHub Actions) actively run tests located in skrub/tests and ensure that changes in the Core Library and reporting tools maintain quality.
- Documentation pulls source information (docstrings, examples, and API references from the code) to generate developer and user guides using Sphinx.

##### Architectural Patterns and Design Principles
- Modularity – The project is organized into self-contained, well-defined modules (e.g., separate modules for joiners, encoders, reporting) that promote maintainability and reusability.
- Separation of Concerns – There is a clear separation between data preparation (encoding/joining), data visualization (reporting), examples, and testing/CI processes.
- Extensibility – Many modules (e.g., the reporting and encoding utilities) are designed to be extendable, permitting easy integration of additional functionality.
- Integration with external systems – Emphasis on CI/CD pipelines, automated documentation generation, and sample benchmarks shows an ecosystem-wide approach toward reliable software development.

##### Relevant Technologies, Frameworks, or Libraries
- Python language features (with emphasis on modular packages).
- Sphinx – Used for documentation generation as indicated in the “doc” folder.
- Testing frameworks – Likely pytest or similar for tests found in skrub/tests.
- CI/CD tools – CircleCI, GitHub Actions, and configuration files for scheduling builds and code coverage (codecov, etc.).
- Other external integrations (e.g., dependencies managed via pyproject.toml and pre-commit hooks).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Nodes
    DATA["Datasets Module"]:::datasets
    CORE["Core Library Module"]:::core
    REPORT["Reporting/Visualization"]:::reporting
    EX["Examples"]:::examples
    BENCH["Benchmarks"]:::benchmarks
    subgraph "Testing & CI/CD"
        TESTS["Tests"]:::testing
        CIRCLECI["CircleCI Config"]:::testing
        GHA["GitHub Actions"]:::testing
    end
    DOCS["Documentation Engine"]:::docs
    BUILD["Build & Config Tools"]:::build

    %% Relationships
    DATA -->|"providesData"| CORE
    EX -->|"consumes"| CORE
    BENCH -->|"evaluates"| CORE
    TESTS -->|"verifies"| CORE
    CIRCLECI -->|"verifies"| CORE
    GHA -->|"verifies"| CORE
    DOCS -->|"pulls"| CORE
    CORE -->|"transforms"| REPORT
    BUILD -->|"supports"| CORE

    %% Click Events
    click CORE "https://github.com/skrub-data/skrub/tree/main/skrub"
    click DATA "https://github.com/skrub-data/skrub/tree/main/skrub/datasets"
    click REPORT "https://github.com/skrub-data/skrub/tree/main/skrub/_reporting"
    click BENCH "https://github.com/skrub-data/skrub/tree/main/benchmarks"
    click DOCS "https://github.com/skrub-data/skrub/tree/main/doc"
    click EX "https://github.com/skrub-data/skrub/tree/main/examples"
    click TESTS "https://github.com/skrub-data/skrub/tree/main/skrub/tests"
    click CIRCLECI "https://github.com/skrub-data/skrub/blob/main/.circleci"
    click GHA "https://github.com/skrub-data/skrub/tree/main/.github/workflows"
    click BUILD "https://github.com/skrub-data/skrub/tree/main/build_tools"

    %% Styles
    classDef core fill:#bbf,stroke:#333,stroke-width:2px;
    classDef datasets fill:#cfc,stroke:#333,stroke-width:2px;
    classDef reporting fill:#fcf,stroke:#333,stroke-width:2px;
    classDef benchmarks fill:#cff,stroke:#333,stroke-width:2px;
    classDef docs fill:#ffc,stroke:#333,stroke-width:2px;
    classDef examples fill:#f9f,stroke:#333,stroke-width:2px;
    classDef testing fill:#eef,stroke:#333,stroke-width:2px;
    classDef build fill:#dfd,stroke:#333,stroke-width:2px;
```
